Task execution method, computing system, storage medium, and product

Abstract

Embodiments of the present invention relate to the technical field of distributed computing, and disclosed are a task execution method, a computing system, a storage medium, and a product. In the embodiments of the present application, for a map-reduce task in an application, after executing X first map tasks, a first computing node among allocated M computing nodes executes at least one second reduce task on the X first map tasks in the first computing node, and finally, allocated N computing nodes execute a first reduce task. A reduce operation within a local data range is first performed on a map-reduce task in a computing node before performing a reduce operation within a global data range, such that the amount of network communication between the computing node and a storage system can be reduced, thereby achieving the effect of improving application performance.

Description

Task execution methods, computing systems, storage media and products

[0001] This application claims priority to Chinese Patent Application No. 202410703790.3, filed on May 31, 2024, entitled “Task Execution Method, Computing System, Storage Medium and Product”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of distributed computing technology, and in particular to a task execution method, computing system, storage medium and product. Background Technology

[0003] Currently, Spark-based computing systems, when receiving applications submitted by clients, divide the execution process of MapReduce tasks into two phases: mapping and reduction. Each phase includes a large number of tasks that can be executed in parallel. In the mapping phase, each task repartitions the input data and writes the repartitioned data to a disk file on the I / O device. In the reduction phase, each task reads the repartitioned data belonging to the same partition from the disk file over the network and then performs the corresponding reduction operation (such as a merge operation), thereby improving the execution efficiency of the mapping-reduction task.

[0004] However, during the mapping and reduction tasks in the aforementioned application, when the data volume of the Map task is large, I / O operations are required to read and write data from disk files, which leads to a significant reduction in application performance. This problem is particularly prominent in scenarios with limited computing resources, such as confidential computing scenarios. Summary of the Invention

[0005] This application provides a task execution method, computing system, storage medium, and product, which can reduce the network communication volume between computing nodes and storage systems during the execution of mapping reduction tasks, thereby improving the end-to-end performance of applications. The technical solution is as follows:

[0006] Firstly, a task execution method is provided, in which: a MapReduce task of an application is received; computing nodes are allocated to the mapping task, wherein M computing nodes execute the first mapping task and N computing nodes execute the first reduction task, where M and N are both positive integers; the first computing nodes among the M computing nodes execute the second reduction task at least once for the X first mapping tasks and store the execution result of the at least one second reduction task, where X is a positive integer; and the N computing nodes execute the first reduction task based on the execution result of the at least one second reduction task.

[0007] In this embodiment, for the mapping and reduction task in the application, after the first computing node among the allocated M computing nodes completes X first mapping tasks, it performs at least one second reduction task on the X first mapping tasks within the first computing node. Finally, the allocated N computing nodes execute the first reduction task. This allows for the mapping and reduction task to first perform a local data-scope reduction operation within the computing node, and then perform a global data-scope reduction operation on the mapping and reduction task across different computing nodes. Since the mapping and reduction task is reduced within a local data scope before performing the global data-scope reduction operation, the amount of repartitioned data written to the storage system by the M computing nodes is reduced. Correspondingly, the amount of repartitioned data read from the storage system by the N computing nodes when performing the global data-scope reduction operation is also reduced, thereby reducing the network communication volume between the computing nodes and the storage system and improving application performance.

[0008] The global data range refers to the global data range of the input data for the mapping and reduction task, while the local data range refers to the local data range of the input data for the mapping and reduction task.

[0009] Based on the method provided in the first aspect, in one possible implementation, the first computing node includes Y processes, which are used to execute X first mapping tasks. In this scenario, the implementation process of the first computing node among M computing nodes performing a second reduction task at least once on the X first mapping tasks is as follows: after the Y processes have completed the execution of the X first mapping tasks, the second reduction task is performed on the execution results of the X first mapping tasks by Z processes among the Y processes, where Y and Z are both positive integers.

[0010] The above implementation method allows a second reduction task to be executed between different processes on the same computing node, thereby enabling the mapping reduction task to perform a reduction operation within a local data range within the computing node, which in turn reduces the amount of network communication between the computing node and the storage system.

[0011] Based on the method provided in the first aspect, in one possible implementation, the first computing node includes Y processes, which are used to execute X first mapping tasks. The first process among the Y processes includes A threads, which are used to execute A first mapping tasks. In this scenario, the implementation process of the first computing node among M computing nodes executing a second reduction task at least once on the X first mapping tasks is as follows: after the A threads have completed the execution of the A first mapping tasks, the second reduction task is executed on the execution results of the A first mapping tasks by B threads among the A threads.

[0012] By the implementation manner, the second reduction task is executed once in the same process of the same computing node, so that the reduction operation in the local data range is performed on the mapping reduction task in the process of the computing node, and the network communication amount between the computing node and the storage system is reduced.

[0013] In a possible implementation manner of the method provided in the first aspect, the first computing node includes Y processes, and the Y processes are used to execute X first mapping tasks, wherein a first process in the Y processes includes A threads, and the A threads are used to execute A first mapping tasks; in this case, the implementation process of the first computing node in the M computing nodes for executing the second reduction task at least once on the X first mapping tasks executed is that, after the A threads execute the A first mapping tasks, the B threads in the A threads execute the second reduction task once on the execution results of the A first mapping tasks; and the Z processes in the Y processes execute the second reduction task once more on the results of the second reduction task executed by the Y processes.

[0014] By the implementation manners, the first second reduction task is executed once in the same process of the same computing node, and the second second reduction task is executed between different processes of the same computing node, so that the reduction operation in the local data range is performed on the mapping reduction task twice in the computing node, and the network communication amount between the computing node and the storage system is reduced.

[0015] In a possible implementation manner of the method provided in the first aspect, in the method, after the M computing nodes are allocated for the mapping reduction task, the number of times of execution of the second reduction task is set for each of the M computing nodes, and the M computing nodes are instructed to execute the second reduction task according to the specified number of times of execution of the second reduction task.

[0016] In order to facilitate subsequent scheduling of the computing nodes, the number of times of execution of the second reduction task is set for each of the M computing nodes, so as to improve the efficiency of subsequent scheduling of the computing nodes.

[0017] In a possible implementation manner of the method provided in the first aspect, the first computing node includes a trusted execution environment (TEE) and a rich execution environment (REE); in this case, in the method, the data required by the first computing node for executing the first mapping task is decrypted after being moved from the REE to the TEE; and the execution result of the at least one second reduction task executed by the first computing node is encrypted and then moved from the TEE to the REE and stored.

[0018] In the scenario that the first computing node has the TEE, the application is exemplarily a confidential computing application. In this scenario, since the first computing node moves the execution result of the at least one second reduce task from the TEE to the REE after the execution of the at least one second reduce task is completed, and then stores the encrypted execution result in the storage system connected to the first computing node through the REE. Relative to the scenario that the first computing node moves the execution result of the first map task from the TEE to the REE and stores the encrypted execution result after the execution of the first map task is completed, the method provided in the embodiment of the present application not only reduces the amount of data stored in the storage system by the first computing node in the REE, that is, reduces the network communication amount between the first computing node and the storage system, but also reduces the amount of data moved from the TEE to the REE by the first computing node, that is, reduces the communication amount between the TEE and the REE in the first computing node. Therefore, the method provided in the embodiment of the present application has a more significant effect in the confidential computing application.

[0019] In a second aspect, a computing system is provided, which includes a management node and a plurality of computing nodes.

[0020] The management node is configured to receive a map-reduce MapReduce task of an application, and assign the map-reduce task to the plurality of computing nodes, wherein M computing nodes execute a first map task, and N computing nodes execute a first reduce task, M and N are positive integers; a first computing node in the M computing nodes is configured to execute at least one second reduce task on X first map tasks executed by the first computing node, and store an execution result of the at least one second reduce task, X is a positive integer; and the N computing nodes are configured to execute the first reduce task on the execution result of the at least one second reduce task.

[0021] In a possible implementation manner based on the system provided in the second aspect, the first computing node includes Y processes, and the Y processes are configured to execute X first map tasks; in this scenario, the first computing node is configured to:

[0022] After the Y processes execute the X first map tasks, the first computing node is configured to execute a second reduce task on the execution result of the X first map tasks through Z processes in the Y processes, Y and Z are positive integers.

[0023] In a possible implementation manner based on the system provided in the second aspect, the first computing node includes Y processes, and the Y processes are configured to execute X first map tasks, wherein a first process in the Y processes includes A threads, and the A threads are configured to execute A first map tasks; in this scenario, the first computing node is configured to:

[0024] After the A threads execute the A first map tasks, the first computing node is configured to execute a second reduce task on the execution result of the A first map tasks through B threads in the A threads.

[0025] In a possible implementation manner of the system provided in the second aspect, the first computing node includes Y processes, and the Y processes are used to execute X first mapping tasks, wherein a first process in the Y processes includes A threads, and the A threads are used to execute A first mapping tasks; in this case, the first computing node is configured to:

[0026] After the A threads complete execution of the A first mapping tasks, B threads in the A threads execute a second reduction task on results of the execution of the A first mapping tasks;

[0027] Z processes in the Y processes execute a second reduction task on results of the execution of the second reduction task by the Y processes.

[0028] In a possible implementation manner of the system provided in the second aspect, the management node is further configured to:

[0029] After the M computing nodes are allocated for the mapping and reduction tasks, the management node sets a number of times of execution of the second reduction task for each of the M computing nodes, and instructs the M computing nodes to execute the second reduction task according to the specified number of times of execution of the second reduction task.

[0030] In a possible implementation manner of the system provided in the second aspect, the first computing node includes a trusted execution environment (TEE) and a rich execution environment (REE). In this case, the first computing node is further configured to: decrypt data required by the first computing node to execute the first mapping tasks after the data is moved from the REE to the TEE; and encrypt results of the execution of the at least one second reduction task after the first computing node completes the execution of the at least one second reduction task, and move the results from the TEE to the REE and store the results.

[0031] In a third aspect, a computer-readable storage medium is provided, which stores program instructions, and when the program instructions are executed by a plurality of computing devices, the plurality of computing devices execute the task execution method according to any one of the first aspect.

[0032] In a fourth aspect, a computer program product is provided, which, when executed on a plurality of computing devices, causes the plurality of computing devices to implement the task execution method according to any one of the first aspect.

[0033] The technical effects obtained by the corresponding technical means in the above second aspect to fourth aspect are similar, and will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0034] FIG. 1 is an architecture schematic diagram of a TEE-based confidential computing system provided by an embodiment of the present application;

[0035] FIG. 2 is a schematic diagram of an architecture of a PPML according to an embodiment of the present application;

[0036] FIG. 3 is a schematic diagram of a task designed for a certain mapping reduction task in the PPML shown in FIG. 2 according to an embodiment of the present application;

[0037] FIG. 4 is a schematic diagram of an architecture of a computing system according to an embodiment of the present application;

[0038] FIG. 5 is a schematic diagram of an architecture of a Spark-based computing system according to an embodiment of the present application;

[0039] FIG. 6 is a workflow diagram in the computing system shown in FIG. 5 according to an embodiment of the present application;

[0040] FIG. 7 is a flowchart of a task execution method according to an embodiment of the present application;

[0041] FIG. 8 is a flowchart of task distribution according to an embodiment of the present application;

[0042] FIG. 9 is a schematic diagram of stage division according to an embodiment of the present application;

[0043] FIG. 10 is a schematic diagram of stage relationship in a DAG according to an embodiment of the present application;

[0044] FIG. 11 is a schematic diagram of an architecture of another computing system according to an embodiment of the present application;

[0045] FIG. 12 is a schematic diagram of a structure of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0046] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0047] Before the embodiments of the present application are explained in detail, the application scenarios involved in the embodiments of the present application will be explained.

[0048] When the computing system receives an application including a mapping reduction task, the computing system can complete the computation of the mapping reduction task through mapping operation and reduction operation. When the computing nodes in the computing system have a trusted execution environment (TEE), the computing system can implement the computation of the mapping reduction task of a confidential computing application. When the computing nodes in the computing system do not have a TEE, i.e., only a rich execution environment (REE) is available, the computing system can implement the computation of the mapping reduction task of a plaintext computing application.

[0049] The confidential computing application refers to that, when a computing node executes a task in an application, data read from a storage system is encrypted data, the encrypted data needs to be carried from a REE of the computing node to a TEE, decrypted in the TEE, and then calculated, and the calculation result is encrypted and carried from the TEE to the REE, and then the encrypted calculation result is stored to the storage system through the REE.

[0050] The plaintext computing application refers to that, when a computing node executes a task in an application, data read from a storage system is plaintext data without encryption, the plaintext data can be directly calculated in the REE, and the calculation result is directly stored to the storage system through the REE.

[0051] The application scenarios of the embodiments of the present application are exemplarily described below taking a computing system with a TEE as an example.

[0052] At present, the total amount of data in the world presents exponential growth, however, the data can be managed by multiple organizations and institutions respectively and is subject to privacy protection laws and policies, that is, there are multiple parties of privacy data. Because the privacy data of different parties cannot be circulated with each other, a "data island" is formed, which seriously affects the play of the value of privacy data. In some scenarios, confidential computing can be used to realize the full-link secure calculation of multiple parties of privacy data, and ensure the security of privacy data in the storage, communication and calculation stages, so that the cross-domain joint analysis of privacy data becomes possible, which can help customers improve the accuracy of data analysis and bring greater business success to customers.

[0053] Among them, the confidential computing technology based on the trusted execution environment (TEE) is an important technical direction of confidential computing. The confidential computing technology based on the TEE makes the privacy data safe in the central processing unit (CPU) operation stage through hardware encryption and hardware isolation methods. Moreover, the confidential computing technology based on the TEE has the characteristics of high operation efficiency, flexible deployment and high scalability.

[0054] FIG. 1 is an architecture schematic diagram of a confidential computing system based on a TEE provided by an embodiment of the present application. As shown in FIG. 1, the computing system includes a client, a confidential computing cluster and multiple participating parties, such as input / output (I / O) devices such as disks or network cards. The confidential computing cluster includes multiple computing nodes and a management node, and each computing node has a TEE.

[0055] In the confidential computing system shown in FIG. 1, a user submits a confidential computing application to a management node of a confidential computing cluster through a client, and the management node schedules at least one computing node to execute a task in the confidential computing application. Wherein, when executing the task in the confidential computing application, any computing node needs to obtain ciphertext data from an I / O device, and completes data decryption and data analysis in a TEE of the computing node, and finally obtains a computing result.

[0056] Since the TEE itself does not have the ability to operate the external I / O device, if the computing node wants to operate the external I / O device in the TEE, it needs to pass the I / O instruction and data in the TEE to the rich execution environment (REE) side by means of the user mode library operating system (LibOS), and then operate the external I / O device by the REE side, which will increase the additional REE / TEE data transmission and data encryption and decryption, thereby increasing the time consumption between the user submitting the confidential computing application and the client obtaining the computing result of the confidential computing application, that is, the end-to-end performance of the confidential computing application is poor.

[0057] Currently, examples of confidential computing technology based on TEE include TrustZone technology, software guard extensions (SGX) technology, and secure encrypted virtualization (SEV) technology, etc.

[0058] Among them, PPML is a distributed privacy preserving machine learning platform based on SGX technology, which can protect the data of end-to-end confidential computing applications at various stages such as data input, data analysis, and machine learning. For example, in PPML, open source big data analysis engines such as Spark can be directly run in the TEE, thereby realizing the protection of private data.

[0059] FIG. 2 is an architecture diagram of a PPML provided by an embodiment of the present application. As shown in FIG. 2, the Spark in the PPML includes a Spark driver process (Driver) and a Spark execution process (Executor). For a confidential computing application submitted by a client, the computing system starts a Spark driver process and multiple Spark execution processes corresponding to the confidential computing application. The Spark driver process is used to schedule the Spark execution processes to execute tasks in the confidential computing application.

[0060] As shown in FIG. 2, the Spark driver process and the Spark executor process are both running in the TEE, the Spark driver process can perform ciphertext communication with the Spark executor process through the LibOS, and the Spark executor process can perform ciphertext communication with the I / O device through the LibOS, thereby ensuring the security of data in the confidential computing application.

[0061] The Spark is a distributed computing engine based on a MapReduce (MR) computing mode. For any MapReduce task in the confidential computing application, the execution process of the MapReduce task is divided into a mapping stage and a reducing stage. The mapping stage and the reducing stage each include a large number of tasks that can be executed in parallel. Each task in the mapping stage is used to map input data corresponding to the task into a key-value pair and determine a partition identifier of the key-value pair. This process can be referred to as repartitioning the input data. In the embodiments of the present application, the output of the task in the mapping stage (including the key-value pair and the partition identifier) is collectively referred to as repartitioned data of the input data. After obtaining the repartitioned data of the input data, the task in the mapping stage writes the repartitioned data into a disk file of the I / O device. Each task in the reducing stage corresponds to a partition identifier. Each task in the reducing stage is used to read, through a network, repartitioned data including the corresponding partition identifier in the disk file, and then perform a reducing operation (such as a merging operation) corresponding to the MapReduce task to implement reducing of key-value pairs belonging to the same partition, thereby improving the execution efficiency of the MapReduce task.

[0062] In the embodiments of the present application, the task in the mapping stage is referred to as a mapping task (map task), and the task in the reducing stage is referred to as a reducing task (reduce task).

[0063] FIG. 3 is a schematic diagram of a task designed for a certain MapReduce task in the PPML shown in FIG. 2 according to an embodiment of the present application. As shown in FIG. 3, one or two processes are respectively running on each computing node, and each process includes two threads. The process is, for example, a Spark executor process. Each thread is used to execute a mapping task or a reducing task in the MapReduce task.

[0064] As shown in FIG. 3, each mapping task is used for: the computing node reading the input data assigned to the mapping task from the disk file in the REE, and moving the read input data from the REE to the TEE (this step is not shown in FIG. 3), performing the mapping operation on the input data assigned to the mapping task after decryption in the TEE, and moving the result of the mapping operation, i.e., the repartitioned data, from the TEE to the REE after encryption, and then writing the encrypted repartitioned data to the disk file in the REE. Each reduction task is used for: the computing node reading the repartitioned data belonging to the same partition from all disk files through the network card in the REE, moving the read repartitioned data from the REE to the TEE, performing the reduction operation on the read repartitioned data after decryption in the TEE, and obtaining the output result of the reduction task.

[0065] In the above process of performing the mapping and reduction tasks, not only the computing node needs to perform I / O operations to read and write data from and to the disk file, but also the computing node needs to move data between the REE and the TEE, so in the case that the input data amount of the mapping and reduction tasks is large, the performance of the confidential computing application is greatly lagging behind the performance of the plaintext computing application.

[0066] Therefore, in the embodiments of the present application, for the mapping and reduction tasks in the application, the first computing node in the M computing nodes assigned to perform the X first mapping tasks, and then perform at least one second reduction task on the X first mapping tasks in the first computing node, and finally perform the first reduction task by the N computing nodes assigned to perform the first reduction task. In this way, the reduction operation in the local data range can be performed on the mapping and reduction tasks in the computing node first, and then the reduction operation in the global data range can be performed on the mapping and reduction tasks between different computing nodes. Since the reduction operation in the local data range is performed on the mapping and reduction tasks before the reduction operation in the global data range, the amount of repartitioned data written to the storage system by the M computing nodes is reduced, and accordingly the amount of repartitioned data read from the storage system by the N computing nodes when performing the reduction operation in the global data range is also reduced, thereby reducing the network communication amount between the computing nodes and the storage system, and achieving the effect of improving the performance of the application.

[0067] The global data range refers to the global data range of the input data of the mapping and reduction tasks, and the local data range refers to the local data range of the input data of the mapping and reduction tasks.

[0068] It should be noted that the method provided by the embodiments of the present application can be applied to confidential computing applications, and can also be applied to plaintext computing applications. When applied to confidential computing applications, each computing node has a TEE. In this scenario, since the first computing node moves the execution result of the at least one second reduction task from the TEE to the REE after performing the at least one second reduction task, and then stores the encrypted execution result in the storage system connected to the first computing node through the REE. Relative to the first computing node, after performing the first mapping task, the execution result of the first mapping task is directly encrypted, moved from the TEE to the REE and stored, the method provided by the embodiments of the present application not only can reduce the amount of data stored by the first computing node in the REE to the storage system, that is, reduce the network communication amount between the first computing node and the storage system, but also can reduce the amount of data moved from the TEE to the REE of the first computing node, that is, reduce the communication amount between the TEE and the REE in the first computing node. Therefore, the method provided by the embodiments of the present application has more significant effect in confidential computing applications.

[0069] The architecture of the computing system and the task execution method provided by the embodiments of the present application will be explained and described below.

[0070] FIG. 4 is a schematic diagram of the architecture of a computing system provided by an embodiment of the present application. As shown in FIG. 4, the computing system includes a management node 10, a plurality of computing nodes 20 and a storage system 30. The management node 10 is connected to each computing node 20 for communication, and each computing node 20 is also connected to the storage system 30 for communication.

[0071] As shown in FIG. 4, the management node 10 is also connected to a client outside the computing system for communication. The client is used to submit an application to the management node 10, which can be a confidential computing application or a plaintext computing application. In the scenario where the application is a confidential computing application, each computing node 20 also has a TEE to execute the tasks of the confidential computing application in the TEE. The computing system shown in FIG. 4 exemplarily includes the confidential computing cluster and each participant shown in FIG. 1 in this scenario.

[0072] When the management node 10 receives the application from the client, it first determines which computing nodes 20 can be used to execute the tasks of the application according to the resource requirement information of the application, that is, the computing nodes 20 that can execute the application.

[0073] Among them, the management node 10 can first determine the remaining available resources of each computing node 20 in the plurality of computing nodes 20, and then determine which computing nodes 20 to use to execute the application according to the resource requirement information of the application and the remaining available resources of each computing node 20. The related implementation manner is described in detail in the subsequent method embodiments, which will not be expanded here.

[0074] The resource requirement information is used to indicate the specification of the computing resource required for executing the application. For example, the resource requirement information includes the total number of processes required by the application, the process specification of each process, such as the number of threads in the process and the memory size required by the process. In this case, the remaining available resource of each computing node 20 can be, for example, the remaining executable process on the computing node.

[0075] In addition, the resource requirement information of the application can be obtained by analyzing the application by the user. For example, the user can analyze each task in the code corresponding to the application to obtain the resource requirement information of each task. The resource requirement information of the task includes the number of processes required for executing the task, the number of threads required by each process, and the size of the memory required by a single process. Then, the resource requirement information with the highest requirement is selected from the resource requirement information of each task as the resource requirement information of the application. In this way, it can be ensured that the determined resource requirement information can cover the requirement of each task. Alternatively, the user can also determine the resource requirement information of the application by other ways, which will not be illustrated one by one herein.

[0076] After the management node 10 applies for the computing node 20 for executing the application, the management node 10 can formulate a task flowchart of the application according to the remaining available resource on the applied computing node 20. The task flowchart indicates the execution order of each task and which thread on which process of which computing node executes each task. Subsequently, the management node 10 can schedule the applied computing node 20 to execute each task in turn according to the task flowchart, thereby obtaining the computing result of the application.

[0077] In the embodiment of the application, the task of the application submitted by the user to the client is a mapping induction task, and therefore the computing system shown in FIG. 4 can also be referred to as a MapReduce distributed computing engine or a MapReduce distributed computing framework in the embodiment of the application. The basic principle of the mapping reduction has been described before, which will not be described herein again.

[0078] In addition, the storage system 30 in FIG. 4 can be implemented by a centralized storage node, or alternatively, can be implemented by different storage nodes deployed in a distributed manner, which is not limited in the embodiment of the application. For example, the storage system 30 can be a storage system composed of at least one disk. Each computing node can be connected to the storage system through an Ethernet card.

[0079] In addition, the computing system shown in FIG. 4 can be a Spark-based computing system. Alternatively, it can also be a computing system based on other mapping reduction computing technologies, such as a computing system based on Hadoop MapReduce (an open source distributed computing framework), which will not be illustrated one by one herein.

[0080] The computing system shown in FIG. 4 is further explained below by taking a Spark-based computing system as an example.

[0081] FIG. 5 is a schematic diagram of an architecture of a Spark-based computing system according to an embodiment of the present application. As shown in FIG. 5, the computing system includes a Spark client, a Spark master, a plurality of Spark workers, and Hadoop distributed file systems (HDFS) 1 and 2.

[0082] The computing system shown in FIG. 5 can be applied to processing plaintext computing applications. Alternatively, the computing system shown in FIG. 5 can also be used to process confidential computing applications. In this scenario, as shown in FIG. 5, the computing system further includes a remote attestation service.

[0083] The working principle of the computing system shown in FIG. 5 is explained below by taking the scenario in which the computing system processes confidential computing applications as an example.

[0084] In the scenario in which the computing system shown in FIG. 5 is used to process confidential computing applications, when the computing system is initialized, i.e., in the deployment stage of the computing system, the Spark master and the Spark workers are started in a TEE environment, and the remote attestation service, the HDFS 1 and the HDFS 2 are also in an available state.

[0085] In the distributed computing system shown in FIG. 5, the execution process of an application is as follows:

[0086] (1) A user submits a data analysis application to the Spark master at the Spark client, and the data analysis application is a confidential computing application.

[0087] (2) When the Spark master receives the data analysis application, it starts a Spark driver process for the data analysis application. The Spark driver process can be deployed on the Spark master or on a Spark worker. FIG. 5 takes the deployment on the Spark master as an example. The Spark driver process initiates a resource application request for the data analysis application to the resource scheduler built in the Spark master. When the resource scheduler determines which Spark workers can execute the data analysis application, it sends the information of the applied Spark workers to the Spark driver process. The Spark driver process starts a Spark execution process in the applied Spark workers, so as to execute the tasks of the data analysis application through the Spark execution processes in the Spark workers subsequently.

[0088] (3) The Spark driver process and the Spark executor processes ensure that the program in the data analytics application has not been modified and is currently in the TEE through the remote attestation service.

[0089] (4) The Spark executor processes obtain the encrypted data of the parties through the Hadoop distributed file systems (HDFS) HDFS1 and HDFS2, decrypt the data in the TEE, complete the mapping tasks and reduction tasks specified in the data analytics application, and save the calculation results in the form of ciphertext to a persistent medium such as HDFS1 and HDFS2 or a local disk.

[0090] In the computing system shown in FIG. 5, the Spark client is configured to implement the function of the client in FIG. 4, the Spark worker node with the Spark executor process is configured to implement the function of the computing node shown in FIG. 4, the Spark manager node and the Spark driver process are configured to implement the function of the manager node in FIG. 4, and HDFS1 and HDFS2 are configured to implement the function of the storage system in FIG. 4.

[0091] FIG. 6 is a workflow diagram of the computing system shown in FIG. 5 according to an embodiment of the present application.

[0092] As shown in FIG. 6, the Spark manager node includes a resource scheduler, the Spark worker node includes a resource controller, the Spark driver process includes an MR execution plan generator and an MR task scheduler, and the Spark executor process includes an MR task executor.

[0093] The resource scheduler is configured to apply for the Spark worker node for executing the application based on the resource requirement information of the application, and send the information of the applied Spark worker node to the Spark driver process. The Spark driver process starts the Spark executor process through the resource controller in the Spark worker node according to the information of the applied Spark worker node.

[0094] After the Spark executor process is started, the MR execution plan generator in the Spark driver process can generate a DAG according to the application and the Spark executor processes in each Spark worker node. The DAG includes multiple stages, each stage includes multiple tasks, and the DAG also indicates which thread of which Spark executor process in which Spark worker node executes each task. The MR task scheduler schedules the threads in the Spark executor process to execute the tasks in the order of the tasks in the DAG, thereby completing the calculation of the application.

[0095] In the computing system shown in FIG. 5, for a map-reduce task of an application initiated by a Spark client, a local data range reduction operation is performed on the map-reduce task within a Spark worker node first, and then a global data range reduction operation is performed on the map-reduce task in an external storage system. The MR execution plan generator in the Spark driver process and the resource scheduler in the Spark management node are improved to implement the execution method of the application provided in the embodiments of the present application. The functions of the improved MR execution plan generator and resource scheduler can be referred to the subsequent method embodiments, and will not be described in detail here.

[0096] FIG. 7 is a flowchart of a task execution method provided in the embodiments of the present application. As shown in FIG. 7, the method comprises the following steps.

[0097] Step 701: receiving a map-reduce task of an application.

[0098] In some embodiments, in the scenario of the computing system shown in FIG. 4, the management node receives an application from a client, and the application comprises a map-reduce task.

[0099] For example, when the computing system is a Spark-based computing system, the application can also be referred to as a Spark application, and in this scenario, the Spark application is received by the Spark management node from the Spark client in step 701.

[0100] In some embodiments, each of the plurality of computing nodes is provided with a TEE, and in this scenario, the application in step 701 is a confidential computing application, so as to improve the performance of the confidential computing application by the method provided in the embodiments of the present application.

[0101] Alternatively, each of the plurality of computing nodes can also not be provided with a TEE, and in this scenario, the application in step 701 is a plaintext computing application, so as to improve the performance of the plaintext computing application by the method provided in the embodiments of the present application.

[0102] For the convenience of subsequent description, the application in step 701 is referred to as a target application.

[0103] When the target application is received, the management node can first apply for computing resources for the target application, that is, apply for which computing nodes in the computing system to execute the tasks in the target application.

[0104] In some embodiments, the management node can determine the remaining available resources of each of the plurality of computing nodes; sort the plurality of computing nodes according to the sizes of the remaining available resources of each of the plurality of computing nodes, the size of the remaining available resources of a computing node ranked earlier being larger than the size of the remaining available resources of a computing node ranked later; and select the computing nodes from the sorted plurality of computing nodes in a front-to-back order to execute the target application according to the resource requirement information of the target application, the resource requirement information being used to indicate the specifications of the computing resources required to execute the target application.

[0105] After the computing nodes to execute the target application are selected in the above manner, the mapping tasks in the reduction tasks in the subsequent target application can be implemented by as few computing nodes as possible. In this way, most of the input data can be first completed in a reduction operation within the same computing node, thereby reducing the amount of data written to the storage system and the amount of data read from the storage system when performing the reduction operation in the global data range through the storage system.

[0106] For example, the resource requirement information includes the process specifications of the processes required to execute the target application, and the process specifications can include the number of threads included in a process, the size of the memory required by the process, and the like. In this scenario, for any computing node of the plurality of computing nodes, the management node first determines the number of remaining available cores and the size of the remaining available memory of each computing node, and then determines the number of processes that can be started on each computing node according to the process specifications, i.e., the number of remaining executable processes. That is, the size of the remaining available resources is the number of remaining executable processes.

[0107] Alternatively, for different target applications submitted by the client, the process specifications of the processes required by different target applications are default specifications. In this scenario, the resource requirement information can not include the process specifications of the processes required to execute the target application. The default process specifications are pre-stored in the management node, so that the management node can determine the number of remaining executable processes of each computing node according to the default process specifications.

[0108] After the management node determines the number of remaining executable processes of each computing node, the implementation manner of the management node to sort the plurality of computing nodes according to the sizes of the remaining available resources of each of the plurality of computing nodes can be: sorting the plurality of computing nodes according to the number of remaining executable processes of each of the plurality of computing nodes, the number of remaining executable processes of a computing node ranked earlier being greater than the number of remaining executable processes of a computing node ranked later.

[0109] In this scenario, the resource requirement information can include a total number of processes required for executing the target application. Accordingly, the implementation of selecting computing nodes from the sorted plurality of computing nodes in a front-to-back order to execute the target application according to the resource requirement information of the target application can be: selecting an i-th computing node from the sorted plurality of computing nodes in a front-to-back order, and performing the following operations on the i-th computing node:

[0110] If the total number of processes is greater than 0 and exceeds the number of remaining executable processes of the i-th computing node, all the remaining executable processes on the i-th computing node are selected to execute the target application, and the total number of processes is reduced by the number of remaining executable processes of the i-th computing node to obtain an updated total number of processes. If the total number of processes is greater than 0 and does not exceed the number of remaining executable processes of the i-th computing node, part of the remaining executable processes on the i-th computing node are selected to execute the target application, and the total number of processes is updated to 0. The updated total number of processes is used to determine whether to select the remaining executable processes on the i+1-th computing node to execute the target application.

[0111] In the above case where the total number of processes is greater than 0 and does not exceed the number of remaining executable processes of the i-th computing node, assuming that the total number of processes is n, n remaining executable processes of all the remaining executable processes on the i-th computing node are selected to execute the target application.

[0112] In addition, the process of determining which computing nodes are used to execute the target application can be implemented by a resource scheduler embedded in the management node. For example, in a Spark-based computing system, this process can be implemented by a resource scheduler in the Spark management node shown in FIG. 6, which is exemplarily a Spark Standalone. Alternatively, this process can also be implemented by a resource scheduler external to the management node, which is not limited in the embodiments of the present application, such as a YARN (a big data resource scheduling framework) or a Kubernetes (an open source container orchestration engine) resource scheduler.

[0113] The implementation of applying for computing resources is exemplarily illustrated below with the computing system shown in FIG. 6.

[0114] For example, a user submits a target application to a Spark management node in a computing system, and provides resource requirement information of the target application, which is also referred to as resource configuration information, including total number of processes required by the target application, core number of a single process, and memory size required by the single process, etc. Wherein, the core number of the single process is the number of threads included in the single process. For the convenience of subsequent description, the total number of processes required by the target application in the resource requirement information is marked as requiredNumExec, the core number of the single process is marked as execCores, and the memory size required by the single process is marked as execMem.

[0115] The Spark management node starts a Spark driver process corresponding to the target application in response to the received target application. The Spark driver process applies for TEE resources to a built-in resource scheduler in the Spark management node, that is, applies for Spark worker nodes (i.e. computing nodes) capable of executing the target application. For any Spark worker node i, the resource scheduler obtains the remaining available core number cores i and the total remaining available memory mem i of the Spark worker node i, and then calculates the maximum number of processes maxNumExec i that can be started on the Spark worker node i in theory according to the core number of the single process and the memory size of the single process in the resource requirement information (i.e. the number of remaining executable processes). i All Spark worker nodes are arranged in descending order of the number of remaining executable processes maxNumExec i , and the Spark worker nodes i are selected in turn. If the total number of processes currNumExec allocated at the current time is less than requiredNumExec, the remaining executable processes on the Spark worker node i are selected to execute the target application, and the number of selected remaining executable processes = min(maxNumExeci, requiredNumExec-currNumExec). When currNumExec is equal to requiredNumExec, the resource allocation is stopped, that is, the selection of the Spark worker nodes is stopped. Wherein, the initial value of currNumExec is 0.

[0116] The resource scheduler sends information of the selected process set ExecSet to the Spark driver process, and the information of the process set ExecSet includes the identification nodeID j of the Spark worker node where each process j is located, wherein j∈ExecSet.

[0117] Step 702: allocating computing nodes for the mapping-reduce task, wherein M computing nodes execute the first mapping task and N computing nodes execute the first reduce task, and M and N are positive integers.

[0118] Since the mapping-reduce task needs to be implemented through two operations, one is a mapping operation for repartitioning the input data, and the other is a reduce operation for merging all the repartitioned data, the management node needs to design the first mapping task and the first reduce task for the mapping-reduce task after receiving the mapping-reduce task. Specifically, the first mapping task refers to a task of performing a mapping operation on the input data of the mapping-reduce task. The first reduce task refers to a task of performing a reduce operation in the global data range of the input data of the mapping-reduce task.

[0119] The management node further allocates computing nodes for the mapping-reduce task in the target application after applying for the computing nodes for executing the target application, to determine which computing nodes execute the first mapping task and which computing nodes execute the first reduce task.

[0120] In some embodiments, the implementation of allocating M computing nodes for executing the first mapping task can be that all the applied computing nodes are used as the M computing nodes for executing the first mapping task. That is, M is the total number of the applied computing nodes, so that all the computing resources in the applied computing nodes can be fully utilized.

[0121] In this scenario, the management node first divides the input data of the mapping-reduce task into multiple data blocks according to the number of remaining executable processes of the applied computing nodes and the number of threads included in each remaining executable process, and each data block is used as the input data of a first mapping task. The total number of the divided data blocks is greater than or equal to the total number of threads of all the remaining executable processes in the applied computing nodes, so as to avoid idle threads in the applied computing nodes in the mapping stage.

[0122] For example, the applied computing nodes include computing node 1 and computing node 2, the number of remaining executable processes in computing node 1 is 1, the number of remaining executable processes in computing node 2 is 2, and the number of threads included in each remaining executable process is 2. The management node can divide the input data of the mapping-reduce task into 6 data blocks or more data blocks, to design at least one first mapping task for each thread in the applied computing nodes. In other words, the first mapping task is executed by the applied computing node 1 and the applied computing node 2.

[0123] Optionally, in the scenario where the user specifies the total number of data blocks for dividing the input data of the map-reduce task, the management node directly divides the input data of the map-reduce task according to the total number of data blocks specified by the user, and designs a first map task for each data block after the division, and distributes the designed first map task to the M computing nodes in the applied computing nodes. In this scenario, M can be less than the total number of applied computing nodes.

[0124] In addition, in some embodiments, the implementation of allocating N computing nodes to execute the first reduce task can be: determining the number of first reduce tasks, and selecting N computing nodes from the applied computing nodes to execute the first reduce task according to the number of first reduce tasks. For example, the total number of all threads of the remaining executable processes in the N computing nodes is less than or equal to the number of first reduce tasks, so as to avoid the existence of idle threads in the N computing nodes in the reduce phase.

[0125] Optionally, the number of first reduce tasks can also be automatically configured by the computing system, which is not limited in the embodiments of the present application.

[0126] In the embodiments of the present application, in addition to designing the aforementioned first map task and first reduce task, the management node also designs at least one second reduce task implemented in each computing node in the M computing nodes, so as to implement the reduce operation of the map-reduce task in the computing node to obtain the calculation result of the map-reduce task in the local data range, thereby reducing the network communication amount between the computing node and the storage system, and achieving the effect of improving the performance of the target application.

[0127] For example, for any computing node in the M computing nodes executing the first map task, such as a first computing node, assuming that the first computing node is used to execute X first map tasks, the management node also designs at least one second reduce task for the first computing node, which is used to perform at least one reduce operation on the output result of the X first map tasks executed by the first computing node. To implement at least one second reduce task in the first computing node after executing the X first map tasks.

[0128] The at least one second reduce task can include one or more second reduce tasks. When the at least one second reduce task includes one second reduce task, the first computing node performs one second reduce task after executing the X first map tasks. When the at least one second reduce task includes multiple second reduce tasks, the first computing node sequentially performs multiple second reduce tasks after executing the X first map tasks. Different reduce tasks are used to implement reduce operations in different local data ranges.

[0129] For example, when at least one second reduction task includes one second reduction task, that is, when the execution count of at least one second reduction task is 1, the first computing node executes one second reduction task after executing X first mapping tasks. In this scenario, the second reduction task executed by the first computing node can be a process-level second reduction task or a node-level second reduction task.

[0130] The process-level second reduction task can be understood as follows: after a thread within the same process on the same computing node completes its first mapping task, it executes a second reduction task once within that process. Taking the first computing node as an example, suppose the first computing node includes Y processes, each of which executes X first mapping tasks. The first process within the Y processes includes A threads, each of which executes A first mapping tasks. Then, for any process on the first computing node, such as the first process itself, after the A threads within the first process complete A first mapping tasks, B threads within the A threads execute a second reduction task based on the results of the A first mapping tasks. Here, A and B are both positive integers, and B can be equal to or less than A.

[0131] To enable threads within the same process to execute a second reduction task after completing the first mapping task, a first memory space needs to be configured for that process on the compute node. Different threads within this process can share this first memory. In other words, the first memory is shared memory among different threads within the same process on the same compute node. For example, taking the first process on the first compute node as an example, a first memory space is configured for the first process. After A threads in the first process complete A first mapping tasks, these A threads store the execution results of the first mapping tasks (i.e., the repartitioned data) in the first memory. Since the A threads in the first process share the first memory, subsequent threads B among the A threads can read the repartitioned data from the first memory and execute a second reduction task.

[0132] The node-level second reduction task can be understood as follows: after executing the first mapping task, threads within different processes of the same computing node execute the second reduction task once within that computing node. Taking the first computing node as an example, suppose the first computing node includes Y processes, and these Y processes are used to execute X first mapping tasks. Then, after the Y processes in the first computing node execute the X first mapping tasks, the second reduction task is executed by Z processes within the Y processes based on the execution results of the X first mapping tasks. Y and Z are both positive integers, and Z can be equal to or less than Y.

[0133] To implement the threads in different processes in the same computing node to perform the second reduction task once inside the computing node after the threads in different processes in the same computing node complete the first mapping task, the second memory needs to be configured in the computing node for all processes, and the threads in different processes in the computing node can share the second memory. In other words, the second memory is the memory shared between different processes in the same computing node. For example, taking the first computing node as an example, the second memory is configured for the first computing node, and after the threads in the Y processes of the first computing node complete the X first mapping tasks, the threads in the Y processes store the execution results (i.e., the repartition data) of the first mapping tasks in the second memory. Since all processes in the first computing node share the second memory, the threads in the Z processes of the Y processes can read the repartition data from the second memory and perform the second reduction task once.

[0134] For example, taking the first computing node as an example, the second memory is configured for all processes in the first computing node, and the threads in all processes can share the second memory, and one second memory is configured for each process in the first computing node, and different threads in each process can share the corresponding second memory. Assuming that the first computing node includes Y processes, and the Y processes are used to execute X first mapping tasks, wherein the first process in the Y processes includes A threads, and the A threads are used to execute A first mapping tasks. Then, for any process in the first computing node, such as the first process, after the A threads in the first process complete the A first mapping tasks, the A threads store the execution results (i.e., the repartition data) of the first mapping tasks in the first memory corresponding to the first process, so that the B threads in the A threads can read the repartition data from the first memory corresponding to the first process and perform the first second reduction task. After the Y processes complete the first second reduction task, the Y processes store the execution results of the first second reduction task in the second memory through a mapping operation, and then the threads in the Z processes read the repartition data from the second memory and perform the second second reduction task.

[0135] For example, taking the first computing node as an example, the second memory is configured for all processes in the first computing node, and the threads in all processes can share the second memory, and one second memory is configured for each process in the first computing node, and different threads in each process can share the corresponding second memory. Assuming that the first computing node includes Y processes, and the Y processes are used to execute X first mapping tasks, wherein the first process in the Y processes includes A threads, and the A threads are used to execute A first mapping tasks. Then, for any process in the first computing node, such as the first process, after the A threads in the first process complete the A first mapping tasks, the A threads store the execution results (i.e., the repartition data) of the first mapping tasks in the first memory corresponding to the first process, so that the B threads in the A threads can read the repartition data from the first memory corresponding to the first process and perform the first second reduction task. After the Y processes complete the first second reduction task, the Y processes store the execution results of the first second reduction task in the second memory through a mapping operation, and then the threads in the Z processes read the repartition data from the second memory and perform the second second reduction task.

[0136] wherein the different second reduction tasks and the first reduction task differ in that the different second reduction tasks are used to perform reduction operations in different local data ranges of the input data of the mapping induction task, and the first reduction task is used to perform statistics in a global data range of the input data of the mapping induction task.

[0137] For example, when the input data of the mapping reduction task is divided into multiple data blocks, and each data block is taken as the input data of the first mapping task of a thread in the M computing nodes, for any computing node, such as the first computing node, assuming that the first computing node includes Y processes, and the Y processes include a first process, taking the first process as an example, the process-level second reduction task performed in the first computing node is used to implement: performing a reduction operation on all data blocks assigned to the thread in the first process. The node-level second reduction task performed in the first computing node is used to implement: performing a reduction operation on all data blocks assigned to the thread in the Y processes in the first computing node.

[0138] In addition, the number of reduction tasks in each second reduction task can be specified by a user or automatically configured by the computing system, and the embodiments of the present application are not limited in this regard.

[0139] It should be noted that the above is an example of illustrating multiple second reduction tasks by taking the process-level second reduction task and the node-level second reduction task as an example. In the embodiments of the present application, the multiple second reduction tasks can be implemented in other ways, which will not be illustrated one by one here.

[0140] In addition, in some embodiments, after the management node allocates the M computing nodes for the mapping reduction task, the management node can also set the number of executions of the second reduction task for each of the M computing nodes, and instruct the M computing nodes to execute the second reduction task according to the specified number of executions of the second reduction task. To improve the efficiency of subsequent scheduling of computing nodes.

[0141] For example, after the management node allocates computing nodes for the mapping reduction task in the target application, the management node can also determine a task flowchart of the target application, which can indicate the number of executions of the second reduction task for each of the M computing nodes, and the M computing nodes executing the second reduction task according to the specified number of executions of the second reduction task. Subsequently, the management node can directly schedule each computing node according to the task flowchart to implement the target application.

[0142] When the computing system is a Spark-based computing system, the task flowchart can be generated by an MR execution plan generator in the Spark driver process shown in FIG. 6, and the task flowchart can also be referred to as a directed acyclic graph (DAG).

[0143] The management node determines the sub-task flowchart of each task in the target application in sequence according to the execution order of the tasks in the target application, and then connects the sub-task flowcharts of the tasks according to the execution order of the tasks to obtain the task flowchart for the target application. The task flowchart of the target application is described below by taking the sub-task flowchart of the mapping reduction task in the target application as an example.

[0144] In addition, it should be noted that in the embodiment of the present application, at least one second reduction task is added between the first mapping task and the first reduction task, and a mapping operation needs to be performed once before each reduction task, therefore, after designing the at least one second reduction task, the management node also needs to design a mapping task performed between different second reduction tasks, and a mapping task performed between the last second reduction task and the first reduction task.

[0145] For the convenience of subsequent description, the first mapping task, the at least one second reduction task, and the mapping task performed between different second reduction tasks are collectively referred to as a first type of MR task, and the first reduction task and the mapping task performed between the last second reduction task and the first reduction task are collectively referred to as a second type of MR task.

[0146] For example, in a scenario where the at least one second reduction task includes two second reduction tasks, the first type of MR task includes a first type of sub-MR task and a second type of sub-MR task, and the output of the first type of sub-MR task is the input of the second type of sub-MR task. The first type of sub-MR task includes a first mapping task and a process-level second reduction task; and the second type of sub-MR task includes a node-level second reduction task and a mapping task performed between the node-level second reduction task and the process-level second reduction task.

[0147] Taking the first computing node as an example, it is assumed that the first computing node includes Y processes, and the Y processes are used to execute X first mapping tasks, wherein the first process in the Y processes includes A threads, and the A threads are used to execute A first mapping tasks. The first type of sub-MR task includes A first mapping tasks executed by A threads in the first process in each process in the first computing node, such as the first process, and a first second reduction task executed by B threads in the A threads. The second type of sub-MR task includes a mapping task executed by each process in the first computing node, such as the first process, after the first second reduction task is executed, and a second second reduction task executed by Z processes in the Y processes.

[0148] By the first type of sub-MR task and the second type of sub-MR task, the mapping reduction task can be first reduced within the process of the computing node to obtain the calculation result of the mapping reduction task in a smaller local data range, and then the calculation result of the previous step is reduced between different processes of the same computing node to obtain the calculation result of the mapping reduction task in an intermediate local data range, thereby reducing the network communication amount between the computing node and the storage system, and achieving the effect of improving the performance of the target application.

[0149] Further exemplarily, in a scenario where the at least one second reduction task includes one second reduction task, such as the at least one second reduction task includes a process-level second reduction task, the first type of MR task only includes the first type of sub-MR task and does not include the second type of sub-MR task. In this scenario, the mapping reduction task is only reduced once within the process of the computing node and is not further reduced between different processes, which also reduces the network communication amount between the computing node and the storage system, and achieves the effect of improving the performance of the target application.

[0150] Taking the first computing node as an example, it is assumed that the first computing node includes Y processes, and the Y processes are used to execute X first mapping tasks. The first process in the Y processes includes A threads, and the A threads are used to execute A first mapping tasks. The first type of MR task only includes the first type of sub-MR task, and the first type of sub-MR task includes the A first mapping tasks executed by the A threads in the first process in the first computing node and one second reduction task executed by B threads in the A threads.

[0151] Further exemplarily, in a scenario where the at least one second reduction task includes one second reduction task, such as the at least one second reduction task includes a process-level second reduction task, the first type of MR task only includes the first type of sub-MR task and does not include the second type of sub-MR task. In this scenario, the mapping reduction task is only reduced once within the process of the computing node and is not further reduced between different processes, which also reduces the network communication amount between the computing node and the storage system, and achieves the effect of improving the performance of the target application.

[0152] Taking the first computing node as an example, it is assumed that the first computing node includes Y processes, and the Y processes are used to execute X first mapping tasks. The first type of MR task only includes the first type of sub-MR task and does not include the second type of sub-MR task. The second type of sub-MR task includes the X first mapping tasks executed by the Y processes in the first computing node and one second reduction task executed by Z processes in the Y processes.

[0153] The following takes the first type of MR task including the first type of sub-MR task and the second type of sub-MR task as an example to describe in detail the process of designing the mapping task and the reduction task for implementing the mapping reduction task by the management node.

[0154] For the mapping reduction task, the management node can design the mapping task and the reduction task for implementing the mapping reduction task in the following three levels.

[0155] The first level: the first type of sub-MR task, wherein the first type of sub-MR task includes the first mapping task and the second reduction task at the process level.

[0156] When designing the MR task at the first level, for any computing node in the M computing nodes, such as the first computing node, at least one first mapping task is allocated to each thread in each remaining executable process in the first computing node, and the input data of each first mapping task is a data block obtained by dividing the input data of the mapping reduction task.

[0157] For any process in the first computing node, such as the first process, assuming that the first process includes A threads, after the first mapping task is allocated to the A threads in the first process, the second reduction task is allocated to B threads in the A threads, and these second reduction tasks form a second reduction task at the process level.

[0158] The number of the second reduction tasks allocated to the first process can be specified by the user or automatically configured by the computing system. For example, the number of the second reduction tasks allocated to the first process can be directly configured to be equal to the number of the first mapping tasks allocated to the first process, which is not limited in the embodiments of the present application.

[0159] For example, as shown in FIG. 8, the M computing nodes allocated to execute the first mapping task include the first computing node, and the first computing node has only one remaining executable process, which is marked as the first process. The management node allocates a first level MR identifier to the first process, such as sub-MR0, and designs n first mapping tasks and n second reduction tasks for sub-MR0, wherein n can be greater than or equal to the number of threads included in the first process. The n first mapping tasks and the n second reduction tasks are allocated to the threads in the first process, that is, the executors of each first mapping task and each second reduction task are determined. In FIG. 8, only two mapping tasks in the n first mapping tasks and two reduction tasks in the n second reduction tasks are shown.

[0160] For example, as shown in FIG. 8, the M computing nodes assigned to execute the first mapping tasks include a second computing node, in which there are two remaining executable processes, labeled as a second process and a third process. The management node assigns a first-level MR identifier to the second process, such as labeled as sub-MR1, and designs m first mapping tasks and m second reduction tasks for sub-MR1, where m can be greater than or equal to the number of threads included in the second process, and assigns the m first mapping tasks and the m second reduction tasks to the threads in the second process. Only 2 of the m first mapping tasks and 2 of the m second reduction tasks are shown in FIG. 8. Meanwhile, the management node assigns a first-level MR identifier to the third process, such as labeled as sub-MR2, and designs w first mapping tasks and w second reduction tasks for sub-MR2, where w can be greater than or equal to the number of threads included in the third process, and assigns the w first mapping tasks and the w second reduction tasks to the threads in the third process. Only 2 of the w first mapping tasks and 2 of the w second reduction tasks are shown in FIG. 8.

[0161] The second level: the second-type sub-MR tasks, where the second-type sub-MR tasks include node-level second reduction tasks and mapping tasks executed between the node-level second reduction tasks and process-level second reduction tasks.

[0162] In designing the second-level MR tasks, for any of the M computing nodes, if the number of remaining executable processes in the computing node is greater than 1, the management node can design the second-level MR tasks for the computing node. Correspondingly, if the number of remaining executable processes in the computing node is equal to 1, the management node can not need to design the second-level MR tasks for the computing node.

[0163] For example, as shown in FIG. 8, the M computing nodes assigned to execute the first mapping tasks include a second computing node, in which there are two remaining executable processes, labeled as a second process and a third process. The management node assigns a first-level MR identifier to the second process, such as labeled as sub-MR1, and designs m first mapping tasks and m second reduction tasks for sub-MR1, where m can be greater than or equal to the number of threads included in the second process, and assigns the m first mapping tasks and the m second reduction tasks to the threads in the second process. Only 2 of the m first mapping tasks and 2 of the m second reduction tasks are shown in FIG. 8. Meanwhile, the management node assigns a first-level MR identifier to the third process, such as labeled as sub-MR2, and designs w first mapping tasks and w second reduction tasks for sub-MR2, where w can be greater than or equal to the number of threads included in the third process, and assigns the w first mapping tasks and the w second reduction tasks to the threads in the third process. Only 2 of the w first mapping tasks and 2 of the w second reduction tasks are shown in FIG. 8.

[0164] For the sub-MR3, p second reduction tasks are designed, p is a positive integer, and the p second reduction tasks are allocated to threads in the second process and the third process, so that the second process and the third process re-execute the node-level second reduction task once again on the execution result of the process-level second reduction task stored in the second memory. Only 4 reduction tasks in the p second reduction tasks are shown in FIG. 8.

[0165] The third level: the second type of MR task, the second type of MR task includes the first reduction task, and the mapping task executed between the node-level second reduction task and the first reduction task.

[0166] For example, as shown in FIG. 8, the M computing nodes allocated to execute the first mapping task include the first computing node and the second computing node, and the N computing nodes allocated to execute the first reduction task include the third computing node and the fourth computing node. There is only one remaining executable process in the first computing node, which is marked as the first process, and there are two remaining executable processes in the second computing node, which are marked as the second process and the third process. There is only one remaining executable process in the third computing node, which is marked as the third process, and there are two remaining executable processes in the fourth computing node, which are marked as the fifth process and the sixth process. The management node allocates a third level MR identifier to all computing nodes (i.e., the M computing nodes and the N computing nodes), as shown in FIG. 8, and the third level MR identifier is marked as sub-MR4.

[0167] For the sub-MR4, a mapping task is designed for the node-level second reduction task executed by each thread in the second process and the third process, so that each thread continues to execute a mapping task after executing the node-level second reduction task once, so as to store the execution result of the node-level second reduction task in the storage system (for example, a disk in FIG. 8) shared by all computing nodes. In addition, a mapping task is designed for the process-level second reduction task executed by each thread in the first process, so that each thread continues to execute a mapping task after executing the process-level second reduction task once, so as to store the execution result of the process-level second reduction task in the storage system shared by all computing nodes.

[0168] Then, for the sub-MR4, q first reduction tasks are designed, q is a positive integer, and the q first reduction tasks are allocated to threads in the fourth process, the fifth process and the sixth process, so that the fourth process, the fifth process and the sixth process re-execute the reduction operation in the global data range on the execution result of the second reduction task stored in the storage system. Only 6 reduction tasks in the q first reduction tasks are shown in FIG. 8.

[0169] In addition, in the scenario where the computing system is a Spark-based computing system, after designing the MR tasks of different levels, the management node can further divide the MR tasks of different levels into stages, so that the management node controls the computing nodes to execute the MR tasks in the corresponding stages in sequence according to the order of the stages.

[0170] For example, in the scenario shown in FIG. 8, the embodiment of the present application provides a stage division diagram shown in FIG. 9.

[0171] Specifically, the mapping tasks (i.e., first mapping tasks) in the first type of sub-MR tasks executed by the threads in the same process are divided into the same stage. As shown in FIG. 9, the mapping tasks in the first type of sub-MR tasks executed by the threads in the first process belong to the same stage, which is marked as stage0 in FIG. 9. The mapping tasks in the first type of sub-MR tasks executed by the threads in the second process belong to the same stage, which is marked as stage1 in FIG. 9. The mapping tasks in the first type of sub-MR tasks executed by the threads in the third process belong to the same stage, which is marked as stage3 in FIG. 9.

[0172] The reduction tasks (i.e., second reduction tasks at the process level) in the first type of sub-MR tasks executed by a process and the mapping tasks in the second type of sub-MR tasks executed by the threads in the same process are divided into the same stage. As shown in FIG. 9, the reduction tasks in the first type of sub-MR tasks executed by the threads in the second process and the mapping tasks in the second type of sub-MR tasks executed by the threads in the second process belong to the same stage, which is marked as stage2 in FIG. 9. The reduction tasks in the first type of sub-MR tasks executed by the threads in the third process and the mapping tasks in the second type of sub-MR tasks executed by the threads in the third process belong to the same stage, which is marked as stage4 in FIG. 9.

[0173] The reduction tasks (i.e., second reduction tasks at the process level) in the first type of sub-MR tasks executed by a process and the mapping tasks in the second type of sub-MR tasks executed by the threads in the same process are divided into the same stage. As shown in FIG. 9, the reduction tasks in the first type of sub-MR tasks executed by the threads in the second process and the mapping tasks in the second type of sub-MR tasks executed by the threads in the second process belong to the same stage, which is marked as stage2 in FIG. 9. The reduction tasks in the first type of sub-MR tasks executed by the threads in the third process and the mapping tasks in the second type of sub-MR tasks executed by the threads in the third process belong to the same stage, which is marked as stage4 in FIG. 9.

[0174] Finally, the first reduction task executed by all threads in the second type of MR task is divided into the same stage. As shown in FIG. 9, the reduction task executed by the threads in the third process, the reduction task executed by the threads in the fourth process, and the reduction task executed by the threads in the fifth process in the second type of MR task belong to the same stage, which is marked as stage 7 in FIG. 9.

[0175] It should be noted that for the first computing node, since only the first process exists in the first computing node, no second level MR task, i.e., the second sub-type of MR task, is designed for the first computing node. In this scenario, the reduction task executed by the threads in the first process in the first sub-type of MR task and the map task executed by the threads in the first process in the second type of MR task are divided into the same stage. As shown in FIG. 8 and FIG. 9, the reduction task executed by the threads in the first process in the first sub-type of MR task and the map task executed by the threads in the first process in the second type of MR task belong to the same stage, which is marked as stage 5 in FIG. 9.

[0176] After the stages are divided in the above manner, the stage relationship in the DAG shown in FIG. 10 can be obtained. In addition, the stage relationship in the sub-task flowchart of the map-reduce task designed according to the association technique is also shown in FIG. 10, where the association technique refers to that the threads in all computing nodes directly write the repartitioned data to the storage system when performing the map task, so as to directly perform the reduce operation of the map-reduce task in the global data range.

[0177] As shown in FIG. 10, in the association technique, since all threads directly perform the MR operation in the global data range, the MR task designed for the map-reduce task only has the first map task and the first reduce task, and correspondingly, the stages only have two stages, which are the map stage and the reduce stage.

[0178] However, in the embodiment of the present application, for any computing node, the reduce operation in the local data range is first performed within the same process of the computing node, then the reduce operation in the local data range is performed between different processes of the computing node, and finally the reduce operation in the global data range is performed between all computing nodes. Therefore, as shown in FIG. 10, for any computing node, the DAG includes one or more stage chains corresponding to the computing node, and each stage chain includes at least three stages in series.

[0179] As can be known from FIG. 9 and FIG. 10, the first computing node corresponds to a stage chain, which includes stage 0, stage 5, and stage 7 in series. The second computing node corresponds to two stage chains, which are the stage chain of stage 1, stage 2, stage 6, and stage 7 in series, and the stage chain of stage 3, stage 4, stage 6, and stage 7 in series.

[0180] wherein the number of stages included in the stage chain corresponding to each computing node indicates the number of second reduction tasks that the computing node needs to perform. Specifically, when the number of stages included in the stage chain is 3, it indicates that the computing node needs to perform one second reduction task, and when the number of stages included in the stage chain is 4, it indicates that the computing node needs to perform two second reduction tasks.

[0181] And for any stage chain corresponding to a computing node, all the tasks inside the stages other than the last stage in the stage chain are only executed inside the computing node, so as to reduce the network communication amount between the computing node and the storage system.

[0182] It should be noted that the above stage division manner is used for example illustration. For example, in the scenario where the computing system is a Spark-based computing system, the tasks can be classified into stages by the above manner, and thus the connection relationship between the stages is directly displayed in the DAG.

[0183] Optionally, in the embodiments of the present application, the stages can also not be divided in the above manner. Or the stages can not be divided, and the execution order of the tasks in the task flowchart can be directly displayed.

[0184] In addition, the above is an example of three levels to illustrate how to design the MR tasks. Optionally, in the embodiments of the present application, only the MR tasks of the first level and the third level can be designed. Or only the MR tasks of the second level and the third level can be designed. Here, the example illustration is not repeated. Or in the scenario where the computing node is only one computing node, only the MR tasks of the first level and the MR tasks of the second level need to be designed.

[0185] After obtaining the task flowchart, the management node can schedule the threads in the multiple computing nodes to execute the tasks in the task flowchart according to the task flowchart. When the computing system is a Spark-based computing system, the process can be implemented by the MR task scheduler in the Spark driver process shown in FIG. 6.

[0186] For example, for the task flow chart shown in FIG. 9, the mapping tasks in stages 0, 1 and 2 can be executed in parallel. After the computing nodes finish executing the mapping tasks in the three stages, the management node schedules the computing nodes to execute the tasks in stage 5, the tasks in stage 2 and the tasks in stage 4, where the tasks in the three stages can be executed in parallel. Then the management node schedules the computing nodes to execute the tasks in stage 6. Finally, the management node schedules the computing nodes to execute the tasks in stage 7.

[0187] The management node can schedule a computing node to execute a task in the task flow chart by issuing a task execution instruction. Details are not described herein.

[0188] In the embodiments of the present application, the operations performed by the computing nodes after the management node schedules the computing nodes according to the task flow chart can be explained by steps 703 and 704.

[0189] Step 703: The first computing node in the M computing nodes executes at least one second reduction task on the X first mapping tasks executed by the first computing node, and stores the execution result of the at least one second reduction task, where X is a positive integer.

[0190] Since the management node also designs at least one second reduction task implemented in each computing node in the M computing nodes, for any computing node in the M computing nodes, such as the first computing node, after the first computing node finishes executing the X first mapping tasks, the first computing node executes at least one second reduction task on the X first mapping tasks executed by the first computing node, so as to implement the operation of first reducing the mapping tasks in the first computing node to obtain the calculation result of the mapping tasks in the local data range.

[0191] Before the first computing node executes the X first mapping tasks, the first computing node needs to read the input data required by the X first mapping tasks from the storage system. For example, in the scenario where the application is a confidential computing application and the computing nodes in the computing system have TEEs, the first computing node reads the input data required by the X first mapping tasks from the storage system in the REE, then moves the read input data from the REE to the TEE, and executes the X first mapping tasks after decrypting the read data in the TEE.

[0192] The first computing node stores the execution result of the at least one second reduction task in a storage system in communication with the first computing node, such as a disk, through a mapping task.

[0193] For example, in a scenario in which the application is a confidential computing application, the computing nodes in the computing system have TEEs. In this scenario, the first computing node executes the at least one second reduction task in the TEE, moves the execution result of the at least one second reduction task from the TEE to the REE after encrypting the execution result in the TEE, and stores the encrypted execution result of the at least one second reduction task in the REE in the storage system.

[0194] Step 704: The N computing nodes execute a first reduction task on the execution result of the at least one second reduction task.

[0195] Since each of the M computing nodes stores the execution result of the at least one second reduction task in a storage system in communication with the corresponding computing node, the N computing nodes can read the repartitioned data in the global data range from the storage system and execute the first reduction task to obtain the output result of the mapping reduction task.

[0196] For example, in a scenario in which the application is a confidential computing application, the computing nodes in the computing system have TEEs. In this scenario, each of the N computing nodes first reads the repartitioned data in the global data range from the storage system in the REE, then moves the read data from the REE to the TEE, and executes the first reduction task after decryption in the TEE.

[0197] In a scenario in which the at least one second reduction task includes a node-level second reduction task, taking the first computing node as an example, it is assumed that the first computing node includes Y processes, and the Y processes are used to execute X first mapping tasks. The implementation manner of the first computing node in the M computing nodes executing the X first mapping tasks to execute the at least one second reduction task can be that, after the Y processes execute the X first mapping tasks, Z processes in the Y processes execute the second reduction task on the execution result of the X first mapping tasks, and Y and Z are positive integers.

[0198] The specific implementation manner has been described in detail in the foregoing embodiments, and will not be described here again.

[0199] Optionally, in the scenario that the at least one second reduction task comprises a process-level second reduction task, taking the first computing node as an example, it is assumed that the first computing node comprises Y processes, and the Y processes are used to execute X first mapping tasks, wherein a first process in the Y processes comprises A threads, and the A threads are used to execute A first mapping tasks. In this scenario, the implementation manner of the first computing node in the M computing nodes performing the at least one second reduction task on the X first mapping tasks executed can be that, after the A threads execute the A first mapping tasks, the second reduction task is executed by B threads in the A threads.

[0200] The specific implementation manner has been described in detail in the foregoing embodiments, and will not be described here.

[0201] Optionally, in the scenario that the at least one second reduction task comprises a process-level second reduction task and a node-level second reduction task, taking the first computing node as an example, it is assumed that the first computing node comprises Y processes, and the Y processes are used to execute X first mapping tasks, wherein a first process in the Y processes comprises A threads, and the A threads are used to execute A first mapping tasks. In this scenario, the implementation manner of the first computing node in the M computing nodes performing the at least one second reduction task on the X first mapping tasks executed can be that, after the A threads execute the A first mapping tasks, the second reduction task is executed by B threads in the A threads; the result of the second reduction task executed by the Z processes in the Y processes on the Y processes is used to execute the second reduction task again.

[0202] To further understand the embodiments of the present application, the steps 701-704 are exemplarily described below taking FIG. 8 as an example.

[0203] In the embodiments of the present application, the management node designs the MR task in the following manner:

[0204] (1) For a single mapping-reduction task in the target application submitted by a user, a first-level MR task, i.e., a first-type sub-MR task, is constructed, which comprises a first mapping task and a process-level second reduction task.

[0205] Specifically, an independent first-level MR identifier is allocated to each remaining executable process j in each computing node in the M computing nodes allocated; the number of mapping tasks and the number of reduction tasks designed for each first-level MR identifier are the same, which can be set as the number of threads included in the remaining executable process j, i.e., execCores, or an integer greater than execCores.

[0206] (2) Then, a second-level MR task, i.e., a second type of sub-MR task, is constructed, including a second reduction task at the node level, and a mapping task executed between the second reduction task at the process level and the second reduction task at the node level.

[0207] Specifically, the identifier nodeID of the computing node to which all the processes j belong is traversed j , nodeID j is removed, and a set of computing nodes NodeIDSet (i.e., a set of M computing nodes) executing the mapping reduction task is obtained. An independent second-level MR identifier is assigned to each computing node k in the set of computing nodes NodeIDSet, where k∈NodeIDSet; the second-level MR identifier designed for the computing node k corresponds to the same number of mapping tasks and reduction tasks, which can be set to execCores*numExec, for example. k , where numExec k is the number of remaining executable processes on the computing node k, which can also be set to an integer greater than execCores*numExec. k

[0208] (3) A third-level MR task, i.e., a second type of MR task, is constructed, including a first reduction task, and a mapping task executed between the second reduction task at the node level and the first reduction task.

[0209] A third-level MR identifier is assigned to all the processes, and the third-level MR identifier corresponds to the same number of mapping tasks and reduction tasks, which can be set to requiredNumExec*execCores, or an integer greater than requiredNumExec*execCores.

[0210] The MR tasks at each level can be obtained through steps (1)-(3). The specific content of each task in steps (1)-(3) is referred to the related content in the foregoing step 702, which will not be described here.

[0211] (4) The mapping task in each level of the MR task is combined with the reduction task in the MR task at the upstream level into a stage (stage), and the mapping task corresponding to each process in the first-level MR task generates a stage alone, and all the reduction tasks corresponding to all the processes in the last-level MR task also generate a stage alone; finally, a DAG is obtained.

[0212] (5) According to the dependency relationship of the tasks specified in the DAG, the tasks in the upstream stage are preferentially scheduled for execution by the computing nodes, until all the tasks in all the stages are completed.

[0213] ​Specifically, the flow of each computing node executing a task is as follows:

[0214] a) For the MR task of the first level, multiple threads in the same process of a computing node cooperatively execute the reduction operation of the local data range, and the data exchange is completed by efficient "inter-thread communication".

[0215] The data exchange in step a) can be understood as follows: different threads in a certain process write the output of the mapping task, i.e., the repartitioned data, into the first memory corresponding to the process, and then different threads in the process read the repartitioned data belonging to the same partition from the first memory when executing the second reduction task at the process level.

[0216] b) For the MR task of the second level, multiple threads in multiple processes of the same computing node cooperatively execute the reduction operation of a larger local data range, and the data exchange is completed by "inter-process communication".

[0217] The data exchange in step b) can be understood as follows: for a certain computing node, threads in all processes of the computing node write the output of the mapping task, i.e., the repartitioned data, into the second memory of the computing node, and then threads in all processes read the repartitioned data belonging to the same partition from the second memory when executing the second reduction task at the node level.

[0218] c) For the MR task of the third level, all threads in all processes across multiple computing nodes cooperatively execute the reduction operation of the global data range, and the data exchange is completed by "external storage system and network adapter". The computing process of the MR task of the third level is the same as that of the associated computing MR in FIG. 10; however, the input data changes, and the amount of input data of the MR task of the third level is greatly reduced after the local merging of the previous two levels.

[0219] The data exchange in step c) can be understood as follows: for all computing nodes, threads in different computing nodes write the output of the mapping task, i.e., the repartitioned data, into the storage system, and then threads in all computing nodes read the repartitioned data belonging to the same partition from the storage system when executing the first reduction task.

[0220] In summary, in the embodiment of the present application, for a MapReduce task in an application, the first computing node of the M computing nodes assigned performs X first mapping tasks, then performs at least one second reduction task on the X first mapping tasks within the first computing node, and finally the N computing nodes assigned perform the first reduction task. In this way, the reduction operation in the local data range is performed on the MapReduce task within the computing node first, and then the reduction operation in the global data range is performed on the MapReduce task between different computing nodes. Since the reduction operation in the local data range is performed on the MapReduce task before the reduction operation in the global data range, the amount of repartitioned data written to the storage system by the M computing nodes is reduced, and accordingly the amount of repartitioned data read from the storage system by the N computing nodes when performing the reduction operation in the global data range is also reduced, thereby reducing the network communication amount between the computing nodes and the storage system, and achieving the effect of improving the performance of the application.

[0221] In addition, as shown in FIG. 11, the embodiment of the present application also provides a computing system, and the computing system 00 includes a management node 10 and a plurality of computing nodes 20.

[0222] The management node 10 is configured to receive a MapReduce task of an application, and the specific implementation manner refers to step 701 in the embodiment of FIG. 7.

[0223] The management node 20 is further configured to assign computing nodes for the MapReduce task, wherein M computing nodes perform first mapping tasks, N computing nodes perform a first reduction task, and M and N are both positive integers; and the specific implementation manner refers to step 702 in the embodiment of FIG. 7.

[0224] The first computing node of the M computing nodes is configured to perform at least one second reduction task on the X first mapping tasks, and store the execution result of the at least one second reduction task, and X is a positive integer; and the specific implementation manner refers to step 703 in the embodiment of FIG. 7.

[0225] The N computing nodes are configured to perform the first reduction task on the execution result of the at least one second reduction task. The specific implementation manner refers to step 704 in the embodiment of FIG. 7.

[0226] Optionally, the first computing node includes Y processes, and the Y processes are configured to perform the X first mapping tasks; and the first computing node is configured to perform the second reduction task on the execution result of the X first mapping tasks through Z processes of the Y processes after the Y processes perform the X first mapping tasks, and Y and Z are both positive integers.

[0227] Optionally, the first computing node comprises Y processes, and the Y processes are used to execute X first mapping tasks, wherein a first process in the Y processes comprises A threads, and the A threads are used to execute A first mapping tasks; and the first computing node is configured to: after the A threads execute the A first mapping tasks, execute a second reduction task on execution results of the A first mapping tasks by B threads in the A threads.

[0228] Optionally, the first computing node comprises Y processes, and the Y processes are used to execute X first mapping tasks, wherein a first process in the Y processes comprises A threads, and the A threads are used to execute A first mapping tasks; and the first computing node is configured to: after the A threads execute the A first mapping tasks, execute a second reduction task on execution results of the A first mapping tasks by B threads in the A threads; and execute the second reduction task again on results of executing the second reduction task by Z processes in the Y processes.

[0229] Optionally, the management node is further configured to: after allocating M computing nodes for the mapping reduction task, set a number of times of executing the second reduction task for each of the M computing nodes, and instruct the M computing nodes to execute the second reduction task according to the specified number of times of executing the second reduction task.

[0230] Optionally, the first computing node comprises a trusted execution environment (TEE) and a rich execution environment (REE); and the first computing node is further configured to: decrypt data required by the first computing node for executing the first mapping tasks after the data is moved from the REE to the TEE; and encrypt execution results of at least one second reduction task after the first computing node executes the at least one second reduction task, and move the execution results from the TEE to the REE and store the execution results.

[0231] In summary, in the embodiments of the present application, for the mapping reduction task in an application, a first computing node in M computing nodes allocated for the mapping reduction task executes X first mapping tasks, executes at least one second reduction task on the X first mapping tasks in the first computing node, and finally executes a first reduction task by N computing nodes allocated for the first reduction task. In this way, the mapping reduction task is first executed in a local data range in the computing node, and then executed in a global data range between different computing nodes. Since the mapping reduction task is executed in the local data range before being executed in the global data range, the amount of repartitioned data written to the storage system by the M computing nodes is reduced, and accordingly the amount of repartitioned data read from the storage system by the N computing nodes when performing the reduction operation in the global data range is also reduced, thereby reducing the network communication amount between the computing nodes and the storage system, and achieving the effect of improving the performance of the application.

[0232] It should be noted that the computing system provided in the foregoing embodiments is only used as an example to illustrate the division of the nodes in performing the task, and in actual application, the above-mentioned function distribution can be completed by different nodes according to the needs, that is, the internal structure of the system is divided into different nodes to complete all or part of the functions described above. In addition, the computing system and the task execution method provided in the foregoing embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be described here.

[0233] FIG. 12 is a structural schematic diagram of a computer device provided in an embodiment of the present application. The management node and the computing node in the computing system in the foregoing embodiments can be implemented by one or more computer devices shown in FIG. 12. Referring to FIG. 12, the computer device includes at least one processor 1201, a communication bus 1202, a memory 1203, and at least one communication interface 1204.

[0234] The processor 1201 can be a general central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of programs of the present application.

[0235] The communication bus 1202 can include a path for transmitting information between the above-mentioned components.

[0236] The memory 1203 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, and can also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, an optical disk storage (including a compact disk, a laser disk, an optical disk, a digital versatile disk, a Blu-ray disk, etc.), a magnetic disk or other magnetic storage device, or any other medium capable of carrying or storing desired program codes in the form of instructions or data structures and capable of being accessed by a computer, but is not limited to this. The memory 1203 can exist independently and be connected to the processor 1201 through the communication bus 1202. The memory 1203 can also be integrated with the processor 1201.

[0237] The memory 1203 is configured to store program codes for implementing the solutions of the present application, and the processor 1201 is configured to execute the program codes stored in the memory 1203. The program codes can include one or more software modules. The management node and the computing node in the foregoing embodiments can determine the data for developing the application by using the processor 1201 and one or more software modules in the program codes in the memory 1203.

[0238] The communication interface 1204 is configured to communicate with other devices or communication networks, such as an Ethernet, a radio access network (RAN), a wireless local area networks (WLAN), and the like, by using any transceiver-like device.

[0239] In specific implementations, as an example, the computer device can include multiple processors, such as the processor 1201 and the processor 1205 shown in FIG. 12. Each of the processors can be a single-CPU processor or a multi-CPU processor. The processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0240] In specific implementations, as an example, the computer device can further include an output device 1206 and an input device 1207. The output device 1206 communicates with the processor 1201 and can display information in various ways. For example, the output device 1206 can be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, a projector, or the like. The input device 1207 communicates with the processor 1201 and can receive user input in various ways. For example, the input device 1207 can be a mouse, a keyboard, a touch screen device, a sensing device, or the like.

[0241] The computer device described above can be a general-purpose computer device or a special-purpose computer device. In specific implementations, the computer device can be a desktop computer, a laptop computer, a network server, a personal digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. The embodiments of the present application do not limit the type of the computer device.

[0242] In the above embodiments, all or part of the steps can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the steps can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (for example: coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example: infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media sets. The available media can be magnetic media (for example: floppy disk, hard disk, magnetic tape), optical media (for example: digital versatile disc (DVD)), or semiconductor media (for example: solid state disk (SSD)) and the like.

[0243] A person of ordinary skill in the art can understand that all or part of the steps of the above embodiments can be completed by hardware, or by a program instructing related hardware, and the program can be stored in a computer readable storage medium, and the storage medium mentioned above can be a read only memory, a magnetic disk or an optical disk, etc.

[0244] The above is not intended to limit the embodiments of the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiments of the present application shall be included in the protection scope of the embodiments of the present application.

Claims

1. A task execution method characterized by, The method comprises: receiving a MapReduce task of an application; allocating computing nodes for the MapReduce task, wherein M computing nodes perform a first mapping task and N computing nodes perform a first reduction task, M and N are positive integers; a first computing node of the M computing nodes performs at least one second reduction task on X first mapping tasks performed by the first computing node, and stores the execution result of the at least one second reduction task, X is a positive integer; the N computing nodes perform the first reduction task on the execution result of the at least one second reduction task.

2. The method of claim 1, wherein, The first computing node comprises Y processes for executing the X first mapping tasks; the first computing node of the M computing nodes performs at least one second reduction task on X first mapping tasks performed by the first computing node, comprising: after the Y processes complete the execution of the X first mapping tasks, the Z processes of the Y processes perform the second reduction task on the execution result of the X first mapping tasks, Y and Z are positive integers.

3. The method of claim 1, wherein, The first computing node comprises Y processes for executing the X first mapping tasks, wherein a first process of the Y processes comprises A threads for executing A first mapping tasks; the first computing node of the M computing nodes performs at least one second reduction task on X first mapping tasks performed by the first computing node, comprising: after the A threads complete the execution of the A first mapping tasks, the B threads of the A threads perform the second reduction task on the execution result of the A first mapping tasks.

4. The method of claim 1, wherein, The first computing node comprises Y processes for executing the X first mapping tasks, wherein a first process of the Y processes comprises A threads for executing A first mapping tasks; the first computing node of the M computing nodes performs at least one second reduction task on X first mapping tasks performed by the first computing node, comprising: after the A threads complete the execution of the A first mapping tasks, the B threads of the A threads perform the second reduction task on the execution result of the A first mapping tasks; the Y processes perform the second reduction task on the result of the second reduction task performed by the Z processes of the Y processes.

5. The method according to any one of claims 1 to 4, characterized in that, The method further comprises: after the M computing nodes are allocated for the MapReduce task, the number of times of performing the second reduction task is set for each of the M computing nodes, and the M computing nodes are instructed to perform the second reduction task according to the specified number of times of performing the second reduction task.

6. The method of any one of claims 1-5, wherein, The first computing node comprises a trusted execution environment (TEE) and a rich execution environment (REE); the method further comprises: decrypting the data required by the first computing node to perform the first mapping task after the data is moved from the REE to the TEE; after the first computing node completes the execution of the at least one second reduction task, the execution result is encrypted and moved from the TEE to the REE and stored.

7. A computing system, comprising: The computing system comprises a management node and a plurality of computing nodes; The management node is configured to receive a MapReduce task of an application; The management node is further configured to allocate computing nodes for the MapReduce task, wherein M computing nodes execute a first mapping task and N computing nodes execute a first reduction task, and M and N are positive integers; A first computing node in the M computing nodes is configured to execute at least one second reduction task on X first mapping tasks executed by the first computing node, and store an execution result of the at least one second reduction task, and X is a positive integer; The N computing nodes are configured to execute the first reduction task on the execution result of the at least one second reduction task.

8. The computing system of claim 7, wherein, The first computing node comprises Y processes configured to execute the X first mapping tasks; The first computing node is configured to: After the Y processes execute the X first mapping tasks, execute the second reduction task on the execution result of the X first mapping tasks by Z processes in the Y processes, and Y and Z are positive integers.

9. The computing system of claim 7, wherein, The first computing node comprises Y processes configured to execute the X first mapping tasks, wherein a first process in the Y processes comprises A threads configured to execute A first mapping tasks; The first computing node is configured to: After the A threads execute the A first mapping tasks, execute the second reduction task on the execution result of the A first mapping tasks by B threads in the A threads.

10. The computing system of claim 7, wherein, The first computing node comprises Y processes configured to execute the X first mapping tasks, wherein a first process in the Y processes comprises A threads configured to execute A first mapping tasks; The first computing node is configured to: After the A threads execute the A first mapping tasks, execute the second reduction task on the execution result of the A first mapping tasks by B threads in the A threads; Execute the second reduction task on the result of executing the second reduction task by Z processes in the Y processes.

11. The computing system of any of claims 7-10, wherein, The management node is further configured to: After the M computing nodes are allocated for the MapReduce task, set the number of times of executing the second reduction task for each of the M computing nodes, and instruct the M computing nodes to execute the second reduction task according to the specified number of times of executing the second reduction task.

12. The computing system of any of claims 7-11, wherein, The first computing node comprises a trusted execution environment (TEE) and a rich execution environment (REE); and the first computing node is further configured to: Decrypt data required by the first computing node for executing the first mapping task after the data is moved from the REE to the TEE; After the first computing node executes the at least one second reduction task, encrypt the execution result, move the encrypted execution result from the TEE to the REE, and store the encrypted execution result.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that, when executed by a plurality of computing devices, cause the plurality of computing devices to perform the method of any of claims 1-6.

14. A computer program product, characterised in that, The computer program product, when executed on a plurality of computing devices, causes the plurality of computing devices to implement the method of any of claims 1-6.

Images