Task execution method, computing system, storage medium, and product

Abstract

Embodiments of the present invention relate to the technical field of distributed computing, and disclosed are a task execution method, a computing system, a storage medium, and a product. In the embodiments of the present application, for a map-reduce task in an application, after executing X first map tasks, a first computing node among allocated M computing nodes executes at least one second reduce task on the X first map tasks in the first computing node, and finally, allocated N computing nodes execute a first reduce task. A reduce operation within a local data range is first performed on a map-reduce task in a computing node before performing a reduce operation within a global data range, such that the amount of network communication between the computing node and a storage system can be reduced, thereby achieving the effect of improving application performance.

Description

Task execution methods, computing systems, storage media and products

[0001] This application claims priority to Chinese Patent Application No. 202410703790.3, filed on May 31, 2024, entitled “Task Execution Method, Computing System, Storage Medium and Product”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of distributed computing technology, and in particular to a task execution method, computing system, storage medium and product. Background Technology

[0003] Currently, Spark-based computing systems, when receiving applications submitted by clients, divide the execution process of MapReduce tasks into two phases: mapping and reduction. Each phase includes a large number of tasks that can be executed in parallel. In the mapping phase, each task repartitions the input data and writes the repartitioned data to a disk file on the I / O device. In the reduction phase, each task reads the repartitioned data belonging to the same partition from the disk file over the network and then performs the corresponding reduction operation (such as a merge operation), thereby improving the execution efficiency of the mapping-reduction task.

[0004] However, during the mapping and reduction tasks in the aforementioned application, when the data volume of the Map task is large, I / O operations are required to read and write data from disk files, which leads to a significant reduction in application performance. This problem is particularly prominent in scenarios with limited computing resources, such as confidential computing scenarios. Summary of the Invention

[0005] This application provides a task execution method, computing system, storage medium, and product, which can reduce the network communication volume between computing nodes and storage systems during the execution of mapping reduction tasks, thereby improving the end-to-end performance of applications. The technical solution is as follows:

[0006] Firstly, a task execution method is provided, in which: a MapReduce task of an application is received; computing nodes are allocated to the mapping task, wherein M computing nodes execute the first mapping task and N computing nodes execute the first reduction task, where M and N are both positive integers; the first computing nodes among the M computing nodes execute the second reduction task at least once for the X first mapping tasks and store the execution result of the at least one second reduction task, where X is a positive integer; and the N computing nodes execute the first reduction task based on the execution result of the at least one second reduction task.

[0007] In this embodiment, for the mapping and reduction task in the application, after the first computing node among the allocated M computing nodes completes X first mapping tasks, it performs at least one second reduction task on the X first mapping tasks within the first computing node. Finally, the allocated N computing nodes execute the first reduction task. This allows for the mapping and reduction task to first perform a local data-scope reduction operation within the computing node, and then perform a global data-scope reduction operation on the mapping and reduction task across different computing nodes. Since the mapping and reduction task is reduced within a local data scope before performing the global data-scope reduction operation, the amount of repartitioned data written to the storage system by the M computing nodes is reduced. Correspondingly, the amount of repartitioned data read from the storage system by the N computing nodes when performing the global data-scope reduction operation is also reduced, thereby reducing the network communication volume between the computing nodes and the storage system and improving application performance.

[0008] The global data range refers to the global data range of the input data for the mapping and reduction task, while the local data range refers to the local data range of the input data for the mapping and reduction task.

[0009] Based on the method provided in the first aspect, in one possible implementation, the first computing node includes Y processes, which are used to execute X first mapping tasks. In this scenario, the implementation process of the first computing node among M computing nodes performing a second reduction task at least once on the X first mapping tasks is as follows: after the Y processes have completed the execution of the X first mapping tasks, the second reduction task is performed on the execution results of the X first mapping tasks by Z processes among the Y processes, where Y and Z are both positive integers.

[0010] The above implementation method allows a second reduction task to be executed between different processes on the same computing node, enabling the mapping reduction task to perform a reduction operation within a local data range within the computing node, thereby reducing the amount of network communication between the computing node and the storage system.

[0011] Based on the method provided in the first aspect, in one possible implementation, the first computing node includes Y processes, which are used to execute X first mapping tasks. The first process among the Y processes includes A threads, which are used to execute A first mapping tasks. In this scenario, the implementation process of the first computing node among M computing nodes executing a second reduction task at least once on the X first mapping tasks is as follows: after the A threads have completed the execution of the A first mapping tasks, the second reduction task is executed on the execution results of the A first mapping tasks by B threads among the A threads.

[0012] The above implementation method allows a second reduction task to be executed once within the same process on the same computing node. This enables the reduction operation within a local data range to be performed on the mapping reduction task within the process of the computing node, thereby reducing the amount of network communication between the computing node and the storage system.

[0013] Based on the method provided in the first aspect, in one possible implementation, the first computing node includes Y processes, which are used to execute X first mapping tasks. The first process among the Y processes includes A threads, which are used to execute A first mapping tasks. In this scenario, the implementation process of the first computing node among M computing nodes executing at least one second reduction task on the X first mapping tasks is as follows: after the A threads finish executing the A first mapping tasks, the B threads among the A threads execute a second reduction task on the execution result of the A first mapping tasks; and the Z processes among the Y processes execute the second reduction task on the result of the second reduction task executed by the Y processes execute another second reduction task.

[0014] The above implementation method allows for the execution of the first second reduction task within the same process on the same computing node, followed by the execution of the second second reduction task between different processes on the same computing node. This enables the mapping reduction task to perform two consecutive reduction operations within a local data range within the computing node, thereby reducing the network communication volume between the computing node and the storage system.

[0015] Based on the method provided in the first aspect, in one possible implementation, after allocating M computing nodes for the mapping reduction task, the method further sets the number of times the second reduction task is executed for each of the M computing nodes, and instructs the M computing nodes to execute the second reduction task according to the specified number of times the second reduction task is executed.

[0016] To facilitate subsequent scheduling of computing nodes, the number of times the second reduction task can be executed can be set for each of the M computing nodes, thereby improving the efficiency of subsequent scheduling of computing nodes.

[0017] Based on the method provided in the first aspect, in one possible implementation, the first computing node includes a Trusted Execution Environment (TEE) and a Rich Execution Environment (REE). In this scenario, in the method, the data required by the first computing node to execute the first mapping task is moved from the REE to the TEE and then decrypted. After the first computing node completes at least one second reduction task, the execution result is encrypted and then moved from the TEE to the REE for storage.

[0018] In scenarios where the first computing node has a TEE (Transmission Equipment), the application example is a confidential computing application. In this scenario, the first computing node encrypts and transfers the execution results of at least one second protocol task from the TEE to the REE (Resource Element), and then stores the encrypted execution results in the storage system connected to the first computing node via the REE, only after completing at least one second protocol task. Compared to the first computing node directly encrypting and transferring the execution results of the first mapping task from the TEE to the REE for storage after completing the first mapping task, the method provided in this application not only reduces the amount of data stored by the first computing node in the REE to the storage system, i.e., reduces the network communication volume between the first computing node and the storage system, but also reduces the amount of data transferred from the TEE to the REE, i.e., reduces the communication volume between the TEE and the REE within the first computing node. Therefore, the method provided in this application is more effective in confidential computing applications.

[0019] Secondly, a computing system is provided, which includes a management node and multiple computing nodes.

[0020] The system comprises: a management node for receiving MapReduce tasks from the application; a management node for allocating compute nodes to the mapping task, wherein M compute nodes execute the first mapping task and N compute nodes execute the first reduction task, where M and N are both positive integers; the first compute node among the M compute nodes is used to execute at least one second reduction task on the X first mapping tasks and store the execution result of at least one second reduction task, where X is a positive integer; and the N compute nodes are used to execute the first reduction task on the execution result of at least one second reduction task.

[0021] Based on the system provided in the second aspect, in one possible implementation, the first computing node includes Y processes, which are used to execute X first mapping tasks; in this scenario, the first computing node is used for:

[0022] After Y processes have completed X first mapping tasks, Z processes out of the Y processes execute the second reduction task based on the execution results of the X first mapping tasks, where Y and Z are both positive integers.

[0023] Based on the system provided in the second aspect, in one possible implementation, the first computing node includes Y processes, which are used to execute X first mapping tasks. Each of the Y processes includes A threads, which are used to execute A first mapping tasks. In this scenario, the first computing node is used for:

[0024] After A threads complete A first mapping tasks, B threads among A threads execute second reduction tasks based on the execution results of A first mapping tasks.

[0025] Based on the system provided in the second aspect, in one possible implementation, the first computing node includes Y processes, which are used to execute X first mapping tasks. Each of the Y processes includes A threads, which are used to execute A first mapping tasks. In this scenario, the first computing node is used for:

[0026] After A threads complete A first mapping tasks, B threads among A threads execute a second reduction task based on the execution results of A first mapping tasks.

[0027] The second reduction task is then executed again using the result of the second reduction task performed by Z processes out of Y processes.

[0028] Based on the system provided in the second aspect, in one possible implementation, the management node is also used for:

[0029] After allocating M computing nodes for the mapping reduction task, the execution count of the second reduction task is set for each of the M computing nodes, and the M computing nodes are instructed to execute the second reduction task according to the specified execution count of the second reduction task.

[0030] Based on the system provided in the second aspect, in one possible implementation, the first computing node includes a Trusted Execution Environment (TEE) and a Rich Execution Environment (REE). In this scenario, the first computing node is also used to: move the data required by the first computing node to execute the first mapping task from the REE to the TEE and then decrypt it; and after the first computing node has executed at least one second reduction task, encrypt the execution result and then move it from the TEE to the REE for storage.

[0031] Thirdly, a computer-readable storage medium is provided that stores program instructions, which, when executed by a plurality of computing devices, cause the plurality of computing devices to perform any of the task execution methods described in the first aspect.

[0032] Fourthly, a computer program product is provided, which, when run on multiple computing devices, causes the multiple computing devices to implement any of the task execution methods described in the first aspect.

[0033] The technical effects achieved by the corresponding technical means in the second to fourth aspects mentioned above are similar, and will not be repeated here. Attached Figure Description

[0034] Figure 1 is a schematic diagram of the architecture of a confidential computing system based on TEE provided in an embodiment of this application;

[0035] Figure 2 is a schematic diagram of the architecture of PPML provided in an embodiment of this application;

[0036] Figure 3 is a schematic diagram of a task designed for a certain mapping reduction task in PPML as shown in Figure 2, provided by an embodiment of this application;

[0037] Figure 4 is a schematic diagram of the architecture of a computing system provided in an embodiment of this application;

[0038] Figure 5 is a schematic diagram of the architecture of a Spark-based computing system provided in an embodiment of this application;

[0039] Figure 6 is a flowchart of a computing system shown in Figure 5 provided in an embodiment of this application;

[0040] Figure 7 is a flowchart of a task execution method provided in an embodiment of this application;

[0041] Figure 8 is a task distribution flowchart provided in an embodiment of this application;

[0042] Figure 9 is a schematic diagram of a stage division provided in an embodiment of this application;

[0043] Figure 10 is a schematic diagram of the stage relationship in a DAG provided in an embodiment of this application;

[0044] Figure 11 is a schematic diagram of the architecture of another computing system provided in an embodiment of this application;

[0045] Figure 12 is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0047] Before providing a detailed explanation of the embodiments of this application, the application scenarios involved in the embodiments of this application will be explained first.

[0048] When a computing system receives an application that includes a mapping-induction task, it can complete the computation of that task through mapping and reduction operations. Specifically, when the computing nodes in the system have a trusted execution environment (TEE), the system can perform the computation of mapping-induction tasks for confidential computing applications. When the computing nodes in the system do not have a TEE, i.e., only a rich execution environment (REE), the system can perform the computation of mapping-induction tasks for plaintext computing applications.

[0049] Among them, confidential computing applications refer to the following: when a computing node performs a task in an application, the data read from the storage system is encrypted. This encrypted data needs to be moved from the REE of the computing node to the TEE, decrypted in the TEE, and then calculated. After the calculation, the calculation result is encrypted and moved from the TEE to the REE. Finally, the encrypted calculation result is stored in the storage system through the REE.

[0050] Plaintext computing applications refer to computing nodes reading unencrypted plaintext data from the storage system when performing tasks in the application. This plaintext data can be directly computed in the REE, and the computation results are directly stored in the storage system through the REE.

[0051] The following example uses a computing system with a TEE to illustrate the application scenarios of this application.

[0052] The total amount of global data is growing exponentially. However, this data may be managed by multiple organizations and institutions and is subject to privacy laws and policies, resulting in multi-party privacy data. Because privacy data from different parties cannot be shared, "data silos" are formed, severely hindering the realization of the value of privacy data. In some scenarios, confidential computing can be used to achieve secure computation of multi-party privacy data across the entire chain, ensuring the security of privacy data in the storage, communication, and computation stages. This makes cross-domain collaborative analysis of privacy data possible, helping clients improve the accuracy of data analysis and bring greater business success.

[0053] Among these, confidential computing technology based on a Trusted Execution Environment (TEE) is an important technological direction in confidential computing. TEE-based confidential computing technology uses methods such as hardware encryption and hardware isolation to securely protect private data during the central processing unit (CPU) computation phase. Furthermore, TEE-based confidential computing technology possesses characteristics such as high computational efficiency, flexible deployment, and high scalability.

[0054] Figure 1 is a schematic diagram of the architecture of a confidential computing system based on a TEE provided in an embodiment of this application. As shown in Figure 1, the computing system includes a client, a confidential computing cluster, and multiple participants, which may be input / output (I / O) devices such as disks or network cards. The confidential computing cluster includes multiple computing nodes and a management node, and each computing node is equipped with a TEE.

[0055] In the confidential computing system shown in Figure 1, users submit confidential computing applications to the management node of the confidential computing cluster via a client. The management node schedules at least one computing node to execute tasks within the confidential computing application. When executing a task within the confidential computing application, each computing node needs to retrieve encrypted data from I / O devices, perform data decryption and analysis within its TEE (Transparent Environment Equipment), and ultimately obtain the computation result.

[0056] Since the TEE itself does not have the ability to operate external I / O devices, if a computing node wants to operate external I / O devices within the TEE, it needs to rely on the user-space library operating system (LibOS) to transfer the I / O instructions and data within the TEE to the rich execution environment (REE) side, and then the REE side will operate the external I / O devices. This will increase the additional REE / TEE data transmission and data encryption / decryption, resulting in an increase in the time from when the user submits the confidential computing application to when the client obtains the computing result of the confidential computing application, that is, poor end-to-end performance of the confidential computing application.

[0057] Currently, examples of confidential computing technologies based on TEE include: TrustZone technology, Software Guard Extensions (SGX) technology, and Secure Encrypted Virtualization (SEV) technology.

[0058] PPML is a distributed privacy-preserving machine learning platform based on SGX technology. This platform protects confidential computing data at every stage of the end-to-end process, including data input, data analysis, and machine learning. For example, PPML allows open-source big data analytics engines like Spark to run directly within a TEE (Transactional Environment), thus protecting privacy data.

[0059] Figure 2 is a schematic diagram of the architecture of PPML provided in an embodiment of this application. As shown in Figure 2, Spark in PPML includes a Spark driver process and Spark executors. For a confidential computing application submitted by a client, the computing system starts the Spark driver process and multiple Spark executors corresponding to the confidential computing application. The Spark driver process is used to schedule the Spark executors to execute tasks in the confidential computing application.

[0060] As shown in Figure 2, both the Spark driver process and the Spark execution process run in TEE. The Spark driver process can communicate with the Spark execution process in encrypted form through LibOS, and the Spark execution process can communicate with I / O devices in encrypted form through LibOS, thereby ensuring the security of data in confidential computing applications.

[0061] Spark is a distributed computing engine based on the MapReduce (MR) computing model. For any MapReduce task in a confidential computing application, the execution process is divided into two stages: mapping and reduction. Each stage includes a large number of tasks that can be executed in parallel. In the mapping stage, each task maps the input data corresponding to that task into key-value pairs and determines the partition identifier of the key-value pairs; this process can be called repartitioning the input data. In this embodiment, the output results of the mapping stage tasks (including key-value pairs and partition identifiers) are collectively referred to as the repartitioned data of the input data. After obtaining the repartitioned data of the input data, the mapping stage tasks write the repartitioned data to the disk file of the I / O device. In the reduction stage, each task corresponds to a partition identifier. Each task in the reduction stage reads the repartitioned data including the corresponding partition identifier from the disk file via the network and then performs the reduction operation (such as a merge operation) corresponding to the mapping reduction task to reduce key-value pairs belonging to the same partition, thereby improving the execution efficiency of the mapping reduction task.

[0062] In this embodiment of the application, the tasks in the mapping phase are referred to as mapping tasks, and the tasks in the reduction phase are referred to as reduction tasks.

[0063] Figure 3 is a schematic diagram of a task designed for a specific mapping and reduction task in PPML as shown in Figure 2, according to an embodiment of this application. As shown in Figure 3, one or two processes run on each computing node, and each process includes two threads. This process is, for example, a Spark execution process. Each thread is used to execute the mapping task or reduction task in the mapping and reduction task.

[0064] As shown in Figure 3, each mapping task involves: the compute node reading the input data assigned to the mapping task from the disk file in the REE, moving the read input data from the REE to the TEE (this step is not shown in Figure 3), decrypting the input data assigned to the mapping task in the TEE, performing the mapping operation, encrypting the result of the mapping operation (i.e., the repartitioned data), moving it from the TEE to the REE, and then writing the encrypted repartitioned data to the disk file in the REE. Each reduction task involves: the compute node reading the repartitioned data belonging to the same partition from all disk files in the REE via the network card, moving the read repartitioned data from the REE to the TEE, decrypting the read repartitioned data in the TEE, performing the reduction operation, and obtaining the output result of the reduction task.

[0065] During the aforementioned mapping and reduction task, the computing nodes not only need to perform I / O operations to read and write data from disk files, but also need to move data between the REE and TEE. Therefore, when the input data volume of the mapping and reduction task is large, the performance of confidential computing applications lags significantly behind that of plaintext computing applications.

[0066] Based on this, this application provides a task execution method. In this application embodiment, for a mapping and reduction task in an application, after the first computing node among the allocated M computing nodes completes X first mapping tasks, it performs at least one second reduction task on the X first mapping tasks within the first computing node. Finally, the allocated N computing nodes execute the first reduction task. This allows for the mapping and reduction task to first perform a local data-scope reduction operation within the computing node, and then perform a global data-scope reduction operation on the mapping and reduction task across different computing nodes. Since the mapping and reduction task is reduced within a local data scope before performing the global data-scope reduction operation, the amount of repartitioned data written to the storage system by the M computing nodes is reduced. Correspondingly, the amount of repartitioned data read from the storage system by the N computing nodes when performing the global data-scope reduction operation is also reduced, thereby reducing the network communication volume between the computing nodes and the storage system and improving application performance.

[0067] The global data range refers to the global data range of the input data for the mapping and reduction task, while the local data range refers to the local data range of the input data for the mapping and reduction task.

[0068] It should be noted that the method provided in this application can be applied to both confidential computing applications and plaintext computing applications. When applied to confidential computing applications, each computing node has a TEE (Transaction Execution Environment). In this scenario, the first computing node encrypts and transfers the execution result of at least one second reduction task from the TEE to the REE (Resource Element), and then stores the encrypted execution result in the storage system connected to the first computing node via the REE, only after completing at least one second reduction task. Compared to the first computing node directly encrypting and transferring the execution result of the first mapping task from the TEE to the REE for storage after completing the first mapping task, the method provided in this application not only reduces the amount of data stored by the first computing node in the REE to the storage system (i.e., reduces network communication between the first computing node and the storage system), but also reduces the amount of data transferred from the TEE to the REE (i.e., reduces communication between the TEE and the REE within the first computing node). Therefore, the method provided in this application is more effective in confidential computing applications.

[0069] The architecture of the computing system and the task execution method provided in the embodiments of this application will be explained below.

[0070] Figure 4 is a schematic diagram of the architecture of a computing system provided in an embodiment of this application. As shown in Figure 4, the computing system includes a management node 10, multiple computing nodes 20, and a storage system 30. The management node 10 is connected to each computing node 20 for communication, and each computing node 20 is also connected to the storage system 30 for communication.

[0071] As shown in Figure 4, the management node 10 also connects to clients outside the computing system for communication. Clients are used to submit applications to the management node 10; these applications can be either confidential or plaintext computing applications. In the scenario where the application is a confidential computing application, each computing node 20 also possesses a TEE (Transaction Execution Environment) to execute tasks of the confidential computing application within the TEE. In this scenario, the computing system illustrated in Figure 4 typically includes the confidential computing cluster shown in Figure 1 and the various participating parties.

[0072] When the management node 10 receives an application from the client, it first determines which computing nodes 20 can be used to execute the application's tasks based on the application's resource requirements information, that is, it requests computing nodes 20 that can execute the application.

[0073] In this process, the management node 10 can first determine the remaining available resources of each computing node 20 among the multiple computing nodes 20, and then determine which computing nodes 20 are used to execute the application based on the application's resource requirements and the remaining available resources of each computing node 20. The relevant implementation methods are described in detail in the subsequent method embodiments, and will not be elaborated here.

[0074] The resource requirement information indicates the specifications of the computing resources required to execute the application. For example, the resource requirement information includes the total number of processes required by the application, the process specifications for each process, such as the number of threads within the process and the memory size required by the process. In this scenario, the remaining available resources for each compute node 20 can, for example, be the remaining executable processes on the compute node.

[0075] Additionally, application resource requirements can be obtained by the user through application analysis. For example, the user can analyze each task in the application's code to obtain the resource requirements for each task. The task's resource requirements include the number of processes required to execute the task, the number of threads required per process, and the memory size required by a single process. Then, the resource requirements with the highest demand are selected as the application's resource requirements. This ensures that the determined resource requirements cover the needs of each task. Optionally, users can also determine the application's resource requirements through other methods, which will not be illustrated here.

[0076] After obtaining a computing node 20 for executing the application, management node 10 can formulate a task flowchart for the application based on the remaining available resources on the requested computing node 20. The task flowchart indicates the execution order of each task and which thread on which process of which computing node each task will be executed. Subsequently, management node 10 can schedule the requested computing node 20 to execute each task sequentially according to the task flowchart, thereby obtaining the computation results of the application.

[0077] In this embodiment, the application task submitted by the user to the client is a mapping and induction task. Therefore, in this embodiment, the computing system shown in Figure 4 can also be called a MapReduce distributed computing engine or a MapReduce distributed computing framework. The basic principles of mapping reduction have been explained above and will not be repeated here.

[0078] Furthermore, the storage system 30 in Figure 4 can be implemented by a single centralized storage node, or alternatively, by different storage nodes deployed in a distributed manner; this embodiment of the application does not limit this. For example, the storage system 30 can be a storage system consisting of at least one disk. Each computing node can be connected to the storage system via an Ethernet network interface card (NIC).

[0079] Furthermore, the computing system shown in Figure 4 can be a Spark-based computing system. Alternatively, it can be a computing system based on other mapping reduction computing technologies, such as a computing system based on Hadoop MapReduce (an open-source distributed computing framework), which will not be illustrated here.

[0080] The following section will further elaborate on the computing system shown in Figure 4, using a Spark-based computing system as an example.

[0081] Figure 5 is a schematic diagram of the architecture of a Spark-based computing system provided in an embodiment of this application. As shown in Figure 5, the computing system includes a Spark client, a Spark management node (master), multiple Spark worker nodes, and Hadoop distributed file system (HDFS)1 and HDFS2.

[0082] The computing system shown in Figure 5 can be applied to plaintext computing applications. Optionally, the computing system shown in Figure 5 can also be used to process confidential computing applications. In this scenario, as shown in Figure 5, the computing system also includes a remote authentication service.

[0083] The working principle of the computing system will be explained below using the example of the computing system shown in Figure 5 processing confidential computing applications.

[0084] In the scenario shown in Figure 5, where the computing system is used to process confidential computing applications, the Spark management node and Spark worker nodes are started in the TEE environment during the initialization of the computing system, that is, during the deployment phase of the computing system. The remote proof service, HDFS1, and HDFS2 are also in an available state.

[0085] In the distributed computing system shown in Figure 5, the application execution flow is as follows:

[0086] (1) A user submits a data analysis application to the Spark management node in the Spark client. This data analysis application is a confidential computing application.

[0087] (2) When the Spark management node receives the data analysis application, it starts the Spark driver process for the data analysis application. The Spark driver process can be deployed on the Spark management node or on a Spark worker node. Figure 5 shows the deployment on the Spark management node as an example. The Spark driver process sends a resource request for the data analysis application to the resource scheduler built into the Spark management node. When the resource scheduler determines which Spark worker nodes can execute the data analysis application, it sends the information of the requested Spark worker nodes to the Spark driver process. The Spark driver process starts the Spark execution process on the requested Spark worker node so that the tasks of the data analysis application can be executed through the Spark execution process on the Spark worker node later.

[0088] (3) The Spark driver process and the Spark execution process, through remote verification service, ensure that the program in the data analysis application has not been modified and is currently in TEE.

[0089] (4) The Spark execution process obtains the encrypted data from each party through the Hadoop Distributed File System (HDFS) HDFS1 and HDFS2, decrypts the data in the TEE, and completes the mapping and reduction tasks specified by the data analysis application. The calculation results are saved in encrypted form to persistent media, such as HDFS1 and HDFS2, or stored on the local disk.

[0090] In the computing system shown in Figure 5, the Spark client is used to implement the functions of the client in Figure 4, the Spark worker nodes with Spark execution processes are used to implement the functions of the computing nodes in Figure 4, the Spark management nodes and Spark driver processes are used to implement the functions of the management nodes in Figure 4, and HDFS1 and HDFS2 are used to implement the functions of the storage system in Figure 4.

[0091] Figure 6 is a flowchart of a computing system shown in Figure 5 provided in an embodiment of this application.

[0092] As shown in Figure 6, the Spark management node includes a resource scheduler, the Spark worker node includes a resource controller, the Spark driver process includes a MapReduce execution plan generator and a MapReduce task scheduler, and the Spark execution process includes a MapReduce task executor.

[0093] The resource scheduler is used to request Spark worker nodes for executing the application based on the application's resource requirements, and sends the information of the requested Spark worker nodes to the Spark driver process. The Spark driver process starts the Spark execution process through the resource controller in the Spark worker node according to the information of the requested Spark worker node.

[0094] After the Spark execution process is started, the MapReduce execution plan generator in the Spark driver process generates a Directed Acyclic Graph (DAG) based on the application and the Spark execution processes in each Spark worker node. The DAG consists of multiple stages, each containing multiple tasks. The DAG also indicates which thread in which Spark execution process on which Spark worker node will execute each task. The MapReduce task scheduler schedules the threads in the Spark execution processes to execute tasks sequentially according to the order of execution of the tasks in the DAG, thereby completing the computation for the application.

[0095] In the computing system shown in Figure 5, for the mapping and reduction task initiated by the Spark client, in order to first perform a local data-scope reduction operation on the mapping and reduction task within the Spark worker node, and then perform a global data-scope reduction operation on the mapping and reduction task in the external storage system, improvements have been made to the MR execution plan generator in the Spark driver process and the resource scheduler in the Spark management node to implement the application execution method provided in this application embodiment. The functionality of the improved MR execution plan generator and resource scheduler can be referred to in the subsequent method embodiments, and will not be described in detail here.

[0096] Figure 7 is a flowchart of a task execution method provided in an embodiment of this application. As shown in Figure 7, the method includes the following steps.

[0097] Step 701: Receive the application's mapping specification task.

[0098] In some embodiments, in the scenario of the computing system shown in Figure 4, the management node receives an application from a client, which includes a mapping protocol task.

[0099] For example, when the computing system is a Spark-based computing system, the application can also be called a Spark application. In this scenario, in step 701, the Spark management node receives the Spark application from the Spark client.

[0100] In some embodiments, each of the multiple computing nodes has a TEE. In this scenario, the application in step 701 is a confidential computing application, so as to improve the performance of the confidential computing application through the method provided in the embodiments of this application.

[0101] Optionally, each of the multiple computing nodes may not have a TEE. In this scenario, the application in step 701 is a plaintext computing application, so as to improve the performance of the plaintext computing application through the method provided in the embodiments of this application.

[0102] For ease of explanation later, the application in step 701 will be referred to as the target application.

[0103] When a management node receives a target application, it can first request computing resources for the target application, that is, request which computing nodes in the computing system will execute the tasks in the target application.

[0104] In some embodiments, the management node may determine the remaining available resources of each of the multiple computing nodes; sort the multiple computing nodes according to the size of the remaining available resources of each computing node, wherein the size of the remaining available resources of the computing nodes ranked higher exceeds the size of the remaining available resources of the computing nodes ranked lower; and select computing nodes from the sorted multiple computing nodes in order from front to back to execute the target application according to the resource requirement information of the target application, wherein the resource requirement information is used to indicate the specifications of the computing resources required to execute the target application.

[0105] By selecting which compute nodes to execute the target application using the methods described above, the mapping tasks in the subsequent mapping and reduction tasks of the target application can be performed by as few compute nodes as possible. In this way, most of the input data can undergo a reduction operation within the same compute node first, thereby reducing the amount of data written to and read from the storage system when performing global data-wide reduction operations later.

[0106] For example, resource requirement information includes the process specifications of the processes required to execute the target application. Process specifications may include the number of threads included in the process, the amount of memory required by the process, etc. In this scenario, for any compute node among multiple compute nodes, the management node first determines the remaining available cores and remaining available memory size of each compute node, and then determines the number of processes that can be started on each compute node based on the process specifications, i.e., the number of remaining executable processes. In other words, the size of the remaining available resources is equal to the number of remaining executable processes.

[0107] Optionally, for different target applications submitted by the client, the process specifications required by each target application are the default specifications. In this scenario, the resource requirement information may not include the process specifications required to execute the target application. This default process specification is pre-stored in the management node so that the management node can determine the number of remaining executable processes on each compute node based on the default process specification.

[0108] After determining the number of remaining executable processes on each computing node, the management node can sort the computing nodes according to the amount of remaining available resources on each computing node. This can be achieved by sorting the computing nodes according to the number of remaining executable processes on each computing node, with the number of remaining executable processes on the computing nodes ranked higher being greater than the number of remaining executable processes on the computing nodes ranked lower.

[0109] In this scenario, resource requirement information can include the total number of processes required to execute the target application. Accordingly, the implementation of selecting computing nodes from a sorted list of computing nodes in order of priority to execute the target application, based on the resource requirement information, can be as follows: Select the i-th computing node from the sorted list of computing nodes in order of priority, and perform the following operation on the i-th computing node:

[0110] If the total number of processes is greater than 0 and exceeds the number of remaining executable processes on the i-th compute node, then all remaining executable processes on the i-th compute node are selected to execute the target application, and the total number of processes is subtracted from the number of remaining executable processes on the i-th compute node to obtain the updated total number of processes. If the total number of processes is greater than 0 and does not exceed the number of remaining executable processes on the i-th compute node, then some of the remaining executable processes on the i-th compute node are selected to execute the target application, and the total number of processes is updated to 0. The updated total number of processes is used to determine whether to select the remaining executable processes on the (i+1)-th compute node to execute the target application.

[0111] If the total number of processes is greater than 0 and does not exceed the number of remaining executable processes on the i-th computing node, assuming the total number of processes is n, then n remaining executable processes from all remaining executable processes on the i-th computing node are selected to execute the target application.

[0112] Furthermore, the process of determining which computing nodes are used to execute the target application can be implemented through a resource scheduler embedded in the management node. For example, in a Spark-based computing system, this process can be implemented by the resource scheduler in the Spark management node shown in Figure 6, which can be Spark Standalone, for example. Alternatively, this process can also be implemented through a resource scheduler external to the management node. This application embodiment does not limit this, for example, the resource scheduler can be YARN (a big data resource scheduling framework) or Kubernetes (an open-source container orchestration engine), etc.

[0113] The following example, using the computing system shown in Figure 6, illustrates how the above-mentioned request for computing resources can be implemented.

[0114] For example, a user submits a target application to the Spark management node in the computing system, providing its resource requirements, also known as resource configuration information. This information includes the total number of processes required by the target application, the number of cores per process, and the memory size required per process. The number of cores per process is the number of threads within that process. For ease of explanation later, the total number of processes required by the target application in the resource requirements will be labeled as `requiredNumExec`, the number of cores per process as `execCores`, and the memory size required per process as `execMem`.

[0115] In response to a received target application, the Spark management node starts the corresponding Spark driver process. The Spark driver process requests TEE resources from the resource scheduler built into the Spark management node, which means requesting Spark worker nodes (i.e., compute nodes) capable of executing the target application. For any Spark worker node i, the resource scheduler obtains the remaining available cores for that Spark worker node. i and the total remaining available memory mem i Then, based on the number of cores and memory size of a single process in the resource requirement information, the theoretical maximum number of processes that Spark worker node i can start, maxNumExec, is calculated. i (i.e., the number of remaining executable processes). Based on the remaining executable processes maxNumExec for each Spark worker node i. i All Spark worker nodes are sorted in reverse order. Spark worker node i is selected sequentially. If the total number of allocated processes, currNumExec, is less than requiredNumExec, then the remaining executable processes on Spark worker node i are selected to execute the target application. The number of remaining executable processes selected is equal to min(maxNumExeci, requiredNumExec - currNumExec). When currNumExec equals requiredNumExec, resource allocation stops, i.e., Spark worker node selection stops. The initial value of currNumExec is 0.

[0116] The resource scheduler sends the selected process set ExecSet information to the Spark driver process. The ExecSet information includes the nodeID, the identifier of the Spark worker node where each process j resides. j , where j∈ExecSet.

[0117] Step 702: Assign computing nodes to the mapping and reduction tasks, where M computing nodes execute the first mapping task and N computing nodes execute the first reduction task, where M and N are both positive integers.

[0118] Since the mapping and reduction task requires two operations: a mapping operation to repartition the input data and a reduction operation to merge all repartitioned data, the management node, upon receiving the mapping and reduction task, must first design a first mapping task and a first reduction task. Specifically, the first mapping task refers to the task that performs the mapping operation on the input data of the mapping and reduction task. The first reduction task refers to the task that performs the reduction operation within the global data scope of the input data of the mapping and reduction task.

[0119] After the management node obtains the computing nodes to execute the target application, it further allocates computing nodes to the mapping and reduction tasks in the target application to determine which computing nodes will execute the first mapping task and which computing nodes will execute the first reduction task.

[0120] In some embodiments, allocating M computing nodes to execute the first mapping task can be implemented by using all the requested computing nodes as the M computing nodes to execute the first mapping task. That is, M is the total number of computing nodes requested, which can make full use of all the computing resources in the requested computing nodes.

[0121] In this scenario, the management node first divides the input data of the mapping reduction task into multiple data blocks based on the number of remaining executable processes on the allocated compute nodes and the number of threads in each remaining executable process. Each data block serves as the input data for a first mapping task. The total number of data blocks is greater than or equal to the total number of threads in all remaining executable processes on the allocated compute nodes to avoid idle threads on the allocated compute nodes during the mapping phase.

[0122] For example, if the allocated computing nodes include Computing Node 1 and Computing Node 2, and Computing Node 1 has 1 remaining executable process, while Computing Node 2 has 2 remaining executable processes, and each remaining executable process includes 2 threads, then the management node can divide the input data of the mapping reduction task into 6 or more data blocks to design at least one first mapping task for each thread in the allocated computing nodes. In other words, the first mapping task is executed by the allocated computing nodes 1 and 2.

[0123] Optionally, in scenarios where the user specifies the total number of data blocks for the input data of the mapping and induction task, the management node directly divides the input data for the mapping and induction task according to the total number of data blocks specified by the user, designs a first mapping task for each data block, and assigns the designed first mapping tasks to M computing nodes among the applied computing nodes. In this scenario, M may be less than the total number of applied computing nodes.

[0124] In some embodiments, the allocation of N computing nodes to execute the first reduction task can be implemented as follows: determine the number of first reduction tasks, and select N computing nodes from the allocated computing nodes to execute the first reduction tasks based on the number of first reduction tasks. For example, the total number of threads in all remaining executable processes across the N computing nodes is less than or equal to the number of first reduction tasks to avoid idle threads on the N computing nodes during the reduction phase.

[0125] The number of the first protocol tasks can be specified by the user. Alternatively, it can be automatically configured by the computing system; this embodiment of the application does not limit this.

[0126] In this embodiment, in addition to the aforementioned first mapping task and first reduction task, the management node also designs at least one second reduction task implemented within each of the M computing nodes. This is to perform the reduction operation of the mapping and reduction task within the computing node first, and obtain the calculation result of the mapping and reduction task within the local data range, thereby reducing the network communication volume between the computing node and the storage system and achieving the effect of improving the performance of the target application.

[0127] For example, for any one of the M computing nodes executing the first mapping task, such as the first computing node, assuming the first computing node is used to execute X first mapping tasks, the management node also designs at least one second reduction task for the first computing node. This at least one second reduction task is used to perform at least one reduction operation on the output results of the X first mapping tasks executed by the first computing node. This ensures that after the first computing node has completed executing X first mapping tasks, it performs at least one second reduction task within the first computing node itself.

[0128] The at least one second reduction task may include one or more second reduction tasks. When the at least one second reduction task includes one second reduction task, the first computing node performs one second reduction task after executing X first mapping tasks. When the at least one second reduction task includes multiple second reduction tasks, the first computing node performs multiple second reduction tasks sequentially after executing X first mapping tasks. Different reduction tasks are used to implement reduction operations within different local data ranges.

[0129] For example, when at least one second reduction task includes one second reduction task, that is, when the execution count of at least one second reduction task is 1, the first computing node executes one second reduction task after executing X first mapping tasks. In this scenario, the second reduction task executed by the first computing node can be a process-level second reduction task or a node-level second reduction task.

[0130] The process-level second reduction task can be understood as follows: after a thread within the same process on the same computing node completes its first mapping task, it executes a second reduction task once within that process. Taking the first computing node as an example, suppose the first computing node includes Y processes, each of which executes X first mapping tasks. The first process within the Y processes includes A threads, each of which executes A first mapping tasks. Then, for any process on the first computing node, such as the first process itself, after the A threads within the first process complete A first mapping tasks, B threads within the A threads execute a second reduction task based on the results of the A first mapping tasks. Here, A and B are both positive integers, and B can be equal to or less than A.

[0131] To enable threads within the same process to execute a second reduction task after completing the first mapping task, a first memory space needs to be configured for that process on the compute node. Different threads within this process can share this first memory. In other words, the first memory is shared memory among different threads within the same process on the same compute node. For example, taking the first process on the first compute node as an example, a first memory space is configured for the first process. After A threads in the first process complete A first mapping tasks, these A threads store the execution results of the first mapping tasks (i.e., the repartitioned data) in the first memory. Since the A threads in the first process share the first memory, subsequent threads B among the A threads can read the repartitioned data from the first memory and execute a second reduction task.

[0132] The node-level second reduction task can be understood as follows: after executing the first mapping task, threads within different processes of the same computing node execute the second reduction task once within that computing node. Taking the first computing node as an example, suppose the first computing node includes Y processes, and these Y processes are used to execute X first mapping tasks. Then, after the Y processes in the first computing node execute the X first mapping tasks, the second reduction task is executed by Z processes within the Y processes based on the execution results of the X first mapping tasks. Y and Z are both positive integers, and Z can be equal to or less than Y.

[0133] To enable threads within different processes on the same compute node to execute a second reduction task within the compute node after completing the first mapping task, a second memory needs to be configured for all processes on the compute node. Different processes on the compute node can share this second memory. In other words, the second memory is shared memory among different processes on the same compute node. For example, taking the first compute node as an example, if a second memory is configured for the first compute node, after threads in Y processes on the first compute node complete X first mapping tasks, these Y processes store the execution results of the first mapping tasks (i.e., the repartitioned data) in the second memory. Since all processes on the first compute node share the second memory, threads in Z processes out of the subsequent Y processes can read the repartitioned data from the second memory and execute a second reduction task.

[0134] For example, when at least one second reduction task includes two second reduction tasks, that is, when the execution count of at least one second reduction task is 2, the first computing node executes two second reduction tasks after executing X first mapping tasks. In this scenario, the two second reduction tasks executed by the first computing node are a process-level second reduction task and a node-level second reduction task, respectively.

[0135] For example, taking the first computing node as an example, a second memory is configured for all processes in the first computing node. Threads within all processes can share this second memory. Each process in the first computing node is configured with its own second memory, and different threads within each process can share their corresponding second memory. Assume the first computing node includes Y processes, which execute X first mapping tasks. The first process within the Y processes includes A threads, which execute A first mapping tasks. Then, for any process in the first computing node, such as the first process, after the A threads in the first process complete A first mapping tasks, these A threads store the execution results of the first mapping tasks (i.e., repartitioned data) in the first memory corresponding to the first process. This allows B threads within the A threads to read the repartitioned data from the first memory corresponding to the first process and execute the first second reduction task. After the Y processes complete the first second reduction task, they store the execution results of the first second reduction task in the second memory through mapping operations. Then, threads in the Z processes read the repartitioned data from the second memory and execute the second second reduction task.

[0136] The difference between the different levels of the second reduction task and the first reduction task is that the different levels of the second reduction task are used to perform reduction operations within different local data ranges of the input data of the mapping induction task, while the first reduction task is used to perform statistics within the global data range of the input data of the mapping induction task.

[0137] For example, when dividing the input data of a mapping reduction task into multiple data blocks and using each data block as the input data for the first mapping task of a thread in M ​​computing nodes, for any computing node, such as the first computing node, assuming the first computing node includes Y processes, and the Y processes include the first process, then, taking the first process as an example, the process-level second reduction task performed within the first computing node is used to: perform a reduction operation on all data blocks allocated to threads in the first process. The node-level second reduction task performed within the first computing node is used to: perform a reduction operation on all data blocks allocated to threads in the Y processes within the first computing node.

[0138] In addition, the number of protocol tasks in each second protocol task can be specified by the user or automatically configured by the computing system, and this application embodiment does not limit this either.

[0139] It should be noted that the above examples illustrate multiple second reduction tasks using process-level and node-level second reduction tasks. In the embodiments of this application, multiple second reduction tasks can be implemented in other ways, which will not be illustrated here.

[0140] In addition, in some embodiments, after allocating M computing nodes for the mapping reduction task, the management node can also set the number of executions of the second reduction task for each of the M computing nodes, and instruct the M computing nodes to execute the second reduction task according to the specified number of executions. This improves the efficiency of subsequent scheduling of computing nodes.

[0141] For example, after the management node allocates computing nodes for the mapping reduction task in the target application, it can also determine the task flowchart of the target application. This task flowchart can indicate the number of times the second reduction task is executed by each of the M computing nodes, and the number of times the M computing nodes execute the second reduction task according to the specified number of times the second reduction task is executed. Subsequently, the management node can directly schedule each computing node according to the task flowchart to realize the target application.

[0142] When the computing system is a Spark-based computing system, the task flowchart can be implemented by the MR execution plan generator in the Spark-driven process shown in Figure 6. The task flowchart can also be called a directed acyclic graph (DAG).

[0143] When determining the task flowchart, the management node sequentially identifies the sub-task flowcharts corresponding to each task according to the execution order of the tasks in the target application. Then, it connects the sub-task flowcharts according to the task execution order to obtain the task flowchart for the target application. The following explanation uses the sub-task flowchart of the mapping and reduction task in the target application as an example to illustrate the task flowchart of the target application.

[0144] Additionally, it should be noted that in this embodiment of the application, at least one second reduction task is added between the first mapping task and the first reduction task. Since a mapping operation needs to be performed before each reduction task, after designing at least one second reduction task, the management node also needs to design mapping tasks to be performed between different second reduction tasks, as well as mapping tasks to be performed between the last second reduction task and the first reduction task.

[0145] For ease of explanation, the first mapping task, at least one second reduction task, and the mapping tasks executed between different second reduction tasks are collectively referred to as the first type of MR task, and the first reduction task and the mapping tasks executed between the last second reduction task and the first reduction task are collectively referred to as the second type of MR task.

[0146] For example, in a scenario where at least one second reduction task includes two second reduction tasks, the first type of MR task includes a first type of sub-MR task and a second type of sub-MR task, with the output of the first type of sub-MR task serving as the input of the second type of sub-MR task. Specifically, the first type of sub-MR task includes a first mapping task and a process-level second reduction task; the second type of sub-MR task includes a node-level second reduction task and a mapping task executed between the node-level second reduction task and the process-level second reduction task.

[0147] Taking the first compute node as an example, assume it comprises Y processes, each executing X first mapping tasks. The first process within the Y processes includes A threads, each executing A first mapping tasks. Then, the first type of sub-MR tasks includes the A first mapping tasks executed by each process (e.g., the A threads within the first process) and the first second reduction task executed by the B threads within the A threads. The second type of sub-MR tasks includes the mapping task executed by each process within the first compute node after completing the first second reduction task, and the second second reduction task executed by the Z processes within the Y processes.

[0148] By using the first and second types of sub-MR tasks mentioned above, the mapping and reduction task can be performed within the process of the computing node to obtain the calculation result of the mapping and reduction task within a smaller local data range. Then, the calculation result of the previous step can be reduced between different processes of the same computing node to obtain the calculation result of the mapping and reduction task within an intermediate local data range. This reduces the network communication volume between the computing node and the storage system, thereby improving the performance of the target application.

[0149] For example, in scenarios where at least one second reduction task includes another second reduction task, such as at least one second reduction task including a process-level second reduction task, then the first type of MR task only includes the first type of sub-MR task and not the second type of sub-MR task. In this scenario, the mapping reduction task is only performed once within the process of the compute node, without further reduction operations between different processes. This can also reduce the network communication volume between the compute node and the storage system, thereby improving the performance of the target application.

[0150] Taking the first computing node as an example, assume that the first computing node includes Y processes, and these Y processes are used to execute X first mapping tasks. The first process among the Y processes includes A threads, and these A threads are used to execute A first mapping tasks. The first type of MR task only includes first type sub-MR tasks. The first type of sub-MR task includes, for example, the A first mapping tasks executed by the A threads in each process of the first computing node, and a second reduction task executed by B threads within the A threads.

[0151] For example, in scenarios where at least one second reduction task includes another second reduction task, such as at least one second reduction task including a node-level second reduction task, then the first type of MR task only includes the second type of sub-MR task, and not the first type of sub-MR task. In this scenario, directly performing the reduction operation between different processes on the compute node, without first performing the reduction operation within the compute node's process, can also reduce the network communication volume between the compute node and the storage system, thereby improving the performance of the target application.

[0152] Taking the first computing node as an example, assume that the first computing node includes Y processes, and these Y processes are used to execute X first mapping tasks. The first type of MR task only includes the second type of sub-MR task, and does not include the first type of sub-MR task. The second type of sub-MR task includes the X first mapping tasks executed by the Y processes in the first computing node, and one second reduction task executed by Z processes in the Y processes.

[0153] The following section uses the first type of MR task, which includes the first type of sub-MR task and the second type of sub-MR task, as an example to explain in detail the process of designing the mapping task and the reduction task for the management node to implement the mapping and reduction task.

[0154] For mapping and specification tasks, the management node can design mapping and specification tasks to implement mapping and specification tasks in the following three levels.

[0155] First level: First type of sub-MR task, which includes first mapping task and process-level second reduction task.

[0156] When designing the first-level MR task, for any of the M computing nodes, such as the first computing node, at least one first mapping task is assigned to each thread in each remaining executable process of the first computing node. The input data of each first mapping task is a data block after the input data of the mapping induction task is divided.

[0157] For any process in the first computing node, such as the first process, assuming the first process includes A threads, after assigning the first mapping task to thread A in the first process, the second reduction task is assigned to thread B in these A threads. These second reduction tasks constitute a process-level second reduction task.

[0158] The number of second reduction tasks allocated to the first process can be specified by the user or automatically configured by the computing system. For example, the number of second reduction tasks allocated to the first process can be directly configured to be equal to the number of first mapping tasks allocated to the first process; this embodiment of the application does not limit this.

[0159] For example, as shown in Figure 8, the M computing nodes allocated for executing the first mapping task include the first computing node, which has only one remaining executable process, labeled as the first process. The management node assigns a first-level MR identifier to the first process, for example, labeled as sub-MR0, and designs n first mapping tasks and n second reduction tasks for sub-MR0, where n can be greater than or equal to the number of threads included in the first process. The n first mapping tasks and n second reduction tasks are then assigned to the threads within the first process, thus determining the executor of each first mapping task and each second reduction task. Figure 8 only illustrates two mapping tasks out of the n first mapping tasks and two reduction tasks out of the n second reduction tasks.

[0160] For example, as shown in Figure 8, the M computing nodes allocated for executing the first mapping task include a second computing node. The second computing node has two remaining executable processes, labeled as the second process and the third process. The management node assigns a first-level MR identifier to the second process, for example, labeled as sub-MR1, and designs m first mapping tasks and m second reduction tasks for sub-MR1. m can be greater than or equal to the number of threads included in the second process. The m first mapping tasks and m second reduction tasks are then assigned to the threads within the second process. Figure 8 only illustrates two of the m first mapping tasks and two of the m second reduction tasks. Simultaneously, the management node assigns a first-level MR identifier to the third process, for example, labeled as sub-MR2, and designs w first mapping tasks and w second reduction tasks for sub-MR2. w can be greater than or equal to the number of threads included in the third process. The w first mapping tasks and w second reduction tasks are then assigned to the threads within the third process. Figure 8 only illustrates two mapping tasks out of w first mapping tasks and two reduction tasks out of w second reduction tasks.

[0161] Second level: Second type of sub-MR tasks, which include node-level second reduction tasks and mapping tasks executed between node-level second reduction tasks and process-level second reduction tasks.

[0162] When designing a second-level MapReduce task, for any of the M compute nodes, if the number of remaining executable processes in that compute node is greater than 1, then the management node can design a second-level MapReduce task for that compute node. Conversely, if the number of remaining executable processes in that compute node is equal to 1, then it is not necessary to design a second-level MapReduce task for that compute node.

[0163] For example, as shown in Figure 8, the M computing nodes allocated for executing the first mapping task include a second computing node. The second computing node contains two remaining executable processes, labeled as the second process and the third process. The management node assigns a second-level MR identifier to the second computing node, for example, labeled as sub-MR3. For sub-MR3, a mapping task is designed for each process-level second reduction task executed by each thread in the second and third processes. This allows each thread to execute a mapping task after executing a process-level second reduction task, storing the result of this process-level second reduction task in a second memory shared by both threads in the second and third processes.

[0164] For sub-MR3, p second reduction tasks are designed, where p is a positive integer. These p second reduction tasks are assigned to threads in the second and third processes, so that the second and third processes can execute node-level second reduction tasks again based on the execution results of the process-level second reduction tasks stored in the second memory. Figure 8 only shows four of the p second reduction tasks.

[0165] The third level: the second type of MR task, which includes the first reduction task and the mapping task executed between the second reduction task and the first reduction task at the node level.

[0166] For example, as shown in Figure 8, the M computing nodes allocated for executing the first mapping task include the first computing node and the second computing node, and the N computing nodes allocated for executing the first reduction task include the third computing node and the fourth computing node. The first computing node has only one remaining executable process, labeled as the first process; the second computing node has two remaining executable processes, labeled as the second process and the third process; the third computing node has only one remaining executable process, labeled as the third process; and the fourth computing node has two remaining executable processes, labeled as the fifth process and the sixth process. The management node then assigns a third-level MR identifier to all computing nodes (i.e., the M and N computing nodes), as shown in Figure 8, and labels this third-level MR identifier as sub-MR4.

[0167] For sub-MR4, a mapping task is designed for each node-level second reduction task executed by each thread in the second and third processes. This allows each thread to execute a mapping task after performing a node-level second reduction task, storing the result of this node-level second reduction task in a shared storage system (Figure 8 uses a disk as an example). Additionally, a mapping task is designed for each process-level second reduction task executed by each thread in the first process. This allows each thread to execute a mapping task after performing a process-level second reduction task, also storing the result of this process-level second reduction task in a shared storage system.

[0168] Then, for subMR4, q first reduction tasks are designed, where q is a positive integer. These q first reduction tasks are assigned to threads in the fourth, fifth, and sixth processes, so that the fourth, fifth, and sixth processes can then perform global data-wide reduction operations on the execution results of the second reduction tasks stored in the storage system. Figure 8 only shows 6 of the q first reduction tasks.

[0169] In addition, in scenarios where the computing system is based on Spark, after designing different levels of MapReduce tasks, the management node can also divide the MapReduce tasks of different levels into stages, so that the management node can control the computing nodes to execute the MapReduce tasks in the corresponding stages in sequence.

[0170] For example, in the scenario shown in Figure 8, an embodiment of this application provides a phase division diagram as shown in Figure 9.

[0171] Specifically, the mapping tasks executed by threads within the same process in the first type of sub-MR tasks (i.e., the first mapping tasks) are divided into the same stage. As shown in Figure 9, the mapping tasks executed by threads in the first process in the first type of MR tasks belong to the same stage, which is marked as stage0 in Figure 9. The mapping tasks executed by threads in the second process in the first type of MR tasks belong to the same stage, which is marked as stage1 in Figure 9. The mapping tasks executed by threads in the third process in the first type of MR tasks belong to the same stage, which is marked as stage3 in Figure 9.

[0172] The reduction tasks (i.e., process-level second reduction tasks) executed by a process in the first type of sub-MR tasks and the mapping tasks executed by threads within the same process in the second type of sub-tasks are grouped into the same stage. As shown in Figure 9, the reduction tasks executed by threads in the second process in the first type of MR tasks and the mapping tasks executed by threads in the second process in the second type of MR tasks belong to the same stage, which is marked as stage2 in Figure 9. Similarly, the reduction tasks executed by threads in the third process in the first type of MR tasks and the mapping tasks executed by threads in the third process in the second type of MR tasks belong to the same stage, which is marked as stage4 in Figure 9.

[0173] The reduction tasks (i.e., node-level second reduction tasks) executed by a certain computing node in the second type of sub-MR tasks and the mapping tasks executed by the same computing node in the second type of MR tasks are grouped into the same stage. As shown in Figure 9, the reduction tasks executed by threads in the second process and the reduction tasks executed by threads in the third process in the second type of MR tasks, as well as the mapping tasks executed by threads in the second process and the mapping tasks executed by threads in the third process in the second type of MR tasks, belong to the same stage, which is marked as stage6 in Figure 9.

[0174] Finally, the first reduction task executed by all threads in the second type of MR task is divided into the same stage. As shown in Figure 9, the reduction tasks executed by the thread in the third process, the thread in the fourth process, and the thread in the fifth process in the second type of MR task belong to the same stage, which is marked as stage7 in Figure 9.

[0175] It should be noted that, for the first compute node, since it only contains the first process, no second-level MapReduce tasks, i.e., second sub-class MapReduce tasks, are designed for it. In this scenario, the reduction task executed by a thread in the first process within the first sub-class MapReduce task and the mapping task executed by a thread in the first process within the second sub-class MapReduce task are grouped into the same stage. As shown in Figures 8 and 9, the reduction task executed by a thread in the first process within the first sub-class MapReduce task and the mapping task executed by a thread in the first process within the second sub-class MapReduce task belong to the same stage, which is marked as stage 5 in Figure 9.

[0176] After dividing the stages according to the above method, we can obtain the stage relationship diagram in the DAG shown in Figure 10. In addition, Figure 10 also shows the stage relationship in the subtask flowchart of the mapping and reduction task designed according to the association technique. The association technique refers to the fact that when the threads in all the computing nodes that have been applied for execute the mapping task, they directly write the repartitioned data into the storage system, so as to directly perform the reduction operation of the mapping and reduction task in the global data scope.

[0177] As shown in Figure 10, in the association technique, since all threads directly perform MR operations within the global data scope, the MR tasks designed for the mapping and reduction tasks are only the first mapping task and the first reduction task. Correspondingly, there are only two stages: the mapping stage and the reduction stage.

[0178] However, in this embodiment, for any computing node, a local data range reduction operation is first performed within the same process of the computing node, then a local data range reduction operation is performed between different processes of the computing node, and finally a global data range reduction operation is performed between all computing nodes. Therefore, as shown in Figure 10, for any computing node, the DAG includes one or more stage chains corresponding to that computing node, and each stage chain includes at least three stages connected in series.

[0179] As shown in Figures 9 and 10, there is a stage chain corresponding to the first computing node, which includes stage 0, stage 5, and stage 7 connected in series. There are two stage chains corresponding to the second computing node: one is a stage chain consisting of stage 1, stage 2, stage 6, and stage 7 connected in series, and the other is a stage chain consisting of stage 3, stage 4, stage 6, and stage 7 connected in series.

[0180] The number of stages in the stage chain corresponding to each computing node indicates the number of times the computing node needs to perform the second reduction task. Specifically, when the number of stages in the stage chain is 3, it indicates that the computing node needs to perform the second reduction task 1 time; when the number of stages in the stage chain is 4, it indicates that the computing node needs to perform the second reduction task 2 times.

[0181] Furthermore, for any stage chain corresponding to any compute node, except for the last stage in the stage chain, all tasks within the remaining stages of the stage chain are executed only within the compute node, thereby reducing the amount of network communication between the compute node and the storage system.

[0182] It should be noted that the above stage division method is for illustrative purposes only. For example, in a scenario where the computing system is based on Spark, tasks can be categorized into various stages using the above method, so that the DAG directly displays the connection relationships between the stages.

[0183] Alternatively, in this embodiment, the stages may not be divided in the manner described above. Alternatively, stages may not be divided, and the task flowchart can directly show the execution order of each task.

[0184] Furthermore, the above explanation uses three levels as an example to illustrate how to design MR tasks. Optionally, in this embodiment, only the first and third level MR tasks may be designed. Alternatively, only the second and third level MR tasks may be designed. These will not be illustrated further here. Alternatively, in a scenario where only one computing node is available, only the first and second level MR tasks need to be designed.

[0185] After obtaining the task flowchart, the management node can schedule threads from multiple compute nodes to execute the tasks in the flowchart. When the computing system is a Spark-based computing system, this process can be implemented by the MapReduce task scheduler in the Spark driver process shown in Figure 6.

[0186] For example, in the task flowchart shown in Figure 9, the mapping tasks labeled stage 0, stage 1, and stage 2 can be executed in parallel. After the management node schedules the compute nodes to execute the mapping tasks in these three stages, it then schedules the compute nodes to execute the tasks in stage 5, stage 2, and stage 4. The tasks in stages 5, 2, and 4 can be executed in parallel. Then, the management node schedules the compute nodes to execute the tasks in stage 6. Finally, the management node schedules the compute nodes to execute the tasks in stage 7.

[0187] When a management node schedules a computing node to execute a task in the task flowchart, it can do so by issuing task execution instructions. This will not be explained in detail here.

[0188] In this embodiment of the application, after the management node schedules each computing node according to the task flowchart, the operations performed by each computing node can be explained by the following steps 703 and 704.

[0189] Step 703: The first computing node among the M computing nodes performs at least one second reduction task on the X first mapping tasks and stores the execution result of at least one second reduction task, where X is a positive integer.

[0190] Since the management node is also designed to implement at least one second reduction task within each of the M computing nodes, for any computing node among the M computing nodes, such as the first computing node, after the first computing node has executed X first mapping tasks, the first computing node will execute at least one second reduction task on the executed X first mapping tasks, so as to first perform the reduction operation of the mapping reduction task within the first computing node and obtain the calculation result of the mapping reduction task within the local data range.

[0191] Before executing X first mapping tasks, the first computing node needs to read the input data required for the X first mapping tasks from the storage system. For example, in a scenario where the application is a confidential computing application and the computing nodes in the computing system have a TEE, the first computing node reads the input data required for the X first mapping tasks from the storage system in the REE, then moves the read input data from the REE to the TEE, decrypts the read data in the TEE, and then executes the X first mapping tasks.

[0192] In this process, after the first computing node executes at least one second reduction task for the X first mapping tasks, in order to enable subsequent reduction operations within the global data scope, it needs to store the execution results of at least one second reduction task in a storage system, such as a disk, that communicates with the first computing node through a mapping task.

[0193] For example, in a scenario where the application is confidential computing, the computing nodes in the computing system have a TEE (Transaction Execution Environment). In this scenario, the first computing node executes at least one second reduction task in the TEE, encrypts the execution result of the at least one second reduction task in the TEE, and then moves it to the REE (Resource Execution Environment) so that the encrypted execution result of the at least one second reduction task can be stored in the storage system in the REE.

[0194] Step 704: N computing nodes execute the first reduction task based on the execution result of at least one second reduction task.

[0195] Since each of the M computing nodes stores the execution result of the second reduction task at least once in the storage system that communicates with the corresponding computing node, the N computing nodes can read the repartitioned data within the global data range from the storage system and then execute the first reduction task to obtain the output result of the mapping reduction task.

[0196] For example, in a scenario where the application is confidential computing, the computing nodes in the computing system have a TEE (Transmission Equipment Environment). In this scenario, each of the N computing nodes first reads the repartitioned data within the global data scope from the storage system in the REE, then moves the read data from the REE to the TEE, decrypts it within the TEE, and then executes the first reduction task.

[0197] In a scenario where at least one second reduction task includes a node-level second reduction task, taking the first computing node as an example, assuming the first computing node includes Y processes, and the Y processes are used to execute X first mapping tasks; then the implementation of the first computing node among M computing nodes executing at least one second reduction task for the X first mapping tasks can be as follows: after the Y processes have completed the execution of X first Map tasks, the second reduction task is executed by Z processes among the Y processes based on the execution results of the X first mapping tasks, where Y and Z are both positive integers.

[0198] The specific implementation method has been described in detail in the foregoing embodiments, and will not be repeated here.

[0199] Optionally, in scenarios where at least one second reduction task includes a process-level second reduction task, taking the first computing node as an example, suppose the first computing node includes Y processes, which are used to execute X first mapping tasks. The first process among the Y processes includes A threads, which are used to execute A first mapping tasks. In this scenario, the implementation method for the first computing node among M computing nodes to execute at least one second reduction task for the X first mapping tasks can be: after A threads have completed executing A first mapping tasks, the second reduction task is executed by B threads among the A threads.

[0200] The specific implementation method has been described in detail in the foregoing embodiments, and will not be repeated here.

[0201] Optionally, in scenarios where at least one second reduction task simultaneously includes both process-level and node-level second reduction tasks, taking the first computing node as an example, assume the first computing node includes Y processes, which execute X first mapping tasks. The first process among the Y processes includes A threads, which execute A first mapping tasks. In this scenario, the implementation of the first computing node among M computing nodes performing at least one second reduction task on the X first mapping tasks can be as follows: after A threads complete A first mapping tasks, a second reduction task is executed once by B threads among the A threads; and a second reduction task is then executed again by Z processes among the Y processes on the result of the second reduction tasks executed by the Y processes.

[0202] To further understand the embodiments of this application, steps 701-704 are illustrated below using Figure 8 as an example.

[0203] In this embodiment of the application, the process of designing MR tasks by the management node is as follows:

[0204] (1) For a single mapping reduction task in the target application submitted by the user, construct the first-level MR task, that is, the first type of sub-MR task, including the first mapping task and the process-level second reduction task.

[0205] Specifically, each remaining executable process j in each of the M allocated computing nodes is assigned an independent first-level MR identifier; the number of mapping tasks and reduction tasks designed for each first-level MR identifier are the same, for example, it can be set to the number of threads included in the remaining executable process j, execCores, or it can be set to an integer larger than execCores.

[0206] (2) Next, construct the second level of MR tasks, namely the second type of sub-MR tasks, including the node-level second reduction tasks and the mapping tasks executed between the process-level second reduction tasks and the node-level second reduction tasks.

[0207] Specifically, iterate through the nodeIDs of all compute nodes to which process j belongs. j For nodeID j After deduplication, the set of compute nodes executing the mapping reduction tasks, NodeIDSet, is obtained (i.e., a set of M compute nodes). Each compute node k in NodeIDSet is assigned a unique second-level MR identifier, where k∈NodeIDSet. The number of mapping tasks and reductions corresponding to the second-level MR identifier designed for compute node k is the same as the number of tasks, for example, it can be set to execCores*numExec. k , where numExec k To calculate the number of remaining executable processes on node k, it can also be set to be greater than execCores * numExec k Larger integers.

[0208] (3) Construct the third-level MR task, namely the second type of MR task, including the first reduction task and the mapping task executed between the second reduction task and the first reduction task at the node level.

[0209] Assign a third-level MR identifier to all processes. The number of mapping tasks and reduction tasks corresponding to the third-level MR identifier are generally set to requiredNumExec*execCores, but can also be set to a larger integer than requiredNumExec*execCores.

[0210] The MR tasks at each level can be obtained through steps (1)-(3). The specific content of each task in steps (1)-(3) is as described in step 702 above, and will not be repeated here.

[0211] (4) Merge the mapping tasks in each level of MR tasks with the reduction tasks in the upstream level of MR tasks into a stage. In the first level of MR tasks, the mapping tasks corresponding to each process are generated into a separate stage, and in the last level of MR tasks, the reduction tasks corresponding to all processes are also generated into a separate stage; finally, a DAG is obtained.

[0212] (5) According to the task dependencies specified in the DAG, prioritize scheduling computing nodes to execute tasks in the upstream stage until all tasks in all stages are completed.

[0213] Specifically, the task execution process for each computing node is as follows:

[0214] a) For first-level MR tasks, multiple threads in the same process on the compute node work together to perform local data range reduction operations, and data exchange is accomplished by efficient "inter-thread communication".

[0215] The data exchange in step a) can be understood as follows: different threads in a process write the output of the mapping task, i.e., the repartitioned data, into the first memory corresponding to the process. Subsequently, when different threads in the process execute the second reduction task at the process level, they read the repartitioned data belonging to the same partition from the first memory.

[0216] b) For the second-level MR task, multiple threads in multiple processes on the same compute node work together to perform reduction operations on a larger local data range, and data exchange is accomplished by "inter-process communication".

[0217] The data exchange in step b) can be understood as follows: For a certain computing node, all threads in the process of that computing node write the output of the mapping task, that is, the repartitioned data, into the second memory of that computing node. Subsequently, when all threads in the process execute the second reduction task at the node level, they read the repartitioned data belonging to the same partition from the second memory.

[0218] c) For the third-level MR task, all threads within all processes across multiple computing nodes collaboratively execute global data-scope reduction operations, with data exchange handled by the external storage system and network adapter. The computation process of the third-level MR task is the same as that of the associated MR task in Figure 10; however, the input data changes. After the local merging of the first two levels, the amount of input data for the third-level MR task is significantly reduced.

[0219] The data exchange in step c) can be understood as follows: For all computing nodes, threads in different computing nodes write the output of the mapping task, i.e., the repartitioned data, into the storage system. Subsequently, when all threads in the computing nodes execute the first reduction task, they read the repartitioned data belonging to the same partition from the storage system.

[0220] In summary, in this embodiment, for the mapping and reduction task in the application, after the first computing node among the allocated M computing nodes completes X first mapping tasks, it performs at least one second reduction task on the X first mapping tasks within the first computing node. Finally, the allocated N computing nodes execute the first reduction task. This allows for the mapping and reduction task to first perform a local data-scope reduction operation within the computing node, and then perform a global data-scope reduction operation on the mapping and reduction task across different computing nodes. Since the mapping and reduction task is reduced within a local data scope before performing the global data-scope reduction operation, the amount of repartitioned data written to the storage system by the M computing nodes is reduced. Correspondingly, the amount of repartitioned data read from the storage system by the N computing nodes when performing the global data-scope reduction operation is also reduced, thereby reducing the network communication volume between the computing nodes and the storage system and improving application performance.

[0221] In addition, as shown in Figure 11, this application embodiment also provides a computing system, which includes a management node 10 and multiple computing nodes 20.

[0222] Among them, the management node 10 is used to receive the MapReduce task of the application's mapping specification; for the specific implementation method, refer to step 701 in the embodiment of Figure 7.

[0223] The management node 20 is also used to allocate computing nodes for the mapping and reduction tasks, wherein M computing nodes execute the first mapping task and N computing nodes execute the first reduction task, where M and N are both positive integers; for specific implementation, refer to step 702 in the embodiment of Figure 7.

[0224] The first computing node among the M computing nodes is used to execute at least one second reduction task on the X first mapping tasks and store the execution results of at least one second reduction task, where X is a positive integer; for specific implementation, refer to step 703 in the embodiment of Figure 7.

[0225] N computing nodes are used to execute the first reduction task based on the execution result of at least one second reduction task. For specific implementation details, refer to step 704 in the embodiment shown in Figure 7.

[0226] Optionally, the first computing node includes Y processes, which are used to execute X first mapping tasks; the first computing node is used to: after the Y processes have completed the execution of the X first mapping tasks, execute a second reduction task on the execution results of the X first mapping tasks through Z processes among the Y processes, where Y and Z are both positive integers.

[0227] Optionally, the first computing node includes Y processes, which are used to execute X first mapping tasks. The first process among the Y processes includes A threads, which are used to execute A first mapping tasks. The first computing node is used to: after the A threads have completed the execution of A first mapping tasks, execute a second reduction task on the execution results of A first mapping tasks through B threads among the A threads.

[0228] Optionally, the first computing node includes Y processes, which are used to execute X first mapping tasks. The first process among the Y processes includes A threads, which are used to execute A first mapping tasks. The first computing node is used to: after the A threads have completed the execution of A first mapping tasks, execute a second reduction task once using B threads among the A threads on the execution results of the A first mapping tasks; and execute a second reduction task again using Z processes among the Y processes on the results of the second reduction tasks executed by the Y processes.

[0229] Optionally, the management node is also configured to: after allocating M computing nodes for the mapping reduction task, set the number of times the second reduction task is executed for each of the M computing nodes, and instruct the M computing nodes to execute the second reduction task according to the specified number of times the second reduction task is executed.

[0230] Optionally, the first computing node includes a Trusted Execution Environment (TEE) and a Rich Execution Environment (REE); the first computing node is also used to: move the data required by the first computing node to execute the first mapping task from the REE to the TEE and then decrypt it; after the first computing node has executed at least one second reduction task, encrypt the execution result and then move it from the TEE to the REE for storage.

[0231] In summary, in this embodiment, for the mapping and reduction task in the application, after the first computing node among the allocated M computing nodes completes X first mapping tasks, it performs at least one second reduction task on the X first mapping tasks within the first computing node. Finally, the allocated N computing nodes execute the first reduction task. This allows for the mapping and reduction task to first perform a local data-scope reduction operation within the computing node, and then perform a global data-scope reduction operation on the mapping and reduction task across different computing nodes. Since the mapping and reduction task is reduced within a local data scope before performing the global data-scope reduction operation, the amount of repartitioned data written to the storage system by the M computing nodes is reduced. Correspondingly, the amount of repartitioned data read from the storage system by the N computing nodes when performing the global data-scope reduction operation is also reduced, thereby reducing the network communication volume between the computing nodes and the storage system and improving application performance.

[0232] It should be noted that the computing system provided in the above embodiments is only illustrated by the division of the nodes described above when executing tasks. In practical applications, the above functions can be assigned to different function nodes as needed, that is, the internal structure of the system can be divided into different nodes to complete all or part of the functions described above. In addition, the computing system and the task execution method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0233] Figure 12 is a schematic diagram of a computer device provided in an embodiment of this application. The management node and computing node in the computing system of the foregoing embodiments can both be implemented using one or more computer devices as shown in Figure 12. Referring to Figure 12, the computer device includes at least one processor 1201, a communication bus 1202, a memory 1203, and at least one communication interface 1204.

[0234] The processor 1201 may be a general-purpose central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application.

[0235] The communication bus 1202 may include a path for transmitting information between the aforementioned components.

[0236] The memory 1203 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or it may be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), a magnetic disk or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. The memory 1203 may exist independently and be connected to the processor 1201 via the communication bus 1202. The memory 1203 may also be integrated with the processor 1201.

[0237] The memory 1203 stores program code for executing the scheme of this application, and its execution is controlled by the processor 1201. The processor 1201 executes the program code stored in the memory 1203. The program code may include one or more software modules. In the foregoing embodiments, both the management node and the computing node can determine the data used for application development through the processor 1201 and one or more software modules in the program code in the memory 1203.

[0238] Communication interface 1204 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc.

[0239] In a specific implementation, as one embodiment, the computer device may include multiple processors, such as processor 1201 and processor 1205 shown in FIG. 12. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. Here, a processor may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0240] In a specific implementation, as one embodiment, the computer device may further include an output device 1206 and an input device 1207. The output device 1206 communicates with the processor 1201 and can display information in various ways. For example, the output device 1206 may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device 1207 communicates with the processor 1201 and can receive user input in various ways. For example, the input device 1207 may be a mouse, keyboard, touchscreen device, or sensing device, etc.

[0241] The aforementioned computer device can be a general-purpose computer device or a special-purpose computer device. In specific implementations, the computer device can be a desktop computer, a portable computer, a network server, a handheld computer (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. This application does not limit the type of computer device.

[0242] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0243] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0244] The above content is not intended to limit the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of the embodiments of this application.

Claims

1. A task execution method characterized by, The method comprises: receiving a MapReduce task of an application; allocating computing nodes for the MapReduce task, wherein M computing nodes perform a first mapping task and N computing nodes perform a first reduction task, M and N are positive integers; a first computing node of the M computing nodes performs at least one second reduction task on X first mapping tasks performed by the first computing node, and stores the execution result of the at least one second reduction task, X is a positive integer; the N computing nodes perform the first reduction task on the execution result of the at least one second reduction task.

2. The method of claim 1, wherein, The first computing node comprises Y processes for executing the X first mapping tasks; the first computing node of the M computing nodes performs at least one second reduction task on X first mapping tasks performed by the first computing node, comprising: after the Y processes complete the execution of the X first mapping tasks, the Z processes of the Y processes perform the second reduction task on the execution result of the X first mapping tasks, Y and Z are positive integers.

3. The method of claim 1, wherein, The first computing node comprises Y processes for executing the X first mapping tasks, wherein a first process of the Y processes comprises A threads for executing A first mapping tasks; the first computing node of the M computing nodes performs at least one second reduction task on X first mapping tasks performed by the first computing node, comprising: after the A threads complete the execution of the A first mapping tasks, the B threads of the A threads perform the second reduction task on the execution result of the A first mapping tasks.

4. The method of claim 1, wherein, The first computing node comprises Y processes for executing the X first mapping tasks, wherein a first process of the Y processes comprises A threads for executing A first mapping tasks; the first computing node of the M computing nodes performs at least one second reduction task on X first mapping tasks performed by the first computing node, comprising: after the A threads complete the execution of the A first mapping tasks, the B threads of the A threads perform the second reduction task on the execution result of the A first mapping tasks; the Y processes perform the second reduction task on the result of the second reduction task performed by the Z processes of the Y processes.

5. The method according to any one of claims 1 to 4, characterized in that, The method further comprises: after the M computing nodes are allocated for the MapReduce task, the number of times of performing the second reduction task is set for each of the M computing nodes, and the M computing nodes are instructed to perform the second reduction task according to the specified number of times of performing the second reduction task.

6. The method of any one of claims 1-5, wherein, The first computing node comprises a trusted execution environment (TEE) and a rich execution environment (REE); the method further comprises: decrypting the data required by the first computing node to perform the first mapping task after the data is moved from the REE to the TEE; after the first computing node completes the execution of the at least one second reduction task, the execution result is encrypted and moved from the TEE to the REE and stored.

7. A computing system, comprising: The computing system comprises a management node and a plurality of computing nodes; The management node is configured to receive a MapReduce task of an application; The management node is further configured to allocate computing nodes for the MapReduce task, wherein M computing nodes execute a first mapping task and N computing nodes execute a first reduction task, and M and N are positive integers; A first computing node in the M computing nodes is configured to execute at least one second reduction task on X first mapping tasks executed by the first computing node, and store an execution result of the at least one second reduction task, and X is a positive integer; The N computing nodes are configured to execute the first reduction task on the execution result of the at least one second reduction task.

8. The computing system of claim 7, wherein, The first computing node comprises Y processes configured to execute the X first mapping tasks; The first computing node is configured to: After the Y processes execute the X first mapping tasks, execute the second reduction task on the execution result of the X first mapping tasks by Z processes in the Y processes, and Y and Z are positive integers.

9. The computing system of claim 7, wherein, The first computing node comprises Y processes configured to execute the X first mapping tasks, wherein a first process in the Y processes comprises A threads configured to execute A first mapping tasks; The first computing node is configured to: After the A threads execute the A first mapping tasks, execute the second reduction task on the execution result of the A first mapping tasks by B threads in the A threads.

10. The computing system of claim 7, wherein, The first computing node comprises Y processes configured to execute the X first mapping tasks, wherein a first process in the Y processes comprises A threads configured to execute A first mapping tasks; The first computing node is configured to: After the A threads execute the A first mapping tasks, execute the second reduction task on the execution result of the A first mapping tasks by B threads in the A threads; Execute the second reduction task on the result of executing the second reduction task by Z processes in the Y processes again.

11. The computing system of any of claims 7-10, wherein, The management node is further configured to: After the M computing nodes are allocated for the MapReduce task, set the number of times of executing the second reduction task for each of the M computing nodes, and instruct the M computing nodes to execute the second reduction task according to the specified number of times of executing the second reduction task.

12. The computing system of any of claims 7-11, wherein, The first computing node comprises a trusted execution environment (TEE) and a rich execution environment (REE); and the first computing node is further configured to: Decrypt data required by the first computing node for executing the first mapping task after the data is moved from the REE to the TEE; After the first computing node executes the at least one second reduction task, encrypt the execution result, move the encrypted execution result from the TEE to the REE, and store the encrypted execution result.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that, when executed by a plurality of computing devices, cause the plurality of computing devices to perform the method of any of claims 1-6.

14. A computer program product, characterised in that, The computer program product, when executed on a plurality of computing devices, causes the plurality of computing devices to implement the method of any of claims 1-6.

Images