Parallel task processing method and related device

Abstract

The invention discloses a parallel task processing method and a related device, and belongs to the technical field of computers. The method comprises the following steps: receiving a task allocation message sent by an MPI scheduler; the at least one MPI process is started, and the distributed tasks are executed through the at least one MPI process; selecting a target MPI process from the plurality of MPI processes; and if the target MPI process is one MPI process in the at least one MPI process, and whenever any one MPI process in the plurality of MPI processes is processed by the allocated task, performing post-processing on a task processing result of any one MPI process through the target MPI process. Due to the fact that the target MPI process is the process with the fastest task processing speed, the first computing node can carry out post-processing on the task processing result of the MPI process through the target MPI process whenever any MPI process is processed by the distributed task, and the parallel task processing speed can be increased.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a parallel task processing method and related devices. Background Art

[0002] The message passing interface (MPI) has the characteristics of high performance, scalability, and portability, and is widely used in high-performance computing such as parallel task processing.

[0003] In related technologies, the MPI scheduler dynamically detects the remaining central processing unit (CPU) and memory resources of each computing node in a computing cluster. Based on these resources, it assigns tasks to each computing node, allowing the multiple computing nodes to process tasks in parallel. After the multiple computing nodes have completed their parallel tasks, the task processing results are aggregated and processed.

[0004] However, when the CPU computing power (i.e., computing capacity) on different computing nodes is different, processing parallel tasks according to the above method will result in a long processing time for the entire parallel task and low task processing efficiency. Summary of the Invention

[0005] This application provides a parallel task processing method and related devices, which can speed up the processing of parallel tasks. The technical solution is as follows:

[0006] In a first aspect, a parallel task processing method is provided, which is applied to a first computing node included in a computing cluster, wherein the first computing node is one of multiple computing nodes in the computing cluster currently used to process parallel tasks; the method includes: receiving a task assignment message sent by an MPI scheduler, the task assignment message indicating at least one MPI process in the first computing node currently used to process the parallel task and the task assigned to each MPI process; starting the at least one MPI process, and executing the respective assigned tasks through the at least one MPI process; selecting a target MPI process from multiple MPI processes, wherein the multiple MPI processes are MPI processes in the multiple computing nodes used to process the parallel tasks, and the target MPI process is the process with the fastest task processing speed among the multiple MPI processes; if the target MPI process is one of the at least one MPI process, and whenever the task processing assigned to any one of the multiple MPI processes is completed, the task processing result of the any one MPI process is post-processed by the target MPI process.

[0007] Since the target MPI process is the process with the fastest task processing speed, and whenever any of the multiple MPI processes completes the assigned task processing, no matter whether the execution process of other MPI processes is in the task calculation stage or the result output stage, the first computing node will post-process the task processing results of the MPI process that has completed the task processing through the target MPI process, thereby realizing the simultaneous execution of the task calculation stage and / or the result output stage and the post-processing, which can speed up the parallel task processing speed.

[0008] Optionally, the method also includes: in the process of starting the at least one MPI process, running I / O middleware in the at least one MPI process; selecting the target MPI process from multiple MPI processes includes: obtaining the processor computing power of the multiple computing nodes through the I / O middleware, and the processor computing power of the multiple computing nodes is obtained through the I / O middleware running in the MPI processes included in each of them; selecting the target computing node with the strongest processor computing power from the multiple computing nodes; and selecting the target MPI process from the MPI processes belonging to the target computing node in the multiple MPI processes.

[0009] Since a computing node with a higher processor power has a higher task processing speed for the MPI process in that computing node, and a computing node with a lower processor power has a lower task processing speed for the MPI process in that computing node, after obtaining the processor power of the multiple computing nodes through the I / O middleware, the computing node with the highest processor power among the multiple computing nodes can be used as the target computing node, enabling the target MPI process to be determined in advance, thereby improving the speed and accuracy of determining the target MPI process.

[0010] Optionally, the method also includes: in the process of starting the at least one MPI process, running input / output I / O middleware in the at least one MPI process; selecting the target MPI process from multiple MPI processes includes: synchronizing the task processing status of the multiple MPI processes through the I / O middleware, the task processing status indicating whether the corresponding MPI process has processed the assigned tasks; based on the task processing status of the multiple MPI processes, selecting the target MPI process from the multiple MPI processes.

[0011] Since the I / O middleware has a communication function, the first computing node can obtain the task processing status of the MPI process included in each computing node among the multiple computing nodes, and can not only determine whether at least one MPI process included in itself has processed the assigned task, but also determine whether the MPI processes in other computing nodes have processed the assigned tasks. If any task processing status indicates that a certain MPI process has processed the assigned task, it means that the task processing speed of the MPI process is the fastest. Therefore, the MPI process can be determined as the target MPI process. In other words, the target MPI process can be determined in the process of the multiple MPI processes processing parallel tasks, and when there are some hardware acceleration units in the computing nodes that are not taken into account in the processor computing power, it can avoid the situation where the target MPI process is not the fastest in task processing due to misjudgment of the processor computing power.

[0012] Optionally, before synchronizing the task processing status of the multiple MPI processes through the I / O middleware, the method further includes: if the total number of the multiple MPI processes is not greater than the number threshold, executing the step of synchronizing the task processing status of the multiple MPI processes through the I / O middleware.

[0013] If the total number of the multiple MPI processes is not greater than the number threshold, it indicates that the cost of synchronizing the task processing states of the multiple processes through the I / O middleware is low. If the total number of the multiple MPI processes is greater than the number threshold, it indicates that the cost of synchronizing the task processing states of the multiple MPI processes through the I / O middleware is high. In other words, only when the total number of the multiple MPI processes is not greater than the number threshold, that is, when the cost is low, will the step of synchronizing the task processing states of the multiple MPI processes through the I / O middleware be performed, thereby saving costs.

[0014] Optionally, the multiple MPI processes respectively correspond to a storage file, and the storage file is a storage area allocated to the corresponding MPI process in the storage node; the method also includes: for each MPI process in the at least one MPI process, if the task processing assigned to the MPI process is completed, the I / O middleware running in the MPI process writes its own task processing result into the storage file corresponding to itself; whenever the task processing assigned to any one of the multiple MPI processes is completed, the task processing result of any one of the multiple MPI processes is post-processed by the target MPI process, including: whenever the task processing assigned to any one of the multiple MPI processes is completed, the task processing result is obtained from the storage file corresponding to the any one MPI process through the target MPI process, and the obtained task processing result is post-processed.

[0015] Since the multiple MPI processes correspond to a storage file respectively, and the I / O middleware has a writing function, after each MPI process completes the processing of the assigned task, it can write its own task processing results into the corresponding storage file through the I / O middleware running in the MPI process. In this way, the multiple MPI processes can write multiple task processing results into the corresponding storage file at the same time. Compared with the case where the multiple MPI processes correspond to the same storage file and can only write one task processing result into the storage file at the same time, the efficiency of the result output of the multiple MPI processes can be improved, thereby further accelerating the processing speed of parallel tasks.

[0016] In a second aspect, a parallel task processing apparatus is provided, wherein the parallel task processing apparatus has the function of implementing the behavior of the parallel task processing method in the first aspect. The parallel task processing apparatus includes at least one module, wherein the at least one module is used to implement the parallel task processing method provided in the first aspect.

[0017] In a third aspect, a computing cluster is provided, comprising a plurality of computing nodes, each computing node comprising a processor and a memory, the memory being configured to store a computer program for executing the parallel task processing method provided in the first aspect. The processor is configured to execute the computer program stored in the memory to implement the parallel task processing method described in the first aspect.

[0018] Optionally, each computing node may further include a communication bus, which is used to establish a connection between the processor and the memory.

[0019] In a fourth aspect, a computer-readable storage medium is provided, wherein the storage medium stores instructions. When the instructions are executed on a computing node, the computing node executes the steps of the parallel task processing method described in the first aspect.

[0020] In a fifth aspect, a computer program product comprising instructions is provided. When the instructions are executed on a computing cluster, the computing nodes execute the steps of the parallel task processing method described in the first aspect.

[0021] The technical effects obtained in the above-mentioned second, third, fourth and fifth aspects are similar to the technical effects obtained by the corresponding technical means in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a schematic diagram of the structure of an implementation environment provided by an embodiment of the present application;

[0023] Figure 2 This is a schematic diagram of another implementation environment provided by an embodiment of the present application;

[0024] Figure 3 is a structural diagram of a computing device provided in an embodiment of the present application;

[0025] Figure 4 This is a flowchart of a parallel task processing method provided by an embodiment of the present application;

[0026] Figure 5 This is a schematic diagram of the execution process of an MPI process provided in an embodiment of the present application;

[0027] Figure 6 This is a schematic diagram of an MPI process providing an embodiment of the present application for writing task processing results into a storage file;

[0028] Figure 7 This is a flowchart of another parallel task processing method provided by an embodiment of the present application;

[0029] Figure 8 This is a flowchart of another parallel task processing method provided by an embodiment of the present application;

[0030] Figure 9 It is a structural diagram of a parallel task processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0032] Before explaining in detail the parallel task processing method provided in the embodiment of the present application, the terminology, application scenarios and implementation environment involved in the embodiment of the present application are first introduced.

[0033] First, the terms involved in the embodiments of the present application are introduced.

[0034] MPI: MPI is a cross-language communication protocol for programming parallel computers. It supports both point-to-point and broadcast communication. MPI's goals are high performance, scalability, and portability. MPI remains the dominant model for high-performance computing today.

[0035] Heterogeneous system: Multiple processors of different types are used to handle parallel tasks. Each component in the heterogeneous system has its own autonomy. While achieving data sharing, it still retains its own application characteristics, integrity characteristics, and security characteristics.

[0036] Process (or rank): is the execution process of a program, the basic unit for task allocation by the MPI scheduler, and the basic execution unit.

[0037] Next, the application scenarios involved in the embodiments of this application are introduced.

[0038] In related technologies, the MPI scheduler needs to dynamically perceive the remaining CPU and memory resources of each computing node in the computing cluster. Based on these remaining CPU and memory resources, it assigns tasks to each computing node, allowing multiple computing nodes to process tasks in parallel. Only after the parallel tasks of the multiple computing nodes have been processed are the task processing results of the multiple computing nodes aggregated and processed.

[0039] However, when the CPU computing power (i.e., computing capacity) on different computing nodes is different or the multiple computing nodes are in a heterogeneous system, the CPU with stronger computing power needs to wait for the CPU with weaker computing power to complete the parallel task processing after the parallel task processing is completed. Only after the CPU with weaker computing power completes the parallel task processing will the task processing results of the multiple computing nodes be post-processed. Therefore, processing parallel tasks according to the above method will result in a longer processing time for the entire parallel task and lower task processing efficiency. In addition, the workload of the tasks assigned to each computing node by the MPI scheduler is usually the same, and the MPI scheduler cannot perceive the time spent by each computing node to process the parallel tasks. Therefore, it is difficult to reasonably assign tasks to the multiple computing nodes based on the strength of the CPU computing power.

[0040] Based on this, the present application provides a parallel task processing method, which selects a target MPI process from multiple MPI processes. When the task assigned to any one of the multiple MPI processes is completed, the computing node will post-process the task processing result through the target MPI process, instead of waiting until all tasks are processed before post-processing, thereby realizing task processing and post-processing at the same time, thereby shortening the processing time of the entire parallel task and greatly improving the efficiency of task processing.

[0041] Finally, the implementation environment involved in the embodiments of this application is introduced.

[0042] Figure 1 This is a schematic diagram of the structure of an implementation environment provided by the embodiment of this application. Figure 1 The implementation environment includes an MPI scheduler 101 and multiple computing nodes 102. Each computing node 102 includes a CPU, each CPU includes at least one core, and each computing node 102 includes at least one MPI process 103. When executing a task, the at least one process 103 runs on the core included in the CPU. The MPI scheduler 101 can communicate with the multiple computing nodes 102. The communication connection can be a wired connection or a wireless connection, which is not limited in this embodiment of the application.

[0043] The MPI scheduler 101 is used to obtain parallel tasks submitted by users, and based on the parallel tasks submitted by users, select computing nodes 102 for executing the parallel tasks, thereby sending task allocation messages to the selected computing nodes 102 .

[0044] The computing node 102 is used to receive the task assignment message sent by the MPI scheduler 101, and start at least one MPI process 103, and execute the respective assigned tasks through at least one MPI process 103; select a target MPI process from multiple MPI processes 103, if the target MPI process is an MPI process in at least one MPI process 103, and whenever the task processing assigned to any one of the multiple MPI processes 103 is completed, the task processing result of any one of the MPI processes is post-processed through the target MPI process.

[0045] In some embodiments, please refer to Figure 2 The at least one MPI process 103 also has an I / O middleware 104. The I / O middleware 104 runs in the at least one MPI process when the computing node 102 starts the at least one MPI process 103.

[0046] exist Figure 2 In the illustrated implementation, compute node 102 is further configured to obtain the processor computing power of multiple compute nodes 102 via I / O middleware 104, select a target compute node with the strongest processor computing power from the multiple compute nodes 102, and select a target MPI process from the multiple MPI processes 103 that belong to the target compute node. Alternatively, compute node 102 is further configured to synchronize the task processing status of multiple MPI processes 103 via I / O middleware 104, and select a target MPI process from the multiple MPI processes 103 based on the task processing status of the multiple MPI processes 103.

[0047] In some embodiments, the above implementation environment is implemented based on the MPI framework, so the parallel task can also be called an MPI parallel task.

[0048] Please refer to Figure 3 , Figure 3 is a schematic diagram of a computing device according to an embodiment of the present application. The computing device may be Figure 1 The computing node 102 shown in FIG. The computing device includes at least one processor 301 , a communication bus 302 , a memory 303 and at least one communication interface 304 .

[0049] The processor 301 may be a general-purpose central processing unit (CPU), a network processor (NP), a microprocessor, or one or more integrated circuits for implementing the solution of the present application, such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0050] Communication bus 302 is used to transmit information between the above components. Communication bus 302 can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or one type of bus.

[0051] The memory 303 may be a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), an optical disc (including a compact disc read-only memory (CD-ROM), a compact disc, a laser disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 303 may exist independently and be connected to the processor 301 via the communication bus 302. The memory 303 may also be integrated with the processor 301.

[0052] The communication interface 304 uses any device, such as a transceiver, for communicating with other devices or communication networks. The communication interface 304 includes a wired communication interface and may also include a wireless communication interface. For example, the wired communication interface may be an Ethernet interface. The Ethernet interface may be an optical interface, an electrical interface, or a combination thereof. The wireless communication interface may be a wireless local area network (WLAN) interface, a cellular network communication interface, or a combination thereof.

[0053] As an example, the processor 301 may include one or more CPUs, such as Figure 3 CPU0 and CPU1 are shown in the figure.

[0054] As an example, a computing device may include multiple processors, such as Figure 3 301 and processor 305 are shown in FIG. Each of these processors can be a single-core processor or a multi-core processor. A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0055] In some embodiments, the computing device may further include an output device and an input device. The output device communicates with the processor 301 and can display information in a variety of ways. For example, the output device may be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. The input device communicates with the processor 301 and can receive user input in a variety of ways. For example, the input device may be a mouse, a keyboard, a touch screen device, or a sensor device.

[0056] In some embodiments, the memory 303 is used to store the program code 310 for executing the solution of the present application, and the processor 301 can execute the program code 310 stored in the memory 303. The program code 310 may include one or more software modules, and the computing device can implement the following by using the processor 301 and the program code 310 in the memory 303. Figure 4 The embodiment provides a parallel task processing method.

[0057] Next, the parallel task processing method provided in the embodiment of the present application is explained in detail.

[0058] Figure 4This is a flowchart of a parallel task processing method provided by an embodiment of the present application. The method is applied to a first computing node included in a computing cluster. The first computing node is one of the multiple computing nodes currently used to process parallel tasks in the computing cluster. Figure 4 , the method includes the following steps.

[0059] Step 401: Receive a task allocation message sent by an MPI scheduler, where the task allocation message indicates at least one MPI process currently used to process parallel tasks in a first computing node and the tasks allocated to each MPI process.

[0060] In some embodiments, the MPI scheduler dynamically senses the remaining memory resources and number of MPI processes of each computing node in the computing cluster, and based on the remaining memory resources and number of MPI processes of each computing node in the computing cluster, as well as the memory resources and number of MPI processes required to process the parallel task, determines from the computing cluster the multiple computing nodes currently used to process the parallel task. In other words, the remaining memory resources of the multiple computing nodes currently used to process the parallel task must be greater than or equal to the memory resources required to process the parallel task, and similarly, the remaining number of MPI processes of the multiple computing nodes must also be greater than or equal to the number of MPI processes required to process the parallel task.

[0061] For example, assuming that the computing cluster includes a first computing node, a second computing node, and a third computing node, if the MPI scheduler dynamically perceives that the remaining memory resources in the first computing node are X and the remaining number of MPI processes is A, the remaining memory resources in the second computing node are Y and the remaining number of MPI processes is B, the remaining memory resources in the third computing node are Z and the remaining number of MPI processes is C, and the memory resources required to process parallel tasks are V and the required number of MPI processes is D. If V is less than or equal to the sum of X, Y, and Z, and E is also less than or equal to the sum of A, B, and C, it means that the remaining memory resources and the number of MPI processes of the first computing node, the second computing node, and the third computing node all meet the requirements for processing the parallel task. In this way, the MPI scheduler can determine the first computing node, the second computing node, and the third computing node as the multiple computing nodes currently used to process the parallel task; if V is less than or equal to the sum of X and Y, and E is also less than or equal to the sum of A and B, it means that the remaining memory resources and the number of MPI processes of the first computing node and the second computing node can meet the requirements for processing the parallel task. In this way, the MPI scheduler can determine the first computing node and the second computing node as the multiple computing nodes currently used to process the parallel task.

[0062] After determining the multiple computing nodes currently used to process the parallel task, the MPI scheduler can assign tasks to each of the multiple computing nodes based on the remaining memory resources and number of MPI processes of the multiple computing nodes. In other words, the memory resources and number of MPI processes required by the computing node to process the assigned task must be less than or equal to the remaining memory resources and number of MPI processes in the computing node. Then, a corresponding task assignment message is sent to each computing node. Since the first computing node is one of the multiple computing nodes currently used to process the parallel task, the first computing node can receive the task assignment message sent by the MPI scheduler.

[0063] In some embodiments, since the first computing node includes multiple MPI processes, and the multiple MPI processes may or may not all be currently used to process parallel tasks, that is, at least one MPI process among the multiple MPI processes in the first computing node is currently used to process parallel tasks, the first computing node can determine the at least one MPI process currently used to process parallel tasks and the tasks assigned to each MPI process based on the at least one MPI process currently used to process parallel tasks and the tasks assigned to each MPI process indicated in the task assignment message.

[0064] In some embodiments, the MPI scheduler can run the MPI framework through the MPI program startup command, and the multiple computing nodes are run based on the MPI framework. Since the number of MPI processes currently used to process parallel tasks in the multiple computing nodes may be different, the tasks assigned to each MPI process may also be different. In other words, the task assignment message sent by the MPI scheduler to each computing node in the multiple computing nodes may be different. Therefore, the MPI scheduler can add an information sending parameter to the MPI program startup command, and send the MPI program startup command with the added information sending parameter to the corresponding computing node, thereby sending the task assignment message to the corresponding computing node. That is, the task assignment message is implemented through the MPI program startup command, and the information sending parameter in the MPI program startup command indicates at least one MPI process currently used to process parallel tasks in the corresponding computing node and the tasks assigned to each MPI process.

[0065] For example, the MPI program startup command is "mpirun -". The MPI scheduler can add the "-hostfile" parameter to the above command to send the task assignment message to the corresponding computing node. Of course, in actual applications, other methods can also be used to enable the MPI scheduler to send the task assignment message to the corresponding computing node, and the embodiments of the present application are not limited to this.

[0066] In some embodiments, before the first computing node receives the task assignment message sent by the MPI scheduler, the MPI scheduler can provide a configuration interface in which the user can submit task description information. The task description information is used to indicate the memory resources occupied by the parallel tasks to be processed and the number of MPI processes required to process the parallel tasks. In this way, the MPI scheduler can determine the multiple computing nodes currently used to process the parallel tasks based on the parallel task description information.

[0067] Step 402: Start the at least one MPI process, and execute the assigned tasks through the at least one MPI process.

[0068] In some embodiments, after receiving a task assignment message sent by the MPI scheduler, at least one MPI process can be started based on the at least one MPI process currently used to process the parallel task indicated by the task assignment message, and the task assignment message also indicates the tasks assigned to each MPI process, so that the respective assigned tasks can be executed by the at least one MPI process.

[0069] In some embodiments, since the MPI scheduler can run the MPI framework through the MPI program startup command, and the first computing node runs based on the MPI framework, after the MPI framework runs, the first computing node will also start running, that is, the MPI program startup command is also used to start the at least one MPI process in the first computing node. Based on the above description, it can be seen that at least one MPI process in the first computing node is currently used to process parallel tasks. Therefore, the MPI scheduler can add a process quantity parameter to the MPI program startup command to enable the first computing node to start a specified number of MPI processes. That is, the information sending parameter includes the process quantity parameter. For example, the MPI scheduler can add a "-N" parameter to the MPI startup command to enable the first computing node to start at least one MPI process currently used to process parallel tasks. Of course, in actual applications, other methods can also be used to enable the first computing node to start a corresponding number of MPI processes, and the embodiments of the present application are not limited to this.

[0070] In some embodiments, as Figure 5As shown, when each MPI process executes its assigned task, the execution process is generally divided into two stages: task calculation and result output. Task calculation involves calculating the assigned task to obtain the task processing result for that task; result output involves performing optimization operations such as aggregation, concatenation, and reorganization on the task processing results. The output task processing results can be stored in a storage node. This storage node is used to store the task processing results of at least one MPI process in the first computing node and also to store the task processing results of at least one MPI process in other computing nodes. In other words, this storage node is the storage node where the task processing results of multiple computing nodes currently processing parallel tasks are stored together.

[0071] In some embodiments, the parallel task includes multiple iterative tasks, each iterative task includes multiple subtasks that need to be processed simultaneously. The multiple MPI processes currently used to process parallel tasks can simultaneously process multiple subtasks in the same iterative task. After the multiple MPI processes complete the processing of the first iterative task, they will process the second iterative task. The parallel task is considered to be completed only when all the multiple iterative tasks are completed.

[0072] Step 403: Select a target MPI process from the multiple MPI processes, where the multiple MPI processes are MPI processes in the multiple computing nodes for processing parallel tasks, and the target MPI process is the process with the fastest task processing speed among the multiple MPI processes.

[0073] In some embodiments, the multiple MPI processes are MPI processes in the multiple computing nodes used to process and execute tasks. That is, the multiple MPI processes include at least one MPI process in the first computing node and at least one MPI process in other computing nodes. Therefore, the target MPI process can be selected from the multiple MPI processes by the first computing node or by another computing node other than the first computing node. However, no matter which computing node selects the target MPI process from the multiple MPI processes, the goal is to select the process with the fastest task processing speed from the multiple MPI processes. Therefore, the target MPI process selected from the multiple MPI processes by the first computing node or another computing node is the same.

[0074] Since the target process is the process with the fastest task processing speed among the multiple MPI processes, in some embodiments, there is only one process with the fastest task processing speed among the multiple MPI processes, and the process with the fastest task processing speed is determined as the target MPI process; however, in other embodiments, there are multiple processes with the same and fastest task processing speed among the multiple MPI processes, and the process with the same and fastest task processing speed is determined as the target MPI process. In other words, one MPI process can be selected as the target MPI process, and at least two MPI processes can also be selected as the target MPI process.

[0075] In the embodiments of the present application, a target MPI process can be selected in a variety of ways, such as selecting a target MPI process based on processor computing power, or selecting a target MPI process by synchronizing task processing status, etc. Next, the target MPI process selection process will be described using the following two implementation methods, using processor computing power and synchronizing task processing status as examples.

[0076] The first implementation method is to run I / O middleware in the at least one MPI process during the process of starting the at least one MPI process; the implementation process of selecting the target MPI process from the multiple MPI processes includes: obtaining the processor computing power of the multiple computing nodes through the I / O middleware, and the processor computing power of the multiple computing nodes is obtained through the I / O middleware running in the MPI processes included in each of them; selecting the target computing node with the strongest processor computing power from the multiple computing nodes; and selecting the target MPI process from the MPI processes belonging to the target computing node in the multiple MPI processes.

[0077] In some embodiments, the MPI scheduler can add middleware operating parameters to the MPI program startup command, and the middleware operating parameters are used to indicate the operation of the I / O middleware, that is, the information sending parameters include the middleware operating parameters, so that when the first computing node starts the at least one MPI process, the I / O middleware can be run in each of the at least one MPI process. For example, the MPI scheduler can add the "LD_PRELOAD" parameter to the MPI program startup command to enable the first computing node to run the I / O middleware in the at least one MPI process. Of course, in actual applications, other methods can also be used to run the I / O middleware in the at least one MPI process, and the embodiments of the present application are not limited to this. Similarly, other computing nodes except the first computing node can also run the I / O middleware into each MPI process through the above method. In other words, each MPI process in each of the multiple computing nodes runs the I / O middleware.

[0078] The I / O middleware has communication and write functions. It can communicate with other I / O middleware through the communication function and optimize the task processing results of the MPI process and store them in a storage area through the write function. Because each MPI process in each of the multiple compute nodes runs an I / O middleware, each MPI process can obtain the processor computing power of its corresponding compute node through the I / O middleware running on it.

[0079] In some embodiments, the I / O middleware running in each MPI process can obtain the processor computing power of the computing node where it is located through computing power acquisition instructions. For example, assuming that I / O middleware is running in MPI process 1 in the first computing node, the I / O middleware can obtain the processor computing power of the first computing node through the "Iscpu" command, and can also obtain the processor computing power of the first computing node through the "cat / sys / devices / system / cpu / cpu* / cpufreq / scaling_cur_freq" command. Of course, in actual applications, the I / O middleware can also obtain the processor computing power of the corresponding computing node through other methods, and the embodiments of the present application do not limit this.

[0080] In some embodiments, since at least one MPI process in the first computing node is currently used to process parallel tasks, and each of the at least one MPI process runs I / O middleware, any one of the at least one MPI processes can be selected so that the MPI process obtains the processor computing power of the first computing node through the I / O middleware running on it. Of course, each of the at least one MPI process can also obtain the processor computing power of the first computing node through the I / O middleware running on it. Similarly, computing nodes other than the first computing node can also obtain processor computing power using the above method.

[0081] For example, assume that a computing cluster includes a first computing node and a second computing node, both of which are currently used to process parallel tasks. The first computing node includes MPI processes 1, 2, and 3, and the second computing node includes MPI processes 4, 5, and 6. The first computing node can select MPI process 1 from among MPI processes 1, 2, and 3 to obtain the processor computing power of the first computing node through its own running I / O middleware. Similarly, the second computing node can select MPI process 6 from among MPI processes 4, 5, and 6 to obtain the processor computing power of the second computing node through its own running I / O middleware. Of course, MPI processes 1, 2, and 3 in the first computing node can also obtain the processor computing power of the first computing node through their own running I / O middleware. Similarly, MPI processes 4, 5, and 6 in the second computing node can also obtain the processor computing power of the second computing node through their own running I / O middleware.

[0082] Continuing with the above description, since the processor computing power of the first computing node is obtained through the I / O middleware running in the MPI process included in the first computing node, and the I / O middleware has a communication function, after the I / O middleware obtains the processor computing power of the first computing node, it can send the processor computing power of the first computing node to other I / O middleware and also obtain the processor computing power of other computing nodes. The other I / O middleware is the I / O middleware running in the MPI process included in other computing nodes other than the first computing node.

[0083] In this way, the first computing node can not only obtain its own processor computing power, but also obtain the processor computing power of other computing nodes. Other computing nodes can also obtain their own processor computing power and the processor computing power of the first computing node.

[0084] In other embodiments, the technician can specify any one computing node to obtain the processor computing power of the multiple computing nodes. For example, the technician can specify the first computing node to obtain the processor computing power of the multiple computing nodes. That is to say, the I / O middleware running in the MPI process included in the first computing node sends a computing power acquisition request to other I / O middleware, and the other I / O middleware only needs to reply the processor computing power of its corresponding computing node to the first computing node. In this way, the first computing node can obtain the processor computing power of the multiple computing nodes through the I / O middleware.

[0085] The stronger the processor computing power of a computing node, the faster the task processing speed of the MPI process in the computing node; the weaker the processor computing power of a computing node, the slower the task processing speed of the MPI process in the computing node. Therefore, after obtaining the processor computing power of the multiple computing nodes, the computing node with the strongest processor computing power among the multiple computing nodes can be used as the target computing node. In some embodiments, only one computing node among the multiple computing nodes has the strongest processor computing power, and this computing node is used as the target computing node; in other embodiments, at least two computing nodes among the multiple computing nodes have the same processor computing power and the strongest processor computing power, and the at least two computing nodes are used as the target computing nodes. That is to say, in an embodiment of the present application, the target computing node may include one computing node or at least two computing nodes.

[0086] For example, the processor computing power of the first computing node is X, and the processor computing power of the second computing node is Y. If X is greater than Y, the first computing node is used as the target computing node; if X is less than Y, the second computing node is used as the target computing node; if X is equal to Y, both the first computing node and the second computing node are used as target computing nodes.

[0087] Based on the above description, it can be seen that each of the multiple computing nodes can obtain the processor computing power of the multiple computing nodes, or the technician specifies one of the multiple computing nodes to uniformly obtain the processor computing power of the multiple computing nodes. Therefore, each of the multiple computing nodes can determine the target computing node, or the computing node designated by the technician determines the target computing node. However, since the determined target computing node is the node with the strongest processor computing power among the multiple computing nodes, whether each computing node determines the target computing node or one computing node determines the target computing node, the determined target computing node is the same.

[0088] Continuing with the above description, after determining the target computing node, a target MPI process can be selected from the MPI processes in the target computing node. If the target computing node includes one computing node, some or all of the MPI processes can be selected from at least one MPI process in the target computing node as the target MPI process, for example, any one of the MPI processes can be selected as the target MPI process. If the target computing node includes at least two computing nodes, any one of the at least two computing nodes can be selected, and some or all of the MPI processes from the computing node can be selected as the target MPI process. Alternatively, a computing node greater than one can be selected from the at least two computing nodes, and some or all of the MPI processes from the multiple MPI processes included in the selected computing node can be selected as the target MPI process.

[0089] For example, if the target computing node is the first computing node, then any MPI process can be selected from at least one MPI process of the first computing node as the target MPI process, and all MPI processes in the first computing node can also be selected as target MPI processes; if the target computing node is the first computing node and the second computing node, then the second computing node can be selected from the above two computing nodes, and then some or all MPI processes can be selected from at least one MPI process of the second computing node as the target MPI process, and some or all MPI processes can be selected from multiple MPI processes included in the first computing node and the second computing node as the target MPI process.

[0090] The second implementation method is to run I / O middleware in the at least one MPI process during the process of starting the at least one MPI process; the implementation process of selecting the target MPI process from the multiple MPI processes includes: synchronizing the task processing status of the multiple MPI processes through the I / O middleware, and the task processing status indicates whether the corresponding MPI process has processed the assigned tasks; based on the task processing status of the multiple MPI processes, selecting the target MPI process from the multiple MPI processes.

[0091] In the second implementation, in the process of starting the at least one MPI process, the method of running the I / O intermediate in the at least one MPI process is the same as the method in the first implementation mentioned above, which will not be repeated here. Please refer to the above content.

[0092] In some embodiments, since the I / O middleware has a communication function, the first computing node can interact with the I / O middleware corresponding to other computing nodes through its own I / O middleware to synchronize the task processing status of multiple MPI processes in the multiple computing nodes. That is, the first computing node can obtain the task processing status of the MPI process included in each computing node in the multiple computing nodes, and can not only determine whether at least one MPI process included in itself has processed the assigned task, but also determine whether the MPI process in other computing nodes has processed the assigned task. Similarly, other computing nodes except the first node can also obtain the task processing status of the MPI process included in each computing node in the multiple computing nodes in the above manner.

[0093] Continuing with the above description, after the first synchronization, that is, after the first computing node interacts for the first time with the I / O middleware of other computing nodes through its own I / O middleware, if it is determined through the acquired task processing status that the MPI process in each computer node in the multiple computing nodes has not completed processing the assigned tasks, then the first computing node can perform a second synchronization after a period of time until there is an MPI process in the MPI process of a computing node among the multiple computing nodes that has completed processing the assigned tasks.

[0094] For example, assuming that the computing nodes currently used to process parallel tasks in the computing cluster include a first computing node, a second computing node, and a third computing node, the first computing node, the second computing node, and the third computing node can interact through their own I / O middleware to synchronize the task processing states of multiple MPI processes in the three nodes. In other words, the first computing node can interact with the I / O middleware in the second computing node through its own I / O middleware to synchronize the task processing states of the MPI processes included in the second computing node. Similarly, the first computing node can also interact with the I / O middleware in the third computing node through its own I / O middleware to synchronize the task processing states of the MPI processes included in the third node. After the first synchronization, if the MPI processes in the second and third computing nodes have not completed processing the assigned tasks, and the MPI processes in the first computing node have not completed processing the assigned tasks, then the first computing node can perform a second synchronization. Assuming that after the second synchronization, at least one MPI process in the second computing node has completed processing the assigned task, at this time, the first computing node can stop synchronizing the task processing states of the multiple MPI processes.

[0095] Continuing with the above description, after synchronizing the task processing status of the multiple MPI processes, a target MPI process can be selected from the multiple MPI processes based on the task processing status of the multiple MPI processes. That is, if at least one MPI process among the multiple MPI processes has completed processing the assigned tasks, it means that the task processing speed of the at least one MPI process is the fastest. Therefore, some or all of the MPI processes in the at least one MPI process can be used as target MPI processes. For example, if MPI process 1, MPI process 2, and MPI process 3 in the first computing node have completed processing the assigned tasks, while the MPI processes in other computing nodes have not completed processing the assigned tasks, it means that the MPI process 1, MPI process 2, and MPI process 3 in the first computing node have the fastest task processing speed. Therefore, one or more MPI processes can be selected from the above-mentioned MPI process 1, MPI process 2, and MPI process 3 as target MPI processes.

[0096] In some embodiments, before synchronizing the task processing status of the multiple MPI processes through the I / O middleware, the method also includes: if the total number of the multiple MPI processes is not greater than the number threshold, executing the step of synchronizing the task processing status of the multiple MPI processes through the I / O middleware.

[0097] That is, if the total number of the multiple MPI processes is not greater than the quantity threshold, it means that the cost of synchronizing the task processing states of the multiple processes through the I / O middleware is relatively low, and therefore, the task processing states of the multiple MPI processes can be synchronized through the I / O middleware; if the total number of the multiple MPI processes is greater than the quantity threshold, it means that the cost of synchronizing the task processing states of the multiple MPI processes through the I / O middleware is relatively high, and therefore, the task processing states of the multiple MPI processes are not synchronized through the I / O middleware. For example, assuming that the quantity threshold is 1000, if the total number of MPI processes is 900, then the task processing states of the multiple MPI processes can be synchronized through the I / O middleware; if the total number of MPI processes is 1100, it means that the cost of synchronizing the task processing states of the multiple MPI processes through the I / O middleware is relatively high, and then the above steps are not performed.

[0098] It should be noted that the above description is based on the number threshold being 1000. Of course, in actual applications, the number threshold can be set according to actual conditions, and the embodiments of the present application do not limit this.

[0099] Step 404: If the target MPI process is an MPI process in the at least one MPI process, and whenever any one of the multiple MPI processes completes processing of a task assigned to it, the target MPI process performs post-processing on the task processing result of the any one MPI process.

[0100] In some embodiments, if the target MPI process is one of at least one MPI process included in the first computing node, and whenever any one of the multiple MPI processes included in the multiple computing nodes completes processing of a task assigned to it, the first computing node performs post-processing on the task processing result of any one of the multiple MPI processes through the target MPI process. In other words, as long as processing of a task assigned to one of the multiple MPI processes is completed, regardless of whether the execution process of the other MPI processes is in the task calculation phase or the result output phase, the first computing node will perform post-processing on the task processing result of the MPI process that has completed task processing through the target MPI process, thereby achieving the simultaneous execution of task calculation and / or result output and post-processing during the processing of parallel tasks, thereby accelerating the processing of parallel tasks.

[0101] In other embodiments, if the target MPI process is not an MPI process included in at least one MPI process of the first computing node, then the first computing node does not need to post-process the task processing results of any MPI process. As long as each MPI process in the first computing node completes the assigned task processing, it means that the first computing node has completed the corresponding parallel task processing.

[0102] In some embodiments, since multiple MPI processes currently processing parallel tasks store their task processing results in a storage node during the result output phase, the target MPI process can obtain any task processing result from the storage node and perform post-processing on any task processing result by performing an aggregation operation on the task processing result. The task processing result is then the task processing result of any MPI process. In this way, after all assigned tasks of all MPI processes are processed and all task processing results are stored in the storage node, the target MPI process can combine the task processing results of all MPI processes into a complete task processing result.

[0103] In other embodiments, the multiple MPI processes respectively correspond to a storage file, which is a storage area allocated to the corresponding MPI process in the storage node; the method also includes: for each MPI process in the at least one MPI process, if the task processing assigned to the MPI process is completed, the I / O middleware running in the MPI process writes its own task processing result into its own corresponding storage file; whenever the task processing assigned to any one of the multiple MPI processes is completed, the task processing result of any one of the multiple MPI processes is post-processed by the target MPI process. The implementation process includes: whenever the task processing assigned to any one of the multiple MPI processes is completed, the task processing result is obtained from the storage file corresponding to any one of the MPI processes through the target MPI process, and the obtained task processing result is post-processed.

[0104] Since the multiple MPI processes correspond to a storage file respectively, and based on the above description, it can be known that the I / O middleware has a write function. Therefore, after each MPI process completes the processing of the assigned task, it can write its own task processing results into the corresponding storage file through the I / O middleware running in the MPI process. In this way, the multiple MPI processes can write multiple task processing results into the corresponding storage file at the same time. Figure 6 As shown, compared with the case where the multiple MPI processes correspond to the same storage file and can only write one task processing result into the storage file at the same time, the efficiency of the multiple MPI processes in outputting results can be improved, thereby speeding up the processing speed of parallel tasks.

[0105] Continuing with the above description, since each MPI process will write the task processing results into its own corresponding storage file after completing the processing of the assigned task, whenever any MPI process among the multiple MPI processes completes the processing of the assigned task, the first computing node will obtain the task processing results from the storage file corresponding to any MPI process through the target MPI process, and post-process the obtained task processing results.

[0106] Next, the parallel task processing method provided in the embodiment of the present application will be generally introduced.

[0107] If the target MPI process is determined by the first implementation method in step 403, then the flowchart of the parallel task processing method provided in the embodiment of the present application is as follows: Figure 7 As shown. Figure 7 As can be seen in the figure, the computing nodes currently used to process parallel tasks are the first computing node and the second computing node. After the MPI scheduler sends the task allocation message, the first computing node can obtain the processor computing power of the first computing node and the processor computing power of the second node through the I / O middleware running in MPI process 1. Similarly, the second computing node can also obtain the processor computing power of the first computing node and the processor computing power of the second computing node through the I / O middleware running in MPI process 2, thereby determining that the target MPI process is MPI process 1; after MPI process 1 and MPI process 2 complete the processing of the assigned tasks, they will both write the task processing results to the storage node through the I / O middleware running on themselves; since MPI process 1 is the target MPI process, it means that MPI process 1 has the fastest task processing speed. Therefore, MPI process 1 will obtain the stored task processing results from the storage node through the I / O middleware running on itself and post-process the task processing results.

[0108] If the target MPI process is determined by the second implementation method in step 403, then the flowchart of the parallel task processing method provided in the embodiment of the present application is as follows: Figure 8 As shown. Figure 8 It can be seen that the computing nodes currently used to process parallel tasks are the first computing node and the second computing node. After the MPI scheduler sends the task allocation message, the MPI process 1 in the first computing node will write the task processing results to the storage node through the I / O middleware running on it. Similarly, the MPI process 2 in the second computing node will also write the task processing results to the storage node through the I / O middleware running on it. And both MPI process 1 and MPI process 2 will synchronize the task processing status of MPI process 1 and MPI process 2 through the I / O middleware. Figure 8As can be seen in the figure, the task synchronization status of MPI process 1 indicates that the task assigned to MPI process 1 has been processed. Therefore, MPI process 1 can be used as the target MPI process; then, MPI process 1 will obtain the stored task processing results from the storage node through the I / O middleware running on itself, and post-process the task processing results.

[0109] In an embodiment of the present application, since the target MPI process is the process with the fastest task processing speed, and whenever any one of the multiple MPI processes is assigned a task to complete processing, regardless of whether the execution process of other MPI processes is in the task calculation stage or the result output stage, the first computing node will post-process the task processing results of the MPI process that has completed the task processing through the target MPI process, thereby achieving the simultaneous execution of the task calculation stage and / or the result output stage and the post-processing, which can speed up the parallel task processing speed. Moreover, not only can the target MPI process be selected based on the processor computing power of the computing node, but the target MPI process can also be selected based on the task processing status of the multiple MPI processes, thereby improving the flexibility of selecting the target MPI process and making the application scenarios more extensive. In addition, the multiple MPI processes correspond to a storage file respectively. After the task processing assigned to each MPI process is completed, the I / O middleware running in the MPI process can write its own task processing results into its corresponding storage file, thereby improving the efficiency of result output during the MPI execution process and further speeding up the processing speed of parallel tasks.

[0110] Figure 9 This is a schematic diagram of the structure of a parallel task processing device provided by an embodiment of the present application. The parallel task processing device can be implemented by software, hardware, or a combination of both to become part or all of the first computing node. The first computing node can be Figure 3 The computing device shown. Figure 9 The device includes: a receiving module 901, a starting module 902, a selecting module 903 and a post-processing module 904.

[0111] Receiving module 901 is used to receive the task allocation message sent by the MPI scheduler, which indicates at least one MPI process currently used to process parallel tasks in the first computing node and the tasks assigned to each MPI process; for the detailed implementation process, please refer to the above embodiments and will not be repeated here.

[0112] The starting module 902 is used to start the at least one MPI process and execute the assigned tasks through the at least one MPI process. For the detailed implementation process, please refer to the above embodiments and will not be repeated here.

[0113] The selection module 903 is used to select a target MPI process from multiple MPI processes. The multiple MPI processes are the MPI processes used to process parallel tasks in the multiple computing nodes. The target MPI process is the process with the fastest task processing speed among the multiple MPI processes. For the detailed implementation process, please refer to the above embodiments and will not be repeated here.

[0114] A post-processing module 904 is configured to, if the target MPI process is one of the at least one MPI process, perform post-processing on the task processing result of any one of the multiple MPI processes by the target MPI process whenever the task processing assigned to any one of the multiple MPI processes is completed. The detailed implementation process is described in the above embodiments and will not be repeated here.

[0115] Optionally, the device further comprises:

[0116] A first running module is configured to run an input / output (I / O) middleware in the at least one MPI process during the process of starting the at least one MPI process;

[0117] The selection module 903 includes:

[0118] An acquisition submodule, configured to acquire the processor computing power of the plurality of computing nodes through the I / O middleware, wherein the processor computing power of the plurality of computing nodes is acquired through the I / O middleware running in the MPI process included in each of the plurality of computing nodes;

[0119] A first selection submodule is configured to select a target computing node with the strongest processor computing power from the multiple computing nodes;

[0120] The second selection submodule is configured to select a target MPI process from the MPI processes belonging to the target computing node in the multiple MPI processes.

[0121] Optionally, the device further comprises:

[0122] A second running module is configured to run an input / output (I / O) middleware in the at least one MPI process during the process of starting the at least one MPI process;

[0123] The selection module 903 includes:

[0124] A synchronization submodule is used to synchronize the task processing status of the multiple MPI processes through the I / O middleware, where the task processing status indicates whether the corresponding MPI process has completed processing the assigned task;

[0125] The third selection submodule is configured to select a target MPI process from the multiple MPI processes based on the task processing status of the multiple MPI processes.

[0126] Optionally, the device further comprises:

[0127] The trigger module is used to trigger the synchronization submodule to synchronize the task processing status of the multiple MPI processes through the I / O middleware if the total number of the multiple MPI processes is not greater than the number threshold.

[0128] Optionally, each of the multiple MPI processes corresponds to a storage file, and the storage file is a storage area allocated to the corresponding MPI process in the storage node; the device further includes:

[0129] a writing module configured to write, for each MPI process in the at least one MPI process, a task processing result of the MPI process into a corresponding storage file of the MPI process through the I / O middleware running in the MPI process if the task processing assigned to the MPI process is completed;

[0130] The post-processing module 904 is specifically used for:

[0131] Whenever any one of the multiple MPI processes completes processing of a task assigned to it, the target MPI process obtains the task processing result from the storage file corresponding to the any one of the MPI processes and performs post-processing on the obtained task processing result.

[0132] In an embodiment of the present application, since the target MPI process is the process with the fastest task processing speed, and whenever any one of the multiple MPI processes is assigned a task to complete processing, regardless of whether the execution process of other MPI processes is in the task calculation stage or the result output stage, the first computing node will post-process the task processing results of the MPI process that has completed the task processing through the target MPI process, thereby achieving the simultaneous execution of the task calculation stage and / or the result output stage and the post-processing, which can speed up the parallel task processing speed. Moreover, not only can the target MPI process be selected based on the processor computing power of the computing node, but the target MPI process can also be selected based on the task processing status of the multiple MPI processes, thereby improving the flexibility of selecting the target MPI process and making the application scenarios more extensive. In addition, the multiple MPI processes correspond to a storage file respectively. After the task processing assigned to each MPI process is completed, the I / O middleware running in the MPI process can write its own task processing results into its corresponding storage file, thereby improving the efficiency of result output during the MPI execution process and further speeding up the processing speed of parallel tasks.

[0133] It should be noted that the parallel task processing device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate the processing of parallel tasks. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the parallel task processing device provided in the above embodiment and the parallel task processing method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0134] An embodiment of the present application further provides a computing cluster comprising a plurality of computing nodes, each of which includes a processor and a memory, wherein the memory is configured to store a computer program for executing the aforementioned parallel task processing method. The processor is configured to execute the computer program stored in the memory to implement the aforementioned parallel task processing method.

[0135] An embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores instructions. When the instructions are executed on a computing node, the computing node executes the above-mentioned parallel task processing method.

[0136] An embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computing cluster, enables computing nodes to execute the above-mentioned parallel task processing method.

[0137] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)). It is worth noting that the computer-readable storage medium mentioned in the embodiments of the present application may be a non-volatile storage medium, in other words, a non-transient storage medium.

[0138] It should be understood that the "plurality" mentioned herein refers to two or more. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is merely a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate a clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0139] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions.

[0140] The above description is an embodiment provided for this application and is not intended to limit this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application should be included in the scope of protection of this application.

Claims

1. A parallel task processing method, characterized in that: The method is applied to a first computing node included in a computing cluster, where the first computing node is one of multiple computing nodes in the computing cluster currently used to process parallel tasks; the method includes: Receive a task assignment message sent by a message passing interface MPI scheduler, where the task assignment message indicates at least one MPI process currently used to process the parallel task in the first computing node and a task assigned to each MPI process; Starting the at least one MPI process and executing the assigned tasks through the at least one MPI process; Selecting a target MPI process from a plurality of MPI processes, wherein the plurality of MPI processes are MPI processes in the plurality of computing nodes for processing the parallel tasks, and the target MPI process is the process with the fastest task processing speed among the plurality of MPI processes; If the target MPI process is an MPI process among the at least one MPI process, and whenever any one of the multiple MPI processes completes processing of a task assigned to it, the target MPI process performs post-processing on the task processing result of the any one MPI process.

2. The method according to claim 1, wherein The method further comprises: In the process of starting the at least one MPI process, running an input / output (I / O) middleware in the at least one MPI process; The step of selecting a target MPI process from a plurality of MPI processes comprises: Obtaining the processor computing power of the plurality of computing nodes through the I / O middleware, wherein the processor computing power of the plurality of computing nodes is obtained through the I / O middleware running in the MPI process included in each of the plurality of computing nodes; Selecting a target computing node with the strongest processor computing power from the multiple computing nodes; The target MPI process is selected from the MPI processes belonging to the target computing node among the multiple MPI processes.

3. The method according to claim 1, wherein The method further comprises: In the process of starting the at least one MPI process, running an input / output (I / O) middleware in the at least one MPI process; The step of selecting a target MPI process from a plurality of MPI processes comprises: Synchronizing task processing status of the plurality of MPI processes through the I / O middleware, wherein the task processing status indicates whether the corresponding MPI process has completed processing the assigned task; The target MPI process is selected from the plurality of MPI processes based on task processing states of the plurality of MPI processes.

4. The method according to claim 3, wherein Before synchronizing the task processing states of the multiple MPI processes through the I / O middleware, the method further includes: If the total number of the multiple MPI processes is not greater than the number threshold, a step of synchronizing the task processing states of the multiple MPI processes through the I / O middleware is performed.

5. The method according to any one of claims 1 to 4, characterized in that The multiple MPI processes each correspond to a storage file, and the storage file is a storage area allocated to the corresponding MPI process in the storage node; the method further includes: For each MPI process in the at least one MPI process, if the task assigned to the MPI process is completed, the I / O middleware running in the MPI process writes the task processing result of the process into the storage file corresponding to the process; Whenever the task processing assigned to any one of the multiple MPI processes is completed, the target MPI process performs post-processing on the task processing result of the any one MPI process, including: Whenever any one of the multiple MPI processes completes processing of a task assigned thereto, the target MPI process obtains a task processing result from a storage file corresponding to the any one MPI process, and performs post-processing on the obtained task processing result.

6. A parallel task processing device, characterized in that: The device is applied to a first computing node included in a computing cluster, where the first computing node is one of multiple computing nodes in the computing cluster currently used to process parallel tasks; the device includes: a receiving module, configured to receive a task assignment message sent by a message passing interface (MPI) scheduler, wherein the task assignment message indicates at least one MPI process currently used to process the parallel task in the first computing node and a task assigned to each MPI process; A startup module, configured to start the at least one MPI process and execute the assigned tasks through the at least one MPI process; A selection module is configured to select a target MPI process from a plurality of MPI processes, wherein the plurality of MPI processes are MPI processes in the plurality of computing nodes for processing the parallel tasks, and the target MPI process is the process with the fastest task processing speed among the plurality of MPI processes; A post-processing module is used to, if the target MPI process is an MPI process among the at least one MPI process, and whenever the task processing assigned to any one of the multiple MPI processes is completed, post-process the task processing result of the any one MPI process through the target MPI process.

7. The device according to claim 6, characterized in that The device further comprises: A first running module is configured to run an input / output (I / O) middleware in the at least one MPI process during the process of starting the at least one MPI process; The selection module includes: an acquisition submodule, configured to acquire the processor computing power of the plurality of computing nodes through the I / O middleware, wherein the processor computing power of the plurality of computing nodes is acquired through the I / O middleware running in the MPI process included in each of the plurality of computing nodes; A first selection submodule is configured to select a target computing node with the strongest processor computing power from the multiple computing nodes; The second selection submodule is configured to select the target MPI process from the MPI processes belonging to the target computing node in the multiple MPI processes.

8. The device according to claim 6, wherein The device further comprises: A second running module is configured to run an input / output (I / O) middleware in the at least one MPI process during the process of starting the at least one MPI process; The selection module includes: a synchronization submodule, configured to synchronize the task processing status of the plurality of MPI processes through the I / O middleware, wherein the task processing status indicates whether the corresponding MPI process has completed processing the assigned task; The third selection module is configured to select the target MPI process from the multiple MPI processes based on the task processing status of the multiple MPI processes.

9. The device according to claim 8, wherein The device further comprises: The trigger module is configured to trigger the synchronization submodule to synchronize the task processing states of the multiple MPI processes through the I / O middleware if the total number of the multiple MPI processes is not greater than a number threshold.

10. The device according to any one of claims 6 to 9, characterized in that Each of the multiple MPI processes corresponds to a storage file, and the storage file is a storage area allocated to the corresponding MPI process in the storage node; The device further comprises: a writing module configured to, for each MPI process in the at least one MPI process, write the task processing result thereof into the storage file corresponding to the MPI process through the I / O middleware running in the MPI process if the task processing assigned to the MPI process is completed; The post-processing module is specifically used for: Whenever any one of the multiple MPI processes completes processing of a task assigned thereto, the target MPI process obtains a task processing result from a storage file corresponding to the any one MPI process, and performs post-processing on the obtained task processing result.

11. A computing cluster, characterized in that: comprising a plurality of computing nodes, each computing node comprising a memory and a processor; The memory is used to store computer instructions, and the processor is used to execute the instructions stored in the memory, so that the computing node executes the method according to any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that The storage medium stores instructions, and when the instructions are executed by a computing node in a computing cluster, the computing node executes the method according to any one of claims 1 to 5.

13. A computer program product comprising instructions, characterized in that When the instruction is executed by a computing node in a computing cluster, the computing node executes the method according to any one of claims 1 to 5.

Images