Method for supporting heterogeneous collaborative parallel computing in portable parallel programming framework

Abstract

The invention relates to a method for supporting heterogeneous collaborative parallel computing in a portable parallel programming framework, which comprises the following steps of: simultaneously generating a code for controlling CPU (Central Processing Unit) computing and a code for controlling GPU (Graphics Processing Unit) computing during code generation, modifying an MPI (Message Passing Interface) communication method of the two types of codes into a heterogeneous collaborative compatible communication method, and starting processes of the two types of codes by utilizing an MPMD (Message Passing Multiple Data) mode, and the load between the two types of processes is balanced, so that the support for the heterogeneous collaborative parallel computing of the CPU and the accelerator is realized. By means of the method, the codes supporting heterogeneous collaborative parallel computing can be automatically generated only by using a portable parallel programming framework to conduct application programming as before, and the purpose of conducting computing through a CPU and an accelerator at the same time is achieved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of parallel computing, and relates to a method for supporting heterogeneous cooperative parallel computing in a portable parallel programming framework. BACKGROUND

[0002] A heterogeneous parallel computing system is composed of a general-purpose multi-core CPU (Central Processing Unit) and an accelerator, the most common of which is a GPU (Graphics Processing Unit), but it can also be a DSP (Digital Signal Processor), a MIC (Many Integrated Core), etc. The computing power of a heterogeneous parallel computing system comes from both the CPU and the accelerator, although the accelerator usually provides greater peak computing power, but the CPU cannot be ignored with the continuous increase in the number of cores and the use of vector components (for example, the AMD fifth-generation EPYC processor released in October 2024 can contain up to 192 computing cores, use AVX-512 vector instructions, and the peak double-precision floating-point performance can reach 13.824 TFLOPS). For the sake of convenience, the following will take GPU to represent the accelerator in the heterogeneous parallel computing system.

[0003] For an application program, parallel programming is needed to utilize the computing power of a heterogeneous computing system, and different parallel programming techniques are usually needed for CPU and GPU in the heterogeneous computing system, for example, OpenMP (Open Multi-Processing) multi-thread parallel is usually used for developing multi-core parallelism for multi-core CPU, and CUDA (Compute Unified Device Architecture) programming is usually used for developing parallelism of a large number of computing cores of GPU. MPI (Message-Passing Interface) is also an important parallel programming mode, which can be applied to developing CPU multi-core parallelism on one hand, and can be used in combination with OpenMP and CUDA on the other hand, that is, MPI+OpenMP hybrid parallel program and MPI+CUDA hybrid parallel program. Moreover, MPI is a necessary means to realize multi-node parallelism, and if a plurality of nodes are included in a heterogeneous parallel computing system, MPI programming is needed to realize multi-node parallel computing. However, a pure OpenMP, MPI or MPI+OpenMP hybrid parallel program can only utilize the computing power of CPU in the heterogeneous computing system (except for the OpenMP heterogeneous parallel extension programming mode); a pure CUDA or MPI+CUDA program usually adopts an accelerated computing mode (that is, the main computing work is given to GPU to do, and the CPU end is only responsible for controlling GPU and processing communication and I / O), and can only utilize the computing power of GPU in the heterogeneous computing system. If an application program wants to utilize CPU and GPU for computing at the same time (that is, heterogeneous collaborative computing), programming of MPI, OpenMP and CUDA usually needs to be combined, and the problem of how to allocate tasks on CPU and GPU to realize load balancing also needs to be involved.

[0004] In addition to parallel programming languages such as OpenMP, MPI, CUDA, etc., a class of portable parallel programming frameworks such as OPS (Oxford Parallel library for Structured mesh solvers), OP2 (Oxford Parallel library for Unstructured mesh solvers), etc. have appeared. These parallel programming frameworks provide special APIs (Application Programming Interface), and programmers describe and refactor applications in a domain-specific language based on these APIs to obtain a set of source codes. Subsequently, the programming framework takes the refactored source codes as input and automatically generates source codes of various parallel modes (such as MPI, OpenMP, CUDA, MPI+OpenMP, MPI+CUDA, etc.) in a source-to-source conversion manner. Finally, the parallel codes are compiled with the support of the backend library of the programming framework, and the executable programs of the corresponding parallel modes are obtained. From the perspective of application development, this kind of portable parallel programming framework technology can support parallel programming of CPU and GPU devices based on a set of source codes, and has certain advantages in programming efficiency and code maintainability.

[0005] Although the existing portable parallel programming framework can generate MPI+OpenMP version code for CPU computing and MPI+CUDA version code for GPU computing, it cannot directly generate parallel code supporting CPU and GPU heterogeneous collaborative computing on a heterogeneous parallel system because it does not consider the problem of compatible processing MPI communication with the other code when generating MPI+OpenMP version or MPI+CUDA version code, nor does it consider the load balancing problem between the two. Therefore, the code generated by it cannot support CPU and GPU heterogeneous collaborative computing. The main goal of using a heterogeneous parallel system for computing is to improve computing performance. If only CPU or GPU in the system is used for computing, the performance of the unused computing device cannot be utilized, resulting in waste of resources. SUMMARY

[0006] In view of the problems in the above-mentioned traditional method, the present application provides a method for supporting heterogeneous collaborative parallel computing in a portable parallel programming framework.

[0007] To achieve the above-mentioned purpose, the embodiments of the present application adopt the following technical solutions: On the one hand, a method for supporting heterogeneous collaborative parallel computing in a portable parallel programming framework is provided, and the method comprises the following steps: In the code generation module of the portable parallel programming framework, a heterogeneous collaborative compatible MPI communication method is designed for parallel codes of two modes, including: MPI+OpenMP mode and MPI+CUDA mode.

[0008] An application code is written by using the API of the portable parallel programming framework.

[0009] A code generation module of the portable parallel programming framework is used to generate parallel codes of two modes.

[0010] The parallel codes of two modes are compiled respectively to obtain executable programs of two modes.

[0011] The executable programs of two modes are started in the MPMD mode in the same task to perform heterogeneous collaborative parallel computing.

[0012] In one embodiment, in the code generation module of the portable parallel programming framework, a heterogeneous collaborative compatible MPI communication method is designed for parallel codes of two modes, including: MPI+OpenMP mode and MPI+CUDA mode, including: An MPI environment is initialized, and each MPI process obtains its own unique number, wherein the process with the number 0 is the root process.

[0013] For parallel programs of two modes, a same grid partition library and partition strategy are used for grid division according to the number of started MPI processes, and the divided grids are distributed to each process.

[0014] All MPI processes send their own process types to the root process through a data collection function, and the root process stores all received MPI process types in an array ProcType; the process with the process type 0 is a process executing the MPI+OpenMP version code, and the process with the process type 1 is a process executing the MPI+CUDA version code.

[0015] The root process broadcasts the array ProcType to each MPI process through an MPI data broadcast function.

[0016] When the current MPI process performs MPI communication with other MPI processes, the type of the other process is retrieved from the ProcType array according to the number of the other process, and a corresponding communication processing algorithm is used according to the type of the other process and the type of the current process.

[0017] In one embodiment, when the current MPI process performs MPI communication with other MPI processes, the current MPI process retrieves the type of the other process from the ProcType array according to the number of the other process, and adopts a corresponding communication processing algorithm according to the type of the other process and the type of the current process, including: If ProcType[my_rank] == ProcType[other_rank], then the communication processing directly uses the original processing method in the portable parallel programming framework, and no special processing is required; wherein my_rank is the number of the current MPI process, other_rank is the number of the other MPI process, ProcType[my_rank] is the type of the current process, and ProcType[other_rank] is the type of the other MPI process except the current process.

[0018] If ProcType[my_rank]!= ProcType[other_rank], then the MPI communication processing uses a heterogeneous cooperative compatible communication algorithm to enable correct communication between MPI processes of the two types.

[0019] In one embodiment, if ProcType[my_rank]!= ProcType[other_rank], then the MPI communication processing uses a heterogeneous cooperative compatible communication algorithm to enable correct communication between MPI processes of the two types, including: When ProcType[my_rank] == 1: Iterate through all the boundary regions of the current process and the process with the number other_rank, copy the updated current boundary region data on the GPU to the CPU side, and then the CPU side calls the non-blocking MPI message sending function MPI_Isend to send the boundary region data to the process with the number other_rank; Iterate through all the boundary regions of the current process and the process with the number other_rank, and the CPU side calls the non-blocking MPI message receiving function MPI_Irecv to receive the boundary region data from the process with the number other_rank; Call the MPI function MPI_Waitall to wait for all the boundary region data between the current process and the process with the number other_rank to be completely received; Iterate through all the boundary regions of the current process and the process with the number other_rank, and copy the updated current boundary region data on the CPU side to the GPU side.

[0020] In one embodiment, if ProcType[my_rank]≠ProcType[other_rank], then the MPI communication process uses a heterogeneous co-sympathetic communication algorithm, so that the two types of MPI processes can communicate correctly, including: When ProcType[my_rank] == 0: Traverse all the boundary zones between the current process and the process numbered other_rank, and call the non-blocking MPI message sending function MPI_Isend to send the boundary zone data to the process numbered other_rank; Traverse all the boundary zones between the current process and the process numbered other_rank, and call the non-blocking MPI message receiving function MPI_Irecv to receive the boundary zone data from the process numbered other_rank; Call the MPI function MPI_Waitall to wait for all the boundary zone data between the current process and the process numbered other_rank to be completely received.

[0021] In one embodiment, the parallel code of the two modes is compiled respectively to obtain executable programs of the two modes, including: Compile the parallel program code of the MPI+OpenMP mode to obtain an executable program that controls the CPU to perform calculation; Compile the parallel program code of the MPI+CUDA mode to obtain an executable program that controls the GPU to perform calculation.

[0022] In one embodiment, the executable programs of the two modes are started in the MPMD mode in the same task to perform heterogeneous co-sympathetic parallel calculation, including: Suppose that the heterogeneous parallel computer system running the program contains blocks of GPUs, and the number of processes of the executable program that controls the GPU to perform calculation is set to ; Suppose that the heterogeneous parallel computer system running the program contains CPU cores, and CPU cores are equivalent to blocks of GPUs in terms of computing power for the application, then the number of processes of the executable program that controls the CPU to perform calculation is set to ; In the same task, M CPU processes are started in the MPI MPMD start mode to execute the executable program that controls the CPU to perform calculation, The GPU process executes the executable program to control the GPU to perform calculation; wherein each GPU process only starts one thread at the CPU end, and one CPU core is used to control the GPU execution and process the CPU end communication and I / O; wherein the number of threads started by each CPU process is .

[0023] One of the technical solutions in the above technical solution has the following advantages and beneficial effects: The above method for supporting heterogeneous collaborative parallel computing in a portable parallel programming framework generates the code for controlling CPU computing and the code for controlling GPU computing at the same time, modifies the MPI communication methods of the two types of codes into heterogeneous collaborative compatible communication methods, starts the processes of the two types of codes using the MPMD mode, and balances the loads between the two types of processes, thereby supporting the heterogeneous collaborative parallel computing of CPU and accelerator. Through the method, the code supporting the heterogeneous collaborative parallel computing can be automatically generated by using the portable parallel programming framework for application programming as before, and the computing capacity of CPU and accelerator can be simultaneously utilized for computing. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0025] Figure 1 A flowchart of the method for supporting heterogeneous collaborative parallel computing in a portable parallel programming framework in one embodiment; Figure 2 An execution situation diagram of the heterogeneous collaborative parallel computing in one embodiment. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the present application more clear, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not intended to limit the present application.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the present application is only for the purpose of describing specific embodiments and is not intended to limit the present application.

[0028] It is to be understood that any reference to an "embodiment" herein does not necessarily refer to the same embodiment, nor is it necessarily mutually exclusive of other embodiments. As such, the term "and / or" as used herein refers to any one or more of the associated listed items, in any combination with one another, and includes all possible combinations of the items. The term "comprising" as used herein is intended to mean that the embodiments include the recited elements, but not excluding others. "Consisting essentially of" when used herein in relation to a combination means that the combination includes the recited elements, and includes additional elements that do not materially affect the basic and novel characteristics of the combination. "Consisting of" when used herein in relation to a combination means that the combination includes the recited elements and no additional elements.

[0029] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0030] In one embodiment, as shown in Figure 1 a method for supporting heterogeneous collaborative parallel computing in a portable parallel programming framework can include the following steps 100 to 108: Step 100: In the code generation module of the portable parallel programming framework, design a heterogeneous collaborative compatible MPI communication method for two modes of parallel code, including: MPI+OpenMP mode and MPI+CUDA mode.

[0031] Step 102: Write application code using the API of the portable parallel programming framework.

[0032] Specifically, the application developer writes a set of program code using the API of the portable parallel programming framework, which is the same as the previous method of programming using the portable parallel programming framework.

[0033] Step 104: Generate two modes of parallel code using the code generation module of the portable parallel programming framework.

[0034] Specifically, based on the source-to-source code generation module of the portable parallel programming framework modified in step 100, the MPI+OpenMP and MPI+CUDA mode parallel codes are generated respectively using the application code written in step 102 as input.

[0035] Step 106: Compile the parallel code of the two modes respectively to obtain executable programs of the two modes.

[0036] Specifically, the MPI+OpenMP mode and MPI+CUDA mode parallel codes generated in step 104 are compiled respectively to obtain executable programs.

[0037] Step 108: Start the executable programs of the two modes in the same task in the MPMD mode to perform heterogeneous collaborative parallel computing.

[0038] The method for supporting heterogeneous cooperative parallel computing in the portable parallel programming framework, by generating the code for controlling CPU computing and the code for controlling GPU computing simultaneously at code generation, modifying the MPI communication methods of the two types of codes into heterogeneous cooperative compatible communication methods, starting the processes of the two types of codes by using the MPMD mode, and balancing the loads between the two types of processes, thereby supporting the heterogeneous cooperative parallel computing of CPU and accelerator. By using the method, the code supporting the heterogeneous cooperative parallel computing can be automatically generated only by using the portable parallel programming framework for application programming as before, and the computing capacity of CPU and accelerator can be simultaneously used for computing.

[0039] Suppose that the computing capacity of GPU in the heterogeneous computing node for an application is 1, and the computing capacity of CPU for the application is 0.2, then the method can obtain a performance improvement of 20% relative to the original method of using only GPU for computing.

[0040] In one embodiment, the step 100 comprises: initializing an MPI environment, each MPI process obtaining its own unique number, wherein the process with the number 0 is the root process; for the two types of parallel programs, using the same grid partition library and partition strategy to perform grid partition according to the number of started MPI processes, and distributing the partitioned grid to each process; all MPI processes send the type of the process to which the process belongs to the root process through a data collection function, and the root process stores all the types of the MPI processes received in an array ProcType; the process with the type 0 is the process for executing the MPI+OpenMP version code, and the process with the type 1 is the process for executing the MPI+CUDA version code; the root process broadcasts the array ProcType to each MPI process through an MPI data broadcast function; when the current MPI process performs MPI communication with other MPI processes, the type of the other process is retrieved from the ProcType array according to the number of the other process, and a corresponding communication processing algorithm is used according to the type of the other process and the type of the current process.

[0041] In one embodiment, when the current MPI process performs MPI communication with other MPI processes, the current MPI process retrieves the type to which the other process belongs from the ProcType array according to the number of the other process, and adopts a corresponding communication processing algorithm according to the type to which the other process belongs and the type to which the current process belongs, including: if ProcType[my_rank] == ProcType[other_rank], then the communication processing directly uses the original processing method in the portable parallel programming framework, and no special processing is required; wherein my_rank is the number of the current MPI process, other_rank is the number of the other MPI process, ProcType[my_rank] is the type of the current process, and ProcType[other_rank] is the type of the other MPI process; if ProcType[my_rank]!= ProcType[other_rank], then the MPI communication processing uses a heterogeneous collaborative compatible communication algorithm, so that the two types of MPI processes can correctly communicate with each other.

[0042] In one embodiment, if ProcType[my_rank]!= ProcType[other_rank], then the MPI communication processing uses a heterogeneous collaborative compatible communication algorithm, so that the two types of MPI processes can correctly communicate with each other, including: when ProcType[my_rank] == 1: traverse all boundary regions between the current process and the process with the number other_rank, copy the updated current boundary region data on the GPU to the CPU end, then the CPU end calls the non-blocking MPI message sending function MPI_Isend to send the boundary region data to the process with the number other_rank; traverse all boundary regions between the current process and the process with the number other_rank, and the CPU end calls the non-blocking MPI message receiving function MPI_Irecv to receive the boundary region data from the process with the number other_rank; call the MPI function MPI_Waitall to wait for all boundary region data between the current process and the process with the number other_rank to be completely received; traverse all boundary regions between the current process and the process with the number other_rank, and copy the updated current boundary region data on the host to the CPU end to the GPU end.

[0043] It should be noted that the method is for a heterogeneous parallel computing system of CPU and accelerator, and the accelerator is most commonly GPU, but can also be other accelerators such as GPDSP.

[0044] In one embodiment, if ProcType[my_rank]≠ProcType[other_rank], then the MPI communication process uses a heterogeneous co-compatible communication algorithm, so that the two types of MPI processes can correctly communicate, including: when ProcType[my_rank] == 0: traverse all the boundary zones of the current process and the process numbered as other_rank, call the non-blocking MPI message sending function MPI_Isend to send the boundary zone data to the process numbered as other_rank; traverse all the boundary zones of the current process and the process numbered as other_rank, call the non-blocking MPI message receiving function MPI_Irecv to receive the boundary zone data from the process numbered as other_rank; call the MPI function MPI_Waitall to wait for all the boundary zone data between the current process and the process numbered as other_rank to be completely received.

[0045] Specifically, in the source-to-source code generation module of the portable parallel programming framework, for MPI+OpenMP mode parallel programs and MPI+CUDA mode parallel codes, a heterogeneous co-compatible MPI communication method is implemented, and the specific implementation process includes: Step 1.1: Initialize the MPI environment, and each MPI process obtains its own unique number. Assuming that a total of N MPI processes are started, the numbers of all MPI processes are 0, 1, 2,..., (N-1) respectively, wherein the process numbered as 0 is the root process.

[0046] Step 1.2: For parallel programs of two modes, according to the number of started MPI processes, use the same grid partition library (for example, PARMETIS) and partition strategy (for example, K-Way graph partitioning for grid cell sets) to perform grid partitioning, and divide the partitioned grid to each process. In this way, even if different processes execute different mode parallel programs in the same task, the grid handled by each process is determined by its number, and the boundary zone between different processes is also determined by their numbers.

[0047] Step 1.3: All processes send which type of process they belong to to the root process through the data collection function MPI_Gather. Here, it is specified that the process executing the MPI+OpenMP version code (i.e., the CPU process) is of type 0, and the process executing the MPI+CUDA version code (i.e., the GPU process) is of type 1. The root process stores all the integers received here into an array ProcType containing N integers, wherein the array element ProcType[i] stores the type of the ith process.

[0048] Step 1.4: The root process broadcasts the array ProcType to each process through the MPI data broadcast function MPI_Bcast.

[0049] Step 1.5: One process (suppose its process number is my_rank) retrieves the type of the other process (suppose its process number is other_rank) from the ProcType array according to the process number of the other process when communicating with the other process, and uses the corresponding communication processing algorithm according to the type of the other process and the type of the process.

[0050] Step 1.5.1: If ProcType[my_rank] == ProcType[other_rank], i.e. the type of the process is the same as the type of the other process, the communication processing directly uses the original processing method in the portable parallel programming framework, and no special processing is needed.

[0051] Step 1.5.2: If ProcType[my_rank]!= ProcType[other_rank], i.e. the type of the process is different from the type of the other process, the MPI communication processing uses a heterogeneous cooperative compatible communication algorithm to enable correct communication between the two versions of processes, which is processed as follows: Step 1.5.2.1: If ProcType[my_rank] == 1, i.e. the process is a GPU process, the following is executed: Step 1.5.2.1.1: Traverse all the boundary regions between the process and the process with the number other_rank, copy the current boundary region data updated on the GPU to the CPU side, and then the CPU side calls the non-blocking MPI message sending function MPI_Isend to send the boundary region data to the process with the number other_rank.

[0052] Step 1.5.2.1.2: Traverse all the boundary regions between the process and the process with the number other_rank, and the CPU side calls the non-blocking MPI message receiving function MPI_Irecv to receive the boundary region data from the process with the number other_rank. Step 1.5.2.1.3: Call the MPI function MPI_Waitall to wait for all the boundary region data between the process and the process with the number other_rank to be received. Step 1.5.2.1.4: Traverse all the boundary regions between the process and the process with the number other_rank, and copy the current boundary region data updated on the host to the GPU side.

[0053] Step 1.5.2.2: If ProcType[my_rank] == 0, meaning this process is a CPU process, then perform the following processing: Step 1.5.2.2.1: Traverse all boundary regions between the current process and the process with the ID "other_rank", and call the non-blocking MPI message sending function MPI_Isend to send the boundary region data to the process with the ID "other_rank"; Step 1.5.2.2.2: Traverse all boundary regions between the current process and the process with the ID "other_rank", and call the non-blocking MPI message receiving function MPI_Irecv to receive boundary region data from the process with the ID "other_rank"; Step 1.5.2.2.3: Call the MPI function MPI_Waitall to wait for all boundary area data between this process and the process numbered other_rank to be received.

[0054] In one embodiment, step 106 includes: compiling parallel program code in MPI+OpenMP mode to obtain an executable program that controls the CPU to perform calculations; and compiling parallel program code in MPI+CUDA mode to obtain an executable program that controls the GPU to perform calculations.

[0055] Specifically, the parallel code generated in step 104 for both MPI+OpenMP and MPI+CUDA modes is compiled to obtain an executable program. The specific process includes: Step 4.1: Compile the parallel program code in MPI+OpenMP mode generated in step 104 to obtain an executable program that can control the CPU to perform calculations, denoted as... .

[0056] Step 4.2: Compile the parallel program code in MPI+CUDA mode generated in step 104 to obtain an executable program that can control the GPU for computation, denoted as... .

[0057] In one embodiment, step 108 includes: setting the number of processes of the executable program that initiates execution control CPU for computation to... Each GPU process starts only one thread on the CPU, with a dedicated CPU core used to control GPU execution and handle CPU-side communication and I / O; M CPU processes are launched and executed in the same task using MPI's MPMD startup mode. , Execute a GPU process .

[0058] Specifically, in the same task, two executable programs in different modes are launched using MPMD to perform heterogeneous collaborative parallel computing. The runtime configuration needs to be determined based on the number of GPUs, the number of CPU cores, and their performance within the heterogeneous parallel computer system. Assume the heterogeneous parallel computer system running the program contains... Block GPU, containing One CPU core. Specific steps include: Step 5.1: Start execution The number of program processes (i.e., GPU processes) is set to .

[0059] Step 5.2: Each GPU process starts only one thread on the CPU side, and a dedicated CPU core is used to control GPU execution and handle CPU-side communication and I / O. Assume the remaining CPU cores in the computer system... The computing power of each CPU core relative to the application Block GPU equivalent ( The value can be obtained by running the experiment in the actual program, then execution will begin. The number of program processes (i.e., CPU processes) is set to Also, set the number of threads per CPU process. For example, you can use the command "export OMP_NUM_THREADS= To achieve this.

[0060] Step 5.3: Start in the same task using MPI's MPMD boot mode Execute a CPU process program, Execute a GPU process The program, in command form, is as follows: mpirun -np M <Parameter List 1>: -np <Parameter List 2> Wherein, parameter list 1 and parameter list 2 are respectively and Command-line arguments used during execution (empty if applicable, otherwise specified). A total of [number] commands were started here. Each MPI process. The execution of this heterogeneous collaborative parallel computing is as follows: Figure 2 As shown, for the sake of simplicity, Figure 2 Only one CPU process and one GPU process are shown in the image. Figure 2The CPU process in the above-mentioned flow

[0061] It should be understood that, although the above-mentioned flow Figure 1 shows the steps in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the above-mentioned flow Figure 1 may include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0062] The technical features of the above-mentioned embodiments can be combined in any manner. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described, but as long as the combination of the technical features does not exist contradictions, it should be considered as the scope of the present application.

[0063] The above embodiments only express several implementation manners of the present application, and the description is specific and detailed, but it should not be understood as a limitation on the protection scope of the present application. It should be pointed out that, for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which all belong to the protection scope of the present application.

Claims

1. A method for supporting heterogeneous collaborative parallel computing in a portable parallel programming framework, characterized in that, The method includes: In the code generation module of the portable parallel programming framework, a heterogeneous collaborative and compatible MPI communication method is designed for two modes of parallel code: MPI+OpenMP mode and MPI+CUDA mode. Write application code using APIs with a portable parallel programming architecture; The code generation module of the portable parallel programming framework generates parallel code in two modes. Compile the parallel code for the two modes separately to obtain executable programs for the two modes; In the same task, two executable programs in different modes are launched using MPMD to perform heterogeneous collaborative parallel computing.

2. The method for supporting heterogeneous collaborative parallel computing in a portable parallel programming framework according to claim 1, characterized in that, In the code generation module of the portable parallel programming framework, a heterogeneous and cooperative MPI communication method is designed for two modes of parallel code: MPI+OpenMP mode and MPI+CUDA mode. The MPI environment is initialized, and each MPI process is assigned a unique number, with the process with number 0 being the root process. For parallel programs in both modes, the same grid partitioning library and partitioning strategy are used to partition the grid according to the number of MPI processes started, and the partitioned grid is distributed to each process. All MPI processes send their process type to the root process through the data collection function. The root process stores the process types of all received MPI processes in an array ProcType. Processes with process type 0 are those executing the MPI+OpenMP version of the code, and processes with process type 1 are those executing the MPI+CUDA version of the code. The root process broadcasts the array ProcType to each MPI process via the MPI data broadcast function; When the current MPI process communicates with other MPI processes, it retrieves the type of the other process from the ProcType array based on the other process's ID, and then uses the appropriate communication processing algorithm based on the type of the other process and the type of its own process.

3. The method for supporting heterogeneous collaborative parallel computing in a portable parallel programming framework according to claim 2, characterized in that, When the current MPI process communicates with other MPI processes, it retrieves the process type from the ProcType array based on the process ID of the other process. Then, based on the type of both the other and current processes, it employs the appropriate communication processing algorithm, including: If ProcType[my_rank] == ProcType[other_rank], then the communication processing directly uses the original processing method in the portable parallel programming framework, without special processing; where my_rank is the number of the current MPI process, other_rank is the number of other MPI processes, ProcType[my_rank] is the type of the current process, and ProcType[other_rank] is the type of the MPI process with the number other_rank; If ProcType[my_rank]≠ProcType[other_rank], then the MPI communication process uses a heterogeneous cooperative and compatible communication algorithm, enabling the two types of MPI processes to communicate correctly.

4. The method for supporting heterogeneous collaborative parallel computing in a portable parallel programming framework according to claim 3, characterized in that, If ProcType[my_rank] ≠ ProcType[other_rank], then MPI communication processing uses a heterogeneous cooperative and compatible communication algorithm to enable correct communication between the two types of MPI processes, including: When ProcType[my_rank]==1: Iterate through all the boundary regions of the current process and the process with the number other_rank, copy the updated current boundary region data from the GPU to the CPU, and then call the non-blocking MPI message sending function MPI_Isend to send the boundary region data to the process with the number other_rank on the CPU. Traverse all boundary regions between the current process and the process numbered other_rank. On the CPU side, call the non-blocking MPI message receiving function MPI_Irecv to receive the boundary region data from the process numbered other_rank. The MPI function MPI_Waitall is called to wait for all boundary region data between the current process and the process numbered other_rank to be received. Iterate through all boundary regions between the current process and the process numbered other_rank, and copy the updated current boundary region data from the CPU to the GPU.

5. The method for supporting heterogeneous collaborative parallel computing in a portable parallel programming framework according to claim 3, characterized in that, If ProcType[my_rank] ≠ ProcType[other_rank], then MPI communication processing uses a heterogeneous cooperative and compatible communication algorithm to enable correct communication between the two types of MPI processes, including: When ProcType[my_rank] == 0: Iterate through all boundary regions between the current process and the process with the ID "other_rank", and call the non-blocking MPI message sending function MPI_Isend to send the boundary region data to the process with the ID "other_rank". Iterate through all boundary regions between the current process and the process numbered other_rank, and call the non-blocking MPI message receiving function MPI_Irecv to receive boundary region data from the process numbered other_rank. The MPI function MPI_Waitall is called to wait for all boundary region data between the current process and the process numbered other_rank to be received.

6. The method for supporting heterogeneous collaborative parallel computing in a portable parallel programming framework according to claim 1, characterized in that, Compile the parallel code for both modes separately to obtain executable programs for each mode, including: Compile the parallel program code in MPI+OpenMP mode to obtain an executable program that controls the CPU to perform calculations; Compile the parallel program code in MPI+CUDA mode to obtain an executable program that controls the GPU to perform computations.

7. The method for supporting heterogeneous collaborative parallel computing in a portable parallel programming framework according to claim 1, characterized in that, In the same task, executable programs in two modes are launched using MPMD to perform heterogeneous collaborative parallel computing, including: Assume the heterogeneous parallel computer system running the program contains Block GPU, set the number of processes that launch the executable program that controls the GPU for computation to be set to ; Assume the heterogeneous parallel computer system running the program contains Each CPU core The computing power of each CPU core relative to the application If the number of GPUs is comparable, then set the number of processes that launch the executable program that controls the CPU for computation to be [number]. ; In the same task, M CPU processes are launched in MPI's MPMD boot mode to execute executable programs that control the CPU for computation. Each GPU process executes an executable program that controls the GPU to perform computations; each GPU process starts only one thread on the CPU side, using a dedicated CPU core for controlling GPU execution and handling CPU-side communication and I / O; the number of threads started by each CPU process is... .

Images