A LBM parallel computing method based on CPU-DCU heterogeneous architecture
By dividing molecular grid blocks on the CPU-DCU heterogeneous architecture and performing asynchronous data transmission and inter-process communication, the problem of low parallel computing performance of LBM models under the CPU-DCU heterogeneous architecture is solved, and a large-scale and high-precision Fluid dynamics simulation and performance improvement.
Patent Information
- Application Number
- CN202410687687.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-30
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-05-30
AI Technical Summary
There is a lack of research on parallel computing of the lattice Boltzmann method (LBM) model for CPU-DCU heterogeneous architecture in the prior art, resulting in too long calculation time for large-scale high-precision fluid dynamics simulations.
By dividing the three-dimensional global grid into npx×npy sub-grid blocks along the xoy plane, the CPU allocates processes and allocates DCU accelerator, and adopts asynchronous data transmission and inter-process communication masking technology to achieve parallel collisions between boundary layer and internal grid and Migration calculation to reduce the number of data exchanges between processes.
It significantly improves LBM computing performance, realizes large-scale and high-precision fluid dynamics simulation, achieves an order of magnitude performance improvement compared to LBM acceleration on the CPU, and realizes load balancing and efficient parallel computing.
Smart Images

Figure CN118519773B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of algorithm optimization, and in particular to an LBM parallel computing method based on a CPU-DCU heterogeneous architecture. Background Art
[0002] The Lattice Boltzmann Method (LBM) is a unique fluid dynamics simulation method in the field of computational fluid dynamics. Different from traditional methods such as the finite volume method (macroscopic scale) and molecular dynamics simulation (microscopic scale), LBM is based on molecular dynamics theory and focuses on simulating fluid flow at the mesoscopic scale.
[0003] LBM discretizes the fluid domain into a uniform grid, and stores the corresponding velocity distribution function at each grid point to describe the particle density at that point in different velocity directions. In each time step, LBM simulates the evolution of the fluid by iteratively executing two main steps: collision and migration. In the collision phase, particles collide at each grid point, and the velocity distribution function is adjusted according to the collision model. In the migration phase, the velocity distribution function propagates to adjacent grid points along the predefined velocity direction, simulating the movement of particles in the flow field.
[0004] The LBM model is widely recognized for its simplicity and convenience for parallel computing, and performs well in complex fluid simulation. Compared with traditional methods, the advantage of LBM is that it is suitable for multi-scale simulation, providing a flexible and efficient way to study complex fluid behaviors including chemical reactions, turbulence, heat transfer, etc.
[0005] Implementing the lattice Boltzmann method includes the following key steps:
[0006] 1. Discretize the fluid domain into a uniform grid and create data structure information for storing flow field information. The data structure includes: a grid distribution function data structure, which stores information such as the particle density of each grid point in different speed directions and the properties of the grid point. For example: solid, fluid, boundary, wall, etc.;
[0007] 2. Perform collision calculation, traverse all grid points, and perform collision calculation according to the collision model;
[0008] 3. Perform migration calculations, traverse all grid points, and complete the process of particles at each grid point propagating to adjacent grid points along different speed directions;
[0009] 4. Repeat 2-3 until the required number of iterations for the simulation is reached.
[0010] In the above steps, collision (step 2) and migration (step 3) are the most time-consuming parts of the lattice Boltzmann method and are also hot spots in the calculation. As the fluid domain increases or the accuracy of the discrete grid increases, the size of the grid increases at a power function level, the memory required to store flow field information increases at a power function level, and the amount of calculation for the collision and migration steps also increases at a power function level. The above reasons make it difficult to perform large-scale high-precision fluid dynamics simulations and are prone to long calculation times.
[0011] The heterogeneous architecture cluster equipped with CPU and DCU (Deep Computing Unit) realizes high-speed interconnection of multiple heterogeneous computing nodes through the Infinite Band network. Each node consists of a CPU chip and four DCU accelerator cards. Compared with the traditional CPU architecture, the heterogeneous architecture based on CPU-DCU is more complex in structure and development environment, so it is more challenging in program design. However, the architecture of this heterogeneous computing cluster provides great potential for high-performance computing, especially when processing computationally intensive tasks, DCU can significantly improve the computing performance of the program. At present, there is still a lack of research on parallel computing of LBM models based on heterogeneous architecture based on CPU-DCU. Summary of the invention
[0012] The present invention provides a parallel computing method of a lattice Boltzmann model based on a CPU-DCU, so as to solve the problem in the prior art that there is a lack of research on parallel computing of an LBM model for a heterogeneous architecture of a CPU-DCU.
[0013] To achieve the above object, the present invention provides a LBM parallel computing method based on a CPU-DCU heterogeneous architecture, comprising the following steps:
[0014] S1. Divide the three-dimensional global grid into npx×npy sub-grid blocks along the xoy plane according to the calculation domain range and grid accuracy. The CPU side allocates a process to each sub-grid block for calculation. The sub-grid block includes an internal grid and a boundary layer grid.
[0015] S2, add one exchange layer grid on each of the four faces in the x and y directions of each grid block for data exchange between processes;
[0016] S3, each process on the CPU side creates subgrid data structure information for storing subgrid information, wherein the subgrid data structure information includes: subgrid number, subgrid size, subgrid property array, internal grid distribution function array, internal grid adjacent grid index array, boundary layer grid distribution function array, boundary layer grid adjacent grid index array, and exchange layer distribution function array;
[0017] S4, each process on the CPU side reads the geometric structure information of the corresponding calculation domain and the initial conditions of the flow field, initializes the sub-grid data structure, and allocates a DCU accelerator to each process on the CPU side, and then each process on the CPU side transmits the sub-grid data structure information to the corresponding DCU side;
[0018] S5. On the DCU side, first traverse the boundary layer grids in the subgrid, assign a hip thread to each boundary layer grid point, perform collision calculation according to the LBM collision model, and after the calculation is completed, assign the boundary layer grid distribution function array to the exchange layer grid distribution function array according to the preset assignment rule, and then start asynchronous copying to transfer the distribution function array corresponding to the exchange layer grid from the DCU side to the CPU side, and after starting the asynchronous copying, assign a thread to each internal grid point, and start traversing the internal grid for parallel collision calculation;
[0019] S6. After completing the data copy from the DCU end to the CPU end, each process first completes the inter-process data communication of the exchange layer distribution function array in the x direction according to the two-dimensional division method of the grid, and then completes the inter-process data communication of the exchange layer distribution function array in the y direction after waiting for the data communication in the x direction to be completed;
[0020] S7, the CPU starts asynchronous transmission to transmit the received sub-grid switching layer data to the switching layer of the DCU terminal grid;
[0021] S8. After completing steps S5-S7, the DCU performs migration calculations of the boundary layer and the internal grid in the subgrid block;
[0022] S9. Repeat steps S5-S8 until the preset number of iterations required by the simulation is reached.
[0023] Specifically, step S1 is as follows: the size of the global grid is (X, Y, Z), then the number of grids in the x direction of the subgrid blocks corresponding to the first (X modnpx) processes along the positive x direction is floor(X / npx)+1, the number of grids in the x direction of the grid blocks corresponding to other processes is floor(X / npx), the number of grids in the y direction of the grid blocks corresponding to the first (Y mod npy) processes along the positive y direction is floor(Y / npy)+1, the number of grids in the y direction of the grid blocks corresponding to other processes is floor(Y / npy), the number of grids in the z direction of the grid blocks corresponding to each process is Z, and mod is the modulo function.
[0024] Specifically, in step S4, a DCU accelerator is allocated to each process on the CPU side, and then each process on the CPU side transmits the sub-grid data structure information to the corresponding DCU side. Specifically, a DCU accelerator is allocated to each process on the CPU side through the hipSetDevice function, and then each process on the CPU side transmits the sub-grid data structure information to the corresponding DCU side through the hipMemcpy function.
[0025] Specifically, the LBM collision model in step S5 includes: a single relaxation model LBGK and a multi-relaxation model MRT.
[0026] Specifically, the preset assignment rule in step S5 is as follows: the boundary layer of the process (x, y) corresponding to the sub-grid block is divided into a first boundary surface, a second boundary surface, a third boundary surface and a fourth boundary surface, and the exchange layer is divided into a first data surface, a second data surface, a third data surface and a fourth data surface, and the data of the four surfaces of the boundary layer are assigned outward to the corresponding positions of the exchange layer, and the edge grid data intersecting between the data surfaces of the exchange layer are assigned to the edge grid data intersecting between the data surfaces of the corresponding boundary layers, wherein the first boundary surface and the first data surface are positive surfaces of the y-axis, the second boundary surface and the second data surface are negative surfaces of the y-axis, the third boundary surface and the third data surface are negative surfaces of the x-axis, and the fourth boundary surface and the fourth data surface are positive surfaces of the x-axis.
[0027] Specifically, after the calculation in step S5 is completed, the hipMemcpyAsync function is used to start asynchronous copying to transfer the distribution function array corresponding to the exchange layer grid from the DCU end to the CPU end.
[0028] Specifically, step S6 is as follows: after the CPU receives the exchange layer grid distribution function array transmitted by the DCU, it first completes the inter-process data communication of the exchange layer grid distribution function array in the x direction through the inter-process communication MPI_Sendrecv_replace function according to the grid division method, and after the data communication in the x direction is completed, completes the inter-process data communication of the exchange layer grid distribution function array in the y direction.
[0029] The data communication rule in step S6 is: when process (x, y) performs inter-process data communication in the x direction, the third data plane data of process (x, y) is sent to process (x-1, y), and the fourth data plane data sent by process (x-1, y) is received; the fourth data plane of process (x, y) is sent to process (x+1, y), and the third data plane data sent by process (x+1, y) is received. After completing the inter-process data communication in the x direction, the inter-process data communication in the y direction is started, the first data plane data of the exchange layer grid is sent to process (x, y+1) and the second data plane data sent by process (x, y+1) is received, the second data plane data is sent to process (x, y-1) and the first data plane data sent by process (x, y-1) is received.
[0030] Specifically, when a process is located at the boundary of the process topology, the corresponding sending and receiving processes are set to empty processes.
[0031] Specifically, step S7 is as follows: the CPU starts asynchronous transmission through the hipMemcpyAsync function to transmit the received sub-grid exchange layer data to the exchange layer of the DCU terminal grid.
[0032] Specifically, during the migration calculation process of step S8, when updating the internal grid distribution function array, it is necessary to rely on the internal grid adjacent grid index array to obtain the adjacent grid point distribution function array, and when updating the boundary grid distribution function array, it is necessary to rely on the boundary grid adjacent grid index array to obtain the adjacent grid point distribution function array.
[0033] The invention discloses an LBM parallel computing method based on a CPU-DCU heterogeneous architecture, comprising the following steps: S1, dividing a three-dimensional global grid into npx×npy subgrid blocks along an xoy plane according to a computing domain range and grid accuracy, and assigning a process to each subgrid block on the CPU side for computing; S2, adding a layer of exchange layer grids on each of the four faces in the x and y directions of each grid block for data exchange between processes; S3, each process on the CPU side creates subgrid data structure information for storing subgrid information; S4, each process on the CPU side reads the geometric structure information of the corresponding computing domain and the initial conditions of the flow field, initializes the subgrid data structure, and assigns a DCU accelerator to each process on the CPU side, and then each process on the CPU side transmits the subgrid data structure information to the corresponding DCU side; S5, first traversing the boundary layer grids in the subgrids on the DCU side, assigning a hip thread to each boundary layer grid point, performing collision computing according to the LBM collision model, and after the computing is completed, assigning the boundary layer grid distribution function array to the exchange layer grid distribution function array according to a preset assignment rule, and then starting the asynchronous Copy the distribution function array corresponding to the exchange layer grid from the DCU end to the CPU end, and after starting the asynchronous copy, assign a thread to each internal grid point, and start traversing the internal grid for parallel collision calculation; S6, after completing the data copy from the DCU end to the CPU end, each process first completes the inter-process data communication of the exchange layer distribution function array in the x direction according to the two-dimensional division method of the grid, and waits for the data communication in the x direction to complete the inter-process data communication of the exchange layer distribution function array in the y direction; S7, the CPU end starts asynchronous transmission to transmit the received sub-grid exchange layer data to the exchange layer of the DCU terminal grid; S8, after completing steps S5-S7, the DCU end performs the migration calculation of the boundary layer and the internal grid in the sub-grid block; S9, repeat steps S5-S8 until the preset number of iterations required by the simulation is reached; This method expands the fluid domain range of the lattice Boltzmann method in fluid dynamics simulation, realizes the lattice Boltzmann calculation method based on CPU-DCU heterogeneous clusters, can perform large-scale parallel simulations on clusters, and can realize large-scale and high-precision fluid dynamics simulations based on the lattice Boltzmann method;
[0034] Furthermore, the present invention utilizes the computing performance of DCU and designs a DCU-based collision and migration parallel computing method. At the same time, through asynchronous data transmission between CPU and DCU and inter-process communication masking technology, data copying between DCU and CPU and time masking of inter-process communication are realized, thereby reducing the computing time of the lattice Boltzmann method in fluid dynamics simulation, greatly improving the performance of LBM calculation, and accelerating the performance by an order of magnitude compared to LBM on CPU.
[0035] In addition, the method for exchanging boundary layer grid data between processes proposed in the present invention can reduce the number of times data is sent and received between processes, thereby reducing the MPI communication time between processes and greatly improving the efficiency of parallel computing.
[0036] In addition, the present invention proposes a method for dividing the global grid into sub-grid blocks according to the number of processes, which can achieve reasonable equalization so that the number of grids calculated by each DCU is basically the same, achieving effective load balancing and improving the efficiency of parallel computing. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 It is a flowchart of a LBM parallel computing method based on a CPU-DCU heterogeneous architecture provided by an embodiment of the present invention;
[0038] Figure 2 It is a schematic diagram of a two-dimensional division mode of a flow field calculation domain provided by an embodiment of the present invention;
[0039] Figure 3 It is a schematic diagram of sub-grid numbers and corresponding process numbers after grid division provided by an embodiment of the present invention;
[0040] Figure 4 is a schematic diagram of a subgrid structure provided by an embodiment of the present invention;
[0041] Figure 5 is a schematic diagram of a subgrid data structure provided by an embodiment of the present invention;
[0042] Figure 6 It is a schematic diagram of data processing at the exchange layer before data communication of process (x, y) provided by an embodiment of the present invention;
[0043] Figure 7 This is a schematic diagram of data exchange in the x direction of a process (x, y) provided by an embodiment of the present invention;
[0044] Figure 8 It is a schematic diagram of data exchange in the y direction of a process (x, y) provided by an embodiment of the present invention;
[0045] Fig. 9 It is a schematic diagram of an LBM parallel computing device based on a CPU-DCU heterogeneous architecture provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments and drawings; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0047] The present invention can be explained in detail through the following embodiments. The purpose of disclosing the present invention is to protect all technical improvements within the scope of the present invention. In the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "front", "back", "left", "right", etc. indicating directions or positional relationships, they only correspond to the drawings of the present application for the convenience of describing the present invention, and do not indicate or imply that the device or element referred to must have a specific direction.
[0048] Embodiment 1
[0049] refer to Figure 1 According to the embodiment, a LBM parallel computing method based on a CPU-DCU heterogeneous architecture is disclosed, comprising the following steps:
[0050] The present invention provides a parallel computing method of a lattice Boltzmann model based on a CPU-DCU, so as to solve the problem in the prior art that there is a lack of research on parallel computing of an LBM model for a heterogeneous architecture of a CPU-DCU.
[0051] To achieve the above object, the present invention provides a LBM parallel computing method based on a CPU-DCU heterogeneous architecture, comprising the following steps:
[0052] S1. Divide the three-dimensional global grid into npx×npy sub-grid blocks along the xoy plane according to the calculation domain range and grid accuracy. The CPU side allocates a process to each sub-grid block for calculation. The sub-grid block includes an internal grid and a boundary layer grid.
[0053] Specifically, step S1 is as follows: the size of the global grid is (X, Y, Z), then the number of grids in the x direction of the subgrid blocks corresponding to the first (X modnpx) processes along the positive x direction is floor(X / npx)+1, the number of grids in the x direction of the grid blocks corresponding to other processes is floor(X / npx), the number of grids in the y direction of the grid blocks corresponding to the first (Y mod npy) processes along the positive y direction is floor(Y / npy)+1, the number of grids in the y direction of the grid blocks corresponding to other processes is floor(Y / npy), the number of grids in the z direction of the grid blocks corresponding to each process is Z, and mod is the modulo function.
[0054] Specifically, according to the computational domain range, the three-dimensional global grid is divided into npx×npy two-dimensional grid blocks along the xoy plane, and npx×npy processes are allocated to complete the calculation of the corresponding grid blocks, such as Figure 2 shown.
[0055] The sub-grids and process numbers after division are as follows Figure 3. If the size of the global grid is (X,Y,Z), then the number of grid blocks in the x direction corresponding to the first (X mod npx) processes along the positive x direction is floor(X / npx)+1, and the number of grid blocks in the x direction corresponding to other processes is floor(X / npx). The number of grid blocks in the y direction corresponding to the first (Y modnpy) processes along the positive y direction is floor(Y / npy)+1, and the number of grid blocks in the y direction corresponding to other processes is floor(Y / npy). That is, for the process (x,y), the number of grids in the x, y, and z dimensions of the subgrids that need to be processed is:
[0056]
[0057] Z sub =Z
[0058] Where X sub is the number of grids in the process subgrid block in the x direction, Y sub is the number of grids in the subgrid block in the y direction, Z sub is the number of grids in the process subgrid block in the z direction. The number of grids in the z direction is consistent with the number of grids in the global grid in the z direction.
[0059] This embodiment proposes a method for dividing the global grid into sub-grid blocks according to the number of processes, which can achieve reasonable equalization, so that the number of grids calculated by each DCU is basically the same, achieving effective load balancing and improving the efficiency of parallel computing.
[0060] S2, add one exchange layer grid on each of the four faces in the x and y directions of each grid block for data exchange between processes;
[0061] Specifically, based on the grid blocks that each process is responsible for calculating, in order to complete the migration step, it is necessary to add a layer of grids on the four faces of each grid block in the x and y directions for data exchange between processes. Therefore, the grid size allocated in each process needs to increase by two layers of grids in the x and y directions, and add a column of edge grids at the intersection of the increased grids. The increased outermost grid is called the exchange layer grid, and the outermost grid without the exchange layer is called the boundary layer grid. The other grids are called internal grids, such as Figure 4 That is, for the process (x, y), the actual number of sub-grids that need to be processed in the three dimensions of x, y, and z is:
[0062]
[0063] Z sub =Z
[0064] S3, each process on the CPU side creates subgrid data structure information for storing subgrid information, wherein the subgrid data structure information includes: subgrid number, subgrid size, subgrid property array, internal grid distribution function array, internal grid adjacent grid index array, boundary layer grid distribution function array, boundary layer grid adjacent grid index array, exchange layer grid distribution function array;
[0065] Each process on the CPU side creates a sub-grid data structure information to store sub-grid information, such as Figure 5 , including: the subgrid number is used to store the number of the subgrid in the global grid; the subgrid size includes the number of grids in the subgrid in the three directions of x, y, and z; the subgrid property array is used to store the properties corresponding to each grid point in the subgrid: solid, fluid, boundary, wall, etc.; the internal grid distribution function array is used to store the particle density of each grid point in the internal grid in different speed directions; the internal grid adjacent grid index array is used to store the index of the adjacent grids of each grid point in the internal grid in different speed directions; the boundary layer grid distribution function array is used to store the particle density of each grid point in the boundary layer grid in different speed directions; the boundary layer grid adjacent grid index array is used to store the index of the adjacent grids of each grid point in the boundary layer grid in different speed directions; the exchange layer grid distribution function array is used to store the boundary layer grid distribution function that requires inter-process data communication.
[0066] S4, each process on the CPU side reads the geometric structure information of the corresponding calculation domain and the initial conditions of the flow field, initializes the sub-grid data structure, and allocates a DCU accelerator to each process on the CPU side, and then each process on the CPU side transmits the sub-grid data structure information to the corresponding DCU side;
[0067] Specifically, in step S4, a DCU accelerator is allocated to each process on the CPU side, and then each process on the CPU side transmits the sub-grid data structure information to the corresponding DCU side. Specifically, a DCU accelerator is allocated to each process on the CPU side through the hipSetDevice function, and then each process on the CPU side transmits the sub-grid data structure information to the corresponding DCU side through the hipMemcpy function.
[0068] S5. On the DCU side, first traverse the boundary layer grids in the subgrid, assign a hip thread to each boundary layer grid point, perform collision calculation according to the LBM collision model, and after the calculation is completed, assign the boundary layer grid distribution function array to the exchange layer grid distribution function array according to the preset assignment rule, and then start asynchronous copying to transfer the distribution function array corresponding to the exchange layer grid from the DCU side to the CPU side, and after starting the asynchronous copying, assign a thread to each internal grid point, and start traversing the internal grid for parallel collision calculation;
[0069] Specifically, the boundary layer grid distribution function array in the subgrid is first traversed on the DCU side, a hip thread is assigned to each boundary layer grid point, collision calculation is performed according to the LBM collision model to complete the update of the boundary layer grid distribution function array, and after the calculation is completed, the boundary layer grid distribution function array is assigned to the exchange layer grid distribution function array according to the preset assignment rule, and then the asynchronous copy is started to transfer the exchange layer grid distribution function array from the DCU side to the CPU side, and after the asynchronous copy is started, a thread is assigned to each internal grid point, and the internal grid distribution function array is started to be traversed for parallel collision calculation;
[0070] After the asynchronous copy is started, the internal mesh is traversed for collision calculation. The internal mesh collision calculation and asynchronous copy are performed simultaneously.
[0071] Specifically, the LBM collision model in step S5 includes: a single relaxation model LBGK and a multi-relaxation model MRT.
[0072] Specifically, after the calculation in step S5 is completed, the hipMemcpyAsync function is used to start asynchronous copying to transfer the distribution function array corresponding to the exchange layer grid from the DCU end to the CPU end.
[0073] Specifically, the preset assignment rule in step S5 is as follows: the boundary layer of the process (x, y) corresponding to the sub-grid block is divided into a first boundary surface, a second boundary surface, a third boundary surface and a fourth boundary surface, and the exchange layer is divided into a first data surface, a second data surface, a third data surface and a fourth data surface, and the data of the four surfaces of the boundary layer are assigned outward to the corresponding positions of the exchange layer, and the edge grid data intersecting between the data surfaces of the exchange layer are assigned to the edge grid data intersecting between the data surfaces of the corresponding boundary layers, wherein the first boundary surface and the first data surface are positive surfaces of the y-axis, the second boundary surface and the second data surface are negative surfaces of the y-axis, the third boundary surface and the third data surface are negative surfaces of the x-axis, and the fourth boundary surface and the fourth data surface are positive surfaces of the x-axis.
[0074] like Figure 6As shown in the figure, the boundary layer of the process (x, y) corresponding to the sub-grid block is divided into four boundary surfaces ①②③④ according to the four directions of the x and y axes, and the exchange layer is also divided into four data surfaces, among which the first boundary surface ① and the first data surface of the exchange layer are the positive surfaces of the y axis, the second boundary surface ② and the second data surface are the negative surfaces of the y axis, the third boundary surface ③ and the third data surface are the negative surfaces of the x axis, and the fourth boundary surface ④ and the fourth data surface are the positive surfaces of the x axis. The four surfaces ①②③④ are copied to the corresponding positions of the data surfaces in the same direction in the exchange layer, and the values of the edge grids at the intersections of the boundary surfaces in the boundary layer are assigned to the values of the edges of the data surfaces in the same direction in the exchange layer. In the edge grid at the intersection; the first data surface contains the data of the first boundary surface and the intersection edge data of the first boundary surface with the third boundary surface and the fourth boundary surface; the second data surface contains the data of the second boundary surface and the intersection edge data of the second boundary surface with the third boundary surface and the fourth boundary surface; the third data surface contains the data of the third boundary surface and the intersection edge data of the third boundary surface with the first boundary surface and the second boundary surface; the fourth data surface contains the data of the fourth boundary surface and the intersection edge data of the fourth boundary surface with the first boundary surface and the second boundary surface. The above operations are completed to realize the transfer of the boundary layer grid distribution function array to the boundary layer.
[0075] S6. After completing the data copy from the DCU end to the CPU end, each process first completes the inter-process data communication of the exchange layer distribution function array in the x direction according to the two-dimensional division method of the grid, and then completes the inter-process data communication of the exchange layer distribution function array in the y direction after waiting for the data communication in the x direction to be completed;
[0076] Specifically, step S6 is as follows: after the CPU receives the exchange layer grid distribution function array transmitted by the DCU, it first completes the inter-process data communication of the exchange layer grid distribution function array in the x direction through the inter-process communication MPI_Sendrecv_replace function according to the grid division method, and after the data communication in the x direction is completed, completes the inter-process data communication of the exchange layer grid distribution function array in the y direction.
[0077] Specifically, the data communication rule in step S6 is: when process (x, y) performs inter-process data communication in the x direction, the third data plane data of process (x, y) is sent to process (x-1, y), and the fourth data plane data sent by process (x-1, y) is received; the fourth data plane of process (x, y) is sent to process (x+1, y), and the third data plane data sent by process (x+1, y) is received. After completing the inter-process data communication in the x direction, the inter-process data communication in the y direction is started, the first data plane data of the exchange layer grid is sent to process (x, y+1) and the second data plane data sent by process (x, y+1) is received, the second data plane data is sent to process (x, y-1) and the first data plane data sent by process (x, y-1) is received.
[0078] like Figure 7 As shown in the figure, when process (x, y) is communicating data between processes in the x direction, the third data plane data of the exchange layer grid is sent to process (x-1, y) and the fourth data plane data sent by process (x-1, y) is received, thereby realizing data exchange between process (x, y) and process (x-1, y) in the x direction; the fourth data plane data of process (x, y) is sent to process (x+1, y) and the third data plane data sent by process (x+1, y) is received, thereby realizing data exchange between process (x, y) and process (x+1, y) in the x direction; after completing the inter-process data communication in the x direction, as shown in the figure, Figure 8 As shown, process (x, y) starts inter-process data communication in the y direction, the first data plane data of the exchange layer grid of process (x, y) is sent to process (x, y+1) and the second data plane data sent by process (x, y+1) is received, thereby realizing data exchange between process (x, y) and process (x, y+1) in the y direction; the second data plane data of process (x, y) is sent to process (x, y-1) and the first data plane data sent by process (x, y-1) is received, thereby realizing data exchange between process (x, y) and process (x, y-1) in the y direction.
[0079] When process (x, y) completes inter-process communication in the x direction, the third data plane of process (x, y) contains the data of the fourth data plane of process (x+1, y), and the third data plane of process (x, y+1) contains the data of the fourth data plane of process (x+1, y). When inter-process communication in the y direction is performed, because the third data plane of process (x, y) and the first data plane have a common edge grid data, after the y direction communication is completed, the edge grid data containing the boundary between process (x+1, y) and process (x, y+1) is transmitted to each other. Similarly, the edge grid data at the boundary between process (x, y) and process (x+1, y+1) is also transmitted to each other.
[0080] After completing the data exchange, the exchange layer data of each process stores the grid information adjacent to the process boundary layer grid in the overall grid. During the LBM calculation process, only the migration calculation requires the data of adjacent grid points. The purpose of data exchange is to complete the subsequent migration calculation.
[0081] Specifically, data between data planes between processes can be exchanged through MPI communication.
[0082] When implementing conventional four-face four-edge data exchange, each process usually needs to send four face data and four edge data, for a total of eight data transmissions, and also needs to receive data eight times. In the four-face inter-process data exchange method proposed in this embodiment, each process only needs to send four face data and receive four face data, reducing the number of data exchanges by half, thereby reducing the inter-process MPI communication time and greatly improving the efficiency of parallel computing.
[0083] Specifically, when a process is located at the boundary of the process topology, the corresponding sending and receiving processes are set to empty processes.
[0084] During the entire calculation process, the boundary layer grid participates in the calculation, while the exchange layer grid does not participate in the calculation and is used to store data transmitted from other processes.
[0085] S7, the CPU starts asynchronous transmission to transmit the received sub-grid switching layer data to the switching layer of the DCU terminal grid;
[0086] Specifically, step S7 is as follows: the CPU starts asynchronous transmission through the hipMemcpyAsync function to transmit the received sub-grid exchange layer data to the exchange layer of the DCU terminal grid.
[0087] refer to Figure 1 , the asynchronous copying of data between the CPU and the DCU in steps S5 and S7 and the inter-process communication in S6 are performed simultaneously with the grid point collision calculation on the DCU to achieve the purpose of covering up the copying and communication time.
[0088] S8. After completing steps S5-S7, the DCU performs migration calculations of the boundary layer and the internal grid in the subgrid block;
[0089] During the migration calculation process, when updating the internal grid distribution function array, it is necessary to rely on the internal grid adjacent grid index array to obtain the adjacent grid point distribution function array. When updating the boundary grid distribution function array, it is necessary to rely on the boundary grid adjacent grid index array to obtain the adjacent grid point distribution function array.
[0090] The data of these adjacent grid points are accessed by obtaining the index position of the grid point adjacent to the current grid point in the boundary layer grid distribution function array through the boundary layer grid adjacent grid index array, and then obtaining the value of the distribution function of the corresponding adjacent grid through this index;
[0091] After completing steps S5-S7, the DCU allocates a thread to each grid point in the subgrid except the switching layer grid to complete the parallel calculation of the migration process.
[0092] S9. Repeat steps S5-S8 until the preset number of iterations required by the simulation is reached.
[0093] The preset number of iteration steps can be determined according to the specific requirements of the simulation calculation and is not limited here. Preferably, the preset number of iteration steps in this embodiment is 20,000.
[0094] The present embodiment discloses an LBM parallel computing method based on a CPU-DCU heterogeneous architecture, comprising the following steps: S1, dividing a three-dimensional global grid into npx×npy sub-grid blocks along the xoy plane according to the computing domain range and grid accuracy, and the CPU side assigns a process to each sub-grid block for computing; S2, adding one exchange layer grid to each of the four faces in the x and y directions of each grid block for data exchange between processes; S3, each process on the CPU side creates sub-grid data structure information for storing sub-grid information; S4, each process on the CPU side reads the corresponding geometric structure information of the computing domain and the initial conditions of the flow field, initializes the sub-grid data structure, and assigns a DCU accelerator to each process on the CPU side, and then each process on the CPU side transmits the sub-grid data structure information to the corresponding DCU side; S5, first traversing the boundary layer grids in the sub-grid on the DCU side, assigning each boundary layer grid point A hip thread performs collision calculation according to the LBM collision model. After the calculation is completed, the boundary layer grid distribution function array is assigned to the exchange layer grid distribution function array according to the preset assignment rule, and then the asynchronous copy is started to transfer the distribution function array corresponding to the exchange layer grid from the DCU end to the CPU end. After starting the asynchronous copy, a thread is assigned to each internal grid point to start traversing the internal grid for parallel collision calculation; S6, after completing the data copy from the DCU end to the CPU end, each process completes the data exchange of the exchange layer grid distribution function array between processes according to the grid division method between processes; S7, the CPU end starts asynchronous transmission to transmit the received sub-grid exchange layer data to the exchange layer of the DCU terminal grid; S8, after completing steps S5-S7, the DCU end performs migration calculations of the boundary layer and the internal grid in the sub-grid block; S9, repeat steps S5-S8 until the preset number of iterations required by the simulation is reached.
[0095] This embodiment expands the fluid domain range of the lattice Boltzmann method in fluid dynamics simulation, implements the lattice Boltzmann calculation method based on CPU-DCU heterogeneous clusters, can perform large-scale parallel simulation on the cluster, and can achieve large-scale, high-precision fluid dynamics simulation based on the lattice Boltzmann method;
[0096] Furthermore, this embodiment utilizes the computing performance of the DCU and designs a DCU-based collision and migration parallel computing method. At the same time, the asynchronous data transmission between the CPU and the DCU and the inter-process communication masking technology are used to realize the time masking of data copying between the DCU and the CPU and the inter-process communication, thereby reducing the computing time of the lattice Boltzmann method in fluid dynamics simulation, greatly improving the performance of LBM calculation, and accelerating the performance by an order of magnitude compared to the LBM on the CPU.
[0097] In addition, this embodiment proposes a method for exchanging boundary layer grid data between processes, which can reduce the number of times data is sent and received between processes, thereby reducing the MPI communication time between processes and effectively improving the efficiency of parallel computing.
[0098] Embodiment 2
[0099] This embodiment discloses an LBM parallel computing device based on a CPU-DCU heterogeneous architecture, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run an LBM parallel computing method based on a CPU-DCU heterogeneous architecture provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.
[0100] Embodiment 3
[0101] refer to Fig. 9 , Fig. 9 : is a structural diagram of an LBM parallel computing device based on a CPU-DCU heterogeneous architecture of this embodiment. An LBM parallel computing device 20 based on a CPU-DCU heterogeneous architecture of this embodiment includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, the steps in the above method embodiment are implemented. Alternatively, when the processor 21 executes the computer program, the functions of each module / unit in the above device embodiments are implemented.
[0102] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the LBM parallel computing device 20 based on the CPU-DCU heterogeneous architecture. For example, the computer program may be divided into the modules in Embodiment 2. For the specific functions of each module, please refer to the working process of the device described in the above embodiment, which will not be repeated here.
[0103] The LBM parallel computing device 20 based on the CPU-DCU heterogeneous architecture may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will appreciate that the schematic diagram is merely an example of an LBM parallel computing device 20 based on the CPU-DCU heterogeneous architecture, and does not constitute a limitation on an LBM parallel computing device 20 based on the CPU-DCU heterogeneous architecture, and may include more or fewer components than shown in the diagram, or a combination of certain components, or different components, for example, the LBM parallel computing device 20 based on the CPU-DCU heterogeneous architecture may also include input and output devices, network access devices, buses, and the like.
[0104] The processor 21 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 21 is the control center of the LBM parallel computing device 20 based on the CPU-DCU heterogeneous architecture, and uses various interfaces and lines to connect the various parts of the entire LBM parallel computing device 20 based on the CPU-DCU heterogeneous architecture.
[0105] The memory 22 can be used to store the computer program and / or module. The processor 21 implements various functions of the LBM parallel computing device 20 based on the CPU-DCU heterogeneous architecture by running or executing the computer program and / or module stored in the memory 22 and calling the data stored in the memory 22. The memory 22 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0106] Wherein, if the module / unit integrated in the LBM parallel computing device 20 based on the CPU-DCU heterogeneous architecture is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor 21, the steps of the above-mentioned method embodiments can be implemented. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electrical carrier signals and telecommunication signals.
[0107] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.
[0108] So far, the technical solutions of the present invention have been described in combination with examples, but it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.
Claims
1. A LBM parallel computing method based on CPU-DCU heterogeneous architecture, characterized in that: The following steps are involved: S1. Divide the three-dimensional global grid into npx×npy sub-grid blocks along the xoy plane according to the calculation domain range and grid accuracy. The CPU side allocates a process to each sub-grid block for calculation. The sub-grid block includes an internal grid and a boundary layer grid. S2, add one exchange layer grid on each of the four faces in the x and y directions of each grid block for data exchange between processes; S3, each process on the CPU side creates subgrid data structure information for storing subgrid information, wherein the subgrid data structure information includes: subgrid number, subgrid size, subgrid property array, internal grid distribution function array, internal grid adjacent grid index array, boundary layer grid distribution function array, boundary layer grid adjacent grid index array, and exchange layer distribution function array; S4, each process on the CPU side reads the geometric structure information of the corresponding calculation domain and the initial conditions of the flow field, initializes the sub-grid data structure, and allocates a DCU accelerator to each process on the CPU side, and then each process on the CPU side transmits the sub-grid data structure information to the corresponding DCU side; S5. On the DCU side, first traverse the boundary layer grids in the subgrid, assign a hip thread to each boundary layer grid point, perform collision calculation according to the LBM collision model, and after the calculation is completed, assign the boundary layer grid distribution function array to the exchange layer grid distribution function array according to the preset assignment rule, and then start asynchronous copying to transfer the distribution function array corresponding to the exchange layer grid from the DCU side to the CPU side, and after starting the asynchronous copying, assign a thread to each internal grid point, and start traversing the internal grid for parallel collision calculation; The preset assignment rule in step S5 is specifically: the boundary layer of the process (x, y) corresponding to the sub-grid block is divided into a first boundary surface, a second boundary surface, a third boundary surface and a fourth boundary surface, and the exchange layer is divided into a first data surface, a second data surface, a third data surface and a fourth data surface, and the data of the four surfaces of the boundary layer are assigned outward to the corresponding positions of the exchange layer, and the edge grid data intersecting between the data surfaces of the exchange layer are assigned to the edge grid data intersecting between the data surfaces of the corresponding boundary layer, wherein the first boundary surface and the first data surface are positive surfaces of the y-axis, the second boundary surface and the second data surface are negative surfaces of the y-axis, the third boundary surface and the third data surface are negative surfaces of the x-axis, and the fourth boundary surface and the fourth data surface are positive surfaces of the x-axis; S6. After completing the data copy from the DCU end to the CPU end, each process first completes the inter-process data communication of the exchange layer distribution function array in the x direction according to the two-dimensional division method of the grid, and then completes the inter-process data communication of the exchange layer distribution function array in the y direction after waiting for the data communication in the x direction to be completed; S7, the CPU starts asynchronous transmission to transmit the received sub-grid switching layer data to the switching layer of the DCU terminal grid; S8. After completing steps S5-S7, the DCU performs migration calculations of the boundary layer and the internal grid in the subgrid block; S9. Repeat steps S5-S8 until the preset number of iterations required by the simulation is reached.
2. The method according to claim 1, characterized in that: Step S1 is specifically as follows: the size of the global grid is (X, Y, Z), then the number of grids in the x direction of the subgrid blocks corresponding to the first (X mod npx) processes along the positive x direction is floor(X / npx)+1, and the number of grids in the x direction of the grid blocks corresponding to other processes is floor(X / npx), the number of grids in the y direction of the grid blocks corresponding to the first (Y mod npy) processes along the positive y direction is floor(Y / npy)+1, and the number of grids in the y direction of the grid blocks corresponding to other processes is floor(Y / npy), and the number of grids in the z direction of the grid blocks corresponding to each process is Z, and mod is the remainder function.
3. The method according to claim 2, characterized in that In step S4, a DCU accelerator is allocated to each process on the CPU side, and then each process on the CPU side transmits the sub-grid data structure information to the corresponding DCU side. Specifically: a DCU accelerator is allocated to each process on the CPU side through the hipSetDevice function, and then each process on the CPU side transmits the sub-grid data structure information to the corresponding DCU side through the hipMemcpy function.
4. The method according to claim 1, characterized in that: After the calculation in step S5 is completed, the hipMemcpyAsync function is used to start asynchronous copying to transfer the distribution function array corresponding to the exchange layer grid from the DCU end to the CPU end.
5. The method according to claim 1, characterized in that Step S6 is specifically as follows: after the CPU receives the exchange layer grid distribution function array transmitted by the DCU, it first completes the inter-process data communication of the exchange layer grid distribution function array in the x direction through the inter-process communication MPI_Sendrecv_replace function according to the grid division method, and after the data communication in the x direction is completed, it completes the inter-process data communication of the exchange layer grid distribution function array in the y direction.
6. The method according to claim 1, characterized in that The data communication rule in step S6 is: when process (x, y) performs inter-process data communication in the x direction, the third data plane data of process (x, y) is sent to process (x-1, y), and the fourth data plane data sent by process (x-1, y) is received; the fourth data plane of process (x, y) is sent to process (x+1, y), and the third data plane data sent by process (x+1, y) is received. After completing the inter-process data communication in the x direction, the inter-process data communication in the y direction is started, the first data plane data of the exchange layer grid is sent to process (x, y+1) and the second data plane data sent by process (x, y+1) is received, the second data plane data is sent to process (x, y-1) and the first data plane data sent by process (x, y-1) is received.
7. The method according to claim 5, characterized in that When a process is located at the boundary of the process topology, the corresponding sending and receiving processes are set to empty processes.
8. The method according to claim 1, characterized in that Step S7 is specifically as follows: the CPU starts asynchronous transmission through the hipMemcpyAsync function to transmit the received sub-grid exchange layer data to the exchange layer of the DCU terminal grid.
9. The method according to claim 1, characterized in that: During the migration calculation process of step S8, when updating the internal grid distribution function array, it is necessary to rely on the internal grid adjacent grid index array to obtain the adjacent grid point distribution function array; when updating the boundary grid distribution function array, it is necessary to rely on the boundary grid adjacent grid index array to obtain the adjacent grid point distribution function array.
