A GPU direct read and write method for large-scale HDF5 data

Abstract

The present invention discloses a GPU direct connection reading and writing method for HDF5 large-scale data. The method comprises the following steps: dividing an HDF5 data set into N zones, storing the data set in M ​​HDF5 grid files on average, and enabling T GPU threads to perform high-bandwidth parallel reading; the GPU has C computing cores, and each GPU thread enables C / T computing cores to perform full-load iterative calculation; setting the iterative step of the calculation result according to demand, and writing the result in parallel to a data set having N zones and M HDF5 flow field files through the corresponding GPU threads.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a GPU direct connection reading and writing method for HDF5 large-scale data. Background Art

[0002] In the current High Performance Computing (HPC) environment, Computer Aided Engineering (CAE) is facing an increasing demand for managing and computing large-scale meshes and flow field data. Early on, due to limitations in programming languages ​​and hardware architecture, high-performance computing research in CAE primarily focused on CPU clusters. However, with the introduction of GPU programming languages ​​like CUDA, the threshold for GPU-accelerated computing has been lowered, and an increasing number of CAE researchers are turning to GPU research. CAE simulation is seeing a surge in computational acceleration research using single GPUs or GPU clusters. Furthermore, to better utilize computing resources, heterogeneous CPU / GPU computing has also emerged, achieving significant acceleration results.

[0003] Although the CPU / GPU heterogeneity is currently optimized to perform multiple operations separately, the GPU still needs to rely on the CPU to provide data and start computing tasks on the GPU when used to accelerate scientific computing. The relevant data needs to be frequently communicated between the GPU memory and the CPU memory for data distribution and aggregation. In the current data flow of GPU-accelerated scientific computing, data is usually first read from the storage system to the CPU memory, then transferred to the GPU memory for calculation, and finally returned to the CPU for aggregation. For example, in NVIDIA's CUDA programming, cudaMemcpy is used to perform explicit data movement between the CPU and GPU. Figure 1 As shown, data is read from the storage device into CPU memory via steps 1-2-3, then distributed to the display memory of one or more GPUs via steps 3-2-4. After each calculation is complete, the data is transferred back to the CPU memory for aggregation via step 4-2-3, and finally serially written back to the storage device via step 3-2-1. During the calculation process, data transmission between GPUs requires frequent execution of steps 4-2-3-2-4, due to the need for data communication in overlapping areas. In this process, the CPU memory acts as a buffer between the storage system and the GPU display memory. Due to the multiple transmissions of large amounts of data, this approach creates an efficiency bottleneck for read and write communication in heterogeneous CPU / GPU computing.

[0004] In the von Neumann computer standard, the CPU and its memory are separate, while the GPU and its memory are integrated. PCIe (Peripheral Component Interconnect Express) is an interface standard for connecting high-speed components. Every server or desktop motherboard has many PCIe slots, which can be used to add multiple GPUs, discrete graphics cards, NVMe solid-state drives, and so on. In traditional parallel computing, the CPU preprocesses data. However, when large-scale accelerated computing relies heavily or entirely on GPUs, temporarily storing data in CPU memory is unnecessary. GPU direct connection technology is an emerging data transmission method designed to optimize the performance of data-parallel applications. By technical means, it bypasses the data channel between the CPU and its memory and establishes a direct path using PCIe between the local storage system or remote file system and the GPU memory. It works by avoiding copying data to the CPU memory and instead transferring data directly between the GPU memory and the file storage device, and between GPU memory and GPU memory.

[0005] Since GPU memory has the characteristics of high bandwidth and good continuity, it is necessary to design specific large-scale data structures for it. HDF5 (Hierarchical Data Format), as an important database, is widely used in data storage and management in science and engineering. In the field of CAE simulation, mesh data is the geometric topological information before iterative calculation, such as points, lines, surfaces, and bodies; flow field data is the physical information distributed on the mesh during or after the iterative calculation, such as velocity, density, pressure, temperature, etc. The HDF5 library provides a flexible and efficient data storage solution. Its hierarchical data format and storage abstraction layer make the management of complex data structures and large-capacity data sets more efficient and scalable. The HDF5 library provides a rich set of APIs and tools that enable users to easily create, read, and write data files, and supports the storage of multiple data types and complex data structures, providing strong support for large-scale data set management in the field of CAE simulation.

[0006] This paper will explore how to leverage the capabilities and features of the HDF5 library, combined with the latest GPU direct connection technology, to achieve efficient data read and write between GPU memory and storage devices. By designing and implementing a large-scale distributed data structure based on HDF5, using GPU direct connection technology to bypass the CPU and its memory for read and write, the paper significantly improves the read and write performance and efficiency of CAE mesh and flow field data files in heterogeneous parallel computing environments. This will provide important technical support for scientific computing and data management in the CAE field, and promote its development and application in HPC environments. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a GPU direct connection reading and writing method for HDF5 large-scale data. The method improves the reading and writing performance of CAE grid and flow field data files in a heterogeneous parallel computing environment by using GPU direct connection technology.

[0008] In order to solve the above technical problems, the present invention is implemented in the following ways:

[0009] A GPU direct reading and writing method for large-scale HDF5 data includes parsing an HDF5 dataset from a storage device and reading it into the memory of a single GPU card or a GPU cluster in parallel through a PCIe channel, thereby achieving a one-time reading of the storage device data to the memory of a single GPU card or a GPU cluster, and direct connection between GPU cluster memories; and directly compressing the HDF5 dataset from the memory of a single GPU card or a GPU cluster and writing it out to the storage device in parallel through a PCIe channel. The GPU direct reading and writing method for large-scale HDF5 data includes the following steps:

[0010] S1. For a dataset divided into N zones and stored in an average of M HDF5 grid files, the direct read-write method enables T GPU threads to perform high-bandwidth parallel reading;

[0011] S2,GPU has C computing cores, each GPU thread uses C / T computing cores to perform full-load iterative calculations;

[0012] S3. Set the iteration step of the calculation results according to the requirements, and write them out in parallel to a dataset with N zones and M HDF5 flow field files through the corresponding GPU threads.

[0013] Furthermore, the HDF5 dataset adopts a distributed storage method, which is as follows:

[0014] 1) The same grid data is divided into N zones, each of which stores geometric topology information and physical flow field data for parallel load balancing;

[0015] 2) N zones are completely split and stored in M ​​HDF5 grid files, where N is greater than or equal to M, so that the amount of network data in each HDF5 file is similar;

[0016] 3) The data in M ​​HDF5 grid files are associated with each other based on the parent-child node information to form a complete grid dataset;

[0017] 4) Flow field data can be stored in the original HDF5 dataset and incrementally written back to the respective grid data partitions so that the flow field data and grid data correspond one to one; or a new HDF5 file can be created for storage, which must be associated with the original grid data.

[0018] Furthermore, the pre-processing of the grid data in step S1 is as follows:

[0019] a) If the CAE mesh file is not in the HDF5 database format, it needs to be converted to another format.

[0020] b) If the grid data is not partitioned, it needs to be zoned and split into files to achieve distributed storage.

[0021] Furthermore, the C / T computing cores of the GPU thread in step S2 together form T computing flows, and all communication between computing flows is completed inside the GPU. The specific communication process is as follows:

[0022] S21, each GPU computing flow receives its own grid data into GPU memory and starts iterative computing;

[0023] S22. Each GPU computing flow completes a CAE iterative calculation and sends the overlapping area data to the adjacent zone;

[0024] S23, each GPU computing flow receives the overlapping area data sent by its adjacent zone in turn, and continues the next iterative calculation;

[0025] S24. Execute steps S22-S23 in a loop until the set convergence condition is reached.

[0026] Furthermore, the specific method of writing in step S3 is as follows:

[0027] If only one set of flow field data needs to be written out, the increments are stored in the original grid data file, or a new HDF5 flow field data file is created and stored separately and associated with the original grid data file;

[0028] If multiple sets of flow field data need to be written out, multiple sets of HDF5 flow field data files are created to store the flow field data of different iteration steps respectively, and all of them are associated with the original grid data files to form data combinations at different times.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] This paper introduces GPU direct connection technology, bypassing CPU memory and using PCIe high-speed channels to achieve direct transmission of grid flow field data within a multi-graphics card cluster, between video memory and storage devices. Using the HDF5 layered data format, a distributed dataset containing multiple data types is designed, establishing a data standard for grid flow fields in GPU direct connection mode. Integrating HDF5 into GPU direct connection technology enables efficient data reading and writing in the field of large-scale CAE scientific computing without requiring explicit data movement between the CPU and GPU. This method demonstrates excellent performance in large-scale CAE data reading, writing, and computing, effectively reducing CPU memory load. With the development of PCIe standards, its performance will be further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Schematic diagram of the HDF5 data flow process for traditional CPU / GPU heterogeneous computing;

[0032] Figure 2 Schematic diagram of the HDF5 data transfer process of the integrated GPU direct connection technology of the present invention;

[0033] Figure 3 This is a schematic diagram of the logical structure of the HDF5 large-scale distributed data set of the present invention;

[0034] Figure 4 This is a schematic diagram of the CAE calculation process of direct reading and writing of GPU based on HDF5 in the present invention. DETAILED DESCRIPTION

[0035] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, the specific embodiments of the present invention are now described with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0036] A GPU direct reading and writing method for large-scale HDF5 data combines the characteristics of HDF5 distributed storage and GPU direct high-speed reading and writing. It is aimed at reading and writing data that requires GPU for scientific computing. It bypasses the CPU and its memory, parses the HDF5 dataset directly from the storage device, and reads it into the GPU single card or GPU cluster memory in parallel through the PCIe channel. It realizes a one-time reading of storage device data to the GPU single card or GPU cluster memory, and direct connection between GPU cluster memories; and directly compresses the HDF5 dataset from the GPU single card or GPU cluster memory and writes it out to the storage device in parallel through the PCIe channel, such as Figure 4 As shown, the GPU direct connection reading and writing method includes the following steps:

[0037] S1. For a dataset divided into N zones, the dataset is stored in M ​​HDF5 grid files according to the index sequence and data volume of the data blocks, and this program method enables T GPU threads to perform high-bandwidth parallel reading;

[0038] S2,GPU has C computing cores. Considering load balancing, each GPU thread enables C / T computing cores to perform full-load iterative calculations;

[0039] S3. Set the iteration step of the calculation results according to the requirements, and write them out in parallel to a dataset with N zones and M HDF5 flow field files through the corresponding GPU threads.

[0040] Furthermore, the HDF5 grid flow field dataset adopts a distributed storage method. In step S1, for a dataset divided into N zones and stored in M ​​HDF5 grid flow field files, as shown in FIG. Figure 3 As shown, its distributed logical structure is designed as follows:

[0041] 1) The same grid data is divided into N grid data partitions, each zone stores geometric topology information and physical flow field data, and performs parallel load balancing adjustment. The grid refers to the topological data of the point, line, surface and body of the CAE simulation model, and the flow field refers to the physical data in the CAE calculation process: such as temperature, velocity, pressure, etc.

[0042] 2) Make N grid data partitions Zone be completely split and stored in M ​​HDF5 grid files, where N is greater than or equal to M, so that the amount of network data in each HDF5 file is similar;

[0043] 3) The data in M ​​HDF5 grid files are associated with each other based on the parent-child node information to form a complete grid dataset;

[0044] 4) Flow field data can be stored in the original HDF5 dataset and incrementally written back to the respective grid data partitions so that the flow field data and grid data correspond one to one; or a new HDF5 file can be created for storage, which must be associated with the original grid data.

[0045] When reading and writing large-scale grid files in other data formats or non-partitioned grid files, it is necessary to perform certain preprocessing on the grid data. The preprocessing of the grid data in step S1 is as follows:

[0046] If the CAE grid file is not in the HDF5 database format, it needs to be converted using CAE field software; if the grid data is not partitioned, it needs to be zoned according to the data size and topological relationship.

[0047] If the grid file is not split or the split is unreasonable, it is necessary to further balance the data size of each zone and split it into multiple HDF5 format files to form a data set to achieve distributed storage.

[0048] Furthermore, the C / T computing cores of the GPU thread in step S2 together form T computing flows, and all communication between computing flows is completed inside the GPU. The specific communication process is as follows:

[0049] S21, each GPU computing flow receives its own grid data into GPU memory and starts iterative computing;

[0050] S22. Each GPU computing flow completes a CAE iterative calculation and sends the overlapping area data to the adjacent zone;

[0051] S23, each GPU computing flow receives the overlapping area data sent by its adjacent zone in turn, and continues the next iterative calculation;

[0052] S24. Execute steps S22-S23 in a loop until the set convergence condition is reached.

[0053] Furthermore, after step S3 reaches the set iterative step that needs to be output, the specific writing method is as follows:

[0054] If only one set of flow field data needs to be written out, the increments are stored in the original grid data file, or a new HDF5 flow field data file is created and stored separately and associated with the original grid data file;

[0055] If multiple sets of flow field data need to be written out, multiple sets of HDF5 flow field data files are created to store the flow field data of different iteration steps respectively, and all of them are associated with the original grid data files to form data combinations at different times.

[0056] The bandwidth of a graphics card typically occupies x16 PCIe lanes, while the bandwidth of an NVMe solid-state drive typically occupies x4 PCIe lanes. To better match the high bandwidth of the graphics card, a PCIe disk array card is required. For devices that store HDF5 data in a distributed manner, increasing the number of NVMe solid-state drives allows bandwidth to be stacked at the physical hardware level, matching the high bandwidth of the graphics card's PCIe, further multiplying the data read and write speeds of direct GPU connections.

[0057] like Figure 2 As shown, the data flow process of the present invention uses GPU direct connection technology. Through steps 1-2, the program reads the storage device data at one time to the memory of a single GPU card or multiple GPU cards, and directly realizes the internal data communication aggregation of the GPU through step 2-1; and for the GPU cluster and between graphics cards, step 3 is used to directly connect the memory of the two graphics cards without the need to return to the CPU memory.

[0058] Using the HDF5 hierarchical data format, we designed a distributed dataset containing multiple data types and established a data standard for mesh flow fields in direct-connection mode with a dedicated graphics card. When applying mesh and flow field data to GPU direct access, we considered specific optimization strategies to accommodate the GPU's direct access requirements. This required optimizing data layout and memory to ensure contiguous data storage within GPU memory, improving GPU data access efficiency. Taking advantage of the GPU's parallel computing capabilities, we implemented performance optimizations and parallel computing strategies to fully utilize the GPU's parallel processing capabilities and improve computational efficiency and performance.

[0059] By utilizing data layering, large-scale scientific computing data is split outside of the HDF5 format file into multiple files distributed across various paths in the storage system. These files are interconnected and can be read and written based on child and parent nodes. Within the HDF5 format file, data is divided into different zones, each storing geometric topology information and physical flow field data. Geometric topology information includes various composite data types such as points, faces, and cells, while physical flow field data includes various data types such as velocity, pressure, density, and temperature. Due to the file segmentation, internal data can also be associated; for example, the geometric topology information and physical flow field data for Zone 2 are stored in different files. Furthermore, due to the separation of mesh data and flow field data, when outputting computational results, a single HDF5 mesh file can be matched to multiple HDF5 flow field files with different iteration steps, reducing storage pressure. This distributed data storage method effectively utilizes the high bandwidth of GPUs, further improving data read and write speeds.

[0060] The above description is merely an embodiment of the present invention. It is stated again that, for a person skilled in the art, several improvements can be made to the present invention without departing from the principles of the present invention, and these improvements are also included in the scope of protection of the claims of the present invention.

Claims

1. A GPU direct read and write method for large-scale HDF5 data, characterized by: Parse HDF5 datasets from storage devices and read them into the memory of a single GPU or GPU cluster in parallel through PCIe channels, achieving a one-time read of storage device data to the memory of a single GPU or GPU cluster, with direct connection between GPU cluster memories; And directly compress HDF5 datasets from the memory of a single GPU card or GPU cluster and write them out to storage devices in parallel through PCIe channels; The GPU direct reading and writing method for HDF5 large-scale data includes the following steps: S1, divide the dataset into N zones, store them in M ​​HDF5 grid files on average, and enable T GPU threads to perform high-bandwidth parallel reading; S2,GPU has C computing cores, each GPU thread uses C / T computing cores to perform full-load iterative calculations; S3. Set the iteration step of the calculation results according to the requirements, and write them out in parallel to a dataset with N zones and M HDF5 flow field files through the corresponding GPU threads.

2. The GPU direct read and write method for HDF5 large-scale data according to claim 1, characterized in that: The HDF5 dataset adopts a distributed storage method, which is as follows: 1) The same grid data is divided into N zones, each of which stores geometric topology information and physical flow field data for parallel load balancing; 2) N zones are completely split and stored in M ​​HDF5 grid files, where N is greater than or equal to M, so that the amount of network data in each HDF5 file is similar; 3) The data in M ​​HDF5 grid files are associated with each other based on the parent-child node information to form a complete grid dataset; 4) Flow field data can be stored in the original HDF5 dataset and incrementally written back to the respective grid data partitions so that the flow field data and grid data correspond one to one; or a new HDF5 file can be created for storage and the corresponding association with the original grid data can be made.

3. The GPU direct reading and writing method for HDF5 large-scale data according to claim 1 is characterized in that: In step S2, the C / T computing cores of the GPU thread together form T computing flows. The communication between the computing flows is all completed inside the GPU. The specific communication process is as follows: S21, each GPU computing flow receives its own grid data into GPU memory and starts iterative computing; S22. Each GPU computing flow completes a CAE iterative calculation and sends the overlapping area data to the adjacent zone; S23, each GPU computing flow receives the overlapping area data sent by its adjacent zone in turn, and continues the next iterative calculation; S24. Execute steps S22-S23 in a loop until the set convergence condition is reached.

4. The GPU direct reading and writing method for HDF5 large-scale data according to claim 1, characterized in that: The specific method of writing in step S3 is as follows: If only one set of flow field data needs to be written out, the increments are stored in the original grid data file, or a new HDF5 flow field data file is created and stored separately and associated with the original grid data file; If multiple sets of flow field data need to be written out, multiple sets of HDF5 flow field data files are created to store the flow field data of different iteration steps respectively, and all of them are associated with the original grid data files to form data combinations at different times.

Images