GPU direct connection read-write method for HDF5 large-scale data

Abstract

The invention discloses a GPU (Graphics Processing Unit) direct-connection read-write method for HDF5 large-scale data, which comprises the following steps of: analyzing an HDF5 data set from a storage device, and reading the HDF5 data set into a GPU single card or a GPU cluster memory in parallel through a PCIe (Peripheral Component Interconnect Express) channel, so that the data of the storage device can be read to the GPU single card or the GPU cluster memory at one time; from a GPU single card or a GPU cluster memory, directly compressing an HDF5 data set, and writing the HDF5 data set in parallel to a storage device through a PCIe channel; comprising the following steps: dividing into N Zones, averagely storing the N Zones in a data set of M HDF5 grid files, and starting T GPU threads to execute high-bandwidth parallel read-in; the GPU has C computing cores, and each GPU thread starts C / T computing cores to perform full-load iterative computation; and setting an iteration step of a calculation result according to requirements, and writing the calculation result in a data set with N Zone and M HDF5 flow field files in parallel through a corresponding GPU thread.

Description

Technical Field

[0001] The invention belongs to the field of computer technology, and in particular relates to a GPU direct connection reading and writing method for HDF5 large-scale data. Background Art

[0002] In the current high-performance computing (HPC) environment, Computer Aided Engineering (CAE) has an increasing demand for the management and calculation of large-scale grids and flow field data. In the early days, limited by programming languages ​​and hardware architecture, high-performance computing research in the CAE field was mainly based on CPU clusters. With the introduction of GPU programming languages ​​such as CUDA, the threshold for using GPU for accelerated computing has been lowered, and more and more CAE researchers have begun to turn to GPU research. A large number of computational acceleration studies using single GPU cards or GPU clusters have begun to appear in the CAE simulation field. At the same time, in order to better utilize computing resources, CPU / GPU heterogeneous computing has also begun to appear, and has achieved certain acceleration effects.

[0003] Although the CPU / GPU heterogeneity is currently optimized to perform multiple operations separately, the GPU still needs to rely on the CPU to provide data and start computing tasks on the GPU for accelerating scientific computing. The relevant data needs to be frequently communicated between the GPU video memory and the CPU memory for data distribution and aggregation. In the current data flow of GPU-accelerated scientific computing, data is usually first read from the storage system to the CPU memory, then transferred to the GPU memory for calculation, and finally returned to the CPU for aggregation. For example, in NVIDIA's CUDA programming, cudaMemcpy is used to perform explicit data movement between the CPU and GPU. Figure 1 As shown, the data is read from the storage device into the CPU memory through step 1-2-3, and then distributed to a single or multiple GPU memory through step 3-2-4; after each calculation is completed, it is transferred back to the CPU memory for aggregation through step 4-2-3, and finally serially written back to the storage device through step 3-2-1; during the calculation process, due to the need for communication of data such as overlapping areas, the data transmission between GPUs needs to frequently execute step 4-2-3-2-4. In the above process, the CPU memory acts as a buffer between the storage system and the GPU memory. Due to the multiple communication transmissions of large-scale data, this method causes the read and write communication of CPU / GPU heterogeneous computing to encounter efficiency bottlenecks.

[0004] In the von Neumann computer standard system, the CPU and its memory are separated, while the GPU and its memory are integrated. PCIe (Peripheral Component Interconnect Express) is an interface standard for connecting high-speed components. There are many PCIe slots on each server or desktop computer motherboard, which can be used to add multiple GPU independent graphics cards, Nvme solid-state drives, etc. In traditional parallel computing, the CPU preprocesses the data, but in the case of large-scale accelerated computing that relies heavily or entirely on GPUs, it is unnecessary to temporarily store data in the CPU memory. GPU direct connection technology is an emerging data transmission method designed to optimize the performance of data parallel applications. By bypassing the data channel between the CPU and its memory through technical means, a direct path using PCIE is established directly between the local storage system or remote file system and the GPU memory. Its working principle is to avoid copying data to the CPU memory, but to directly transfer data between the GPU memory and the file storage device, and between the GPU memory and the GPU memory.

[0005] Since GPU memory has the characteristics of high bandwidth and good continuity, it is necessary to design specific large-scale data structures for it. As an important database, HDF5 (Hierarchical Data Format) is widely used in data storage and management in science and engineering. In the field of CAE simulation, mesh data is the geometric topological information before iterative calculation, such as points, lines, surfaces, and bodies; flow field data is the physical information distributed on the mesh during or after iterative calculation, such as velocity, density, pressure, temperature, etc. The HDF5 library provides a flexible and efficient data storage solution. Its hierarchical data format and storage abstraction layer make the management of complex data structures and large-capacity data sets more efficient and scalable. The HDF5 library provides a wealth of APIs and tools that enable users to easily create, read, and write data files, and supports the storage of multiple data types and complex data structures, providing strong support for large-scale data set management in the field of CAE simulation.

[0006] This paper will focus on exploring how to use the functions and characteristics of the HDF5 library, combined with the latest GPU direct connection technology, to achieve efficient data reading and writing between GPU memory and storage devices. By designing and implementing a large-scale distributed data structure based on HDF5, using GPU direct connection technology to bypass the CPU and its memory for reading and writing, the reading and writing performance and efficiency of CAE grid and flow field data files in a heterogeneous parallel computing environment will be greatly improved. This will provide important technical support for scientific computing and data management in the CAE field, and promote its development and application in the HPC environment. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a GPU direct connection reading and writing method for HDF5 large-scale data. The method improves the reading and writing performance of CAE grid and flow field data files in a heterogeneous parallel computing environment by using GPU direct connection technology.

[0008] In order to solve the above technical problems, the present invention is implemented in the following ways:

[0009] A GPU direct connection reading and writing method for HDF5 large-scale data, which parses an HDF5 data set from a storage device, reads it into a GPU single card or GPU cluster memory in parallel through a PCIe channel, realizes a one-time reading of the storage device data to the GPU single card or GPU cluster memory, and directly connects the GPU cluster memories; and directly compresses the HDF5 data set from the GPU single card or GPU cluster memory, and writes it out to the storage device in parallel through the PCIe channel; the GPU direct connection reading and writing method for HDF5 large-scale data includes the following steps:

[0010] S1. For a data set divided into N zones and stored in M ​​HDF5 grid files on average, the direct read-write method enables T GPU threads to perform high-bandwidth parallel read-in;

[0011] S2, GPU has C computing cores, and each GPU thread enables C / T computing cores for full-load iterative calculations;

[0012] S3. Set the iteration step of the calculation results according to the requirements, and write them out in parallel to a data set with N zones and M HDF5 flow field files through the corresponding GPU threads.

[0013] Furthermore, the HDF5 dataset adopts a distributed storage method, which is as follows:

[0014] 1) The same grid data is divided into N zones, each of which stores geometric topology information and physical flow field data for parallel load balancing;

[0015] 2) Make N zones completely split and stored in M ​​HDF5 grid files, where N is greater than or equal to M, so that the amount of network data in each HDF5 file is similar;

[0016] 3) The data in the M HDF5 grid files are associated with each other according to the parent-child node information to form a complete grid data set;

[0017] 4) The flow field data can be stored in the original HDF5 data set and incrementally written back to the respective grid data partitions, so that the flow field data and the grid data correspond one to one; or a new HDF5 file is created for storage, and the corresponding association with the original grid data must be made.

[0018] Furthermore, the preprocessing of the grid data in step S1 is as follows:

[0019] a) If the CAE mesh file is not in the HDF5 database format, it needs to be converted;

[0020] b) If the grid data is not partitioned, it needs to be zoned and the files split to achieve distributed storage.

[0021] Furthermore, the C / T computing cores of the GPU thread in step S2 together form T computing flows, and the communication between the computing flows is all completed inside the GPU. The specific communication process is as follows:

[0022] S21, each GPU computing flow receives its own grid data into the GPU memory and starts iterative computing;

[0023] S22, each GPU computing flow completes a CAE iterative calculation and sends the overlapping area data to the adjacent zone;

[0024] S23, each GPU computing flow receives the overlapping area data sent by its adjacent zone in turn, and continues the next iterative calculation;

[0025] S24, loop through steps S22-S23 until a set convergence condition is reached.

[0026] Furthermore, the specific method of writing in step S3 is as follows:

[0027] If only one set of flow field data needs to be written out, the increments are stored in the original grid data file, or a new HDF5 flow field data file is created and stored separately and associated with the original grid data file;

[0028] If multiple sets of flow field data need to be written out, multiple sets of HDF5 flow field data files are created to store the flow field data of different iteration steps respectively, and they are all associated with the original grid data files to form data combinations at different times.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The present invention bypasses CPU memory by introducing GPU direct connection technology and uses PCIe high-speed channels to achieve direct transmission of grid flow field data within a multi-graphics card cluster, between video memory and storage devices; uses HDF5 layered data format, designs distributed data sets containing multiple data types, and constructs data standards for grid flow fields in GPU direct connection mode; integrates HDF5 in GPU direct connection technology, so that data can be efficiently read and written in the field of CAE large-scale scientific computing, without involving explicit data movement between CPU and GPU. This method shows excellent performance in large-scale CAE data reading, writing and computing, effectively reducing the CPU memory load, and its performance will be further improved with the development of PCIe standards. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a schematic diagram of the HDF5 data flow process for traditional CPU / GPU heterogeneous computing;

[0032] Figure 2 A schematic diagram of the HDF5 data transfer process of the integrated GPU direct connection technology of the present invention;

[0033] Figure 3 A schematic diagram of the logical structure of a large-scale distributed data set of HDF5 according to the present invention;

[0034] Figure 4 This is a schematic diagram of the CAE calculation process of direct GPU reading and writing based on HDF5 in the present invention. DETAILED DESCRIPTION

[0035] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, the specific implementation methods of the present invention are now described with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention generally described and shown in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0036] A GPU direct connection reading and writing method for large-scale HDF5 data combines the characteristics of HDF5 distributed storage and GPU direct connection high-speed reading and writing. For data reading and writing that requires GPU for scientific computing, it bypasses the CPU and its memory, directly parses the HDF5 data set from the storage device, and reads it into the GPU single card or GPU cluster memory in parallel through the PCIe channel, so as to achieve one-time reading of storage device data to the GPU single card or GPU cluster memory, and direct connection between GPU cluster memories; and directly compresses the HDF5 data set from the GPU single card or GPU cluster memory, and writes it out to the storage device in parallel through the PCIe channel, such as Figure 4 As shown, the GPU direct connection reading and writing method includes the following steps:

[0037] S1. For a data set divided into N zones, according to the index sequence and data volume of the data blocks, the data is stored in M ​​HDF5 grid files on average, and this program method enables T GPU threads to perform high-bandwidth parallel reading;

[0038] S2, GPU has C computing cores. Considering load balancing, each GPU thread enables C / T computing cores to perform full-load iterative calculations;

[0039] S3. Set the iteration step of the calculation results according to the requirements, and write them out in parallel to a data set with N zones and M HDF5 flow field files through the corresponding GPU threads.

[0040] Furthermore, the HDF5 grid flow field dataset adopts a distributed storage method. In step S1, for a dataset divided into N zones and stored in M ​​HDF5 grid flow field files, Figure 3 As shown, its distributed logical structure is designed as follows:

[0041] 1) The same grid data is divided into N grid data partitions, each zone stores geometric topology information and physical flow field data, and performs parallel load balancing adjustment. The grid refers to the topological data of the point, line, surface and body of the CAE simulation model, and the flow field refers to the physical data in the CAE calculation process: such as temperature, velocity, pressure, etc.

[0042] 2) Make N grid data partitions Zone be completely split and stored in M ​​HDF5 grid files, where N is greater than or equal to M, so that the amount of network data in each HDF5 file is similar;

[0043] 3) The data in the M HDF5 grid files are associated with each other according to the parent-child node information to form a complete grid data set;

[0044] 4) The flow field data can be stored in the original HDF5 data set and incrementally written back to the respective grid data partitions, so that the flow field data and the grid data correspond one to one; or a new HDF5 file is created for storage, and the corresponding association with the original grid data must be made.

[0045] When it is applicable to reading and writing large-scale grid files in other data formats or unpartitioned, it is necessary to perform certain preprocessing on the grid data. The preprocessing performed on the grid data in step S1 is as follows:

[0046] If the CAE grid file is not in the format of the HDF5 database, it needs to be converted using CAE field software; if the grid data is not partitioned, it needs to be zoned according to the data size and topological relationship.

[0047] If the grid file is not split or the split is unreasonable, it is necessary to further balance the data size of each zone and split it into multiple HDF5 format files to form a data set to achieve distributed storage.

[0048] Furthermore, the C / T computing cores of the GPU thread in step S2 together form T computing flows, and the communication between the computing flows is all completed inside the GPU. The specific communication process is as follows:

[0049] S21, each GPU computing flow receives its own grid data into the GPU memory and starts iterative computing;

[0050] S22, each GPU computing flow completes a CAE iterative calculation and sends the overlapping area data to the adjacent zone;

[0051] S23, each GPU computing flow receives the overlapping area data sent by its adjacent zone in turn, and continues the next iterative calculation;

[0052] S24, loop through steps S22-S23 until a set convergence condition is reached.

[0053] Furthermore, after the step S3 reaches the set iteration step to be output, the specific writing method is as follows:

[0054] If only one set of flow field data needs to be written out, the increments are stored in the original grid data file, or a new HDF5 flow field data file is created and stored separately and associated with the original grid data file;

[0055] If multiple sets of flow field data need to be written out, multiple sets of HDF5 flow field data files are created to store the flow field data of different iteration steps respectively, and they are all associated with the original grid data files to form data combinations at different times.

[0056] The bandwidth of a computing graphics card generally occupies x16 PCIe channels, while the bandwidth of an Nvme solid-state storage generally occupies x4 PCIe channels. In order to better match the high bandwidth of the graphics card, a PCIe hard disk array card is required. For devices that store HDF5 data in a distributed manner, the number of Nvme solid-state storages can be reasonably increased so that bandwidth can be superimposed at the physical hardware level, thereby matching the high bandwidth of the graphics card PCIe and further doubling the data read and write speed of the GPU direct connection.

[0057] like Figure 2 As shown, the data flow process of the present invention uses GPU direct connection technology. Through steps 1-2, the program reads the storage device data at one time to the memory of a single GPU card or multiple GPU cards, and directly implements GPU internal data communication aggregation through step 2-1; and for GPU clusters and graphics cards, step 3 is used to directly connect the memories of two graphics cards without the need to return to the CPU memory.

[0058] Using the HDF5 hierarchical data format, a distributed data set containing multiple data types is designed, and the data standard of the grid flow field in the independent graphics card direct connection mode is constructed. When the grid and flow field data are applied to the GPU direct connection reading, considering the specific optimization strategy to adapt to the direct reading requirements of the GPU, it is necessary to optimize the data layout and memory to ensure that the data is stored in a continuous manner in the GPU memory to improve the GPU data access efficiency. In view of the parallel computing characteristics of the GPU, it is necessary to implement performance optimization and parallel computing strategies to give full play to the parallel processing capabilities of the GPU and improve computing efficiency and performance.

[0059] By using data stratification, for large-scale scientific computing data, outside the HDF5 format file, it is cut into multiple files, distributed in various paths of the storage system, associated with each other, and read and write to each other according to the child and parent nodes. Inside the HDF5 format file, the data is divided into different zones, and each zone stores geometric topological information and physical flow field data. Among them, the geometric topological information has multiple composite data types such as points (Node), faces (Face), and volumes (cell), and the physical flow field data has various data such as speed, pressure, density, and temperature. Due to the segmentation of the file, the internal data also supports association. For example, the geometric topological information and physical flow field data of Zone2 are stored in different files. At the same time, due to the segmentation of grid data and flow field data, when the calculation results are output, an HDF5 grid file can be realized to match multiple groups of HDF5 flow field files with different iteration steps to reduce storage pressure. Through this distributed data storage method, the high bandwidth characteristics of GPU are effectively utilized to further improve the reading and writing speed of data.

[0060] The above description is only an implementation mode of the present invention. It is stated again that for ordinary technicians in this technical field, several improvements can be made to the present invention without departing from the principle of the present invention. These improvements are also included in the protection scope of the claims of the present invention.

Claims

1. A GPU direct reading and writing method for large-scale HDF5 data, characterized by: Parse the HDF5 data set from the storage device and read it into the memory of a single GPU card or a GPU cluster in parallel through the PCIe channel, so that the data of the storage device can be read once and arrive at the memory of a single GPU card or a GPU cluster, and the memory of the GPU cluster can be directly connected. And directly compress the HDF5 dataset from the memory of a single GPU card or GPU cluster and write it out to the storage device in parallel through the PCIe channel; The GPU direct connection reading and writing method of HDF5 large-scale data includes the following steps: S1, divided into N zones, the dataset is stored in M ​​HDF5 grid files on average, and T GPU threads are enabled to perform high-bandwidth parallel reading; S2, GPU has C computing cores, and each GPU thread enables C / T computing cores for full-load iterative calculations; S3. Set the iteration step of the calculation results according to the requirements, and write them out in parallel to a data set with N zones and M HDF5 flow field files through the corresponding GPU threads.

2. The GPU direct connection reading and writing method of HDF5 large-scale data according to claim 1 is characterized in that: The HDF5 dataset adopts a distributed storage method, which is as follows: 1) The same grid data is divided into N zones, each of which stores geometric topology information and physical flow field data for parallel load balancing; 2) Make N zones completely split and stored in M ​​HDF5 grid files, where N is greater than or equal to M, so that the amount of network data in each HDF5 file is similar; 3) The data in the M HDF5 grid files are associated with each other according to the parent-child node information to form a complete grid data set; 4) The flow field data can be stored in the original HDF5 data set and incrementally written back to the respective grid data partitions, so that the flow field data and the grid data correspond one to one; or a new HDF5 file can be created for storage, and the corresponding association with the original grid data can be made.

3. The GPU direct connection reading and writing method of HDF5 large-scale data according to claim 1 is characterized in that: In step S2, the C / T computing cores of the GPU thread together form T computing flows. The communication between the computing flows is all completed inside the GPU. The specific communication process is as follows: S21, each GPU computing flow receives its own grid data into the GPU memory and starts iterative computing; S22, each GPU computing flow completes a CAE iterative calculation and sends the overlapping area data to the adjacent zone; S23, each GPU computing flow receives the overlapping area data sent by its adjacent zone in turn, and continues the next iterative calculation; S24, loop through steps S22-S23 until a set convergence condition is reached.

4. The GPU direct connection reading and writing method of HDF5 large-scale data according to claim 1 is characterized in that: The specific method of writing in step S3 is as follows: If only one set of flow field data needs to be written out, the increments are stored in the original grid data file, or a new HDF5 flow field data file is created and stored separately and associated with the original grid data file; If multiple sets of flow field data need to be written out, multiple sets of HDF5 flow field data files are created to store the flow field data of different iteration steps respectively, and they are all associated with the original grid data files to form data combinations at different times.

Images