Multi-node parallel acceleration methods, devices, equipment, and media based on high-performance computing platforms and RoCE networks.

Abstract

This invention provides a multi-node parallel acceleration method, apparatus, device, and medium based on a high-performance computing platform and RoCE network, relating to the field of cluster computing technology. The method includes the following steps: constructing a communication stack based on the RoCE network to realize an interconnected transmission network between nodes from sender to receiver; executing the computing and communication transmission tasks of the high-performance computing platform using CPU core binding technology; processing memory allocation requests initiated by operating system applications and requests for memory access via virtual addresses based on transparent big-page functionality; and performing preprocessing using multi-threading before the main computing task of the high-performance computing platform begins, and starting open multiprocessing threads to execute the main computing task and initiate the next round of preprocessing tasks in parallel. This scheme achieves parallel acceleration of multi-node cluster computing.

Description

Technical Field

[0001] This invention relates to the field of cluster computing technology, and in particular to a multi-node parallel acceleration method, apparatus, device, and medium based on a high-performance computing platform and RoCE network. Background Technology

[0002] The core idea of ​​multi-node parallel acceleration in cluster computing is to decompose a large computational task into many smaller subtasks, then distribute these subtasks to multiple computing nodes in the cluster for simultaneous execution (parallel computing), and finally aggregate the results from each node, thereby significantly reducing the overall computation time. Achieving the desired acceleration effect is not easy and will encounter the following main challenges:

[0003] 1. The time spent exchanging data between nodes: If communication is too frequent or the data volume is too large, nodes may spend a lot of time waiting for data instead of performing calculations.

[0004] 2. If tasks are not distributed evenly, some nodes will finish their work early and enter an idle state, while other nodes are still busy, which will waste resources.

[0005] 3. Amdahl's Law: The speedup of a system is limited by the proportion of its serial parts.

[0006] 4. With numerous nodes in the cluster, the probability of a single node failing is greatly increased. The failure of one node can cause the entire large-scale computing task to fail. Summary of the Invention

[0007] In view of this, embodiments of the present invention provide a multi-node parallel acceleration method based on a high-performance computing platform and RoCE network to solve the technical problems of long data exchange time between nodes, uneven task decomposition, and high failure rate in the prior art. The method includes:

[0008] Construct a communication stack based on the RoCE network, and build an interconnected transmission network between the sending and receiving ends of the cluster nodes based on the communication stack;

[0009] The computing tasks of the high-performance computing platform are executed through the CPU core binding technology and the non-uniform memory access architecture of the CPU. The communication transmission tasks between the high-performance computing platforms are executed based on the communication stack, the non-uniform memory access architecture and the CPU core binding technology of the RoCE network.

[0010] Enable the transparent big page function in the operating system of the high-performance computing platform, and based on the transparent big page function, process the memory allocation requests initiated by the application of the operating system and the requests of the application of the operating system to access memory through virtual addresses;

[0011] Before the main computing task of the high-performance computing platform begins, the data loading task, format conversion task and initialization computing task in the preprocessing are executed by multiple threads to generate preprocessed data, and open multiprocessing threads are started. The main computing task is executed by multiple open multiprocessing threads, and the next round of preprocessing tasks is started in parallel.

[0012] This invention also provides a multi-node parallel acceleration device based on a high-performance computing platform and RoCE network to solve the technical problems of long data exchange time between nodes, uneven task decomposition, and high failure rate in the prior art. The device includes:

[0013] The network deployment module is used to build a communication stack based on the RoCE network, and to build an interconnected transmission network between the sending and receiving ends of the cluster nodes based on the communication stack.

[0014] The core-binding strategy configuration module is used to execute the computing tasks of the high-performance computing platform through the CPU core-binding technology and the non-uniform memory access architecture of the CPU, and to execute the communication transmission tasks between the high-performance computing platforms based on the communication stack, the non-uniform memory access architecture and the CPU core-binding technology of the RoCE network.

[0015] Enable the transparent big page module to enable the transparent big page function in the operating system of the high-performance computing platform. Based on the transparent big page function, process the memory allocation requests initiated by the application of the operating system and the requests of the application of the operating system to access memory through virtual addresses.

[0016] The architecture adaptation module is used to execute data loading, format conversion and initialization tasks in preprocessing using multi-threading before the main computing task of the high-performance computing platform begins, generate preprocessed data, and start open multiprocessing threads to execute the main computing task and start the next round of preprocessing tasks in parallel.

[0017] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned multi-node parallel acceleration methods based on a high-performance computing platform and RoCE network, in order to solve the technical problems of long data exchange time between nodes, uneven task decomposition, and high failure rate in the prior art.

[0018] This invention also provides a computer-readable storage medium storing a computer program that executes any of the above-described multi-node parallel acceleration methods based on a high-performance computing platform and RoCE network, in order to solve the technical problems of long data exchange time between nodes, uneven task decomposition, and high failure rate in the prior art.

[0019] Compared with the prior art, the beneficial effects that at least one technical solution adopted in the embodiments of this specification can achieve include at least:

[0020] By implementing RoCE high-speed network, CPU core binding, transparent big page, and application parameter adaptation on a high-performance computing platform, a multi-node parallel acceleration framework for CAE simulation of hundreds of millions of machines was constructed. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a multi-node parallel acceleration method based on a high-performance computing platform and RoCE network provided by an embodiment of the present invention;

[0023] Figure 2 This is a structural block diagram of a computer device provided in an embodiment of the present invention;

[0024] Figure 3 This is a structural block diagram of a multi-node parallel acceleration device based on a high-performance computing platform and RoCE network provided in an embodiment of the present invention. Detailed Implementation

[0025] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0026] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] For software development aimed at large-scale CAE simulations involving hundreds of millions of processes, this paper provides a general multi-node parallel acceleration method. This method enables rapid identification of performance bottlenecks in multi-node parallel computing, pinpointing anomalies at the line-of-code level and providing optimization suggestions. It also supports real-time performance analysis of tens of millions of processes.

[0028] The clusters of domestically developed high-performance computing platforms are comprised of underlying hardware (computing nodes, high-speed networks, storage) and basic software (mathematical libraries, compilers, parallel libraries, solvers). The software development process can flexibly utilize domestically developed features. This invention fully integrates the characteristics of domestically produced hardware and software, achieving optimal software performance through this combination. By integrating with domestic hardware, the resource consumption of CPU, memory, network, and disk I / O during performance analysis is minimized, reducing the impact on actual software operation and restoring the software's actual running state.

[0029] This invention closely relies on the domestic high-performance computing software and hardware ecosystem, particularly the computing nodes centered around Huawei's high-performance computing platform processors, the high-speed Ethernet interconnect architecture supporting the RoCE protocol, the domestic Linux operating system, and the accompanying DevKit performance analysis tool. In typical high-performance computing application scenarios such as CAE simulations with grid scales of hundreds of millions, the overall performance of the system is not determined by a single factor, but is comprehensively affected by multiple dimensions such as computation, communication, memory access, and task scheduling. Therefore, the implementation of this invention does not apply a single optimization technique in isolation, but rather systematically integrates several key technical measures to construct an end-to-end collaboratively optimized operating environment, thereby achieving near-theoretical acceleration effects in multi-node parallel computing.

[0030] To achieve efficient multi-node parallel acceleration, this invention focuses on the following four aspects in its implementation: deployment and communication protocol optimization of the RoCE high-speed network, configuration of CPU core binding strategies, activation of Transparent Huge Pages (THP), and deep adaptation of application runtime parameters to the characteristics of the high-performance computing platform architecture. These four technologies operate on the cluster communication layer, process scheduling layer, memory management layer, and application execution layer, respectively, and have clear dependencies and collaborative enhancement mechanisms, forming an organic whole. The following sections will elaborate on the implementation principles, configuration logic, performance impact mechanisms, and positioning of each technology within the overall parallel acceleration framework, emphasizing their inherent connections.

[0031] In this embodiment of the invention, a multi-node parallel acceleration method based on a high-performance computing platform and RoCE network is provided, such as... Figure 1 As shown, the method includes:

[0032] Step S101: Construct a communication stack based on the RoCE network, and construct an interconnection transmission network between the sending and receiving ends of the cluster nodes based on the communication stack;

[0033] Step S102: Execute the computing tasks of the high-performance computing platform through the CPU core binding technology and the non-uniform memory access architecture of the CPU, and execute the communication transmission tasks between the high-performance computing platforms based on the communication stack of the RoCE network, the non-uniform memory access architecture and the CPU core binding technology.

[0034] Step S103: Enable the transparent big page function in the operating system of the high-performance computing platform. Based on the transparent big page function, process the memory allocation requests initiated by the application of the operating system and the requests of the application of the operating system to access memory through virtual addresses.

[0035] Step S104: Before the main computing task of the high-performance computing platform begins, the data loading task, format conversion task and initialization computing task in the preprocessing are executed by multi-threading to generate preprocessed data, and open multi-processing threads are started. The main computing task is executed by multiple open multi-processing threads, and the next round of preprocessing tasks is started in parallel.

[0036] In specific implementation, the following steps are used to construct a communication stack based on the RoCE network, and based on the communication stack, to build an interconnected transmission network between the sending and receiving ends of the cluster nodes:

[0037] A high-performance communication abstraction layer is constructed, and an underlying transmission protocol is selected through the high-performance communication abstraction layer, wherein the underlying transmission protocol is UD mode;

[0038] A communication stack including an application layer, an RDMA transport protocol layer, a network layer, and a data link layer is constructed, and the high-performance communication abstraction layer is inserted between the application layer and the RDMA transport protocol layer;

[0039] The data to be transmitted at the sending end is stored in the RDMA memory area of ​​the sending end, and a UD data packet is constructed using the data to be transmitted; the UD data packet is submitted to the transmission queue of the RoCE network, and the UD data packet is encapsulated into a network data frame supporting the RDMA transmission protocol through the RDMA transmission protocol layer of the communication stack, wherein the network data frame includes a UDP header; the network data frame is sent to the receiving end through the RoCE network, the UD data packet of the network data frame is verified and extracted, and the UD data packet is stored in the RDMA memory area of ​​the receiving end.

[0040] In large-scale MPI (Message Passing Interface) parallel computing, computational tasks are decomposed into multiple subtasks, distributed across different computing nodes in a cluster for parallel execution. However, the frequent exchange of boundary data and the execution of global reduction or synchronization operations between nodes constitute a significant overhead in the entire computation process. Traditional Ethernet communication mechanisms based on the TCP / IP protocol stack face significant bottlenecks in high-performance computing scenarios. First, TCP / IP communication relies on the operating system kernel protocol stack to process network packets, requiring multiple memory copies between user and kernel modes for each transmission and reception, triggering context switches and resulting in high CPU resource consumption. Second, the processing logic of the protocol stack itself (such as verification, retransmission, and congestion control) introduces latency of hundreds of microseconds or even milliseconds, making it difficult to meet the microsecond-level communication response requirements of HPC applications. Third, in high-concurrency, small-message communication scenarios, protocol header overhead accounts for a significant proportion, and the effective bandwidth utilization is far lower than the theoretical value of the physical link. These problems are particularly prominent in CAE simulations involving hundreds of millions of nodes.

[0041] To overcome the aforementioned communication bottlenecks, this invention employs RoCE (RDMA over Converged Ethernet) as a high-speed interconnection network between cluster nodes. RoCE is an implementation of RDMA (Remote Direct Memory Access) technology over standard Ethernet. Its core idea is to allow network cards to directly read and write the application memory of remote nodes, without the involvement of the remote CPU, the operating system kernel, or intermediate data copying.

[0042] Traditional TCP / IP communication requires multiple layers of forwarding: "application → kernel protocol stack → network card driver → network card." RoCE, however, simplifies this process by registering memory regions, allowing the network card to directly initiate read and write operations to remote memory addresses. This mechanism significantly reduces communication latency (down to the microsecond level), improves bandwidth utilization (approaching line speed), and drastically reduces CPU overhead, freeing up more computing resources for actual computational tasks.

[0043] Currently, the mainstream RDMA implementations include InfiniBand, iWARP, and RoCE. InfiniBand is designed specifically for RDMA, offering the best performance but at a high cost, requiring dedicated network interface cards (NICs) and switches. iWARP is based on TCP, and while compatible with ordinary Ethernet switches, a large number of TCP connections consume enormous memory resources during large-scale deployments. RoCE, on the other hand, is based on UDP / IP or the Ethernet link layer, achieving a good balance between performance, cost, and compatibility. RoCE's performance is close to InfiniBand's, its cost is lower than iWARP's, and it can reuse existing Ethernet infrastructure (requiring only switches that support lossless Ethernet features such as PFC and ECN), making it the mainstream choice for domestic HPC clusters.

[0044] RoCE comprises two versions: RoCEv1 operates at the Ethernet link layer (Layer 2) and requires switches to support flow control mechanisms such as Priority Flow Control (PFC) to avoid packet loss; RoCEv2, on the other hand, is encapsulated at the UDP / IP layer (Layer 3), introducing IP addressing, supporting cross-subnet communication, offering greater scalability, and being suitable for large-scale distributed deployments. This invention prioritizes the RoCEv2 solution to balance performance and networking flexibility.

[0045] Simply deploying RoCE hardware cannot automatically activate its performance advantages; the upper-layer communication software stack must be adapted. The specific implementation of this invention includes the following two key steps:

[0046] First, upgrade the MPI library to a version that supports UCX (Unified Communication X). UCX is a high-performance communication abstraction layer that can automatically select the optimal underlying transport protocol (such as RoCE's RC or UD mode). Many domestic CAE software still rely on Intel MPI 2018 or earlier versions, which do not integrate UCX support and still default to TCP / IP communication, thus failing to utilize RoCE capabilities. Therefore, this invention requires replacing the MPI library with Intel MPI 2021 or later, or OpenMPI 4.x or later, to obtain native RoCE support.

[0047] Secondly, the RoCE Unreliable Datagram (UD) transport mode is explicitly specified at runtime. UD mode employs a connectionless design, eliminating the need to maintain end-to-end connection states, resulting in minimal communication latency. Furthermore, connection overhead is linearly proportional to the number of processes, making it ideal for large-scale parallel scenarios with tens of thousands of processes. In contrast, while Reliable Connected (RC) mode provides reliable transmission, it requires maintaining an independent connection for each pair of processes, with the number of connections increasing quadratically (N²) with the number of processes. This leads to significant memory overhead in large-scale deployments, hindering system scalability. Therefore, this invention prioritizes UD mode in the configuration to achieve optimal communication scalability.

[0048] Through the above implementation, MPI communication will bypass the traditional TCP / IP protocol stack and directly access remote memory through the RoCE network card, significantly reducing communication latency and CPU overhead, providing a high-bandwidth, low-latency communication foundation for multi-node parallelism, and fundamentally alleviating the communication bottleneck between multiple nodes.

[0049] In specific implementation, the following steps are used to execute the computing tasks of the high-performance computing platform through CPU core binding technology and the CPU's non-uniform memory access architecture:

[0050] Analyze the topology of the CPU's non-consistent memory access architecture to determine the set of CPU cores and local memory regions corresponding to each node in the non-consistent memory access architecture. Establish a mapping table between message passing interface processes and nodes in the non-consistent memory access architecture, whereby each message passing interface process is mapped one-to-one with a node in the non-consistent memory access architecture. Start the message passing interface process cluster and, according to the mapping table, fix each message passing interface process to a specific CPU core of the target non-consistent memory access architecture node using CPU affinity binding technology. Configure a memory allocation strategy for each message passing interface process so that the message passing interface process uses the local physical memory of the bound node in the non-consistent memory access architecture. Execute the computing tasks of the high-performance computing platform based on the specific CPU core and the memory allocation strategy.

[0051] In specific implementation, the communication stack, the non-uniform memory access architecture, and the CPU core binding technology based on the RoCE network are implemented through the following steps to execute the communication transmission tasks between the high-performance computing platforms:

[0052] The RoCE network is configured in unreliable datagram transmission mode, and each messaging interface process is bound to the local RoCE network interface port of the node in the non-consistent memory access architecture. When communication occurs between the messaging interface processes, the messaging interface process at the sending end obtains the data to be sent from the local memory of the bound node in the non-consistent memory access architecture, encapsulates the data to be sent into a RoCE unreliable datagram packet through the local RoCE network interface port, and sends it. The messaging interface process at the receiving end receives the data to be sent through the local RoCE network interface port of the bound node in the non-consistent memory access architecture, generates received data, and stores the received data in the local memory of the bound node in the non-consistent memory access architecture.

[0053] Even if RoCE resolves the inter-node communication bottleneck, resource scheduling issues within a single computing node can still become new performance constraints. Modern servers (such as nodes equipped with the high-performance computing platform 920 processor) typically employ a multi-NUMA (Non-Uniform Memory Access) architecture. Taking a typical 96-core high-performance computing platform server as an example, it contains multiple NUMA nodes, each integrating several CPU cores, sharing an L3 cache, and directly connecting to local memory channels. If the operating system does not impose affinity constraints on processes / threads, it will lead to loss of cache locality, increased cross-NUMA memory access, and intensified resource contention. Thread migration between different cores can cause frequent L1 / L2 cache invalidations; processes running on NUMA 0 frequently accessing NUMA 1 memory, increasing access latency by 2–3 times; multiple MPI (Message Passing Interface) processes or OpenMP threads competing for the same physical core can trigger context switching overhead. These problems are particularly pronounced in hybrid parallel (MPI + OpenMP) models, directly impacting the computational efficiency of a single node.

[0054] CPU affinity refers to the technique of scheduling processes or threads to a specific CPU core or set of cores. Its core objective is to improve the locality of computational tasks and reduce performance fluctuations caused by scheduling uncertainty. Through affinity, threads can continuously use the private cache of the same core, avoiding cache thrashing; processes can reside on the same NUMA node as their allocated memory, ensuring the locality of memory access; simultaneously, task migration is prohibited, significantly reducing context switching overhead.

[0055] Based on the NUMA topology of the high-performance computing platform, a refined core binding scheme was developed. Each MPI process was bound to a specific NUMA node, and its memory allocation was restricted to local use, ensuring that computation and memory access were completed locally. At the same time, OpenMP threads were bound, preventing thread migration and ensuring that each thread had exclusive access to a physical core, avoiding resource contention (OpenMPI can use the -bind-to core parameter, and other MPIs can use it in conjunction with the numactl command).

[0056] DevKit's NUMA analysis module can be used to monitor the cross-NUMA memory access ratio, L3 cache hit rate, and CPU utilization balance of each core, thus verifying the core binding effect.

[0057] Core binding not only improves single-node computing efficiency but also indirectly optimizes communication performance. Stable CPU affinity allows RoCE network card interrupts (IRQs) to be bound to dedicated cores, avoiding resource contention with computing threads. Simultaneously, the stable scheduling of MPI communication threads enables more timely processing of RoCE events, reducing communication latency jitter. Therefore, core binding provides a stable computing environment foundation for the efficient operation of RoCE.

[0058] In specific implementation, the transparent huge page function in the operating system of the high-performance computing platform is enabled through the following steps. Based on the transparent huge page function, memory allocation requests initiated by applications of the operating system and requests for accessing memory through virtual addresses by applications of the operating system are processed:

[0059] The transparent large page function is enabled in the operating system of the high-performance computing platform, and the mode of the transparent large page is set. When the application of the operating system initiates a memory allocation request, a fixed-size dedicated memory region is determined, and the dedicated memory region is used as the requested memory region. Adjacent regular memory pages are detected, and multiple regular memory pages are merged into a single transparent large page. A direct mapping relationship between the application's virtual address space and the transparent large page is established. When the application of the operating system accesses memory through a virtual address, the memory management unit performs address translation, looks up the physical address through the page table buffer entry of the transparent large page, and accesses the contiguous physical memory region corresponding to the transparent large page.

[0060] Specifically, direct mapping reduces intermediate page table levels. Using the transparent big page's page table buffer entries to look up physical addresses significantly improves TLB hit rate, reduces page table traversal counts, lowers address translation overhead, and quickly obtains physical addresses. Contiguous physical memory regions ensure efficient prefetching, improving cache line filling efficiency.

[0061] High-performance computing applications typically handle extremely large contiguous memory blocks (such as grid data structures with hundreds of millions of records). Linux uses 4KB pages for memory management by default, a mechanism that faces significant bottlenecks in large memory scenarios. First, the number of TLB (Translation Lookaside Buffer) entries is limited (usually several thousand). A 4KB page requires a large number of TLB entries to cover a large memory region, and frequent TLB misses trigger multi-level page table traversals, significantly increasing memory access latency. Second, x86_64 and ARM64 architectures use four-level page tables, resulting in a complex address translation process and significant overhead. Third, page tables themselves occupy a large amount of memory space; for example, 1TB of memory requires approximately 256MB of page tables, wasting valuable memory resources.

[0062] Transparent Large Pages (THP) automatically merges adjacent 4KB pages into 2MB large pages (1GB supported on some platforms). Its advantages include a 512x increase in TLB coverage, significantly reduced TLB misses, fewer page table levels, accelerated virtual-to-physical address translation, reduced page table memory footprint, and increased effective memory bandwidth. THP is transparent to applications, requires no source code modification, and is enabled via kernel parameters.

[0063] This invention enables THP mode (set to always mode) on a high-performance computing platform to maximize memory access efficiency. It's important to note that THP merging relies on contiguous physical memory; long-term operation may lead to memory fragmentation, causing merging failure. Therefore, it is recommended to use a defragmentation strategy (set to always mode) to perform memory defragmentation when necessary. Furthermore, THP is suitable for scenarios involving large, contiguous memory allocations (such as HPC and scientific computing), but may be counterproductive for applications that frequently allocate / release small amounts of memory (such as certain databases). However, CAE applications are highly compatible with THP, and real-world testing shows it can significantly improve memory access performance.

[0064] THP, core binding, and RoCE work in close coordination: core binding ensures that process memory allocation resides on the local NUMA node, improving the availability of local contiguous physical memory and thus increasing the success rate of THP merging; THP's enhanced memory bandwidth and low latency provide strong support for RoCE's high-speed DMA (Direct Memory Access) data transfer; and RoCE's low CPU overhead frees up more computing resources for THP's background memory consolidation. Together, these three components construct a high-bandwidth, low-latency, and highly locale memory subsystem, providing a solid foundation for parallel computing.

[0065] In specific implementation, the following steps are used to generate pre-processed data by executing data loading, format conversion, and initialization tasks in preprocessing before the main computing task of the high-performance computing platform begins:

[0066] A preprocessing thread group independent of the main computing thread pool is created; data loading tasks, format conversion tasks, and initialization computing tasks are executed in parallel through the preprocessing thread group; each preprocessing thread in the preprocessing thread group loads input data from the storage device into the memory buffer through parallel I / O operations; the loaded input data is format-normalized and structurally reorganized in the memory buffer to generate preprocessed data blocks.

[0067] In specific implementation, the following steps are used to enable open multiprocessing threads, execute the main computing task through multiple open multiprocessing threads, and start the next round of preprocessing tasks in parallel:

[0068] Based on the CPU core topology of the high-performance computing platform and the non-uniform memory access architecture, the total number of threads for open multiprocessing is determined; the main computing thread acquires the pre-processed data block through a lock-free synchronization mechanism and decomposes the pre-processed data block into multiple computing subtasks; the total number of threads for open multiprocessing is started, and the computing subtasks are executed in parallel by the multiple threads for open multiprocessing, and the data in memory is accessed through the nodes of the non-uniform memory access architecture; while the current task of the main computing thread is being executed, the next round of pre-processing tasks is started in parallel.

[0069] Specifically, a double buffering mechanism is used to achieve time overlap between data preprocessing and main computation, and to enable the preprocessing thread group and the main computation thread group to form a continuous execution pipeline in the time dimension.

[0070] The embodiments of the present invention achieve deep adaptation between applications and high-performance computing platforms in the following ways:

[0071] First, enable multi-threaded preprocessing. Many CAE software programs use single-threaded processing by default for preprocessing tasks such as mesh interpolation and boundary condition setting, failing to take advantage of the multi-core capabilities of high-performance computing platforms. Enabling parallel preprocessing options can significantly reduce the preparation phase time.

[0072] Secondly, optimize the OpenMP thread configuration. Dynamically set the number of threads based on the number of physical cores to avoid thread overload or resource idleness.

[0073] Application layer adaptation is the final manifestation of the entire optimization chain. RoCE solves the communication bottleneck between nodes, core binding and THP optimize the computing and memory subsystems within nodes, and application parameter adaptation ensures that upper-layer computing tasks can fully utilize the hardware resources optimized at the lower level. These four elements form a full-stack collaborative optimization system of "communication-scheduling-memory-application," none of which can be omitted.

[0074] The key technologies in the embodiments of this invention constitute a hierarchical and mutually reinforcing optimization system:

[0075] RoCE high-speed network serves as the foundation of the communication layer, reducing MPI communication latency from milliseconds to microseconds, eliminating the MPI_Wait bottleneck, and ensuring efficient collaboration among nodes.

[0076] Core binding strategy, as a scheduling layer optimization, improves cache hit rate, ensures NUMA locality, and maximizes single-node computing efficiency by fixing processes / threads to specific cores;

[0077] Transparent big pages are used as an optimization of the memory management layer to reduce TLB misses and page table overhead, providing high-bandwidth, low-latency memory access support for RoCE's high-speed DMA.

[0078] Application parameter adaptation, as an application-layer optimization, transforms the theoretical peak computing power of Kunpeng hardware into actual application performance by enabling multi-threading, link optimization libraries, and compiler vectorization.

[0079] There is a clear dependency and enhancement relationship among the four components: RoCE's efficient operation relies on a low-latency, high-bandwidth memory subsystem, and THP is key to improving memory performance; THP's merging efficiency depends on the NUMA locality and contiguous memory provided by the core binding; the stability of the core binding ensures that the MPI communication thread can handle RoCE events in a timely manner; and finally, the effects of all underlying optimizations need to be fully realized through application-layer adaptation. This closed-loop collaborative mechanism enables the system to achieve near-linear multi-node parallel speedup in complex scenarios such as CAE simulations with hundreds of millions of grid nodes.

[0080] This invention also includes a performance acquisition middleware designed and implemented specifically for domestic platforms by combining domestic software and hardware, targeting large-scale parallel computing scenarios. This middleware utilizes the performance analysis module provided by Huawei's self-developed DevKit performance analysis tool for servers based on high-performance computing platforms. During software development, the performance acquisition middleware module is introduced in advance, and during software runtime, the performance acquisition range can be dynamically adjusted through a unified configuration file.

[0081] The approach combines online and offline data collection and analysis. During online data collection, the use of domestically produced hardware and self-developed basic software (mathematical libraries, compilers, parallel libraries, and solvers) minimizes the consumption of CPU, memory, network, and disk I / O resources. Offline analysis combines large-scale models specific to the AI ​​field, providing precise optimization suggestions based on professional knowledge.

[0082] By integrating with a domestically developed Linux operating system, a kernel-level performance acquisition module is deployed to collect operational characteristics of system calls such as CPU resources, memory access, network communication, and disk I / O during program execution. Combined with a unified data acquisition service, it efficiently and meticulously collects performance data during software operation.

[0083] The main analysis modules include: software hotspot functions, key metrics for OpenMP (Open Multiprocessing) and MPI applications, serial and parallel times for Parallel Regions and Barrier-to-Barrier, memory access analysis, and file I / O analysis.

[0084] The program displays basic runtime information, including execution time, CPU utilization, and CPI, helping customers accurately evaluate the efficiency of program runtime resource utilization. MPI (Message Passing Interface) Wait Rate is an indicator that evaluates the waiting ratio of process communication interfaces, providing a detailed analysis of the blocking situation of program communication interfaces, including communication efficiency, interface ratio, and I / O status.

[0085] Hotspot functions have always been the most effective way to analyze program bottlenecks. However, in MPI scenarios with numerous processes, analysis becomes difficult. DevKit collects hotspots from all processes and summarizes the statistics, making it more suitable for hotspot analysis in MPI scenarios.

[0086] IO Analysis: Provides data collection for basic POSIX and other read / write interfaces, including call efficiency, call stack, and file read / write data. In clusters, storage is typically implemented over a network; analyzing IO issues can reveal network bottlenecks.

[0087] MPI Communication Status: Collect data on the runtime status of the MPI interface in the program, analyze the call efficiency and data volume during the interface call process. Interface issues are the key to MPI issues. Solve communication blockage, adjust task balance, and effectively improve running efficiency.

[0088] The purpose of refined analysis is primarily to construct graphical analysis of runtime programs, providing a global perspective for analyzing MPI execution, and enabling comprehensive analysis from the global to the detailed:

[0089] a) Heatmap: Creates icons to show the access status between ranks, displaying the communication data volume from the horizontal axis rank to the vertical axis rank. Colors are used to mark the access levels to form hot zones. Hot zones are analyzed in detail to solve the communication imbalance problem.

[0090] b) Sequence diagram: All communication interface calls of all ranks are expanded along the same timeline, and color blocks are used to mark the communication area and the computing area. From a global perspective, the communication balance can be analyzed, and long-blocked communication can be quickly identified. Each communication block marks the interface, communication volume, communication object, and call stack. Micro-architectural indicators and hotspots are collected and analyzed in the computing block, which allows for more detailed and in-depth analysis of the problem.

[0091] Existing performance analysis tools for cluster computing run independently (e.g., hpctoolkit). While these tools are highly versatile, they dynamically collect performance data during software runtime. However, this approach can lead to discrepancies between the performance bottlenecks identified and the actual performance bottlenecks encountered during software operation. Furthermore, the data collection process itself can alter the software's efficiency, resulting in the identification of incorrect bottlenecks. In large-scale job scenarios, there are no effective performance tools to aid in performance analysis.

[0092] The method invented in this paper, by combining with domestically produced hardware, reduces the consumption of computing resources such as CPU, memory, network, and disk I / O, can restore the actual performance characteristics of the software, find the precise performance bottleneck, reduce false alarms, and support parallel computing acceleration optimization of E-level clusters.

[0093] Finally, the performance tuning tool Devkit, developed in China, was used to present the results in a graphical interface. Here, we can see the timeline of all processes, and different colors indicate the actual computation time and communication time used by each process.

[0094] In this embodiment, a computer device is provided, such as... Figure 2 As shown, it includes a memory 201, a processor 202, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned multi-node parallel acceleration methods based on high-performance computing platforms and RoCE networks.

[0095] Specifically, the computer device can be a computer terminal, a server, or a similar computing device.

[0096] In this embodiment, a computer-readable storage medium is provided, which stores a computer program that executes any of the above-described multi-node parallel acceleration methods based on a high-performance computing platform and RoCE network.

[0097] Specifically, computer-readable storage media include both permanent and non-permanent, removable and non-removable media, which can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable storage media do not include transient media, such as modulated data signals and carrier waves.

[0098] Based on the same inventive concept, this invention also provides a multi-node parallel acceleration device based on a high-performance computing platform and RoCE network, as described in the following embodiments. Since the principle of the multi-node parallel acceleration device based on a high-performance computing platform and RoCE network is similar to that of the multi-node parallel acceleration method based on a high-performance computing platform and RoCE network, the implementation of the multi-node parallel acceleration device based on a high-performance computing platform and RoCE network can refer to the implementation of the multi-node parallel acceleration method based on a high-performance computing platform and RoCE network; repeated details will not be elaborated further. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0099] Figure 3 This is a structural block diagram of a multi-node parallel acceleration device based on a high-performance computing platform and RoCE network according to an embodiment of the present invention, such as... Figure 3 As shown, it includes: network deployment module 301, core binding policy configuration module 302, transparent big page enable module 303, and architecture adaptation module 304. The structure is described below.

[0100] The network deployment module 301 is used to build a communication stack based on the RoCE network, and to build an interconnected transmission network between the sending end and the receiving end of the cluster nodes based on the communication stack.

[0101] The core-binding strategy configuration module 302 is used to execute the computing tasks of the high-performance computing platform through the CPU core-binding technology and the non-uniform memory access architecture of the CPU, and to execute the communication transmission tasks between the high-performance computing platforms based on the communication stack, the non-uniform memory access architecture and the CPU core-binding technology of the RoCE network.

[0102] Enable the transparent big page module 303 to enable the transparent big page function in the operating system of the high-performance computing platform. Based on the transparent big page function, process the memory allocation requests initiated by the application of the operating system and the requests of the application of the operating system to access memory through virtual addresses.

[0103] The architecture adaptation module 304 is used to execute data loading tasks, format conversion tasks, and initialization calculation tasks in preprocessing using multiple threads before the main computing task of the high-performance computing platform starts, generate preprocessed data, and start open multiprocessing threads to execute the main computing task and start the next round of preprocessing tasks in parallel.

[0104] In one embodiment, the network deployment module includes:

[0105] A communication abstraction layer construction unit is used to construct a high-performance communication abstraction layer and select an underlying transmission protocol through the high-performance communication abstraction layer, wherein the underlying transmission protocol is UD mode;

[0106] A communication construction stack unit is used to construct a communication stack including an application layer, an RDMA transport protocol layer, a network layer, and a data link layer, and to insert the high-performance communication abstraction layer between the application layer and the RDMA transport protocol layer.

[0107] The data storage unit is used to store the data to be transmitted from the transmitting end into the RDMA memory area of ​​the transmitting end, and to construct a UD data packet using the data to be transmitted.

[0108] The first data sending unit is used to submit the UD data packet to the sending queue of the RoCE network, and encapsulate the UD data packet into a network data frame supporting the RDMA transmission protocol through the RDMA transmission protocol layer of the communication stack, wherein the network data frame includes a UDP header;

[0109] The data storage unit is used to send the network data frame to the receiving end through the RoCE network, verify and extract the UD data packet of the network data frame, and store the UD data packet in the RDMA memory area of ​​the receiving end.

[0110] In one embodiment, the core binding strategy configuration module includes:

[0111] The topology analysis unit is used to analyze the topology of the CPU's non-uniform memory access architecture and determine the CPU core set and local memory region corresponding to each node of the non-uniform memory access architecture.

[0112] The mapping table construction unit is used to establish a mapping table between message passing interface processes and nodes of the non-consistent memory access architecture. The mapping table is used to map each message passing interface process to a node of the non-consistent memory access architecture.

[0113] A binding unit is used to start a cluster of message passing interface processes and, according to the mapping table, fix each message passing interface process to a specific CPU core of a node in a target non-uniform memory access architecture using CPU affinity binding technology.

[0114] The allocation strategy configuration unit is used to configure the memory allocation strategy for each of the messaging interface processes, so that the messaging interface processes use the local physical memory of the nodes bound to the non-consistent memory access architecture.

[0115] The task execution unit is used to execute the computing tasks of the high-performance computing platform based on the specific CPU core and the memory allocation strategy.

[0116] In one embodiment, the core binding strategy configuration module further includes:

[0117] The node port binding unit is used to set the RoCE network to unreliable datagram transmission mode and bind each message passing interface process to the local RoCE network card port of the node with the non-consistent memory access architecture.

[0118] The data acquisition unit is used to, when communication is performed between the message passing interface processes, acquire the data to be sent from the local memory of the node bound to the non-consistent memory access architecture, encapsulate the data to be sent into a RoCE unreliable data packet through the local RoCE network card port and send it.

[0119] The second data sending unit is used for the message passing interface process of the receiving end to receive the data to be sent through the local RoCE network card port of the node bound to the non-consistent memory access architecture, generate received data, and store the received data in the local memory of the node bound to the non-consistent memory access architecture.

[0120] In one embodiment, enabling the transparent large page module includes:

[0121] Enable Transparent Large Page Unit, used to enable the transparent large page function in the operating system of the high-performance computing platform and set the mode of the transparent large page;

[0122] The mapping relationship construction unit is used to determine a fixed-size dedicated memory region when the application of the operating system initiates a memory allocation request, use the dedicated memory region as the requested memory region, detect adjacent regular memory pages, merge multiple regular memory pages into a single transparent large page, and establish a direct mapping relationship between the application's virtual address space and the transparent large page.

[0123] The memory access unit is used to perform address translation when the application of the operating system accesses memory through a virtual address. The memory management unit then looks up the physical address through the page table buffer entry of the transparent big page and accesses the contiguous physical memory region corresponding to the transparent big page.

[0124] In one embodiment, the architecture adaptation module includes:

[0125] Create a thread group unit to create a preprocessing thread group that is independent of the main compute thread pool;

[0126] The parallel execution unit is used to execute data loading tasks, format conversion tasks, and initialization calculation tasks in parallel through the preprocessing thread group;

[0127] A data loading unit is used for each preprocessing thread in the preprocessing thread group to load input data from the storage device into a memory buffer through parallel I / O operations.

[0128] The structure reorganization unit is used to perform format standardization and structure reorganization on the loaded input data in the memory buffer to generate a preprocessed data block.

[0129] In one embodiment, the architecture adaptation module further includes:

[0130] The total number of threads determination unit is used to determine the total number of threads in open multiprocessing based on the CPU core topology of the high-performance computing platform and the non-uniform memory access architecture.

[0131] The subtask demarcation unit is used by the main computing thread to obtain the pre-processed data block through a lock-free synchronization mechanism and decompose the pre-processed data block into multiple computing subtasks.

[0132] The execution subtask unit is used to start the total number of threads of the open multiprocessing, execute the computing subtask in parallel through the multiple threads of the open multiprocessing, and access the data in memory through the nodes of the non-consistent memory access architecture;

[0133] The synchronous startup unit is used to start the next round of preprocessing tasks in parallel while the current main computing thread is executing its task.

[0134] The embodiments of the present invention achieve the following technical effects:

[0135] By implementing RoCE high-speed network, CPU core binding, transparent big pages, and application parameter adaptation on a high-performance computing platform, a multi-node parallel acceleration framework for CAE simulations involving hundreds of millions of devices was constructed. This framework fully leverages the multi-core, high-bandwidth, and NUMA architecture characteristics of the high-performance computing platform's processors, combined with the low-latency communication capabilities of the RoCE network, to achieve full-stack performance optimization from the hardware layer to the application layer. Simultaneously, it possesses good reproducibility and portability. The optimization effect can be quantitatively evaluated using the DevKit performance analysis tool, guiding further tuning. The acceleration method of this invention is not only applicable to the CAE field but can also be extended to other large-scale parallel computing scenarios such as climate simulation, bioinformatics, and financial modeling, providing a general technical path for the efficient operation of domestically developed HPC software on independent hardware platforms.

[0136] Obviously, those skilled in the art should understand that the modules or steps of the above-described embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.

[0137] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-node parallel acceleration method based on a high-performance computing platform and RoCE network, characterized in that, include: Construct a communication stack based on the RoCE network, and build an interconnected transmission network between the sending and receiving ends of the cluster nodes based on the communication stack; The computing tasks of the high-performance computing platform are executed through the CPU core binding technology and the non-uniform memory access architecture of the CPU. The communication transmission tasks between the high-performance computing platforms are executed based on the communication stack, the non-uniform memory access architecture and the CPU core binding technology of the RoCE network. Enable the transparent big page function in the operating system of the high-performance computing platform, and based on the transparent big page function, process the memory allocation requests initiated by the applications of the operating system and the requests of the applications of the operating system to access memory through virtual addresses; Before the main computing task of the high-performance computing platform begins, the data loading task, format conversion task and initialization computing task in the preprocessing are executed by multiple threads to generate preprocessed data, and open multiprocessing threads are started. The main computing task is executed by multiple open multiprocessing threads, and the next round of preprocessing tasks is started in parallel.

2. The multi-node parallel acceleration method based on a high-performance computing platform and RoCE network as described in claim 1, characterized in that, Construct a communication stack based on the RoCE network, and build an interconnection transmission network between cluster nodes from sender to receiver based on the communication stack, including: A high-performance communication abstraction layer is constructed, and an underlying transmission protocol is selected through the high-performance communication abstraction layer, wherein the underlying transmission protocol is UD mode; A communication stack including an application layer, an RDMA transport protocol layer, a network layer, and a data link layer is constructed, and the high-performance communication abstraction layer is inserted between the application layer and the RDMA transport protocol layer; The data to be transmitted at the transmitting end is stored in the RDMA memory area of ​​the transmitting end, and a UD data packet is constructed using the data to be transmitted; The UD data packet is submitted to the transmission queue of the RoCE network, and the UD data packet is encapsulated into a network data frame supporting the RDMA transmission protocol through the RDMA transmission protocol layer of the communication stack, wherein the network data frame includes a UDP header; The network data frame is sent to the receiving end through the RoCE network, the UD data packet of the network data frame is verified and extracted, and the UD data packet is stored in the RDMA memory area of ​​the receiving end.

3. The multi-node parallel acceleration method based on a high-performance computing platform and RoCE network as described in claim 1, characterized in that, Executing computational tasks on the high-performance computing platform using CPU core-binding technology and the CPU's non-uniform memory access architecture includes: Analyze the topology of the CPU's non-uniform memory access architecture to determine the CPU core set and local memory region corresponding to each node of the non-uniform memory access architecture; Establish a mapping table between message passing interface processes and nodes of the non-consistent memory access architecture. The mapping table is used to correspond one-to-one between each message passing interface process and a node of the non-consistent memory access architecture. Start the message passing interface process cluster, and fix each message passing interface process to a specific CPU core of the node of the target non-uniform memory access architecture through CPU affinity binding technology according to the mapping table; Configure a memory allocation strategy for each of the messaging interface processes so that the messaging interface processes use the local physical memory of the nodes bound to the non-consistent memory access architecture. The computing tasks of the high-performance computing platform are executed based on the specific CPU core and the memory allocation strategy.

4. The multi-node parallel acceleration method based on a high-performance computing platform and RoCE network as described in claim 1, characterized in that, Based on the communication stack, the non-uniform memory access architecture, and the CPU core binding technology of the RoCE network, the communication transmission tasks between the high-performance computing platforms are executed, including: The RoCE network is set to unreliable datagram transmission mode, and each message passing interface process is bound to the local RoCE network card port of the node in the non-consistent memory access architecture. When communication is performed between the message passing interface processes, the message passing interface process of the sending end obtains the data to be sent from the local memory of the node bound to the non-consistent memory access architecture, encapsulates the data to be sent into a RoCE unreliable data packet through the local RoCE network card port and sends it. The message passing interface process of the receiving end receives the data to be sent through the local RoCE network card port of the node bound to the non-consistent memory access architecture, generates received data, and stores the received data in the local memory of the node bound to the non-consistent memory access architecture.

5. The multi-node parallel acceleration method based on a high-performance computing platform and RoCE network as described in claim 1, characterized in that, Enable the transparent huge page function in the operating system of the high-performance computing platform. Based on the transparent huge page function, process memory allocation requests initiated by applications of the operating system and requests for memory access via virtual addresses by applications of the operating system, including: Enable the transparent large page function in the operating system of the high-performance computing platform and set the mode of the transparent large page; When an application in the operating system initiates a memory allocation request, a fixed-size dedicated memory region is determined, the dedicated memory region is used as the requested memory region, adjacent regular memory pages are detected, multiple regular memory pages are merged into a single transparent large page, and a direct mapping relationship between the application's virtual address space and the transparent large page is established. When an application in the operating system accesses memory via a virtual address, the memory management unit performs address translation, looks up the physical address through the page table buffer entry of the transparent big page, and accesses the contiguous physical memory region corresponding to the transparent big page.

6. The multi-node parallel acceleration method based on a high-performance computing platform and RoCE network as described in any one of claims 1 to 5, characterized in that, Before the main computing task of the high-performance computing platform begins, multi-threaded preprocessing tasks such as data loading, format conversion, and initialization are executed to generate preprocessed data, including: Create a preprocessing thread group that is independent of the main compute thread pool; The preprocessing thread group executes data loading tasks, format conversion tasks, and initialization calculation tasks in parallel. Each preprocessing thread in the preprocessing thread group loads input data from the storage device into the memory buffer through parallel I / O operations; The loaded input data is formatted and restructured in the memory buffer to generate a preprocessed data block.

7. The multi-node parallel acceleration method based on a high-performance computing platform and RoCE network as described in any one of claims 1 to 5, characterized in that, Enable open multiprocessing threads, and execute the main computing task through multiple open multiprocessing threads, while simultaneously launching the next round of preprocessing tasks, including: Based on the CPU core topology of the high-performance computing platform and the non-uniform memory access architecture, determine the total number of threads for open multiprocessing; The main computing thread acquires the pre-processed data block through a lock-free synchronization mechanism and decomposes the pre-processed data block into multiple computing subtasks. The total number of open multiprocessing threads are started, and the computational subtasks are executed in parallel by multiple open multiprocessing threads, and the data in memory is accessed through the nodes of the non-consistent memory access architecture; While the current main computing thread is executing its task, the next round of preprocessing tasks is started in parallel.

8. A multi-node parallel acceleration device based on a high-performance computing platform and RoCE network, characterized in that, include: The network deployment module is used to build a communication stack based on the RoCE network, and to build an interconnected transmission network between the sending and receiving ends of the cluster nodes based on the communication stack. The core-binding strategy configuration module is used to execute the computing tasks of the high-performance computing platform through the CPU core-binding technology and the non-uniform memory access architecture of the CPU, and to execute the communication transmission tasks between the high-performance computing platforms based on the communication stack, the non-uniform memory access architecture and the CPU core-binding technology of the RoCE network. Enable the transparent big page module to enable the transparent big page function in the operating system of the high-performance computing platform. Based on the transparent big page function, process the memory allocation requests initiated by the application of the operating system and the requests of the application of the operating system to access memory through virtual addresses. The architecture adaptation module is used to execute data loading, format conversion and initialization tasks in preprocessing using multi-threading before the main computing task of the high-performance computing platform begins, generate preprocessed data, and start open multiprocessing threads to execute the main computing task and start the next round of preprocessing tasks in parallel.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multi-node parallel acceleration method based on a high-performance computing platform and RoCE network as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that performs the multi-node parallel acceleration method based on a high-performance computing platform and RoCE network as described in any one of claims 1 to 7.

Images