A cross-platform distributed synchronous simulation method using RoCEv2

Abstract

The application discloses a cross-platform distributed synchronous simulation method using RoCEv2 and relates to the technical field of communication simulation, and the method comprises the following steps: configuring switch and node parameter information, and constructing a heterogeneous distributed network of a RoCEv2 enhanced network protocol; deploying a global logical time reference of a master node, calibrating a slave node clock, starting a sliding window to continuously monitor the offset, dynamically detecting the clock offset to trigger adjustment, combining ECN marking to optimize the time step, and dynamically compensating the clock offset of the heterogeneous platform; generating a DAG topology according to the logical timestamp, and preloading high-frequency events; transmitting event data across nodes by RDMA, scheduling by the master node according to the logical timestamp, and realizing cross-platform distributed parallel simulation and consistency checking. The application can improve the processing efficiency and fault tolerance of a communication simulation system for large-scale simulation and can be widely applied to the technical field of communication simulation.

Description

Technical Field

[0001] The present invention relates to the field of communication simulation technology, and in particular to a cross-platform distributed synchronous simulation method using RoCEv2. Background Art

[0002] With the prevalence of heterogeneous computing architectures, such as containerization (Docker / Kubernetes) and hybrid deployments of virtual machines and physical machines, cross-platform collaborative simulation faces the dual challenges of communication efficiency and time synchronization. First, industrial-grade simulations (such as digital twins and traffic simulations) require processing petabytes of discrete event data. Traditional TCP / IP protocol stacks, due to multiple memory copies, result in end-to-end latency of hundreds of microseconds and excessive CPU (central processing unit) resource utilization, making them difficult to meet real-time requirements. Second, physical clock skew across heterogeneous platforms (including lightweight container clocks, virtual machine clocks, and physical machine hardware clocks) causes inconsistent simulation logic time. Existing synchronization mechanisms, such as NTP / PTP, lack dynamic adaptability in cross-platform scenarios and cannot effectively suppress clock jitter at the virtualization layer. Furthermore, existing communication technologies have significant limitations. RoCEv1 only supports intra-subnet communication, and RoCEv2 (Remote Direct Memory Access over Ethernet), while routable, relies on lossless network guarantees. Overall, the system lacks in-depth optimization for heterogeneous protocol compatibility, further hindering the performance and reliability of large-scale simulation systems. Summary of the Invention

[0003] In order to solve the above technical problems, the purpose of the present invention is to provide a cross-platform distributed synchronous simulation method using RoCEv2, which can improve the communication efficiency of cross-platform collaborative simulation, save CPU resources, improve the accuracy of time synchronization, and adapt to more complex communication simulation scenario requirements.

[0004] The present invention adopts a cross-platform distributed synchronous simulation method using RoCEv2, comprising the following steps:

[0005] Configure switch and node parameters to build a heterogeneous distributed network based on the Enhanced RoCEv2 (E-RoCE) protocol.

[0006] Deploy a global logical time base on the master node, calibrate the slave node clocks, enable a sliding window to continuously monitor offsets, dynamically detect clock offsets and trigger adjustments, optimize the time step size in conjunction with ECN marking, and dynamically compensate for clock offsets on heterogeneous platforms.

[0007] Generate DAG topology by sorting events by logical timestamps and preload high-frequency events;

[0008] Cross-node RDMA transmission of event data, the master node sorting and scheduling according to logical timestamps, to achieve cross-platform distributed parallel simulation and consistency verification.

[0009] Further, the step of configuring the switch and node parameter information, and constructing a heterogeneous distributed network of an enhanced RoCEv2 network protocol (E-RoCE) specifically includes:

[0010] Configuring PFC flow control priority at the switch end, allocating the highest bandwidth queue for simulation data, and setting ECN explicit congestion marking threshold;

[0011] Enabling the RoCEv2 protocol stack at the node end, binding the UDP port 4791, and configuring IP routing rules to support IPv4 / IPv6 multi-path forwarding;

[0012] Deploying the GPU Direct RDMA driver, establishing the physical address mapping between the GPU memory and the network card DMA engine, and realizing zero-copy data transmission;

[0013] Configuring the network card hardware offload RDMA semantics (Send / Write operation) to bypass the kernel protocol stack and directly read and write user-mode memory;

[0014] Configuring a multi-path load balancing strategy based on source port hashing algorithm.

[0015] Further, the step of deploying a global logical time reference for the master node, calibrating the slave node clock, starting a sliding window to continuously monitor the offset, dynamically detecting clock offset triggering adjustment, optimizing the time step combined with ECN marking, and dynamically compensating for the clock offset of the heterogeneous platform specifically includes:

[0016] The master node periodically broadcasts Sync messages, and the slave node records the local receiving time T2 and feeds back Delay_Resp messages;

[0017] Calculating the initial offset by the clock deviation formula, and predicting the drift trend by fitting the historical data in the sliding window using the least squares method;

[0018] Dynamically integrating NTP coarse-grained calibration and PTP fine-grained compensation, and setting the monitoring window size to filter transient jitter;

[0019] Real-time monitoring of the clock offset rate in the sliding window, if the deviation exceeds the dynamic threshold, triggering local clock rollback or acceleration;

[0020] Analyzing the ECN marking of network packets, and reducing the time advancing step by a certain proportion when the congestion level reaches a preset level.

[0021] Further, the step of sorting events by logical timestamp to generate a DAG topology, and preloading high-frequency events specifically includes:

[0022] Building a logical timestamp directed acyclic graph (DAG) based on event causality, and generating an event execution sequence through a topological sorting algorithm;

[0023] Analyze historical simulation data, identify high-frequency event types, and preload them into the node's local cache;

[0024] Furthermore, the cross-node RDMA transmission event data, the master node sorting and scheduling according to the logical timestamp, realizes the cross-platform distributed parallel simulation and consistency verification step, which specifically includes:

[0025] The slave node writes local event data to the master node's memory pool through RDMA Write operations, along with a logical timestamp and dependency tag.

[0026] The master node performs multi-way sorting of the event queue based on timestamps to generate a global ordered execution sequence;

[0027] When a causal conflict is detected, roll back to the last conflict-free checkpoint and re-execute;

[0028] At the end of the simulation cycle, the hash summary of the result data of each node is calculated. If there is any inconsistency between the comparisons, the backup queue replay verification is triggered. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a step flow chart of a cross-platform distributed synchronous simulation method using RoCEv2 of the present invention.

[0030] Figure 2 This is a schematic diagram of the framework of a cross-platform distributed synchronous simulation method using RoCEv2 in the present invention.

[0031] Figure 3 This is a logic diagram of a dynamic adaptive time synchronization protocol using a cross-platform distributed synchronization simulation method of RoCEv2 in the present invention. DETAILED DESCRIPTION

[0032] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are provided for ease of description only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted based on the understanding of those skilled in the art.

[0033] Reference Figure 1 The present invention provides a cross-platform distributed synchronous simulation method using RoCEv2, the method comprising the following steps:

[0034] S100, configure switch and node parameter information to build a heterogeneous distributed network based on the Enhanced RoCEv2 (E-RoCE) protocol;

[0035] Specifically, the PFC flow control priority is configured at the switch end, the highest bandwidth queue is allocated for the simulation data, the ECN explicit congestion marking threshold is set, the RoCEv2 protocol stack is enabled at the node end, the UDP port 4791 is bound, the IP routing rule is configured to support IPv4 / IPv6 multi-path forwarding, the GPU Direct RDMA driver is deployed, the physical address mapping between the GPU display memory and the network card DMA engine is established, the zero-copy data transmission is implemented, the network card hardware offload RDMA semantics (Send / Write operation) is configured to bypass the kernel protocol stack to directly read and write the user mode memory, and the multi-path load balancing strategy is configured based on the source port hash algorithm.

[0036] In the embodiment, an RoCEv2 enhanced network protocol (E-RoCE) heterogeneous distributed network is constructed to implement zero-copy data transmission.

[0037] S200, a global logical time reference of a master node is deployed, a slave node clock is calibrated, a sliding window is started to continuously monitor the offset, the clock offset is dynamically detected to trigger adjustment, the time step is optimized in combination with the ECN marking, and the clock offset of the heterogeneous platform is dynamically compensated.

[0038] Specifically, the master node broadcasts a Sync message, the slave node records the local receiving time, and feeds back a Delay_Resp message; the initial offset is calculated by a clock deviation formula, the historical data in the sliding window is fitted by the least square method to predict the drift trend; the NTP coarse-grained calibration and the PTP fine-grained compensation are dynamically fused; the clock offset rate in the sliding window is monitored in real time, and if the deviation exceeds the dynamic threshold, the local clock is rolled back or accelerated; the network packet ECN marking is analyzed, and when the congestion level reaches a preset level, the time advancing step is reduced in proportion to realize dynamic adaptive time synchronization. As shown in Figure 3 The logical diagram of the dynamic adaptive time synchronization protocol includes an initialization stage and a runtime stage.

[0039] S210, dynamic adaptive time synchronization initialization stage;

[0040] Specifically, the master node broadcasts a Sync message, the slave node monitors the data packet in the data link, analyzes the Sync message, feeds back a Delay_Resp message, and calculates the initial offset. The Sync message carries a T1 logical timestamp, and is forwarded with the highest priority in the switch. The slave node records the receiving Sync message event timestamp T2, and generates a Delay_Resp message containing the message processing completion time T3 and the response message sending time T4. The initial offset is obtained based on the two-way time exchange protocol: Δ =((T2 - T1) - (T4 - T3)) / 2.

[0041] S220, dynamic adaptive time synchronization runtime stage;

[0042] Specifically, the last 10 Δ values are saved in the sliding window, and a linear model is fitted using the weighted least squares method: where is the frequency drift rate, is the initial offset, is a random error term. The least squares method determines the parameters by minimizing the sum of squares of residuals, as shown in the following formula:

[0043] ,

[0044] ,

[0045] where, , , represents the number of Δ, represents time, is and the covariance of is the variance of. The drift rate in the sliding window is monitored in real time, and if the drift rate exceeds the dynamic threshold, the local clock is rolled back or accelerated. At the same time, the system clock is aligned periodically (every 10s) through the NTP protocol, and large-scale drift (>1ms) is compensated. In each synchronization period, the clock frequency is fine-tuned through the PTP hardware clock adjustment interface.

[0046] In the receiving end network card, the ECN field (2 bits) in the IPv4 / IPv6 header is parsed to identify congestion indication. If the ECN field is 11, it indicates that the switch buffer exceeds the threshold. The proportion of data packets with ECN field 11 in unit time needs to be counted and mapped to 4 levels , as shown in the following formula:

[0047] ,

[0048] According to the congestion level, the time advancement step is dynamically scaled , as shown in the following formula:

[0049] .

[0050] S300, sort events by logical timestamp to generate DAG topology, and preload high-frequency events;

[0051] Specifically, a logical timestamp directed acyclic graph (DAG) is constructed based on event causality, and an event execution sequence is generated through a topological sorting algorithm; historical simulation data is analyzed to identify high-frequency event types and preloaded to the local cache of the node;

[0052] S400, cross-node RDMA transmission event data, the master node is scheduled according to logical timestamp ordering, realizing cross-platform distributed parallel simulation and consistency check.

[0053] Specifically, the slave node writes local event data into the master node memory pool through the RDMA Write operation, and the logical timestamp and dependency label are attached; the master node performs multi-way ordering on the event queue based on the timestamp, generates a globally ordered execution sequence; when a causal conflict is detected, it is rolled back to the last conflict-free checkpoint for re-execution; at the end of the simulation period, the hash digest of the result data of each node is calculated, and when the inconsistency is found, the backup queue replay verification is triggered.

[0054] To sum up, the embodiment of the application first constructs a heterogeneous distributed network of RoCEv2 enhanced network protocol (E-RoCE), enables the RoCEv2 protocol stack at the node end, binds the UDP port 4791, configures the IP routing rule to support IPv4 / IPv6 multi-path forwarding. Further configure parameter information, configure PFC flow control priority at the switch end, allocate the highest bandwidth queue for simulation data, set the ECN explicit congestion marking threshold, deploy the GPU Direct RDMA driver, establish the physical address mapping between the GPU video memory and the network card DMA engine. Further dynamic adaptive time synchronization, deploy the master node global logical time reference, the master node broadcasts Sync message, the slave node monitors the data packet in the data link, calculates the initial offset to calibrate the slave node clock. Start the sliding window to continuously monitor the offset, dynamically detect the clock offset to trigger adjustment, if the offset rate exceeds the dynamic threshold, trigger the local clock rollback or acceleration, optimize the time step combined with the ECN mark, dynamically scale the time step according to the congestion level, dynamically compensate the clock offset of the heterogeneous platform. Finally, the simulation results of each distributed node are transmitted through RDMA, and the master node is scheduled according to the logical timestamp ordering and consistency verification. That is, as shown in Figure 2 As shown, RoCEv2 enhanced network protocol (E-RoCE) is used, IPv4 / IPv6 routing is supported, and standard Ethernet switches are compatible; GPU Direct RDMA technology is used to realize direct connection between GPU video memory and network card DMA, and the network card directly processes RDMA semantics (Send / Write), bypassing the kernel protocol stack. This method combines the characteristics of RDMA, such as reducing CPU utilization and saving resources, while using dynamic adaptive time synchronization protocol to ensure synchronization of distributed parallel simulation, improving the response speed and processing capacity of the system for large-scale simulation.

[0055] The above is a specific description of the preferred embodiment of the application, but the application is not limited to the described embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A cross-platform distributed synchronous simulation method using RoCEv2, characterized in that: The following steps are involved: Configure switch and node parameters to build a heterogeneous distributed network based on the Enhanced RoCEv2 (E-RoCE) protocol. Deploy a global logical time base on the master node, calibrate the slave node clocks, enable a sliding window to continuously monitor offsets, dynamically detect clock offsets and trigger adjustments, optimize the time step size using explicit congestion notification (ECN) marking, and dynamically compensate for clock offsets on heterogeneous platforms. Sort events by logical timestamp to generate a directed acyclic graph (DAG) topology and preload high-frequency events; Cross-node remote direct memory access (RDMA) transmits event data, and the master node schedules it by logical timestamp, enabling cross-platform distributed parallel simulation and consistency verification. The step of configuring switch and node parameter information to build a heterogeneous distributed network of the RoCEv2 enhanced network protocol E-RoCE specifically includes: Configure priority-based flow control (PFC) on the switch, assign the highest bandwidth queue to the simulation data, and set the ECN explicit congestion marking threshold. Enable the RoCEv2 protocol stack on the node, bind the User Datagram Protocol UDP port 4791, and configure IP routing rules to support IPv4 / IPv6 multipath forwarding; Deploy the GPU Direct RDMA driver to establish physical address mapping between the graphics processing unit (GPU) memory and the network card's direct memory access (DMA) engine, enabling zero-copy data transfer. Configure the network card hardware to offload RDMA technology, which transfers tasks from the computer's processor to the network card, bypassing the kernel protocol stack and directly reading and writing user-mode memory; Configure multipath load balancing policy based on source port hashing algorithm; The steps of deploying a global logical time base on the master node, calibrating the slave node clocks, starting a sliding window to continuously monitor offsets, dynamically detecting clock offsets to trigger adjustments, optimizing the time step in combination with ECN marking, and dynamically compensating for clock offsets on heterogeneous platforms specifically include: The master node periodically broadcasts the synchronization Sync message, and the slave node records the local reception time T2 and feeds back the delayed response Delay_Resp message; The initial offset is calculated using the clock deviation formula, and the drift trend is predicted by fitting historical data within the sliding window using the least squares method. Dynamically integrates Network Time Protocol (NTP) coarse-grained calibration and high-precision time synchronization (PTP) fine-grained compensation, and sets the monitoring window size to filter instantaneous jitter. Real-time monitoring of the clock deviation rate within the sliding window. If the deviation exceeds the dynamic threshold, the local clock is triggered to dial back or accelerate. Parse the ECN marking of network data packets and proportionally reduce the time advancement step when the congestion level reaches the preset level; The step of generating a DAG topology by sorting events by logical timestamps and preloading high-frequency events specifically includes: Construct a logical timestamp directed acyclic graph (DAG) based on event causality, and generate event execution sequences using a topological sorting algorithm; Analyze historical simulation data, identify high-frequency event types, and preload them into the node's local cache; The cross-node RDMA transmission event data, the master node sorting and scheduling according to the logical timestamp, and implementing the cross-platform distributed parallel simulation and consistency verification step specifically include: The slave node writes local event data to the master node's memory pool through RDMA Write operations, along with a logical timestamp and dependency tag. The master node performs multi-way sorting of the event queue based on timestamps to generate a global ordered execution sequence; When a causal conflict is detected, the process rolls back to the last conflict-free checkpoint and re-executes. At the end of the simulation cycle, a hash summary is calculated for the result data of each node. If there is any inconsistency, the backup queue replay verification is triggered.

Images