A parallel fluid simulation synchronization method based on adaptive compression

Abstract

The application belongs to the technical field of fluid simulation, and particularly relates to a parallel fluid simulation synchronization method based on adaptive compression; the method comprises the following steps: storing boundary data generated by simulation into a buffer and switching the buffer state; calculating compression benefits and making compression decisions according to the compression benefits; sending data according to the compression decisions; receiving data and analyzing the data to obtain complete boundary data; and applying the complete boundary data to a calculation domain when the data is received and analyzed completely; the method realizes efficient overlap of the calculation, compression and communication processes, significantly reduces simulation delay, improves system resource utilization, and has good application prospect.

Description

Technical Field

[0001] This invention belongs to the field of fluid simulation technology, specifically relating to a parallel fluid simulation synchronization method based on adaptive compression. Background Technology

[0002] In large-scale parallel simulations, the computational domain is divided into multiple subdomains and assigned to different MPI processes. After completing the computation of a time step, each process must exchange data in the boundary region with the process holding the adjacent subdomain; this process is called "boundary exchange" or "halo exchange." This process introduces synchronization points between processes and is one of the key bottlenecks leading to a decrease in parallel efficiency.

[0003] In early technologies, researchers primarily employed two optimization strategies to reduce the communication costs associated with boundary switching: one was "data compression," which involved lossless or lossy compression of boundary data before transmission to reduce the amount of data transmitted over the network; the other was "non-blocking communication," such as MPI_Isend and MPI_Irecv, to achieve overlap between communication and computation. However, most existing data compression strategies are based on fixed heuristics, such as always compressing or setting thresholds based on experience to determine whether to compress, which cannot adapt to dynamic changes in data distribution, network congestion, CPU load, and other factors during operation. When the compression overhead exceeds the transmission time saved, blind compression not only fails to reduce latency but also increases the overall time cost. Furthermore, traditional non-blocking communication can only hide latency during network transmission, while compression and decompression, as computationally intensive operations, are still executed serially by the main thread, forcing the main computation to pause before compression or decompression is complete.

[0004] As parallelism, simulation accuracy, and solver complexity continue to increase, the scale of boundary data grows significantly, and the dynamics of the operating environment become more unpredictable, including significant fluctuations in data entropy with the evolution of the physical field, unstable network bandwidth between nodes, and intensified competition for CPU and memory resources. Against this backdrop, traditional fixed strategies and partially overlapping mechanisms can no longer meet the demands for efficient synchronization. In real-world systems, compression gains exhibit strong volatility, with significant differences in compressibility between different time steps and even different subdomains. Therefore, an adaptive mechanism is needed that can autonomously determine "whether to compress" based on real-time performance metrics. Simultaneously, to further reduce synchronization waiting costs and fully utilize the spatiotemporal continuity of fluid data (such as the positions and velocities of adjacent particles) to improve compression efficiency, it is necessary to compare the particle position and velocity data with the corresponding data from the previous time step in the cache, generating differential data. This involves separating the compression / decompression process, including differential processing, from the main computation path, achieving true "computation-communication overlap."

[0005] In summary, there is an urgent need for a novel parallel fluid simulation synchronization method that can automatically make decisions based on the operational status and maximize the hiding of communication and compression overhead, in order to break through the performance bottleneck of the traditional MPI halo exchange. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a parallel fluid simulation synchronization method based on adaptive compression, which includes:

[0007] S1: Store the boundary data generated by the simulation into the buffer and switch the buffer state;

[0008] S2: Calculate the compression benefits and make compression decisions based on the compression benefits;

[0009] S3: Send data based on compression decisions;

[0010] S4: Receive and parse the data to obtain complete boundary data;

[0011] S5: Once the data has been received and parsed, apply the complete boundary data to the computation domain.

[0012] Preferably, step S1 specifically includes:

[0013] S11: The main computing process retrieves the transmit buffers with the status marked as free from multiple pre-allocated independent buffer pools according to the category of boundary data;

[0014] S12: Write the current class boundary data generated by the current simulation time step into the corresponding send buffer;

[0015] S13: Update the status flag of the transmit buffer to ready;

[0016] S14: Return to step S12 to execute the next type of boundary data storage; until all types of boundary data have been marked as ready.

[0017] Preferably, step S2 specifically includes:

[0018] S21: Use an independent asynchronous worker thread to randomly extract a data subsample from the ready buffer; compress the subsample to obtain the sample compression ratio and sample compression time;

[0019] S22: By linearly extrapolating the compression ratio and compression time of the subsamples, the compressed size and compression time of the entire boundary data are predicted.

[0020] S23: Calculate the total expected compression delay based on the compressed size and compression time of the boundary data, the currently measured effective network bandwidth, and the decompression task time at the receiving end; calculate the original total expected delay based on the original size of the boundary data, the currently effective network bandwidth, and the additional network protocol delay.

[0021] S24: If the original expected total delay is less than the compressed expected total delay, the decision is to use the compressed mode; otherwise, the decision is to use the original mode.

[0022] Furthermore, the formula for calculating the total expected delay of the compression is as follows:

[0023]

[0024]

[0025] in, This indicates the expected total delay after compression. Indicates the compression time of boundary data. Indicates the transmission time of the compressed boundary data. This indicates the compressed size of the boundary data. Indicates the currently available network bandwidth. This indicates the time taken for the decompression task at the receiving end.

[0026] Furthermore, the formula for calculating the original expected total delay is:

[0027]

[0028]

[0029] in, This represents the original expected total delay. Indicates the transmission time of the original data. Indicates the original size of the boundary data. Indicates the currently available network bandwidth. This indicates additional latency in the network protocol.

[0030] Preferably, step S3 specifically includes:

[0031] S31: If the compression decision is compression mode, the asynchronous worker thread performs compression operation on the boundary data; if the compression decision is original mode, no compression operation is performed; the data field data is obtained.

[0032] S32: Construct a message header, including the data pattern identifier, the original data size, and the compressed data size; and encapsulate the message header and the data field data to obtain a complete message unit;

[0033] S33: Use a non-blocking message passing interface to send the complete message unit to the target process and update the status flag of the buffer to "sending".

[0034] Preferably, step S4 specifically includes:

[0035] The receiving process obtains messages through a non-blocking receive interface and parses the message header to determine the data pattern;

[0036] If the data mode is compressed, the message payload is submitted to a dedicated restore and decompression thread pool, triggering an asynchronous decompression task;

[0037] When the asynchronous decompression task is completed, or when the data is in its original state, the decompressed data or the original data is marked as ready for use by the main computing process.

[0038] Preferably, step S5 specifically includes

[0039] When the main computing process at the receiving end reaches the synchronization point that depends on this boundary data, the application already has the boundary data in a ready state.

[0040] The asynchronous worker thread on the sending end checks the completion status of the non-blocking send operation to confirm whether the data has been sent.

[0041] Once the sending end confirms that the transmission is complete, it resets the corresponding send buffer status flag to idle and returns it to the buffer pool for reuse.

[0042] The beneficial effects of this invention are as follows:

[0043] 1. Maximizing resource utilization and overlap efficiency: By asynchronously and parallelizing the computation, compression, communication, and decompression operations of multiple types of data, this invention greatly hides communication latency and compression computation overhead. The main computing process is almost unblocked, and computing resources, network bandwidth, and compression computation resources can be used efficiently simultaneously, achieving extremely high system throughput.

[0044] 2. Intelligent Adaptive Decision Making: The compression decision-making mechanism based on a latency model proposed in this invention can intelligently select the optimal communication strategy for each type of data according to the real-time changing data characteristics and system state. This avoids the performance loss of fixed compression strategies in unsuitable scenarios and ensures the robustness and efficiency of the method under diverse simulation environments and network conditions.

[0045] 3. Fine-grained parallel processing: By establishing independent buffers and processing flows for each type of data, this invention achieves finer-grained parallelism than traditional batch processing. Various types of data can be "computed immediately upon completion and sent immediately upon readiness," forming a highly efficient pipeline and further compressing the overall completion time of a single time step.

[0046] 4. Significantly Improved Simulation Performance: Combining the above advantages, this invention effectively reduces the synchronization waiting time introduced by boundary exchanges in parallel fluid simulations, thereby significantly shortening the wall clock time for each simulation time step. For large-scale, long-running scientific simulations, this will bring a qualitative leap in overall computational efficiency, accelerating the process of scientific discovery and engineering applications. Attached Figure Description

[0047] Figure 1 This is a flowchart of the parallel fluid simulation synchronization method based on adaptive compression in this invention;

[0048] Figure 2 This is a system architecture diagram of the parallel fluid simulation synchronization method based on adaptive compression in this invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] This invention proposes a parallel fluid simulation synchronization method based on adaptive compression, aiming to address the problem that traditional fixed compression strategies cannot minimize end-to-end latency in dynamic simulation environments. This method aims to provide an intelligent adaptive approach and system. Its core objective is to effectively reduce the network transmission time of boundary data through dynamic decision-making and asynchronous execution, and to overlap the computational overhead of compression and decompression with network communication and other computations as much as possible, thereby significantly reducing the end-to-end latency of a single simulation time step and improving overall simulation efficiency. Figure 1 , Figure 2 As shown, the method includes the following:

[0051] S1: Store the boundary data generated by the simulation into the buffer and switch the buffer state.

[0052] First, multiple types of boundary data are prepared and buffers are switched: After the main computation of the current simulation time step is completed, the generated multiple types of boundary data (including but not limited to particle positions, physical field densities and velocities, and forces) are stored in their corresponding independent transmission buffers, and these buffers are immediately marked as pending. Then, the main computation process enters the next type of data computation, and enters a waiting state after the computation is completed until enough boundary data is received; the specific processing includes the following steps:

[0053] S11: The main computing process retrieves the transmit buffers with the status marked as free from multiple pre-allocated independent buffer pools according to their categories;

[0054] S12: Write the current class boundary data generated by the current simulation time step into the corresponding send buffer;

[0055] S13: Update the status flag of the transmit buffer to ready;

[0056] S14: Return to step S12 to execute the next type of boundary data storage; until all types of boundary data have been marked as ready.

[0057] S2: Calculate compression benefits and make compression decisions based on compression benefits.

[0058] Next, asynchronous compression benefit estimation and decision-making are performed: an independent asynchronous task processing thread takes over the buffer to be processed and performs a fast compression benefit estimation on the boundary data within it. Based on the estimation result and the current network bandwidth, a dynamic decision is made on whether to use compressed mode or original mode for this transmission. Specifically, the following processing steps are included:

[0059] S21. Lightweight Sampling Estimation: An independent asynchronous worker thread randomly extracts a data subsample from the ready buffer and compresses it. If it is position and velocity data, differential processing is required before compression to obtain the sample compression ratio and sample compression time; specifically:

[0060] An asynchronous task thread randomly selects a data subsample from the boundary data to be processed, typically choosing a small data block (a random fragment of 4KB to 64KB) as a sample for compression performance. The sample data is compressed using the LZ4 compression algorithm, and the size of the compressed data is recorded. If the data is location or velocity, differential processing is required before compression. The compression ratio (the ratio of the compressed data size to the original data size) and compression time (the time required to complete the compression of the sample) are quickly calculated. The compression ratio is calculated using the following formula:

[0061]

[0062] The above process avoids the computational and time overhead of fully compressing the entire dataset. Through this lightweight estimation, a preliminary value representing compression efficiency and time consumption can be obtained quickly.

[0063] S22. Global performance prediction: By linear extrapolating the sample compression ratio and sample compression time, predict the compressed size and compression time of the entire boundary data;

[0064] After obtaining the sample compression ratio and compression time, the asynchronous thread then uses a linear extrapolation method to predict the compression performance of the boundary data based on the sample performance. Based on the sample compression ratio and the size of the entire boundary data, it predicts the compressed size of the entire dataset. Assuming the sample compression ratio is... The predicted compressed size of the entire boundary data is then... It can be calculated as follows:

[0065]

[0066] in, It is the original size of the boundary data to be processed. Indicates the compressed size of the sample. This is the predicted compressed size.

[0067] Then, based on the compression time of the samples and the size of the data, the asynchronous task thread predicts the total time required to compress the entire boundary data by proportional extrapolation. The time taken to compress the sample is... , Given the sample data size, the total data compression time is... It can be predicted that:

[0068]

[0069] Through this process, the asynchronous task thread can estimate the time required to compress the complete boundary data and the size of the compressed data, providing data support for subsequent decision-making.

[0070] The computational model is expressed as follows:

[0071]

[0072]

[0073]

[0074]

[0075] in, This represents the compression time at the sender. The network transmission time representing compressed data. This represents the decompression time that cannot be hidden on the receiving end. and These are the predicted compression and decompression times, respectively. and These are the estimated overlap time windows for the transmitter and receiver, respectively. The effective network bandwidth is currently measured. S23. Latency Modeling: Asynchronous task threads need to consider network bandwidth and the overlap time window with the main computation, i.e., the parallelism of communication and computation. For each mode (compression mode and original mode), the expected total latency needs to be calculated, specifically the expected total latency for sending original data, i.e., the expected total latency for compression. And the expected total delay for sending compressed data, i.e., the original expected total delay. Specifically:

[0076] The expected total compression latency is calculated based on the compressed size and compression time of the boundary data, the currently measured effective network bandwidth, and the decompression task time at the receiving end:

[0077]

[0078]

[0079] in, This indicates the expected total delay after compression. Indicates the compression time of boundary data. Indicates the transmission time of the compressed boundary data. This indicates the compressed size of the boundary data. Indicates the currently available network bandwidth; The time taken for the decompression task at the receiving end varies depending on the amount of data. and The estimation was obtained by lightweight sampling and linear extrapolation based on relevant data from the sample.

[0080] Calculate the original expected total latency based on the original size of the boundary data, the current available network bandwidth, and the additional latency of the network protocol:

[0081]

[0082]

[0083] in, This represents the original expected total delay. Indicates the transmission time of the original data. Indicates the original size of the boundary data. Indicates the currently available network bandwidth; This indicates additional latency in network protocols, meaning that the overhead of network protocols (such as message headers, receipt confirmations, etc.) may cause additional delays.

[0084] S24: If the original expected total delay is less than the compressed expected total delay, the decision is to use the compressed mode; otherwise, the decision is to use the original mode.

[0085] like < If the condition is met, the decision is in compressed mode; otherwise, the decision is in original mode.

[0086] S3: Send data based on compression decisions.

[0087] < If the compression process is successful, the compressed data will be sent. Otherwise, if the original data transmission mode is superior, compression will be skipped and the original data will be sent directly. The specific asynchronous data preparation and sending process is as follows:

[0088] S31. Data Preparation: The asynchronous worker thread performs the corresponding data preparation operation according to the decision result of S2: if the decision is compression, the boundary data is compressed; if it is position and velocity data, differential processing is required before compression; if the decision is raw, the compression operation is skipped.

[0089] S32. Message Encapsulation: Construct a message header, which includes a data pattern identifier, the original data size, and the compressed data size; encapsulate the message header and data fields into a complete message unit.

[0090] S33. Asynchronous Sending: Send the encapsulated message unit to the target process through a non-blocking message passing interface, and update the status flag of the buffer to "sending".

[0091] S4: Receive and parse the data to obtain complete boundary data.

[0092] Asynchronous reception and data processing: The receiving process listens for and receives messages through a non-blocking interface. After parsing the metadata, if the data is compressed, a decompression task for that type of data is immediately started asynchronously. If it is position and velocity data, differential restoration is required before decompression. This decompression task is executed concurrently with the main computation of the receiving process and other computational tasks. Specifically:

[0093] The receiving process obtains messages through a non-blocking receive interface and parses the message header to determine the data pattern;

[0094] If the data mode is compressed, the message payload is submitted to a dedicated restore and decompression thread pool, triggering an asynchronous decompression task;

[0095] When the asynchronous decompression task is completed, or when the data is in its original state, the decompressed data or the original data is marked as ready for use by the main computing process.

[0096] S5: Once the data has been received and parsed, apply the complete boundary data to the computation domain.

[0097] Boundary data application: When the main computation process reaches a synchronization point that depends on this complete boundary data, it confirms that all categories of boundary data have been received and processed, and then applies all ready boundary data to the computation domain. Specifically:

[0098] When the main computing process at the receiving end reaches the synchronization point that depends on this boundary data, the application already has the boundary data in a ready state.

[0099] The asynchronous worker thread on the sending end checks the completion status of the non-blocking send operation to confirm whether the data has been sent.

[0100] Once the sending end confirms that the transmission is complete, it resets the corresponding send buffer status flag to idle and returns it to the buffer pool for reuse.

[0101] The present invention also designs a feedback learning mechanism to record the actual performance data of historical decisions and dynamically update the parameters used for global performance prediction and latency modeling based on the actual performance, so as to adaptively simulate the dynamic characteristics of the application.

[0102] Simulation verification of the present invention:

[0103] The present invention was simulated and compared with the uncompressed method. The comparison of various indicators is shown in Table 1 and Table 2.

[0104] Table 1. Comparison of simulation times for DamBreak3D at different resolutions under twisted-pair cabling conditions.

[0105]

[0106] Table 2. Comparison of simulation times for DamBreak3D at different resolutions in an optical network environment.

[0107]

[0108] As can be seen from Tables 1 and 2, the adaptive compression strategy of this invention outperforms the uncompressed scheme in both network environments. Specifically, the performance improvement is particularly significant in bandwidth-constrained twisted-pair environments; while in high-speed optical networks, although the compression gain narrows with increasing particle size, thanks to the intelligent decision-making of the method, its performance remains comparable to or better than the benchmark scheme, ensuring the robustness of the system.

[0109] In summary, this invention addresses the communication bottleneck caused by boundary data exchange in parallel fluid simulation by designing an adaptive compression-based parallel fluid simulation synchronization method. This method achieves efficient overlap of computation, compression, and communication processes, significantly reduces simulation latency, and improves system resource utilization, demonstrating promising application prospects.

[0110] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A parallel fluid simulation synchronization method based on adaptive compression, characterized in that, Includes the following steps: S1: Store the boundary data generated by the simulation into the buffer and switch the buffer state; S2: Calculate the compression benefits and make compression decisions based on the compression benefits; S3: Send data based on compression decisions; S4: Receive and parse the data to obtain complete boundary data; S5: Once the data has been received and parsed, apply the complete boundary data to the computation domain.

2. The parallel fluid simulation synchronization method based on adaptive compression according to claim 1, characterized in that, Step S1 specifically includes: S11: The main computing process retrieves the transmit buffers with the status marked as free from multiple pre-allocated independent buffer pools according to the category of boundary data; S12: Write the current class boundary data generated by the current simulation time step into the corresponding send buffer; S13: Update the status flag of the transmit buffer to ready; S14: Return to step S12 to execute the next type of boundary data storage; until all types of boundary data have been marked as ready.

3. The parallel fluid simulation synchronization method based on adaptive compression according to claim 1, characterized in that, Step S2 specifically includes: S21: Use an independent asynchronous worker thread to randomly extract a data subsample from the ready buffer; compress the subsample to obtain the sample compression ratio and sample compression time; S22: By linearly extrapolating the compression ratio and compression time of the subsamples, the compressed size and compression time of the entire boundary data are predicted. S23: Calculate the total expected compression delay based on the compressed size and compression time of the boundary data, the currently measured effective network bandwidth, and the decompression task time at the receiving end; calculate the original total expected delay based on the original size of the boundary data, the currently effective network bandwidth, and the additional network protocol delay. S24: If the original expected total delay is less than the compressed expected total delay, the decision is to use the compressed mode; otherwise, the decision is to use the original mode.

4. The parallel fluid simulation synchronization method based on adaptive compression according to claim 3, characterized in that, The formula for calculating the total expected compression delay is: ； ； in, This indicates the expected total delay after compression. Indicates the compression time of boundary data. Indicates the transmission time of the compressed boundary data. This indicates the compressed size of the boundary data. Indicates the currently available network bandwidth. This indicates the time taken for the decompression task at the receiving end.

5. The parallel fluid simulation synchronization method based on adaptive compression according to claim 3, characterized in that, The formula for calculating the original expected total delay is: ； ； in, This represents the original expected total delay. Indicates the transmission time of the original data. Indicates the original size of the boundary data. Indicates the currently available network bandwidth. This indicates additional latency in the network protocol.

6. The parallel fluid simulation synchronization method based on adaptive compression according to claim 1, characterized in that, Step S3 specifically includes: S31: If the compression decision is compression mode, the asynchronous worker thread performs compression operation on the boundary data; if the compression decision is original mode, no compression operation is performed; the data field data is obtained. S32: Construct a message header, including the data pattern identifier, the original data size, and the compressed data size; and encapsulate the message header and the data field data to obtain a complete message unit; S33: Use a non-blocking message passing interface to send the complete message unit to the target process and update the status flag of the buffer to "sending".

7. The parallel fluid simulation synchronization method based on adaptive compression according to claim 1, characterized in that, Step S4 specifically includes: The receiving process obtains messages through a non-blocking receive interface and parses the message header to determine the data pattern; If the data mode is compressed, the message payload is submitted to a dedicated restore and decompression thread pool, triggering an asynchronous decompression task; When the asynchronous decompression task is completed, or when the data is in its original state, the decompressed data or the original data is marked as ready for use by the main computing process.

8. The parallel fluid simulation synchronization method based on adaptive compression according to claim 1, characterized in that, Step S5 specifically includes When the main computing process at the receiving end reaches the synchronization point that depends on this boundary data, the application already has the boundary data in a ready state. The asynchronous worker thread on the sending end checks the completion status of the non-blocking send operation to confirm whether the data has been sent. Once the sending end confirms that the transmission is complete, it resets the corresponding send buffer status flag to idle and returns it to the buffer pool for reuse.

Images