Data compression method, device and equipment suitable for mechanical simulation system and medium

Abstract

The application discloses a data compression method, device and equipment suitable for a mechanical simulation system and a medium, relates to the field of computational mechanics, and the mechanical simulation system is a computer system comprising at least two computing nodes. The method comprises the following steps: determining transient field data generated by the at least two computing nodes when performing parallel computation of the mechanical simulation; field variables in the transient field data comprise non-topological field variables and topological field variables; performing data integration based on the continuity physical characteristics of the field variables in the spatial dimension and the time dimension to obtain two-dimensional integrated data; performing lossy compression on the two-dimensional integrated data corresponding to the non-topological field variables to obtain lossy compressed data, and simultaneously performing lossless compression on the two-dimensional integrated data corresponding to the topological field variables to obtain lossless compressed data. Thus, the data integration is implemented by using the continuity physical characteristics of the field variables in the spatial and time dimensions, and the lossy / lossless hybrid parallel compression is combined, so that the compression rate can be improved, and the compression time consumption can be reduced.

Description

Technical Field

[0001] This application relates to the field of computational mechanics, and in particular to a data compression method, apparatus, device, and medium suitable for mechanical simulation systems. Background Technology

[0002] Computational mechanics based on unstructured meshes has been widely applied to numerical simulations of practical industrial problems in aerospace, automotive engineering, and other fields. With the continuous improvement of computational model precision, the amount of field data generated by computation has increased dramatically, especially in transient problems where a single case can generate terabytes (TB) of data, placing enormous pressure on disk storage. Data compression is an effective means of reducing disk consumption. Traditional data compression methods (such as general-purpose compression software) typically compress the raw data directly, ignoring the physical characteristics of similar values ​​in adjacent spatial locations and smooth changes in values ​​between adjacent time steps, resulting in low compression ratios.

[0003] It is evident that improving compression ratio and reducing compression time are problems that need to be solved in this field. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a data compression method, apparatus, device, and medium suitable for mechanical simulation systems. This method utilizes the continuous physical characteristics of field variables in the spatial and temporal dimensions to integrate data, and combines lossy / lossless hybrid parallel compression to improve compression efficiency and reduce compression time. The specific solution is as follows: In a first aspect, this application provides a data compression method suitable for a mechanics simulation system, wherein the mechanics simulation system is a computer system including at least two computing nodes, and the method includes: Determine the transient field data generated when the at least two computing nodes perform a computational mechanics simulation in parallel; the field variables in the transient field data include non-topological field variables and topological field variables; Based on the continuous physical characteristics of the field variables in the transient field data in the spatial and temporal dimensions, the non-topological field variables and the topological field variables are integrated to obtain two-dimensional integrated data; wherein, the data integration includes: sorting the corresponding field variable data in the spatial dimension using a space filling algorithm that prioritizes adjacent positions, and sorting the corresponding field variable data in the temporal dimension according to the time order; The two-dimensional integrated data corresponding to the non-topological field variables are subjected to lossy compression by the at least two computing nodes to obtain lossy compressed data, and the two-dimensional integrated data corresponding to the topological field variables are subjected to lossless compression to obtain lossless compressed data.

[0005] Optionally, the step of sorting the corresponding field variable data using a spatial filling algorithm prioritizing adjacent positions in the spatial dimension includes: The Hilbert space-filling curve is used to perform spatial traversal of the field variable data for the non-topological field variables and the topological field variables, and the corresponding field variable data are sorted based on the traversal order.

[0006] Optionally, sorting the corresponding field variable data in chronological order along the time dimension includes: Based on the ascending order of time, the field variable data of the non-topological field variables and the topological field variables are sorted.

[0007] Optionally, after obtaining the lossy compressed data and the lossless compressed data, the process further includes: Under the root node of the target disk file, child nodes are created for the at least two computing nodes respectively, and under each child node, a field variable data node is created for each field variable in the transient field data; The lossy compressed data and the lossless compressed data are written into the corresponding field variable data nodes.

[0008] Optionally, after sorting the corresponding field variable data using a spatial filling algorithm prioritizing adjacent positions in the spatial dimension, the method further includes: Determine the mapping relationship data corresponding to the field variable data before and after sorting; the mapping relationship data is the data generated during the process of sorting the corresponding field variable data by using a space filling algorithm that prioritizes adjacent positions in the spatial dimension; The mapping relationship data is losslessly compressed so that the corresponding compressed data can be written to the target disk file.

[0009] Optionally, writing the corresponding compressed data to the target disk file includes: A mapping relationship data node is created under each of the child nodes, and the corresponding compressed data is written into the mapping relationship data node.

[0010] Optionally, the method further includes: The decompressed data is verified based on the first data length in the field variable data node and the second data length in the mapping relationship data node. Wherein, the first data length includes the data length of the target field variable data before and after compression, the second data length includes the data length of the mapping relationship data corresponding to the target field variable data before and after compression, and the decompressed data is the data obtained by decompressing the compressed target field variable data.

[0011] Secondly, this application provides a data compression device suitable for a mechanics simulation system, wherein the mechanics simulation system is a computer system including at least two computing nodes, and the device includes: A transient field data determination module is used to determine the transient field data generated by the at least two computing nodes when performing computational mechanics simulations in parallel; the field variables in the transient field data include non-topological field variables and topological field variables; The data integration module is used to integrate the non-topological field variables and the topological field variables respectively based on the continuous physical characteristics of the field variables in the transient field data in the spatial and temporal dimensions to obtain two-dimensional integrated data; wherein, the data integration includes: sorting the corresponding field variable data in the spatial dimension using a space filling algorithm that prioritizes adjacent positions, and sorting the corresponding field variable data in the temporal dimension according to the temporal order; The data compression module is used to perform lossy compression on the two-dimensional integrated data corresponding to the non-topological field variables through the at least two computing nodes to obtain lossy compressed data, and at the same time to perform lossless compression on the two-dimensional integrated data corresponding to the topological field variables to obtain lossless compressed data.

[0012] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor for executing the computer program to implement the data compression method described above for a mechanical simulation system.

[0013] Fourthly, this application provides a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the data compression method applicable to a mechanical simulation system as described above.

[0014] Therefore, the technical solution of this application is applied to a computer system including at least two computing nodes for performing computational mechanics simulations. First, the transient field data generated by the at least two computing nodes during parallel execution of the computational mechanics simulation is determined. The field variables in the transient field data include non-topological field variables and topological field variables. Then, based on the continuous physical characteristics of the field variables in the transient field data in the spatial and temporal dimensions, the non-topological field variables and the topological field variables are integrated to obtain two-dimensional integrated data. The data integration includes: sorting the corresponding field variable data in the spatial dimension using a space-filling algorithm prioritizing adjacent positions, and sorting the corresponding field variable data in the temporal dimension according to chronological order. Then, the at least two computing nodes perform lossy compression on the two-dimensional integrated data corresponding to the non-topological field variables to obtain lossy compressed data, and simultaneously perform lossless compression on the two-dimensional integrated data corresponding to the topological field variables to obtain lossless compressed data. In this way, this application utilizes the continuous physical characteristics of field variables in the spatial and temporal dimensions to integrate data, which can reduce the information entropy of adjacent data. Combined with lossy / lossless hybrid parallel compression, it can significantly improve the compression rate while ensuring the high accuracy of topological data, saving disk storage space and reducing compression time. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0016] Figure 1 This is a flowchart of a data compression method applicable to a mechanical simulation system disclosed in this application; Figure 2 This application discloses a specific data verification flowchart; Figure 3 This application discloses a specific data compression method applicable to mechanical simulation systems, which is shown in the flowchart below. Figure 4 This application discloses a contour map of the distribution of a pressure field variable on a model surface. Figure 5 This application discloses a data length distribution map of a field variable after compression. Figure 6 This is a schematic diagram of a data compression device suitable for a mechanical simulation system disclosed in this application; Figure 7 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] This application provides a data compression method suitable for mechanical simulation systems, applicable to large-scale parallel computational mechanics scenarios based on unstructured meshes, such as numerical simulation of aerodynamic characteristics of aerospace vehicles, aerodynamic optimization of automobile shapes, hydrodynamic analysis of ships, and internal flow simulation of turbomachinery. In these scenarios, the generated unstructured transient field data is enormous, often reaching terabytes (TB) in a single case. Traditional compression methods that directly compress the raw data suffer from low compression ratios and long processing times. This application aims to significantly improve the compression ratio and speed while maintaining high accuracy of the topology data by preprocessing and integrating the field variable data based on physical continuity characteristics in both spatial and temporal dimensions, combining a hybrid lossy and lossless compression strategy, and executing the compression in parallel on multiple computational nodes.

[0019] The methods provided in this application can be applied to parallel computing devices in the field of computational mechanics, including but not limited to aerospace CFD (Computational Fluid Dynamics) simulation systems, automotive industry numerical wind tunnel systems, ship hydrodynamics calculation platforms, and energy and power equipment thermal-fluid coupling simulation systems. They can also be applied to other high-performance computing systems based on unstructured grids that generate large-scale transient field data. This application does not impose any specific limitations on these applications.

[0020] See Figure 1 As shown in the figure, this application discloses a data compression method suitable for a mechanics simulation system, wherein the mechanics simulation system is a computer system including at least two computing nodes, and the method includes: Step S11: Determine the transient field data generated by the at least two computing nodes when performing computational mechanics simulations in parallel; the field variables in the transient field data include non-topological field variables and topological field variables.

[0021] In this application, before initiating the compression process, transient field data to be compressed needs to be generated using a parallel computer system. Specifically, firstly, a job scheduling system (such as Slurm, PBS, or LSF) based on the parallel computer system starts N computing nodes, and parallel computational mechanics simulation software (such as NNW-FlowStar, OpenFOAM, Fluent, etc.) runs on each computing node. The job scheduling system is responsible for allocating the required computing resources, such as the number of CPU (Central Processing Unit) cores and runtime, to the computational tasks, and distributing the executable program and input files of the computational mechanics simulation software to each computing node.

[0022] After each computing node starts up, it loads the computational grid data and initial boundary conditions, initializes the runtime environment, and creates a shared memory data pool in its local memory. Then, the computational mechanics simulation software performs parallel transient calculations on the N computing nodes: each computing node solves the governing equations (such as the Navier-Stokes equations, Euler equations, etc.) at each time step based on its assigned grid sub-region, calculating the numerical distribution of field variables on each grid node. After the calculation is complete, each computing node stores the calculated field data into its shared memory data pool.

[0023] Each computation node's field data includes M field variables. Based on the purpose and accuracy requirements of these variables, the M variables are divided into two categories: the first K variables are non-topological field variables (i.e., lossy compressible field variables), such as density, velocity components in three directions (x-direction velocity, y-direction velocity, z-direction velocity), pressure, laminar viscosity coefficient, and turbulent viscosity coefficient—variables describing the physical state of the flow, typically stored in 64-bit double-precision floating-point format; the latter MK variables are topological field variables (i.e., lossless compressible field variables), such as mesh correction identifiers—variables directly used for topology construction or logical judgment, typically stored in 32-bit integer format. Topological field variables have extremely high accuracy requirements; any data distortion can lead to subsequent topology construction failures or logical judgment errors, therefore lossless compression must be used. Moderate accuracy loss in non-topological field variables has a smaller impact on data display and trend analysis, and lossy compression can be used to achieve higher compression ratios. Let L(p,i) be the length of the i-th field variable (i=1,2,...,M) before compression in the p-th computing node (p=1,2,...,N). Typically, the number of data elements for the M field variables on the same computing node is the same; let Np be the number of data elements for the field variable in the p-th computing node. For transient problems, each field variable contains data for T time steps, where T is greater than or equal to 1.

[0024] In one possible implementation, the shared memory data pool can employ a dynamically allocated memory management strategy. When the amount of field data in a single time step exceeds the preset memory pool size, the memory pool can automatically expand to ensure that all field variable data can be stored completely. The M field variables on each computing node are partitioned in the memory pool according to variable type: the first K non-topological field variables are stored in the floating-point area, and the last MK topological field variables are stored in the integer area. This partitioning method facilitates efficient access and parallel processing by the subsequent compression module. For transient problems, each field variable contains data from T time steps, and the shared memory data pool needs to reserve storage space T times the amount of data in a single time step for each field variable. When the total number of time steps is large and the memory capacity is insufficient to accommodate all time step data simultaneously, a sliding window strategy can be adopted: the memory pool only retains the data from the most recent W time steps (W < T), and the time step data that has been compressed and written is released promptly, thereby supporting the compression of transient field data for long-term sequences under limited memory conditions.

[0025] It should be noted that the classification criteria for field variables—that is, which field variables belong to non-topological field variables and which belong to topological field variables—can be pre-specified through configuration files or command-line parameters before the computational mechanics simulation begins. For example, users can declare in the configuration file that density, velocity components, pressure, laminar viscosity coefficient, and turbulent viscosity coefficient are lossy compressible field variables; and that mesh correction is identified as lossless compressible field variables. This pre-configuration mechanism allows users to flexibly adjust the compression strategy according to the specific computational scenario. In one possible embodiment, for certain field variables whose accuracy sensitivity is between the two (such as the wall distance function), users can also specify them to use lossy compression with higher accuracy or lossless compression, thereby making a more refined trade-off between compressibility and accuracy.

[0026] Understandably, by using a job scheduling system to uniformly manage computing resources, initiate parallel computations on each computing node, and store field data in local memory, the aggregated capacity of distributed memory can be utilized to store large-scale field data. This avoids the I / O (Input / Output) bottleneck caused by centrally transmitting all data to a single node, thus laying the data distribution foundation for subsequent parallel in-situ compression. By classifying field variables into non-topological and topological categories, a clear processing basis can be provided for subsequent hybrid compression strategies.

[0027] Step S12: Based on the continuous physical characteristics of the field variables in the transient field data in the spatial and temporal dimensions, the non-topological field variables and the topological field variables are integrated to obtain two-dimensional integrated data; wherein, the data integration includes: sorting the corresponding field variable data in the spatial dimension using a space filling algorithm that prioritizes adjacent positions, and sorting the corresponding field variable data in the temporal dimension according to the time order.

[0028] Furthermore, considering the continuous physical characteristics of field variables in transient field data in both spatial and temporal dimensions, the non-topological field variables and the topological field variables can be integrated to obtain two-dimensional integrated data; specifically, this involves data integration in both spatial and temporal dimensions.

[0029] In the spatial dimension, a space-filling algorithm prioritizing adjacent positions is used to sort the corresponding field variable data. In unstructured grids, grid node numbers are often arbitrary, leading to potentially large index distances between spatially adjacent grid nodes in the one-dimensional memory array. This spatially discontinuous data arrangement increases the numerical differences (information entropy) between adjacent data, which is detrimental to the compression effect of subsequent compression algorithms. This step reorders the field variable data in the spatial dimension, ensuring that spatially adjacent grid node data are adjacent to each other in the sorted one-dimensional array, thereby effectively reducing information entropy.

[0030] In one possible embodiment, the spatial filling algorithm prioritizing adjacent positions in the spatial dimension for sorting the corresponding field variable data includes: using a Hilbert space-filling curve to perform spatial traversal of the field variable data for both the non-topological and topological field variables, and sorting the corresponding field variable data based on the traversal order. The Hilbert space-filling curve is a fractal curve with good spatial locality preservation properties, ensuring that adjacent points in space maintain their proximity after being mapped to a one-dimensional sequence via the curve when traversing multidimensional space. Specifically, the implementation process of Hilbert curve traversal sorting is as follows: First, determine the bounding box of the unstructured mesh in three-dimensional space, i.e., the minimum and maximum values ​​of all nodes in the mesh in the x, y, and z directions; divide the bounding box into 2 equal parts in the three directions. dA regular Cartesian grid cover is formed by dividing the grid into several elements (d is the order of the Hilbert curve, usually d=8 to 16). For each grid node, its Cartesian grid cell index is determined based on its spatial coordinates (x, y, z). The mapping function of the Hilbert curve is used to convert the three-dimensional cell index into a one-dimensional traversal index along the curve. Finally, the field variable data on each grid node are reordered in ascending order of traversal index. When multiple grid nodes are mapped to the same Cartesian grid cell, they can be reordered according to their order of appearance in the original memory array, or finely ordered according to their spatial position within the cell. The larger the order d of the Hilbert curve, the finer the spatial partitioning and the better the preservation of spatial locality, but the computational cost also increases accordingly. For millions of grid nodes, d=10 to 12 can achieve good results. Let Np be the number of field variables on the p-th computing node, and let f(i') = i be the mapping relationship between the field variables before and after sorting. That is, the original index of the i'-th variable after sorting is i, and the i-th field variable of the p-th computing node after sorting is denoted as V'(p,i).

[0031] In one possible embodiment, the space-filling algorithm can also employ other space-filling curves with spatial locality preservation characteristics. For example, the Z-order curve (Morton curve) achieves spatial traversal by alternating the binary bits of each coordinate component, offering high computational efficiency but slightly weaker locality preservation than the Hilbert curve. In practical applications, the appropriate curve can be selected based on different priorities regarding compression ratio and computational efficiency: the Hilbert curve is preferred when pursuing the highest compression ratio; the Z-order curve can be chosen when a balance needs to be struck between sorting computational overhead and compression ratio. Furthermore, for two-dimensional planar meshes (such as wing surface meshes), two-dimensional Hilbert curves can be used; for three-dimensional volume meshes, three-dimensional Hilbert curves are selected. In one possible embodiment, the traversal results of the space-filling curve can be pre-calculated and cached. When the same mesh is used multiple times for transient field data compression under different operating conditions, the cached results can be directly reused, further reducing sorting overhead.

[0032] It is understandable that spatial reordering of field variable data using Hilbert space-filling curves can significantly reduce the numerical differences between adjacent data. This is because Hilbert curves have excellent spatial locality preservation properties, which can cluster spatially adjacent grid node data into adjacent positions in a one-dimensional sequence. This allows subsequent compression algorithms to more efficiently identify and utilize the spatial continuity features in the data, thereby significantly improving the compression ratio.

[0033] The corresponding field variable data are sorted in chronological order along the time dimension. In one possible embodiment, sorting the corresponding field variable data in chronological order along the time dimension includes: sorting the field variable data of the non-topological field variables and the topological field variables based on the sorting principle of ascending time. Specifically, for each field variable V'(p,i) after spatial reordering, its data at T time steps (i.e., V'(p,i,t=1), V'(p,i,t=2), ..., V'(p,i,t=T)) are assembled in ascending order of time step number t to form the second dimension data. After assembly, the data of each field variable is logically presented as a two-dimensional array: the first dimension is the spatial dimension (Np spatial positions after spatial filling curve sorting), and the second dimension is the time dimension (T time steps). This two-dimensional data structure allows compression algorithms to simultaneously utilize the continuity of data for prediction and encoding in both spatial and temporal dimensions. Compared to compressing data separately at each time step, two-dimensional compression can fully leverage the data correlation and redundancy between different time steps, achieving a higher compression ratio. In one possible embodiment, when the total number of time steps is large, a batch assembly strategy can be adopted to assemble T... all Each time step is divided into several batches for two-dimensional integration and compression.

[0034] It is understandable that, since the evolution of transient physical processes is gradual, the changes in the physical quantities at the same spatial location at adjacent time steps are limited. After arranging the data of the same field variable at multiple time steps in ascending order of time, the numerical differences between adjacent data corresponding to the states that are close in time are greatly reduced. Therefore, the information entropy can be further reduced from the time dimension, thereby improving the compression rate.

[0035] In one possible embodiment, after sorting the corresponding field variable data using a space-filling algorithm prioritizing adjacent positions in the spatial dimension, the method further includes: determining the mapping relationship data corresponding to the field variable data before and after sorting, and performing lossless compression on the mapping relationship data so that the corresponding compressed data can be written to the target disk file. Specifically, the mapping relationship data is f(i')=i, which records the correspondence between the sorted position i' and the original position i before sorting, and is the key basis for restoring the original grid data order during subsequent decompression. The size of the mapping relationship data is proportional to the number of field variable data Np. For a 32-bit integer sequence number, each data element occupies 4 bytes before compression, and the total data length is Np. 4 bytes. Because the mapping relationship data itself has good local continuity (the traversal characteristics of the space-filling curve mean that adjacent sorted positions usually correspond to nearby regions in the grid, and their original indices also have local continuity), even lossless compression can still achieve considerable compression results. In one possible embodiment, the lossless compression of the mapping relationship data uses the DEFLATE compression algorithm. Let the lengths of the mapping relationship data of the p-th computing node before and after compression be Lm(p) and Lmc(p), respectively.

[0036] Understandably, the mapping relationship data is the key basis for restoring the original grid data order after decompression, and it must be ensured to be completely lossless. By performing lossless compression on the mapping relationship data, storage space usage can be reduced without losing any information, thereby ensuring that the decompressed data can be completely and accurately restored to the original grid order.

[0037] Step S13: Perform lossy compression on the two-dimensional integrated data corresponding to the non-topological field variables through the at least two computing nodes to obtain lossy compressed data, and simultaneously perform lossless compression on the two-dimensional integrated data corresponding to the topological field variables to obtain lossless compressed data.

[0038] In this embodiment, the above steps can be used to integrate the field variables of transient field data to obtain two-dimensional integrated data corresponding to the non-topological field variables and the two-dimensional integrated data corresponding to the topological field variables. Furthermore, lossy compression and lossless compression of the two-dimensional integrated data of the two types of field variables can be performed in parallel. It is understood that by using lossy compression on the non-topological field variables, some data accuracy is sacrificed within the user's acceptable accuracy range, thus significantly reducing the amount of non-topological field data. Conversely, by using lossless compression on the topological field variables, a certain degree of compression benefit can still be achieved by utilizing the information entropy reduction effect brought about by data integration while ensuring the absolute accuracy of the topological data.

[0039] For lossy compression of non-topological field data, specifically, N computing nodes sequentially perform parallel compression on the first K field variables (i.e., non-topological field variables) in the shared memory data pool using a lossy compression algorithm that supports two-dimensional data. Let Lc(p,i) be the length of the compressed data of the i-th non-topological field variable (i=1,2,...,K) of the p-th computing node.

[0040] In this application, compression precision refers to the maximum data deviation allowed by lossy compression, usually expressed in the number of significant digits or relative error bound. For example, for engineering visualization, a precision of 6 significant digits is usually sufficient; if more precise quantitative analysis is required later, a higher compression precision (such as 8 or 10 significant digits) can be set. In one possible embodiment, the lossy compression algorithm uses the multidimensional floating-point compression tool SZ (Squeeze). This algorithm utilizes the spatial continuity in multidimensional data for predictive encoding, achieving a high compression ratio within the user-specified error range, and natively supports direct compression of two-dimensional data. Specifically, the core principle of the SZ algorithm is a multi-level prediction model based on the Lorenzo (linear extrapolation of neighboring points in multidimensional data) predictor: first, prediction is performed using the neighborhood values ​​of the data in two-dimensional space to obtain the residual between the predicted value and the true value; then, the residual is quantized and entropy encoded, with the quantization step size controlled by the user-specified error bound. The smaller the error bound, the finer the quantization but the lower the compression ratio. After the data undergoes spatial and temporal two-dimensional integration in step S12, both the spatial and temporal dimensions of the data exhibit good continuity. SZ's predictor can accurately predict the values ​​of most data points, generating very small residuals, thus achieving an extremely high compression rate in the entropy coding stage.

[0041] In one possible embodiment, different compression accuracies can be set for different non-topological field variables. For example, higher accuracies (e.g., 8 significant digits) can be set for variables that have a significant impact on aerodynamic calculations, such as pressure and density, while lower accuracies (e.g., 4 significant digits) can be set for variables that have a higher tolerance for accuracy, such as turbulent viscosity coefficients, to finely balance data accuracy and compression ratio. In another possible embodiment, lossy compression algorithms can also be used, such as ZFP (a floating-point array compression library that supports constant-rate compression) and MGARD (multigrid adaptive data reduction, an error-controllable lossy compression method for scientific data), and other tools that support floating-point data compression. This application does not impose specific limitations on this.

[0042] Understandably, by employing lossy compression on non-topological field variables, and moderately sacrificing some data precision within an acceptable range for the user, the amount of non-topological field data can be significantly reduced. Furthermore, since each computing node compresses its local data in parallel, the computing power of multiple nodes can be fully utilized, significantly shortening the compression time.

[0043] For lossless compression of topological field data, specifically, N computing nodes sequentially perform parallel in-situ compression using a lossless compression algorithm on the last MK field variables (i.e., topological field variables) in the shared memory data pool. Let Lc(p,i) be the length of the compressed data of the i-th topological field variable (i=K+1,...,M) of the p-th computing node. Unlike lossy compression, this compression must ensure that the compressed data can be completely and accurately recovered, without any loss of precision. Topological field variables directly participate in grid topology construction and logical judgment; an error in any bit can lead to serious computational errors. In one possible embodiment, the lossless compression algorithm uses the DEFLATE algorithm, which combines the LZ77 algorithm (a lossless compression algorithm based on a sliding window) and Huffman coding, exhibiting good versatility and stability. Although the information entropy of the topological field variables is reduced after integration in step S12, the compression ratio is usually lower than that of lossy compression because lossless compression requires complete recoverability. However, the proportion of topological field variables in the total field data is usually small, limiting their impact on the overall compression ratio. In one possible embodiment, lossless compression may also employ algorithms such as LZ4 (a lossless compression algorithm characterized by high speed), Zstandard (a lossless compression algorithm with high compression ratio), or LZMA (a lossless compression algorithm with high compression ratio).

[0044] Understandably, by employing lossless compression on topological field variables, a certain degree of compression benefit can still be obtained while ensuring the absolute accuracy of the topological data and leveraging the information entropy reduction effect brought about by data integration. At the same time, it avoids topological construction errors caused by accuracy loss, thereby ensuring the reliability and security of data compression.

[0045] In one possible embodiment, after obtaining the lossy compressed data and the lossless compressed data, the method further includes: creating child nodes for the at least two computing nodes under the root node of the target disk file, and creating a field variable data node for each field variable in the transient field data under each child node; writing the lossy compressed data and the lossless compressed data into the corresponding field variable data nodes. Furthermore, creating a mapping relationship data node under each child node and writing the corresponding compressed data into the mapping relationship data node. Specifically, based on a tree-structured file parallel I / O interface (such as HDF5 (Hierarchical Data Format version 5, a file format and library) parallel I / O interface), N child nodes can be created under the root node of the target disk file, and (M+1) data nodes can be created under each child node, corresponding to M field variables and 1 mapping relationship data, respectively. The compressed data of the i-th field variable of the p-th compute node is written to the i-th data node of the p-th child node; the compressed mapping data of the p-th compute node is written to the M+1-th data node of the p-th child node. The HDF5 parallel I / O interface allows multiple processes to write data to different locations of the same file simultaneously, which can make full use of the aggregate I / O bandwidth of the parallel file system.

[0046] In one possible embodiment, the process may further include: verifying the decompressed data based on a first data length in the field variable data node and a second data length in the mapping relationship data node; wherein the first data length includes the target field variable data lengths L(p,i) and Lc(p,i) before and after compression, and the second data length includes the mapping relationship data corresponding to the target field variable data lengths Lm(p) and Lmc(p) before and after compression, and the decompressed data is the data obtained by decompressing the compressed target field variable data. Specifically, each data length is stored in the form of data node attributes. During decompression, the decompression program reads the length values ​​in the compressed data and attributes, and compares the actual data length with the recorded uncompressed length L(p,i) or Lm(p): if they match, the verification passes; if they do not match, an alarm is triggered or a retry is performed.

[0047] Understandably, by using the tree-structured file parallel I / O interface, each computing node can write to independent data nodes in parallel, avoiding the performance bottleneck of serial writing. Therefore, the aggregated I / O bandwidth of the parallel file system can be fully utilized, significantly shortening the data write time. By embedding data length information before and after compression into each data node, a lightweight yet efficient data integrity verification mechanism can be provided for subsequent decompression, thereby ensuring the reliability of data storage and transmission.

[0048] Therefore, this embodiment achieves parallel in-situ hybrid compression of unstructured transient field data through the above steps. First, the source of the field data is determined, and field variables are classified into non-topological and topological categories, laying the foundation for hybrid compression. Then, the physical continuity of the data is fully explored and utilized from both spatial and temporal dimensions, effectively reducing information entropy and simultaneously performing lossless compression on mapping relationship data. Next, lossy compression is performed on non-topological field variables to obtain a high compression ratio, while lossless compression is performed on topological field variables to ensure the accuracy of key data. Finally, the compressed data is written to a disk file, with data length information embedded for subsequent verification. Because two-dimensional integration preprocessing based on physical characteristics is performed on the data before compression, the compression ratio can be significantly improved. Because a hybrid compression strategy of lossy and lossless compression is used to distinguish data types, the overall compression ratio can be maximized while ensuring the absolute accuracy of topological data. Because each computing node performs compression and write operations in parallel, the overall compression time can be significantly shortened. Therefore, this embodiment can achieve fast and efficient compression of unstructured transient field data while ensuring data accuracy.

[0049] This embodiment, based on the above embodiments, provides a more in-depth description of the data verification mechanism. Figure 2 This is a data verification flowchart provided in an embodiment of this application. For example... Figure 2 As shown in the figure, this embodiment focuses on the recording method of data length information in compressed data and the decompression verification process.

[0050] In this embodiment, to perform subsequent data verification, the data length is selected for recording. Specifically, when the compressed data is written to the disk file, the compression program stores the following as attributes in each field variable data node: the length L(p,i) of the field variable data before compression and the length Lc(p,i) after compression; and in the mapping relationship data node, it stores the following: the length Lm(p) of the mapping relationship data before compression and the length Lmc(p) after compression. Specifically, for the i-th field variable on the p-th computing node, the original data byte length L(p,i) (i.e., the number of data elements multiplied by the number of bytes per element, 8 bytes for double-precision floating-point and 4 bytes for 32-bit integer) is recorded before compression, and the actual byte length Lc(p,i) of the compressed data is recorded after compression. When the compressed data is written to the corresponding data node through the HDF5 parallel IO interface, these length values ​​are written as an attribute of that data node. In one possible embodiment, a checksum (such as a CRC32 or MD5 value) of the data may also be stored. The checksum is calculated by traversing the original data before compression and stored as an attribute along with the length information for more comprehensive integrity verification.

[0051] Furthermore, the decompression program performs data verification according to the following process: First, it locates the data node to be decompressed in the target HDF5 file and reads the compressed data and the lengths of the data before and after compression from the attributes. Second, it performs decompression on the compressed data to obtain the decompressed data, calculates the actual byte length of the decompressed data, and compares it with the length of the data before compression L(p,i) (or Lm(p)) recorded in the attributes. If they match, the decompression is deemed to have been completed correctly, and data integrity is guaranteed; if they do not match, it is determined that data corruption may have occurred during compression, storage, or transmission. The decompression program may issue an alarm, record abnormal information in the error log (including node number p, variable number i, expected length, and actual length, etc.), or reread from other replicas in a multi-replica scenario. Third (optional), if checksum information is stored in the data node, the checksum is recalculated on the decompressed data and compared with the stored value to confirm data integrity with a higher confidence level.

[0052] Understandably, this embodiment provides lightweight data integrity verification capabilities with extremely low additional storage overhead (each data node only needs to store 4 length values, totaling 32 bytes) by embedding data length information before and after compression into the compressed data. Considering that a typical HDF5 file may contain hundreds to thousands of data nodes, the total overhead of the length attributes of all data nodes is only in the range of several kilobytes to tens of kilobytes, which is negligible compared to the GB (gigabyte) size of the compressed field data file.

[0053] In one possible implementation, data length verification can also be linked to other stages of the decompression process. For example, in a streaming decompression scenario (reading and decompressing simultaneously), the decompression program can perform length verification immediately after decompressing each data block. If an anomaly is detected, subsequent decompression operations are prematurely stopped and the problem location information is reported, avoiding wasting computational resources on incomplete data. In another possible implementation, the built-in verification functions of the HDF5 file format itself (such as block checksums of datasets) can be combined to form a multi-layered verification system: HDF5 file-level verification is responsible for detecting data corruption at the file system level, while data length verification is responsible for detecting data integrity issues. The two complement each other, further improving data reliability.

[0054] Therefore, this embodiment realizes the verification process after data decompression by recording the data length. The data length comparison only requires one integer comparison, which is highly efficient. It can realize the integrity verification of each field variable data without affecting the decompression performance. In large-scale parallel IO scenarios, it can effectively detect data anomalies caused by concurrent writing or network transmission, thereby ensuring data reliability.

[0055] like Figure 3As shown in the figure, this embodiment discloses a specific data compression method applicable to mechanical simulation systems, including the following contents.

[0056] In this embodiment, an application instance is configured first, and the High-Lift Common Research Model (HL-CRM), commonly used in computational fluid dynamics, can be selected. The calculation conditions are: incoming Mach number Ma = 0.2, angle of attack α = 17.05°, static pressure P = 17.14 kPa, and static temperature T = 289.4 K. The transient fluid field data is calculated based on the NNW-FlowStar parallel flow field simulation software, using a total of N = 32 calculation nodes, and calculating the flow field data for a total of T = 3 time steps.

[0057] During execution, the Slurm job scheduling system was used to launch 32 parallel computing nodes, running the NNW-FlowStar software. After each node calculated the flow field data, it was stored in a shared memory data pool. The field data included M=8 field variables: density (ρ), velocity components (u, v, w) in the x / y / z directions, pressure (P), laminar viscosity (μl), and turbulent viscosity (μt), all 64-bit double-precision floating-point numbers (K=7 non-topological field variables); and a mesh correction flag, a 32-bit integer (the 8th field variable, i.e., MK=1 topological field variable). Each field variable contained T=3 time steps of data. Figure 4 The diagram shows the distribution contours of the pressure field variables on the model surface at the first time step, with different color shades corresponding to different pressure coefficients. Before compression, the data lengths of the seven double-precision field variables at the same node were equal, while the data length of the integer variables was half that of the double-precision field variables. Subsequently, spatial and temporal two-dimensional integration was performed on the 32 computation nodes. In the spatial dimension, Hilbert space-filling curves were used to spatially traverse and reorder the field variable data, while recording the mapping relationship f(i'). In the temporal dimension, the three time-step data of each reordered field variable were assembled into the second-dimensional data according to their chronological order.

[0058] Furthermore, for the compression process of the field data, given a lossy compression precision of 6 significant digits, 32 nodes respectively use the SZ compression tool for lossy compression of the two-dimensional integrated data of 7 non-topological field variables; the DEFLATE algorithm is used for lossless compression of 1 topological field variable (mesh correction identifier); the DEFLATE algorithm is also used for lossless compression of the mapping relationship data, and the length of the mapping relationship data before compression is Np×4 bytes. Due to the spatial reordering and temporal ascending assembly after Hilbert space-filling curves, the two-dimensional integrated data of each field variable has good continuity in both spatial and temporal dimensions. The Lorenzo predictor in the SZ compression tool can use this continuity to perform high-precision prediction, producing minimal residuals, thus achieving a high compression ratio in the entropy coding stage.

[0059] Finally, based on the HDF5 tree file parallel IO interface, 32 child nodes can be created under the root node, and 9 data nodes (8 field variables + 1 mapping relationship data) can be created under each child node. Compressed data can be written in parallel, and the length of each data before and after compression can be written as an attribute. Figure 5 The data length distribution of 8 field variables across 32 nodes after compression is presented. It can be seen that after the hybrid compression processing of the technical solution in this application, the data volume of non-topological field variables is significantly reduced, the topological field variables also have a certain degree of compression benefit, and the storage occupation of the mapping relationship data is also significantly reduced after lossless compression.

[0060] To verify the effectiveness of this embodiment, a quantitative comparison can be made: Without compression, the original field data file size is 3.25GB. After direct compression using traditional compression software, the file size is 3.02GB, with a compression ratio of only 1.076 and a compression time of approximately 1342 seconds. However, using the hybrid compression method of this application, the file size is reduced to 0.70GB, the compression ratio reaches 4.643, and the compression time is approximately 135 seconds. Compared to the traditional method, the compression ratio is increased to approximately 4.3 times, and the compression time is shortened to approximately 10.1% (135 / 1342≈0.101).

[0061] Therefore, this embodiment employs the Hilbert space-filling curve to ensure that spatially adjacent grid node data are adjacent to each other in memory, significantly reducing spatial dimensional information entropy. Temporal ascending order assembly arranges progressively changing physical quantity data continuously over time, reducing temporal dimensional information entropy. Two-dimensional data integration allows compression algorithms such as SZ to utilize the continuity of both spatial and temporal dimensions for efficient predictive coding, generating minimal residuals for entropy coding. Furthermore, 32 nodes compress local data in parallel, with each node independently completing data sorting, integration, and compression without inter-node data exchange, fully utilizing multi-node computing power. The HDF5 parallel I / O interface allows 32 nodes to simultaneously write data to different data nodes of the same file, avoiding the performance bottleneck of serial writes and fully utilizing aggregated I / O bandwidth.

[0062] like Figure 6 As shown, this embodiment discloses a data compression device suitable for a mechanics simulation system, wherein the mechanics simulation system is a computer system including at least two computing nodes, and the device includes: The transient field data determination module 11 is used to determine the transient field data generated by the at least two computing nodes when performing computational mechanics simulations in parallel; the field variables in the transient field data include non-topological field variables and topological field variables; The data integration module 12 is used to integrate the non-topological field variables and the topological field variables respectively based on the continuous physical characteristics of the field variables in the transient field data in the spatial and temporal dimensions to obtain two-dimensional integrated data; wherein, the data integration includes: sorting the corresponding field variable data in the spatial dimension using a space filling algorithm that prioritizes adjacent positions, and sorting the corresponding field variable data in the temporal dimension according to the time order; The data compression module 13 is used to perform lossy compression on the two-dimensional integrated data corresponding to the non-topological field variables through the at least two computing nodes to obtain lossy compressed data, and at the same time to perform lossless compression on the two-dimensional integrated data corresponding to the topological field variables to obtain lossless compressed data.

[0063] Therefore, it can be seen that by utilizing the continuous physical characteristics of field variables in the spatial and temporal dimensions to implement data integration, the information entropy of adjacent data can be reduced. Combined with lossy / lossless hybrid parallel compression, the compression rate can be significantly improved while ensuring the high accuracy of topological data, saving disk storage space and reducing compression time.

[0064] In one specific embodiment, the data integration module 12 may include: The spatial sorting unit is used to perform spatial traversal of the field variable data of the non-topological field variables and the topological field variables using Hilbert space-filling curves, and to sort the corresponding field variable data based on the traversal order.

[0065] In one specific embodiment, the data integration module 12 may include: The time sorting unit is used to sort the field variable data of the non-topological field variables and the topological field variables based on the sorting principle of ascending time.

[0066] In one specific embodiment, the device may further include: The node creation module is used to create child nodes for the at least two computing nodes under the root node of the target disk file, and to create a field variable data node for each field variable in the transient field data under each child node; The first data writing module is used to write the lossy compressed data and the lossless compressed data into the corresponding field variable data nodes.

[0067] In another specific embodiment, the device may further include: The mapping relationship data determination module is used to determine the mapping relationship data corresponding to the field variable data before and after sorting; the mapping relationship data is the data generated during the sorting of the corresponding field variable data using a space filling algorithm that prioritizes adjacent positions in the spatial dimension; The mapping relationship data compression module is used to perform lossless compression on the mapping relationship data so that the corresponding compressed data can be written to the target disk file.

[0068] In yet another specific embodiment, the mapping relationship data compression module may include: The second data writing module is used to create a mapping relationship data node under each of the child nodes and write the corresponding compressed data into the mapping relationship data node.

[0069] In one specific embodiment, the device may further include: The data verification module is used to verify the decompressed data based on the first data length in the field variable data node and the second data length in the mapping relationship data node; wherein, the first data length includes the data length of the target field variable data before and after compression, the second data length includes the data length of the mapping relationship data corresponding to the target field variable data before and after compression, and the decompressed data is the data obtained by decompressing the compressed target field variable data.

[0070] Furthermore, embodiments of this application also disclose an electronic device, Figure 7 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0071] Figure 7 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the data compression method for mechanical simulation systems disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0072] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0073] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0074] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the data compression method for a mechanical simulation system disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0075] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned data compression method suitable for mechanical simulation systems. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0076] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0077] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0078] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0079] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0080] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data compression method suitable for a mechanics simulation system, wherein the mechanics simulation system is a computer system including at least two computing nodes, characterized in that, The method includes: Determine the transient field data generated when the at least two computing nodes perform a computational mechanics simulation in parallel; the field variables in the transient field data include non-topological field variables and topological field variables; Based on the continuous physical characteristics of the field variables in the transient field data in the spatial and temporal dimensions, the non-topological field variables and the topological field variables are integrated to obtain two-dimensional integrated data; wherein, the data integration includes: sorting the corresponding field variable data in the spatial dimension using a space filling algorithm that prioritizes adjacent positions, and sorting the corresponding field variable data in the temporal dimension according to the time order; The two-dimensional integrated data corresponding to the non-topological field variables are subjected to lossy compression by the at least two computing nodes to obtain lossy compressed data, and the two-dimensional integrated data corresponding to the topological field variables are subjected to lossless compression to obtain lossless compressed data.

2. The data compression method for mechanical simulation systems according to claim 1, characterized in that, The spatial filling algorithm, which prioritizes adjacent positions in the spatial dimension, is used to sort the corresponding field variable data, including: The Hilbert space-filling curve is used to perform spatial traversal of the field variable data for the non-topological field variables and the topological field variables, and the corresponding field variable data are sorted based on the traversal order.

3. The data compression method for mechanical simulation systems according to claim 1, characterized in that, The sorting of the corresponding field variable data in chronological order along the time dimension includes: Based on the ascending order of time, the field variable data of the non-topological field variables and the topological field variables are sorted.

4. The data compression method for mechanical simulation systems according to any one of claims 1 to 3, characterized in that, After obtaining the lossy compressed data and the lossless compressed data, the process further includes: Under the root node of the target disk file, child nodes are created for the at least two computing nodes respectively, and under each child node, a field variable data node is created for each field variable in the transient field data; The lossy compressed data and the lossless compressed data are written into the corresponding field variable data nodes.

5. The data compression method for mechanical simulation systems according to claim 4, characterized in that, After sorting the corresponding field variable data using a spatial filling algorithm that prioritizes adjacent positions in the spatial dimension, the method further includes: Determine the mapping relationship data corresponding to the field variable data before and after sorting; the mapping relationship data is the data generated during the process of sorting the corresponding field variable data using a space filling algorithm that prioritizes adjacent positions in the spatial dimension; The mapping relationship data is losslessly compressed so that the corresponding compressed data can be written to the target disk file.

6. The data compression method for mechanical simulation systems according to claim 5, characterized in that, The step of writing the corresponding compressed data to the target disk file includes: A mapping relationship data node is created under each of the child nodes, and the corresponding compressed data is written into the mapping relationship data node.

7. The data compression method for mechanical simulation systems according to claim 6, characterized in that, Also includes: The decompressed data is verified based on the first data length in the field variable data node and the second data length in the mapping relationship data node. Wherein, the first data length includes the data length of the target field variable data before and after compression, the second data length includes the data length of the mapping relationship data corresponding to the target field variable data before and after compression, and the decompressed data is the data obtained by decompressing the compressed target field variable data.

8. A data compression device suitable for a mechanics simulation system, wherein the mechanics simulation system is a computer system including at least two computing nodes, characterized in that, The device includes: A transient field data determination module is used to determine the transient field data generated by the at least two computing nodes when performing computational mechanics simulations in parallel; the field variables in the transient field data include non-topological field variables and topological field variables; The data integration module is used to integrate the non-topological field variables and the topological field variables respectively based on the continuous physical characteristics of the field variables in the transient field data in the spatial and temporal dimensions to obtain two-dimensional integrated data; wherein, the data integration includes: sorting the corresponding field variable data in the spatial dimension using a space filling algorithm that prioritizes adjacent positions, and sorting the corresponding field variable data in the temporal dimension according to the time order; The data compression module is used to perform lossy compression on the two-dimensional integrated data corresponding to the non-topological field variables through the at least two computing nodes to obtain lossy compressed data, and at the same time perform lossless compression on the two-dimensional integrated data corresponding to the topological field variables to obtain lossless compressed data.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the data compression method for a mechanical simulation system as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the data compression method for a mechanical simulation system as described in any one of claims 1 to 7.

Images