Super-computing-oriented remote collaborative post-processing method, device, equipment and medium

Abstract

The invention provides a super-computing-oriented remote collaborative post-processing method and device, equipment and a medium. The method comprises the following steps: determining a target quantity participating in post-processing, a post-processing type and associated configuration information; and sending the target quantity, the post-processing type and associated configuration information to a super-computing cluster, so that the super-computing cluster reads corresponding simulation data from a matched simulation data file in parallel according to the target quantity, and compresses the read simulation data to obtain data to be rendered, and if the data quantity of the data to be rendered is smaller than a threshold value, the data to be rendered is rendered. And sending the to-be-rendered data to the client, if the data size of the to-be-rendered data is greater than or equal to a threshold value, rendering the to-be-rendered data according to the post-processing type and the associated configuration information to obtain graphic rendering data, and sending the graphic rendering data to the client. The problems that local processing capacity is insufficient and interface display is not smooth are solved.

Description

Technical Field

[0001] The present application relates to the technical field of large-scale simulation data post-processing, and specifically to a remote collaborative post-processing method, apparatus, equipment and medium for supercomputing. Background Art

[0002] As the scale of engineering simulations (such as finite element analysis, computational fluid dynamics, and multiphysics simulation) continues to expand, the amount of simulation data generated is rapidly increasing, reaching terabytes (TBs) or even higher. Simulation data itself cannot intuitively reflect specific physical phenomena. Post-processing is required to transform abstract data into intuitive images to facilitate understanding of complex physical phenomena.

[0003] Current post-processing methods include local post-processing solutions, which download all or part of the simulation data generated on a supercomputing cluster to a local workstation for post-processing and visualization. The main drawbacks of this approach are: transmission bottlenecks. Downloading massive terabyte-level data from the supercomputing cluster to the local workstation via the network is extremely time-consuming and severely impacts efficiency; storage bottlenecks. Local workstations generally lack the capacity to store terabyte-level data, leading to storage resource constraints; and performance bottlenecks. Local workstations, due to performance limitations, cannot effectively meet the requirements for smooth post-processing and high-precision rendering of extremely large datasets.

[0004] In view of this, this application is hereby filed. Summary of the Invention

[0005] The present application provides a remote collaborative post-processing method, apparatus, equipment and medium for supercomputing, aiming to solve the problems of time-consuming downloading of ultra-large-scale simulation data and insufficient local processing and storage capabilities, and enables users to smoothly and automatically utilize supercomputing resources locally for efficient remote post-processing and visualization.

[0006] In a first aspect, an embodiment of the present application provides a remote collaborative post-processing method for supercomputing, which is applied to a client and includes: determining a target quantity involved in post-processing, a post-processing type, and associated configuration information;

[0007] Sending the target amount, the post-processing type, and the associated configuration information to the supercomputing cluster, so that the supercomputing cluster reads the corresponding simulation data from the matching simulation data file in parallel according to the target amount, and compresses the read simulation data according to the post-processing type to obtain data to be rendered. If the data amount of the data to be rendered is less than a threshold, the data to be rendered is sent to the client. If the data amount of the data to be rendered is greater than or equal to the threshold, the data to be rendered is rendered according to the post-processing type and the associated configuration information to obtain graphics rendering data, and the graphics rendering data is sent to the client.

[0008] Receive the data to be rendered sent by the supercomputing cluster, and render and display the data to be rendered according to the post-processing type and associated configuration information; or receive the graphics rendering data sent by the supercomputing cluster, and display the graphics rendering data.

[0009] In a second aspect, an embodiment of the present application further provides a remote collaborative post-processing method for supercomputing, which is applied to a supercomputing cluster, comprising: receiving a simulation data file identifier sent by a client, reading header metadata from a matching simulation data file according to the identifier, and sending the read header metadata to the client, so that the client determines a target amount of post-processing based on the header metadata;

[0010] Receive the target quantity, post-processing type, and related configuration information sent by the client, and read the corresponding simulation data in parallel from the matching simulation data file according to the target quantity through the calculation task;

[0011] Compress the read simulation data according to the post-processing type to obtain data to be rendered;

[0012] If the amount of data to be rendered is less than the threshold, the data to be rendered is sent to the client. If the amount of data to be rendered is greater than or equal to the threshold, the data to be rendered is rendered according to the post-processing type and the associated configuration information to obtain graphics rendering data, and the graphics rendering data is sent to the client.

[0013] In a third aspect, the present application also provides a remote collaborative post-processing device for supercomputing, which is integrated into a client and includes:

[0014] A first determination module is used to determine the target quantity involved in post-processing, the post-processing type and the associated configuration information;

[0015] A first sending module is configured to send a target amount, a post-processing type, and associated configuration information to a supercomputing cluster, so that the supercomputing cluster reads corresponding simulation data from a matching simulation data file in parallel according to the target amount, and compresses the read simulation data according to the post-processing type to obtain data to be rendered; if the amount of the data to be rendered is less than a threshold, the data to be rendered is sent to a client; if the amount of the data to be rendered is greater than or equal to the threshold, the data to be rendered is rendered according to the post-processing type and associated configuration information to obtain graphics rendering data, and the graphics rendering data is sent to the client;

[0016] The first receiving module is used to receive the data to be rendered sent by the supercomputing cluster, and render and display the data to be rendered according to the post-processing type and associated configuration information; or, receive the graphics rendering data sent by the supercomputing cluster, and display the graphics rendering data.

[0017] In a fourth aspect, the present application also provides a remote collaborative post-processing device for supercomputing, which is integrated into a supercomputing cluster and includes:

[0018] A second receiving module is configured to receive a simulation data file identifier sent by a client, read header metadata from a matching simulation data file according to the identifier, and send the read header metadata to the client, so that the client determines a target amount to participate in post-processing according to the header metadata;

[0019] The third receiving module is used to receive the target quantity, post-processing type and related configuration information sent by the client, and read the corresponding simulation data in parallel from the matching simulation data file according to the target quantity through the calculation task;

[0020] A compression processing module is used to compress the read simulation data according to the post-processing type to obtain data to be rendered;

[0021] The second sending module is used to send the data to be rendered to the client if the data volume of the data to be rendered is less than the threshold; if the data volume of the data to be rendered is greater than or equal to the threshold, render the data to be rendered according to the post-processing type and associated configuration information, obtain graphics rendering data, and send the graphics rendering data to the client.

[0022] In a fifth aspect, an embodiment of the present application further provides an electronic device, the electronic device comprising:

[0023] processor and memory;

[0024] The processor is used to execute the steps of the remote collaborative post-processing method for supercomputing in any embodiment by calling the program or instructions stored in the memory.

[0025] In a sixth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a program or instruction, and the program or instruction enables a computer to execute the steps of the remote collaborative post-processing method for supercomputing as in any embodiment.

[0026] In summary, the present application proposes a remote collaborative post-processing method for supercomputers, in which the client communicates with the supercomputer cluster, and the client determines the target amount, post-processing type and associated configuration information involved in the post-processing, and sends it to the supercomputer cluster. The supercomputer cluster reads the corresponding simulation data in parallel from the matching simulation data file according to the target amount, which can improve the reading efficiency of the simulation data, and compresses the read simulation data to obtain the data to be rendered. If the data amount of the data to be rendered is less than the threshold, the data to be rendered is sent to the client. If the data amount of the data to be rendered is greater than or equal to the threshold, the data to be rendered is rendered according to the post-processing type and associated configuration information to obtain graphic rendering data, and the graphic rendering data is sent to the client; the client receives the data to be rendered sent by the supercomputer cluster, and renders and displays the data to be rendered according to the post-processing type and associated configuration information; or, receives the graphic rendering data sent by the supercomputer cluster and displays the graphic rendering data. This method solves the problems of long download time for large-scale simulation data, insufficient local processing capacity and storage capacity, and unsmooth interface display, and enables users to smoothly and automatically use supercomputer resources locally for efficient remote post-processing and visualization. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a process of a remote collaborative post-processing method for supercomputing provided in the embodiment of the present application Figure 1 ;

[0028] Figure 2 This is a process of a remote collaborative post-processing method for supercomputing provided in the embodiment of the present application Figure 2 ;

[0029] Figure 3 This is a schematic diagram of a remote collaborative post-processing architecture for supercomputing provided in an embodiment of the present application;

[0030] Figure 4 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the invention are shown in the accompanying drawings.

[0032] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0033] Example 1

[0034] Figure 1 This is a flowchart of a remote collaborative post-processing method for supercomputing provided by an embodiment of the present application. This method is applied to the client and is suitable for scenarios where simulation data stored in a supercomputing cluster is post-processed by controlling the client, which facilitates user operation and provides users with a smooth interactive experience. Figure 1 , the remote collaborative post-processing method for supercomputing specifically includes the following steps:

[0035] S110: Determine the target quantity involved in post-processing, the post-processing type, and associated configuration information.

[0036] Different engineering simulations (such as finite element analysis, computational fluid dynamics, and multi-physics simulation) generate different types of simulation data. Furthermore, different post-processing types use different types of simulation data.

[0037] For example, target quantities involved in post-processing include but are not limited to temperature, pressure, stress, resistance, speed, and magnetic field strength, etc. Users can flexibly set them according to research needs.

[0038] Post-processing type can also refer to post-processing method, including but not limited to: cloud map, slice, section, vector arrow, streamline, scatter point, volume rendering, etc.

[0039] A cloud map essentially displays the distribution of a physical quantity on a model surface or cross-section using a color gradient. This is useful when you need to quickly visualize the global distribution trend of a physical quantity, such as a pressure cloud map on an aircraft wing.

[0040] Slices and sections essentially slice a 3D model using virtual planes (slices) or surfaces (sections) to display the distribution of physical quantities within the cross-section. This is useful for scenarios where you need to observe changes in physical quantities within a model, such as temperature slicing inside an engine cylinder to visualize the temperature gradient at the center of the combustion chamber.

[0041] "Vector Arrows" use arrows to represent the direction and magnitude of vector physical quantities. They are applicable to scenarios where directional analysis is required, such as a velocity vector diagram of a fluid flowing around an obstacle.

[0042] "Streamline" refers to a curve drawn along a vector field (such as a velocity field). The tangent direction of each point on the curve is consistent with the vector direction of that point. It is often used to represent the motion trajectory of a fluid or field, such as the streamline diagram of the airflow at the rear of a car.

[0043] The essence of "scatter points" is to display the numerical values ​​of physical quantities of discrete points in the form of points. The color and size of the points can map the physical quantities, such as the stress scatter plot of key bolts in a bridge.

[0044] The essence of "volume rendering" is to visualize the entire volume data of a three-dimensional model (rather than its surface or cross-section), and to show the spatial distribution of internal physical quantities (such as the three-dimensional form of smoke, flames, and electromagnetic fields) through transparency and color mapping. For example, volume rendering of the neutron flux in a nuclear reactor can show the three-dimensional distribution of radiation intensity.

[0045] The associated configuration information includes the position and angle of the slice, the range and color of the cloud map, the color, range and transparency of the volume rendering, etc.

[0046] In some embodiments, users can select target quantities for post-processing from physical quantities stored on the client; experienced users can also input target quantities for post-processing themselves; or they can select target quantities from header metadata extracted from simulation data files returned by a supercomputing cluster. This approach is more targeted, more accurate, and less prone to errors.

[0047] S120: Send the target quantity, post-processing type, and associated configuration information to the supercomputing cluster.

[0048] Specifically, the data exchanged between the client and the supercomputing cluster can be transmitted through the SSH tunnel, which increases data security.

[0049] After receiving the target quantity, post-processing type, and associated configuration information, the supercomputing cluster reads the corresponding simulation data from the matching simulation data file in parallel based on the target quantity. Supercomputing clusters are capable of processing large amounts of data, so parallelizing the simulation data reading process improves simulation data reading efficiency.

[0050] Furthermore, the supercomputing cluster compresses the read simulation data through a specific algorithm to further reduce the data volume, balance the relationship between data volume, accuracy, and rendering effect, and eliminate some data (such as redundant data, data with small differences between each other, etc.) while ensuring a certain accuracy to obtain the data to be rendered.

[0051] Compression processing of the read simulation data includes spatial domain compression (including mesh simplification and adaptive sampling to reduce data volume). Mesh simplification uses algorithms (such as edge collapsing and vertex clustering) to reduce the number of vertices and cells in the mesh while preserving the overall shape and key features of the model. For example, simplifying a mesh with 100 million vertices to 10 million directly reduces the data volume by 90%. Adaptive sampling dynamically adjusts the sampling density based on the gradient of the physical field, retaining dense sampling points in areas with drastic data changes and reducing sampling points in areas with gentle changes.

[0052] If the amount of data to be rendered is less than the threshold, the supercomputing cluster will send the data to be rendered to the client, which will be responsible for rendering and display. Since the amount of data is not large, the transmission time at this time usually meets the requirements, and the processing power required for rendering is also matched with the client's capabilities. Therefore, a smooth post-processing effect can be obtained (mainly referring to the display of post-processed graphics on the interface, as well as the interactive smoothness when scaling graphics, rotating graphics, and changing the display angle of graphics).

[0053] If the amount of data to be rendered is greater than or equal to the threshold, in order to avoid long transmission time and avoid the processing power required for rendering exceeding the client's capabilities, resulting in rendering and display jams and unsmoothness, in this case, the supercomputing cluster's superpowers are fully utilized. The supercomputing cluster renders the data to be rendered according to the post-processing type and related configuration information, obtains graphics rendering data, and sends the graphics rendering data to the client for display. That is, the client is not responsible for rendering operations, but only for display, thereby taking into account the client's computing power and ensuring a smooth display effect.

[0054] In particular, in order to further ensure a smaller transmission delay, the supercomputing cluster can render the rendering data through scalable rendering technology, thereby ensuring that the obtained graphics rendering data is lightweight and easy to transmit.

[0055] S130: Receive data to be rendered sent by the supercomputing cluster, and render and display the data to be rendered according to the post-processing type and associated configuration information; or receive graphics rendering data sent by the supercomputing cluster, and display the graphics rendering data.

[0056] The embodiments of this application provide a remote collaborative post-processing method for supercomputers. Users operate on a local client, and all graphical user interface responses are completed locally without any network delays, achieving a smooth and seamless interactive experience like operating local software, and completely solving the lag problem of traditional remote desktop solutions. By fully utilizing supercomputer resources, computationally intensive tasks such as post-processing and rendering are executed on the supercomputer's computing nodes, fully utilizing the supercomputer's powerful parallel computing capabilities, large memory, and high-speed storage, and can handle very large-scale data sets that local workstations cannot handle.

[0057] Example 2

[0058] Based on the above embodiment, this embodiment provides an optional implementation for step S110. Specifically, determining the target quantity for post-processing includes: receiving header metadata sent by the supercomputing cluster and displaying the header metadata, wherein the header metadata is read by the supercomputing cluster from a matching simulation data file, and the matching simulation data file is determined by the supercomputing cluster based on an identifier sent by the client (the identifier includes the file's storage path and file name); and receiving the target quantity for post-processing selected by the user from the header metadata. Since the header metadata is read by the supercomputing cluster from the matching simulation data file, the target quantity for post-processing selected from the header metadata must be the data quantity present in the simulation data file. The target quantity determined in this manner is more accurate and less prone to target quantity setting errors.

[0059] Specifically, the data server deployed in the supercomputing cluster reads the header information of the simulation data file, parses the metadata such as mesh, scalars, timesteps, etc. contained in the simulation data file, and sends these lightweight metadata back to the client to achieve rapid extraction of the header metadata of the simulation data file.

[0060] Furthermore, before determining the target amount involved in post-processing, it also includes: receiving target information set by the user, which is information related to the resources of the supercomputing cluster; sending the target information to the supercomputing cluster; receiving the storage path information of the simulation data file returned by the supercomputing cluster, and displaying the storage path information, wherein the supercomputing cluster determines the storage path information based on the target information when receiving the target information. Usually, different users have different accounts and passwords, and the simulation data file belonging to the user exists under the account. The storage path information of the simulation data file is located according to the account indicated by the target information, and is fed back to the client for the user to select a specific simulation data file from the storage path information (the file is just an identifier, and there is no real data inside it. The real data is stored on the supercomputing cluster, so the speed of interacting with the storage path information between the client and the supercomputing cluster is fast and efficient); receiving a selection instruction for the matching simulation data file identifier triggered by the user based on the storage path information, and sending the selected identifier to the supercomputing cluster.

[0061] Target information includes, but is not limited to, login node address, username, password, number of nodes, number of CPU cores, memory capacity, and partitions. After successful login, the job scheduling adapter automatically launches resource nodes and computing tasks. This built-in job scheduling adapter encapsulates complex Slurm command-line operations within a graphical interface, enabling one-click submission, monitoring, and termination of post-processing jobs. This significantly lowers the barrier to entry for supercomputing systems, allowing users to focus on data analysis rather than tedious system operations.

[0062] Then, the storage path information of the simulation data file is obtained and sent to the client. The client displays the storage path, and the user selects a specific simulation data file based on the displayed storage path. The simulation data file selected by the user is the matching simulation data file. The client sends the identifier of the corresponding file to the supercomputing cluster according to the user's selection operation. The supercomputing cluster finds the real simulation data file based on the identifier, reads the header metadata of the simulation data file, and then sends it to the client for the user to select the target quantity to participate in post-processing.

[0063] In some implementations, a virtual storage path information may also be maintained on the client for the user to select a simulation data file. Exemplarily, a selection instruction for a matching simulation data file identifier is received based on the virtual storage path information, and the selected identifier is sent to the supercomputing cluster. The virtual storage path information is identical to the actual storage path information, but no actual file data exists under the virtual storage path information.

[0064] Example 3

[0065] Based on the above embodiments, Figure 2 A remote collaborative post-processing method for supercomputing is shown, which is applied to a supercomputing cluster and is executed in conjunction with the client of the above embodiment, such as Figure 2 As shown, the following steps are included:

[0066] S210, receiving a simulation data file identifier sent by the client, reading header metadata from a matching simulation data file according to the identifier, and sending the read header metadata to the client, so that the client determines a target amount to participate in post-processing according to the header metadata.

[0067] Before receiving the simulation data file identifier sent by the client, it also includes: in response to receiving the target information sent by the client, determining the storage path information of the simulation data file according to the target information, and starting the matching computing task; sending the storage path information to the client so that the user can select the simulation data file identifier according to the storage path information.

[0068] S220 , receiving the target quantity, post-processing type, and associated configuration information sent by the client, and reading corresponding simulation data in parallel from the matching simulation data file according to the target quantity through the calculation task.

[0069] Specifically, simulation data can be read in parallel by multiple nodes of a supercomputing cluster to improve reading efficiency.

[0070] S230 : compressing the read simulation data to obtain data to be rendered.

[0071] Alternatively, the read simulation data may be compressed in parallel by multiple nodes of a supercomputing cluster, and the data obtained by the parallel compression may be integrated to obtain the final data to be rendered, thereby improving the compression processing efficiency.

[0072] S240. If the amount of the data to be rendered is less than the threshold, the data to be rendered is sent to the client. If the amount of the data to be rendered is greater than or equal to the threshold, the data to be rendered is rendered according to the post-processing type and the associated configuration information to obtain graphics rendering data, and the graphics rendering data is sent to the client.

[0073] Similarly, the rendering data can be rendered in parallel by multiple nodes of a supercomputing cluster, and the parts obtained by parallel rendering can be integrated together to obtain the entire graphics rendering data, thereby improving rendering efficiency.

[0074] Example 4

[0075] Based on the same inventive concept, this embodiment provides a schematic diagram of a remote collaborative post-processing architecture for supercomputing, such as Figure 3 As shown, it includes a client module 310 deployed on the user's local computer, a server module 320 deployed on the supercomputing node, and a job scheduling adaptation and tunnel communication module ( Figure 3 not shown).

[0076] The client module 310 includes a user interface 311 and a rendering window 312. The user interface 311 provides menus, buttons, dialog boxes, etc., allowing users to select files, add physical quantities, set colors, adjust viewing angles, etc. The rendering window 312 is responsible for receiving and displaying the final rendering results (graphics or simplified rendering data). The client module 310 is deployed on the user's local computer (such as a personal workstation or laptop). Its core responsibility is to provide a lightweight, responsive graphical user interface and to receive user interaction instructions (such as data loading, viewing angle change, rendering parameter adjustment, etc.). Through the job scheduling adapter and tunnel communication module, a job request to start the server program is submitted to the supercomputing job management system (such as Slurm).

[0077] The server module 320 mainly includes three parts:

[0078] Component Launcher 321: Component Launcher 321 is typically launched on a remote cluster and communicates with the local client through an SSH tunnel. When other components need to be launched, the component launcher is responsible for invoking and executing them, including launching the parallel computing engine. When launching the parallel computing engine, Component Launcher 321 executes a parallel command with specific parameters that dictate how to launch the engines in parallel. It also maintains the local client's connection to the cluster, eliminating the need to frequently enter passwords when launching remote components.

[0079] Data Server 322: When a client requests a connection to a remote cluster, the component launcher service starts the data server. The data server reads the data file header information, parses the file's metadata, and sends this lightweight metadata back to the client, enabling rapid extraction of simulation data file header metadata.

[0080] Rendering Computing Engine 323: When a client issues a rendering instruction (such as a slice or isosurface), the component launcher controls the launch of Rendering Computing Engine 323. Rendering Computing Engine 323 supports large-scale parallel computing and can be launched in parallel on a supercomputer according to user settings. Each parallel engine process is responsible for reading and processing a portion of the data file. The rendering computing engine not only reads and processes data but also performs rendering, using parallel rendering technology. The engine then synthesizes the partial images or simplified rendering data generated by each process into a complete image or simplified rendering data. After the calculation is complete, the engine sends the final rendering result (usually a compressed image) back to the client's rendering window for display.

[0081] In summary, based on Figure 3 As shown in the architecture diagram, the general process includes: 1. Communication connection; 2. Reading header metadata; 3. Returning header metadata; 4. Sending post-processing instructions; 5. Reading data and rendering; 6. Sending graphics rendering data.

[0082] The above process realizes data scaling, and ultra-large-scale simulation data is passed through the rendering calculation engine to obtain lightweight images or simplified rendering data, further laying the foundation for fast and efficient interaction.

[0083] The job scheduling adaptation and tunnel communication module, serving as a bridge between the client and server, is key to achieving automated and efficient communication. Its core functions include: SSH tunnel transmission. After a job is submitted by the client and assigned to a compute node by the scheduling system, a secure SSH tunnel is automatically established between the local client and that specific compute node. This tunnel is used to transmit post-processing instructions from the client to the server, as well as file information or rendering results from the server to the client. This ensures that the amount of data exchanged is far less than the original simulation data itself and the remote desktop pixel stream, thereby ensuring efficient transmission and smooth interaction. Job scheduling adaptation automatically applies for and manages supercomputing resources based on resource requirements set by the user in the GUI (such as the number of nodes, computation time, and required queues). The client applies for computing resources through the SSH connection, monitors the status of the computational job (queued, running, completed), and automatically cancels the job at the end of the session, releasing resources.

[0084] Specific implementation steps include:

[0085] Step 1: Initialize the client module. Initialize the client module and set the supercomputer connection information, including but not limited to the login node address, account number, password, and supercomputer performance information, including but not limited to the number of nodes, number of CPU cores, memory capacity, partitions, and computing engine path.

[0086] Step 2: Open the simulation data. Step 2.1: According to the supercomputer connection information set in step 1, establish a connection between the client module and the server module through the job scheduling adapter and tunnel communication module, that is, connect the local and supercomputer, and at the same time start the computing task of the server module on the supercomputer, and return whether the task is started. If it is not successfully started, provide a response error message, including but not limited to account password error, cluster partition error, etc. Step 2.2: The client module issues an instruction to open the simulation data, and the server module returns the file path information after receiving the instruction. Step 2.3: The client module sends the simulation file to be opened according to the file path information. The simulation file can be a single file, multiple files of a time series, simulation files of different regions in the same time step, or simulation files of different regions in the time series. Step 2.4: The server module uses parallel reading to open the specified simulation file. Step 2.5: The server module reads the header metadata in the simulation file, including but not limited to the physical quantity name, type, number of grids, and type, and sends the information to the client module.

[0087] Step 3: Post-processing operation. Step 3.1: The client module selects the physical quantity that needs to be post-processed based on the metadata in the simulation file header. Step 3.2: The client module selects the type of post-processing that needs to be performed, including but not limited to cloud maps, slices, sections, vector arrows, streamlines, scatter points, and volume rendering. Step 3.3: The client module sends the physical quantity and type of post-processing, as well as parameter information related to the post-processing type (such as the position and angle of the section, the range and color of the cloud map, the color, range, transparency of the volume rendering, etc.) to the server module. Step 3.4: The server module reads the relevant data in the simulation file according to the post-processing requirements provided by the client module, and analyzes and processes it to obtain the data that needs to be rendered and displayed, that is, simplifies the rendering data.

[0088] Step 4: Rendering and display. There are two ways of rendering and display, one is server-side module rendering, and the other is client-side module rendering. When the amount of rendering display data obtained in step 3.4 is small, the client-side module rendering can be used. When the amount of data is large, the server-side module rendering is used. Step 4.1: Server-side module rendering. The server-side module directly renders the data in step 3.4 off-screen, obtains the rendered graphics, and sends it to the client module for display. Since this method only transmits the rendered graphic information, the rendering process is in the server-side module and is therefore not limited by the size of the data to be rendered. Step 4.2: Client-side module rendering. The server-side module compresses the data in step 3.4 and sends it directly to the client module. The client module renders and displays the data. Since this method requires the data to be rendered to be transferred from the server-side module to the client module, the amount of data cannot be too high, otherwise the transmission time will be long, and it is limited by the performance of the local environment where the client is located, and cannot render too large an amount of data.

[0089] Taking the post-processing and visualization of a specific example as an example, the execution flow of the above solution is explained. First, execute step one and start the client on the local computer. The client includes a user interface and a rendering window. Use the Tianhe new generation supercomputer to configure relevant information in the user interface. Then, execute step two. According to the information set in step one, establish a connection between the client module and the server module through tunnel communication and job scheduling adaptation module, that is, connect the local and supercomputers, and start the computing task on the supercomputer at the same time. According to the requirements, the computing task uses 2000 computing nodes, and each computing node starts 50 MPI processes, totaling 100,000 MPI processes, supporting parallel rendering of hundreds of billions of grids. The client module then issues a command to open the simulation data, and the server module returns the file path information after receiving the command. The client module sends the simulation file to be opened according to the file path information. The simulation files in this example are simulation files for different regions of the time series. Next, the server module uses parallel reading to open the specified simulation file and reads the header metadata in the simulation file. The physical quantities include density, pressure, temperature, component mass fraction, and the grid type is unstructured grid, and sends this information to the client module.

[0090] Then, proceed to step three, browsing the simulation file post-processing list in the user interface. To better display the simulation results, multi-physics volume rendering is used, including the mass fractions of different gas components and temperatures, each with different colors and appropriate transparency settings. The mesh boundaries are displayed as slices to illustrate the complex geometric boundaries of the model. This post-processing information is sent to the server module, which reads the relevant data in the simulation file based on this information and compresses it to obtain the data required for rendering and display, i.e., simplified rendering data.

[0091] Next, step 4 is performed. Since the simplified rendering data obtained in step 3 is large and inconvenient to transmit, the server module rendering method is adopted. The server module directly performs off-screen rendering on the simplified rendering data and sends the rendered graphics data to the client module for display.

[0092] Specific implementation method: Take the post-processing and visualization of a specific instance as an example.

[0093] First, execute step 1 and start the client on the local computer. The client includes a user interface and a rendering window. Use the Tianhe new generation supercomputer and configure relevant information in the user interface.

[0094] Then proceed to step 2. According to the supercomputer information set in step 1, a connection is established between the client module and the server module through tunnel communication and the job scheduling adapter module, that is, the local computer is connected to the supercomputer, and the computing task is started on the supercomputer at the same time. According to the requirements, the computing task uses 2,000 computing nodes, and each computing node starts 50 MPI processes, totaling 100,000 MPI processes, supporting parallel rendering of hundreds of billions of grids. Subsequently, the client module issues an instruction to open the simulation data, and the server module returns the file path information after receiving the instruction. The client module sends the simulation file to be opened according to the file path information. The simulation files of this example are simulation files of different regions of the time series. Immediately afterwards, the server module uses parallel reading to open the specified simulation file and reads the header metadata in the simulation file. The physical quantities include density, pressure, temperature, component mass fraction, and the grid type is an unstructured grid, and sends this information to the client module.

[0095] Then, proceed to step three, browsing the simulation file post-processing list on the user interface. To better display the simulation results, multi-physics volume rendering is used, including the mass fractions of different gas components and temperatures, each with different colors and appropriate transparency settings. The mesh boundaries are displayed as slices to illustrate the model's complex geometric boundaries. This post-processing information is sent to the server module, which reads the relevant data in the simulation file and analyzes it to obtain the data required for rendering and display, i.e., simplified rendering data.

[0096] Next, we proceed to step four. Because the simplified rendering data obtained in step three is large and inconvenient to transmit, we use a server-side rendering method. The server-side module directly renders the simplified rendering data off-screen, then sends the resulting rendered image to the client module for display. Measured rendering and display time totals 120 seconds.

[0097] This solution decomposes the "all-in-one" architecture of traditional post-processing software into a lightweight interactive front-end (client) and a large-scale computational rendering back-end (server). The server and its massive data are located on a supercomputer, enabling highly coordinated data, computation, and rendering, eliminating data transmission bottlenecks. The client only handles UI operations, interaction logic, and result image display, ensuring smooth user interaction regardless of network latency fluctuations. This architecture eliminates the lag problem often associated with traditional remote desktop applications due to the transmission of large desktop image streams.

[0098] The client features deep native integration and automated management with supercomputing job scheduling systems. The client includes built-in interface adapters for mainstream supercomputing job scheduling systems (such as Slurm and PBS). Users can automatically apply for and manage computing resources by setting the required computing resources (such as the number of nodes, cores, and runtime) on the client's graphical interface. The client then automatically submits the server program as a computing task to the supercomputing cluster, where the scheduling system allocates idle computing nodes. The client can query job status and log information in real time and, at the end of a task, automatically issue commands to cancel the job and release computing resources. This enables on-demand resource utilization and flexible allocation, avoiding the drawbacks of traditional methods that abuse login nodes.

[0099] Efficient and secure data lightweighting and transmission. After the server is successfully started on the computing node through the supercomputing scheduling system, it actively connects back to the locally started client and establishes an encrypted, point-to-point dedicated communication channel.

[0100] Transmission data is simplified. Instead of raw 3D models or volumetric data, this channel transmits lightweight control commands, 2D images, and simplified data. This reduces the amount of data transmitted by several orders of magnitude compared to the original data. This lightweight data transmission enables fast and efficient data exchange.

[0101] Ultra-large-scale parallel rendering. This solution supports parallel expansion on the server side. Based on the MPI parallel method, it achieves large-scale parallel rendering through parallel I / O reading of data blocks and CPU parallel rendering algorithms. This solves the problem of insufficient computing power and slow rendering on a single computing node, and maximizes the powerful computing power of supercomputing CPU clusters to achieve real-time interactive rendering of ultra-large-scale data.

[0102] It should be noted that, unless otherwise specified, all operations in the server module are executed in parallel to improve efficiency.

[0103] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown, the electronic device 500 includes one or more processors 501 and a memory 502 .

[0104] The processor 501 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 500 to perform desired functions.

[0105] The memory 502 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on a computer-readable storage medium, and the processor 501 may execute the program instructions to implement the remote collaborative post-processing method for supercomputing of any embodiment of the present application described above and / or other desired functions. Various contents such as initial external parameters, thresholds, etc. may also be stored in the computer-readable storage medium.

[0106] In one example, electronic device 500 may further include an input device 503 and an output device 504, which are interconnected via a bus system and / or other connection mechanisms (not shown). Input device 503 may include, for example, a keyboard, a mouse, etc. Output device 504 may output various information to the outside, including warning information, braking force, etc. Output device 504 may include, for example, a display, a speaker, a printer, a communication network, and remote output devices connected thereto.

[0107] Of course, to simplify, Figure 4 Only some of the components related to the present application in the electronic device 500 are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition, the electronic device 500 may further include any other appropriate components according to specific application scenarios.

[0108] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions. When the computer program instructions are executed by a processor, the processor executes the steps of the remote collaborative post-processing method for supercomputing provided in any embodiment of the present application.

[0109] The computer program product may be written in any combination of one or more programming languages ​​to implement the program code for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0110] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the remote collaborative post-processing method for supercomputing provided in any embodiment of the present application.

[0111] Computer-readable storage media can be any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0112] It should be noted that the terms used in this application are only for describing specific embodiments and are not intended to limit the scope of this application. As shown in the specification and claims of this application, unless the context clearly indicates an exception, the words "one", "an", "a kind of" and / or "the" do not specifically refer to the singular and may also include the plural. The terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method or device comprising a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of further restrictions, the elements defined by the sentence "comprise a..." do not exclude the presence of other identical elements in the process, method or device comprising the elements.

[0113] It should also be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on this application. Unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", etc. should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be a communication between the internal parts of two elements. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0114] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. The above is only the preferred implementation method of this application. It should be pointed out that due to the limitations of textual expression, there are objectively infinite specific structures. For ordinary technicians in this technical field, without departing from the principles of the present invention, they can also make several improvements, modifications or changes, and can also combine the above technical features in an appropriate manner; these improvements, modifications, changes or combinations, or the direct application of the inventive concept and technical solution to other occasions without improvement, should be regarded as the scope of protection of this application.

Claims

1. A remote collaborative post-processing method for supercomputing, applied to a client, characterized in that: include: Determine the target quantity involved in post-processing, post-processing type and associated configuration information; Sending the target amount, the post-processing type, and the associated configuration information to a supercomputing cluster, so that the supercomputing cluster reads corresponding simulation data from a matching simulation data file in parallel according to the target amount, and compresses the read simulation data to obtain data to be rendered; if the data amount of the data to be rendered is less than a threshold, sending the data to be rendered to a client; if the data amount of the data to be rendered is greater than or equal to the threshold, rendering the data to be rendered according to the post-processing type and the associated configuration information to obtain graphics rendering data, and sending the graphics rendering data to the client; Receive the data to be rendered sent by the supercomputing cluster, and render and display the data to be rendered according to the post-processing type and the associated configuration information; or receive the graphics rendering data sent by the supercomputing cluster, and display the graphics rendering data.

2. The method according to claim 1, characterized in that Determining the target amount for post-processing includes: receiving header metadata sent by the supercomputing cluster and displaying the header metadata, wherein the header metadata is read by the supercomputing cluster from a matching simulation data file, and the matching simulation data file is determined by the supercomputing cluster according to an identifier sent by the client; A target amount of participation in post-processing selected by a user from the header metadata is received.

3. The method according to claim 2, characterized in that Before determining the target amount for post-processing, the method further includes: receiving target information set by a user, wherein the target information is information related to resources of the supercomputing cluster; Sending the target information to the supercomputing cluster; receiving storage path information of the simulation data file returned by the supercomputing cluster and displaying the storage path information, wherein the supercomputing cluster determines the storage path information according to the target information when receiving the target information; A selection instruction for the matching simulation data file identifier is received, which is triggered based on the storage path information, and the selected identifier is sent to the supercomputing cluster.

4. The method according to claim 2, characterized in that Before determining the target amount for post-processing, the method further includes: Receive a selection instruction for the matching simulation data file identifier triggered by virtual storage path information, and send the selected identifier to the supercomputing cluster; wherein the virtual storage path information is the same as the real storage path information, but no real file data exists under the virtual storage path information.

5. A remote collaborative post-processing method for supercomputing, applied to a supercomputing cluster, and executed in conjunction with the method according to any one of claims 1 to 4, characterized in that: include: receiving a simulation data file identifier sent by a client, reading header metadata from a matching simulation data file according to the identifier, and sending the read header metadata to the client, so that the client determines a target amount to participate in post-processing according to the header metadata; Receive the target quantity, post-processing type and related configuration information sent by the client, and read the corresponding simulation data in parallel from the matching simulation data file according to the target quantity through the calculation task; Compress the read simulation data to obtain data to be rendered; If the amount of the data to be rendered is less than a threshold, the data to be rendered is sent to the client; if the amount of the data to be rendered is greater than or equal to the threshold, the data to be rendered is rendered according to the post-processing type and associated configuration information to obtain graphics rendering data, and the graphics rendering data is sent to the client.

6. The method according to claim 5, characterized in that Before receiving the simulation data file identifier sent by the client, the method further includes: In response to receiving the target information sent by the client, determining storage path information of the simulation data file according to the target information, and starting a matching computing task; The storage path information is sent to the client, so that the user can select a simulation data file identifier according to the storage path information.

7. A remote collaborative post-processing device for supercomputing, integrated into the client, characterized in that: include: A first determination module is used to determine the target quantity involved in post-processing, the post-processing type and the associated configuration information; a first sending module, configured to send the target amount, the post-processing type, and the associated configuration information to a supercomputing cluster, so that the supercomputing cluster reads corresponding simulation data from a matching simulation data file in parallel according to the target amount, compresses the read simulation data, obtains data to be rendered, sends the data to be rendered to a client if the amount of the data to be rendered is less than a threshold, and renders the data to be rendered according to the post-processing type and the associated configuration information to obtain graphics rendering data, and sends the graphics rendering data to the client if the amount of the data to be rendered is greater than or equal to the threshold; The first receiving module is used to receive the data to be rendered sent by the supercomputing cluster, and render and display the data to be rendered according to the post-processing type and associated configuration information; or, receive the graphics rendering data sent by the supercomputing cluster, and display the graphics rendering data.

8. A remote collaborative post-processing device for supercomputers, integrated into a supercomputer cluster, characterized in that: include: a second receiving module, configured to receive a simulation data file identifier sent by a client, read header metadata from a matching simulation data file according to the identifier, and send the read header metadata to the client, so that the client determines a target amount to participate in post-processing according to the header metadata; A third receiving module is configured to receive a target quantity, a post-processing type, and associated configuration information sent by a client, and read corresponding simulation data in parallel from a matching simulation data file according to the target quantity through a calculation task; A compression processing module is used to compress the read simulation data to obtain data to be rendered; The second sending module is used to send the data to be rendered to the client if the data volume of the data to be rendered is less than a threshold value; if the data volume of the data to be rendered is greater than or equal to the threshold value, render the data to be rendered according to the post-processing type and associated configuration information to obtain graphics rendering data, and send the graphics rendering data to the client.

9. An electronic device, characterized in that: The electronic device comprises: processor and memory; The processor is used to execute the steps of the remote collaborative post-processing method for supercomputing as described in any one of claims 1 to 6 by calling the program or instructions stored in the memory.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program or instruction, which enables a computer to execute the steps of the remote collaborative post-processing method for supercomputing as described in any one of claims 1 to 6.

Images