Supercomputer-oriented remote collaborative post-processing method and device, equipment and medium

Abstract

The application provides a remote cooperative post-processing method and device for supercomputers, equipment and medium. The method comprises the following steps: determining a target quantity participating in post-processing, a post-processing type and associated configuration information; sending the target quantity, the post-processing type and the associated configuration information to a supercomputer cluster, so that the supercomputer cluster reads corresponding simulation data from a matched simulation data file in parallel according to the target quantity, performs compression processing on the read simulation data to obtain to-be-rendered data, sends the to-be-rendered data to a client if the data quantity of the to-be-rendered data is less than a threshold, performs rendering on the to-be-rendered data according to the post-processing type and the associated configuration information to obtain graphic rendering data and sends the graphic rendering data to the client if the data quantity of the to-be-rendered data is greater than or equal to the threshold. The application aims to solve the problems of insufficient local processing capacity and unsmooth interface display.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large-scale simulation data post-processing, in particular to a remote collaborative post-processing method and device for supercomputing, equipment and medium. BACKGROUND

[0002] With the continuous expansion of engineering simulation (such as finite element analysis, computational fluid dynamics, multi-physical field simulation, etc.), the amount of simulation data generated grows rapidly, reaching TB (Terabyte) level or even higher. Simulation data itself cannot intuitively reflect specific physical phenomena, and needs to be converted into intuitive images through "post-processing" to help understand complex physical phenomena.

[0003] The current post-processing method includes a local post-processing scheme, which specifically downloads all or part of the simulation data generated in the supercomputing cluster to the local workstation for post-processing and visualization. The main defects of this scheme are: transmission bottleneck, TB (Terabyte) level massive data is downloaded from the supercomputing cluster to the local workstation through the network, which takes a very long time and seriously affects the efficiency; storage bottleneck, the local workstation usually does not have the ability to store TB (Terabyte) level data, causing storage resource shortage; performance bottleneck, the local workstation is limited by performance constraints and cannot well meet the needs of smooth post-processing and high-precision rendering of super large-scale data sets.

[0004] Therefore, the present application is proposed. SUMMARY

[0005] The present application provides a remote collaborative post-processing method and device for supercomputing, equipment and medium, aiming to solve the problem of long download time, insufficient local processing and storage capacity of super large-scale simulation data, and realize smooth and automatic remote post-processing and visualization using supercomputing resources on the local user side.

[0006] In a first aspect, the present application provides a remote collaborative post-processing method for supercomputing, applied to a client, including: determining a target quantity participating in post-processing, a post-processing type and associated configuration information;

[0007] Sending the target quantity, the post-processing type and the associated configuration information to the supercomputing cluster, so that the supercomputing cluster reads the corresponding simulation data from the matched simulation data file in parallel according to the target quantity, and performs compression processing on the read simulation data according to the post-processing type to obtain rendering data, if the data quantity of the rendering data is less than a threshold, the rendering data is sent to the client, if the data quantity of the rendering data is greater than or equal to the threshold, the rendering data is rendered according to the post-processing type and the associated configuration information to obtain graphic rendering data, and the graphic rendering data is sent to the client;

[0008] receive the to-be-rendered data sent by the supercomputing cluster, and render and display the to-be-rendered data according to the post-processing type and the associated configuration information; or receive the graphic rendering data sent by the supercomputing cluster, and display the graphic rendering data.

[0009] In a second aspect, the embodiments of the present application further provide a remote collaborative post-processing method for supercomputing, applied to a supercomputing cluster, including: receiving a simulation data file identifier sent by a client, reading header metadata from a matched simulation data file according to the identifier, and sending the read header metadata to the client to enable the client to determine a target quantity participating in post-processing according to the header metadata;

[0010] receiving the target quantity, the post-processing type, and the associated configuration information sent by the client, and reading corresponding simulation data from the matched simulation data file in parallel according to the target quantity through a computing task;

[0011] performing compression processing on the read simulation data according to the post-processing type to obtain to-be-rendered data;

[0012] if the data quantity of the to-be-rendered data is less than a threshold, sending the to-be-rendered data to the client, and if the data quantity of the to-be-rendered data is greater than or equal to the threshold, rendering the to-be-rendered data according to the post-processing type and the associated configuration information to obtain graphic rendering data, and sending the graphic rendering data to the client.

[0013] In a third aspect, the embodiments of the present application further provide a remote collaborative post-processing device for supercomputing, integrated in a client, including:

[0014] a first determination module configured to determine a target quantity participating in post-processing, a post-processing type, and associated configuration information;

[0015] a first sending module configured to send the target quantity, the post-processing type, and the associated configuration information to a supercomputing cluster to enable the supercomputing cluster to read corresponding simulation data from a matched simulation data file in parallel according to the target quantity, perform compression processing on the read simulation data according to the post-processing type to obtain to-be-rendered data, send the to-be-rendered data to the client if the data quantity of the to-be-rendered data is less than a threshold, and render the to-be-rendered data according to the post-processing type and the associated configuration information to obtain graphic rendering data if the data quantity of the to-be-rendered data is greater than or equal to the threshold, and send the graphic rendering data to the client;

[0016] a first receiving module configured to receive to-be-rendered data sent by the supercomputing cluster, and render and display the to-be-rendered data according to the post-processing type and the associated configuration information; or receive graphic rendering data sent by the supercomputing cluster, and display the graphic rendering data.

[0017] In a fourth aspect, the embodiments of the present application further provide a remote collaborative post-processing device for supercomputing, integrated in a supercomputing cluster, comprising:

[0018] The second receiving module is configured to receive the simulation data file identifier sent by the client, read the header metadata from the matched simulation data file according to the identifier, and send the read header metadata to the client, so that the client determines the target quantity participating in the post-processing according to the header metadata.

[0019] The third receiving module is configured to receive the target quantity, the post-processing type and the associated configuration information sent by the client, and read the corresponding simulation data from the matched simulation data file according to the target quantity in parallel through the computing task.

[0020] The compression processing module is configured to perform compression processing on the read simulation data according to the post-processing type, to obtain the to-be-rendered data.

[0021] The second sending module is configured to send the to-be-rendered data to the client if the data quantity of the to-be-rendered data is less than the threshold, and perform rendering on the to-be-rendered data according to the post-processing type and the associated configuration information to obtain the graphic rendering data and send the graphic rendering data to the client if the data quantity of the to-be-rendered data is greater than or equal to the threshold.

[0022] In a fifth aspect, the embodiments of the present application further provide an electronic device, which comprises:

[0023] a processor and a memory;

[0024] The processor is configured to execute the steps of the remote collaborative post-processing method for supercomputing according to any one of the embodiments by calling the programs or instructions stored in the memory.

[0025] In a sixth aspect, the embodiments of the present application further provide a computer readable storage medium, which stores programs or instructions, and the programs or instructions make the computer execute the steps of the remote collaborative post-processing method for supercomputing according to any one of the embodiments.

[0026] In summary, the application provides a remote collaborative post-processing method for supercomputing. The client communicates with the supercomputing cluster. The target quantity for participating in post-processing, the post-processing type, and the associated configuration information are determined by the client and sent to the supercomputing cluster. The supercomputing cluster reads the corresponding simulation data in parallel from the matched simulation data file according to the target quantity, which can improve the reading efficiency of the simulation data. The read simulation data is compressed to obtain the to-be-rendered data. If the data quantity of the to-be-rendered data is less than a threshold, the to-be-rendered data is sent to the client. If the data quantity of the to-be-rendered data is greater than or equal to the threshold, the to-be-rendered data is rendered according to the post-processing type and the associated configuration information to obtain the graphic rendering data, and the graphic rendering data is sent to the client. The client receives the to-be-rendered data sent by the supercomputing cluster and renders and displays the to-be-rendered data according to the post-processing type and the associated configuration information. Alternatively, the client receives the graphic rendering data sent by the supercomputing cluster and displays the graphic rendering data. The problems of long download time of super-large-scale simulation data, insufficient local processing and storage capacity, and unsmooth interface display are solved, and the user can smoothly and automatically use supercomputing resources for efficient remote post-processing and visualization on the local. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is a flow of a remote collaborative post-processing method for supercomputing provided by an embodiment of the application Figure 1 ;

[0028] Figure 2 is a flow of a remote collaborative post-processing method for supercomputing provided by an embodiment of the application Figure 2 ;

[0029] Figure 3 is a schematic diagram of a remote collaborative post-processing architecture for supercomputing provided by an embodiment of the application

[0030] Figure 4 is a schematic diagram of the structure of an electronic device provided by an embodiment of the application DETAILED DESCRIPTION

[0031] The application will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. In addition, it should be noted that only parts related to the application are shown in the drawings for ease of description.

[0032] It should be noted that the embodiments in the application and the features in the embodiments can be combined with each other without conflict. The application will be described in detail below with reference to the drawings and embodiments.

[0033] Embodiment one

[0034] Figure 1 is a flow chart of a remote collaborative post-processing method for supercomputing provided by an embodiment of the present application. The method is applied to a client and is suitable for a scenario of controlling post-processing of simulation data stored in a supercomputing cluster through the client, facilitating user operation and providing a smooth operation interactive experience for the user. Referring to Figure 1 , the remote collaborative post-processing method for supercomputing specifically includes the following steps:

[0035] S110, determining target quantities participating in post-processing, post-processing types, and associated configuration information.

[0036] Different engineering simulations (such as finite element analysis, computational fluid dynamics, multi-physical field simulation, etc.) produce different categories of simulation data. Further, different post-processing types use different categories of simulation data.

[0037] For example, the target quantities participating in post-processing include but are not limited to temperature, pressure, stress, resistance, velocity, and magnetic field strength, etc. The user can flexibly set according to research needs.

[0038] The post-processing type can also refer to the post-processing method, including but not limited to: cloud chart, slice, section, vector arrow, streamline, scatter point, volume rendering, etc.

[0039] The essence of "cloud chart" is to display the distribution of a certain physical quantity on the model surface or cross section with a color gradient. The applicable scenarios include scenarios that need to quickly observe the global distribution trend of the physical quantity, such as wing surface pressure cloud chart.

[0040] The essence of "slice and section" is to cut a three-dimensional model through a virtual plane (slice) or curved surface (section) to display the distribution of physical quantities in the cross section. The applicable scenarios include scenarios that need to observe the internal physical quantity changes of the model, such as engine cylinder internal temperature slice, which can view the temperature gradient in the center of the combustion chamber.

[0041] "Vector arrow" refers to indicating the direction and size of a vector physical quantity through an arrow. The applicable scenarios include scenarios that need to analyze the direction, such as the velocity vector diagram of fluid flowing around an obstacle.

[0042] "Streamline" refers to a curve drawn along a vector field (such as flow velocity field), and the tangent direction of each point on the curve is consistent with the vector direction of the point. It is commonly used to represent the motion trajectory of fluid or field, such as the airflow streamline diagram of the tail of a car.

[0043] The essence of "scatter point" is to display the physical quantity values of discrete points in the form of points, and the color and size of the points can map the physical quantity, such as the stress scatter point diagram of the key bolts of a bridge.

[0044] The essence of volume rendering is to visualize the entire volume data of a three-dimensional model (rather than the surface or cross section), and to display the spatial distribution of internal physical quantities (such as the three-dimensional form of smoke, flame, and electromagnetic field) through transparency and color mapping, for example, the volume rendering of the neutron flux in a nuclear reactor can show the three-dimensional distribution of radiation intensity.

[0045] The associated configuration information includes the position, angle, range, color of the cloud chart, the color, range, and transparency of the volume rendering, etc.

[0046] In some embodiments, the user can select the target quantity participating in post-processing from the physical quantities stored in the client; experienced users can also input the target quantity participating in post-processing by themselves; and the target quantity can also be selected from the header metadata extracted from the simulation data file returned by the supercomputing cluster, which is more targeted, more accurate, and less prone to errors.

[0047] S120, send the target quantity, the post-processing type, and the associated configuration information to the supercomputing cluster.

[0048] Specifically, the interaction data between the client and the supercomputing cluster can be transmitted through an SSH tunnel to increase the security of the data.

[0049] Wherein, after receiving the target quantity, the post-processing type, and the associated configuration information, the supercomputing cluster reads the corresponding simulation data in parallel from the matched simulation data file according to the target quantity. The supercomputing cluster has the ability to process large-scale data, so that the reading efficiency of the simulation data can be improved by reading the corresponding simulation data in parallel through the supercomputing cluster.

[0050] Further, the supercomputing cluster compresses the read simulation data through a specific algorithm to further reduce the data volume, balance the relationship between the data volume and the accuracy and rendering effect, and eliminate part of the data (such as redundant data, data with small differences between each other, etc.) under the premise of ensuring a certain accuracy, to obtain the to-be-rendered data.

[0051] The compression processing of the read simulation data includes spatial domain compression (including mesh simplification and adaptive sampling, reducing data volume), wherein the principle of mesh simplification is to reduce the number of vertices and elements of the mesh through an algorithm (such as edge folding and vertex clustering), while preserving the overall shape and key features of the model, for example, simplifying a mesh with 100 million vertices to 10 million vertices, reducing the data volume by 90%.

[0052] If the data amount of the to-be-rendered data is less than the threshold value, the supercomputing cluster sends the to-be-rendered data to the client, and the client is responsible for rendering and display. Since the data amount is not large, the transmission time is generally acceptable at this time, and the processing capability required for rendering is matched with the capability of the client, so that a smooth post-processing effect (mainly referring to the smoothness of the interaction when displaying the post-processed graphics, scaling graphics, rotating graphics, and changing the display angle of graphics) can be obtained.

[0053] If the data amount of the to-be-rendered data is greater than or equal to the threshold value, in order to avoid large transmission time and avoid the processing capability required for rendering exceeding the capability range of the client, resulting in problems such as rendering and display lag and unsmoothness, in this case, the supercomputing cluster is fully utilized, and the to-be-rendered data is rendered according to the post-processing type and associated configuration information to obtain graphics rendering data, and the graphics rendering data is sent to the client for display, that is, the client is not responsible for rendering operation, but only for display, so as to balance the computing capability of the client and ensure smooth display effect.

[0054] In particular, in order to further ensure small transmission delay, the supercomputing cluster can render the to-be-rendered data through scalable rendering technology, so as to ensure that the obtained graphics rendering data is lightweight and convenient for transmission.

[0055] S130, receiving the to-be-rendered data sent by the supercomputing cluster, and rendering and displaying the to-be-rendered data according to the post-processing type and associated configuration information; or, receiving the graphics rendering data sent by the supercomputing cluster, and displaying the graphics rendering data.

[0056] The supercomputing-oriented remote collaborative post-processing method provided by the embodiments of the application is operated by a user on a local client, and all responses of the graphical user interface are completed locally without any network delay, realizing a smooth and smooth interactive experience like operating local software, and completely solving the lag problem of the traditional remote desktop solution. The supercomputing resources are fully utilized, and the post-processing and rendering and other computationally intensive tasks are executed on the supercomputing computing nodes, fully utilizing the powerful parallel computing capability, large memory and high-speed storage of the supercomputing, and being able to process super-large-scale data sets that cannot be handled by a local workstation.

[0057] Embodiment two

[0058] On the basis of the above embodiment, the present embodiment gives an optional implementation of step S110, specifically, determining the target amount participating in post-processing, comprising: receiving the head metadata sent by the super calculation cluster, and displaying the head metadata, wherein the head metadata is read by the super calculation cluster from the matched simulation data file, and the matched simulation data file is determined by the super calculation cluster according to the identification (including the storage path and the file name of the file) sent by the client; receiving the target amount participating in post-processing selected by the user from the head metadata. Since the head metadata is read by the super calculation cluster from the matched simulation data file, the target amount participating in post-processing selected from the head metadata must be the data amount existing in the simulation data file, and the accuracy of the target amount determined by this method is higher, and the target amount setting error is less likely to occur.

[0059] Specifically, the head information of the simulation data file is read by the data server deployed in the super calculation cluster, the metadata such as mesh, scalar, time step, etc. contained in the simulation data file is parsed, and these lightweight metadata are sent back to the client, realizing the rapid extraction of the head metadata of the simulation data file.

[0060] Further, before determining the target amount participating in post-processing, it further comprises: receiving the target information set by the user, the target information is information related to the resources of the super calculation cluster; sending the target information to the super calculation cluster; receiving the storage path information of the simulation data file returned by the super calculation cluster, and displaying the storage path information, wherein the super calculation cluster determines the storage path information according to the target information when receiving the target information, usually different users have different account and password, and there are simulation data files belonging to the user under the account, the storage path information of the simulation data file is located according to the account indicated by the target information, and is fed back to the client, so that the user can select the specific simulation data file (the file is only an identifier, and there is no real data in it, the real data is stored in the super calculation cluster, so the speed of interaction between the client and the super calculation cluster is fast and the efficiency is high) from the storage path information; receiving the selection instruction for the matched simulation data file identifier triggered by the user based on the storage path information, and sending the selected identifier to the super calculation cluster.

[0061] Wherein, the target information includes but is not limited to: login node address, account, password, node number, CPU core number, memory capacity, partition, etc. After successful login, the resource node and the calculation task are automatically started through the job scheduling adaptation module. The built-in job scheduling adaptation module encapsulates the complex Slurm command line operation behind the graphical interface, realizes the function of "one-key" submission, monitoring and termination of post-processing job, greatly reduces the use threshold of super calculation system, so that the user can focus on data analysis itself, rather than tedious system operation.

[0062] Then the storage path information of the simulation data file is acquired and sent to the client, the client displays the storage path, the user selects a specific simulation data file based on the displayed storage path, the simulation data file selected by the user is the matched simulation data file, the client sends the identifier of the corresponding file to the supercomputing cluster according to the selection operation of the user, the supercomputing cluster finds the real simulation data file according to the identifier, and reads the header metadata of the simulation data file and then sends it to the client for the user to select the target quantity participating in post-processing from it.

[0063] In some embodiments, a virtual storage path information can also be maintained on the client for the user to select the simulation data file. For example, a selection instruction for the matched simulation data file identifier triggered based on the virtual storage path information is received, and the selected identifier is sent to the supercomputing cluster; wherein the virtual storage path information is the same as the real storage path information, but there is no real file data under the virtual storage path information.

[0064] Embodiment three

[0065] On the basis of the above-mentioned embodiments, referring to a remote collaborative post-processing method for supercomputing as shown in Figure 2 The method is applied to a supercomputing cluster and cooperates with the client in the above-mentioned embodiments, as shown in Figure 2 The method includes the following steps:

[0066] S210, receiving the simulation data file identifier sent by the client, reading the header metadata from the matched simulation data file according to the identifier, and sending the read header metadata to the client to enable the client to determine the target quantity participating in post-processing according to the header metadata.

[0067] Before receiving the simulation data file identifier sent by the client, it further includes: in response to receiving the target information sent by the client, determining the storage path information of the simulation data file according to the target information, and starting the matched computing task; sending the storage path information to the client to enable the user to select the simulation data file identifier according to the storage path information.

[0068] S220, receiving the target quantity, post-processing type and associated configuration information sent by the client, and reading the corresponding simulation data from the matched simulation data file in parallel through the computing task according to the target quantity.

[0069] Specifically, the simulation data can be read in parallel by multiple nodes of the supercomputing cluster to improve the reading efficiency.

[0070] S230, compressing the read simulation data to obtain the to-be-rendered data.

[0071] The simulation data read by the multiple nodes of the supercomputing cluster can be processed in parallel for compression, and the data obtained through parallel compression can be integrated to obtain the final rendering data, so as to improve the compression processing efficiency.

[0072] S240, if the data amount of the rendering data is less than the threshold value, the rendering data is sent to the client, and if the data amount of the rendering data is greater than or equal to the threshold value, the rendering data is rendered according to the post-processing type and the associated configuration information to obtain the graphic rendering data, and the graphic rendering data is sent to the client.

[0073] Similarly, the rendering data can be rendered in parallel by the multiple nodes of the supercomputing cluster, and the local rendering data obtained through parallel rendering can be integrated together to obtain the entire graphic rendering data, so as to improve the rendering efficiency.

[0074] Embodiment Four

[0075] Based on the same inventive concept, the embodiment provides a remote collaborative post-processing architecture diagram for supercomputing, as shown in Figure 3 The remote collaborative post-processing architecture diagram for supercomputing includes a client module 310 deployed on a user's local computer, a server module 320 deployed on a supercomputing node, and a job scheduling adaptation and tunnel communication module 330 for automatically connecting and managing the client module 310 and the server module 320. Figure 3 (not shown).

[0076] The client module 310 includes a user operation interface 311 and a rendering window 312. The user operation interface 311 provides menus, buttons, dialog boxes, etc., so that the user can select files, add physical quantities, set colors, adjust viewing angles, etc. The rendering window 312 is responsible for receiving and displaying the final rendering result (graphics or simplified rendering data). The client module 310 is deployed on the user's local computer (such as a personal workstation, a notebook computer). Its core responsibility is to provide a lightweight and responsive graphical user interface, and is responsible for receiving user's interactive instructions (such as data loading, viewing angle transformation, rendering parameter adjustment, etc.). Through the job scheduling adaptation and tunnel communication module, a job request for starting the server program is submitted to the supercomputing job management system (such as Slurm).

[0077] The server module 320 mainly includes three parts:

[0078] Component Launcher 321: Component Launcher 321 typically starts on a remote cluster and communicates with the local client via an SSH tunnel. When other components need to be started, the Component Launcher is responsible for invoking and executing them, including the startup process of the parallel computing engine. When starting the parallel computing engine, Component Launcher 321 executes parallel commands with specific parameters indicating how to start the engine in parallel, and it also maintains the connection between the local client and the cluster, avoiding frequent password entries when starting remote components.

[0079] Data Server 322: When a client requests a connection to a remote cluster, the component initiator service starts the data server. The data server is responsible for reading the header information of the data file, parsing out the metadata contained in the file, such as mesh, scalars, and timesteps, and sending this lightweight metadata back to the client, realizing the rapid extraction of metadata from the header of the simulation data file.

[0080] Rendering Calculation Engine 323: When the client issues rendering commands (such as slicing or isosurfaces), the component launcher controls the startup of Rendering Calculation Engine 323. Rendering Calculation Engine 323 supports large-scale parallel computing and can be started in parallel on a supercomputer according to user settings. Each parallel engine process is responsible for reading and processing a portion of the data file. The rendering calculation engine not only needs to read and process data but also performs rendering, employing parallel rendering technology to perform parallel rendering and compositing the local images or simplified rendering data generated by each process into a complete image or simplified rendering data. After the calculation is complete, the engine sends the final rendering result (usually a compressed image) back to the client's rendering window for display.

[0081] General, based on Figure 3 The architecture diagram shown generally includes the following processes: 1. Communication connection; 2. Reading header metadata; 3. Returning header metadata; 4. Sending post-processing instructions; 5. Reading data and rendering; 6. Sending graphics rendering data.

[0082] The above process enables data scaling, transforming massive simulation data into lightweight images or simplified rendering data through a rendering engine, thus laying the foundation for fast and efficient interaction.

[0083] The job scheduling adaptation and tunnel communication module serves as a bridge connecting the client and the server, and is the key to realize automation and efficient communication. Its core functions include: SSH tunnel transmission function, after the client submits a job and the job is allocated to a computing node by the scheduling system, a secure SSH tunnel is automatically established between the local client and the specific computing node, which is used to transmit post-processing instructions from the client to the server, as well as file information or rendering results from the server to the client, so that the data exchange volume is much smaller than the original simulation data itself and the data volume of the remote desktop pixel stream, thereby ensuring the transmission efficiency and the smoothness of interaction. Job scheduling adaptation can automatically apply for and manage supercomputing resources according to the resource requirements (such as node number, computing time, required queue, etc.) set by the user on the GUI. The client applies for computing resources through SSH connection, monitors the computing job status (queuing, running, completion), and automatically cancels the job and releases the resources at the end of the session.

[0084] The specific implementation steps include:

[0085] Step 1: Client module initialization, initialize the client module, set supercomputing connection information, including but not limited to login node address, account, password, and supercomputing performance information, including but not limited to node number, CPU core number, memory capacity, partition, and path of computing engine, etc.

[0086] Step 2: Open simulation data. Step 2.1: According to the supercomputing connection information set in step 1, establish a connection between the client module and the server module through the job scheduling adaptation and tunnel communication module, i.e. connect the local and supercomputing, start the computing task of the server module on the supercomputing, and return whether the task is started, if not successfully started, provide response error information, including but not limited to account password error, cluster partition error, etc. Step 2.2: The client module issues an instruction to open the simulation data, and the server module returns the file path information after receiving the instruction. Step 2.3: The client module sends the simulation file that needs to be opened according to the file path information, which can be a single file, a plurality of files in time sequence, simulation files of different regions at the same time step, or simulation files of different regions in time sequence. Step 2.4: The server module opens the specified simulation file in parallel. Step 2.5: The server module reads the header metadata in the simulation file, including but not limited to physical quantity name, type, grid number, type, and sends the information to the client module.

[0087] Step three: post-processing operation. Step 3.1: the client module selects the physical quantity that needs to be post-processed according to the simulation file header metadata. Step 3.2: the client module selects the post-processing type that needs to be performed, including but not limited to cloud chart, slice, section, vector arrow, streamline, scatter point, and volume rendering. Step 3.3: the client module sends the post-processed physical quantity and type, as well as the parameter information related to the post-processing type (such as the position and angle of the section, the range and color of the cloud chart, the color, range, and transparency of the volume rendering, etc.) to the server module. Step 3.4: the server module reads the relevant data in the simulation file and performs analysis and processing according to the post-processing requirements provided by the client module to obtain the data that needs to be rendered and displayed, i.e., simplified rendering data.

[0088] Step four: rendering and display. Rendering and display includes two ways, one is server module rendering, and the other is client module rendering. When the rendering display data obtained in step 3.4 is small, the client module rendering can be used, and when the data is large, the server module rendering is used. Step 4.1: server module rendering, the server module directly performs off-screen rendering on the data in step 3.4, obtains the rendered graphics, and sends it to the client module for display. Since this method only transmits the rendered graphics information, the rendering process is in the server module, so it is not limited by the size of the data to be rendered. Step 4.2: client module rendering, the server module compresses the data in step 3.4 and directly sends it to the client module, which renders and displays the data. Since this method needs to transfer the data to be rendered from the server module to the client module, the data volume cannot be too high, otherwise the transmission time will be long, and it is limited by the performance of the local environment of the client, and cannot render too large data volume.

[0089] With the post-processing and visualization of a specific example as an example, the execution flow of the above scheme is described. First, step one is executed, and a client is started on a local computer, which includes a user operation interface and a rendering window. A Tianhe new generation supercomputer is used, and relevant information is configured on the user operation interface. Then, step two is executed, and according to the information set in step one, a connection between the client module and the server module is established through the tunnel communication and the job scheduling adaptation module, that is, the local and the supercomputer are connected, and a computing task is started on the supercomputer. According to the demand, the computing task uses 2000 computing nodes, each of which starts 50 MPI processes, a total of 100000 MPI processes, supporting a hundred billion grid parallel rendering. Then the client module sends an instruction to open the simulation data, and the server module returns the file path information after receiving the instruction. The client module sends the simulation file to be opened according to the file path information, and the simulation file in this example is a simulation file of different regions in time sequence. Then, the server module opens the specified simulation file in a parallel reading manner, reads the header metadata in the simulation file, and sends the information to the client module.

[0090] Then step three is performed, and the simulation file post-processing list is browsed on the user operation interface. In order to better display the simulation results, a multi-physical quantity volume rendering method is used, including the component mass fraction and temperature of different gases, different colors are used, and appropriate transparency is set. At the same time, the grid boundary is displayed as a slice to show the complex geometric boundary of the model. The above-mentioned post-processing related information is sent to the server module, and the server module reads the related data in the simulation file according to the information and performs compression processing to obtain the data needed for rendering display, that is, simplified rendering data.

[0091] Then step four is performed, and since the data amount of the simplified rendering data obtained in step three is large and inconvenient to transmit, the server module rendering method is used. The server module directly performs off-screen rendering on the simplified rendering data, and sends the rendering graphics data to the client module for display after obtaining the rendering graphics data.

[0092] DETAILED DESCRIPTION: With the post-processing and visualization of a specific example as an example.

[0093] First, step one is executed, and a client is started on a local computer, which includes a user operation interface and a rendering window. A Tianhe new generation supercomputer is used, and relevant information is configured on the user operation interface.

[0094] Then step two is performed, according to the supercomputer information set in step one, through the tunnel communication and the job scheduling adaptation module, the connection between the client module and the server module is established, that is, the local and the supercomputer are connected, and at the same time the calculation task is started on the supercomputer, according to the demand, the calculation task adopts 2000 computing nodes, each computing node starts 50 MPI processes, a total of 100000 MPI processes, supporting hundreds of billions of grid parallel rendering. Then the client module sends the instruction to open the simulation data, and the server module returns the file path information after accepting the instruction. The client module sends the simulation file that needs to be opened according to the file path information, and the simulation file of this example is the simulation file of different regions in time sequence. Then, the server module opens the specified simulation file in parallel reading mode, reads the header metadata in the simulation file, including density, pressure, temperature, component mass fraction, and non-structured grid type, and sends the information to the client module.

[0095] Then step three is performed, after browsing the simulation file post-processing list in the user operation interface, in order to better display the simulation results, the multi-physical quantity volume rendering mode is adopted, including the component mass fraction and temperature of different gases, different colors are used, and the appropriate transparency is set, and the grid boundary is displayed as a slice to show the complex geometric boundary of the model. The above post-processing information is sent to the server module, and the server module reads the related data in the simulation file according to the information and analyzes and processes to obtain the data needed for rendering display, that is, the simplified rendering data.

[0096] Then step four is performed, because the data amount of the simplified rendering data obtained in step three is large and is not convenient for transmission, the server module rendering mode is adopted. The server module directly performs off-screen rendering on the simplified rendering data, and sends the rendered image to the client module for display. According to the actual measurement, the cumulative time of rendering and display is 120s.

[0097] The scheme decomposes the "integrated" architecture of traditional post-processing software into a lightweight interactive front end (client) and a large-scale computing and rendering back end (server). The server and the massive data are located in the supercomputer, realizing the high coordination of data, calculation and rendering, and eliminating the bottleneck of data transmission. The client only processes UI operation, interactive logic and result image display, so as to ensure the smoothness of user interaction and not be affected by network delay fluctuation. This architecture solves the lag problem caused by the transmission of large desktop image stream in traditional remote desktop.

[0098] Deep native integration and automatic management of supercomputing job scheduling system, the client has an interface adapter with mainstream supercomputing job scheduling systems (such as Slurm, PBS, etc.). After the user sets the required computing resources (such as the number of nodes, the number of cores, and the running time) on the graphical interface of the client, the computing resources can be automatically applied and managed. The client automatically submits the server program to the supercomputing cluster as a computing task, and the scheduling system allocates idle computing nodes to it. The client can query the job status and log information in real time, and can automatically send instructions to cancel the job and release the computing resources when the task is completed. This realizes the on-demand use and elastic allocation of resources, and avoids the disadvantages of the traditional method of abusing the login node.

[0099] Efficient and secure data lightweight and transmission, after the server successfully starts on the computing node through the supercomputing scheduling system, it actively reconnects the locally started client and establishes an encrypted, point-to-point dedicated communication channel.

[0100] Simplified data transmission, the content transmitted in this channel is not the original three-dimensional model or volume data, but lightweight control commands, two-dimensional images and simplified data. The amount of transmission data is reduced by several orders of magnitude compared to the original data. Because lightweight data is transmitted, fast and efficient data interaction can be achieved.

[0101] Large-scale parallel rendering, the server of the present solution supports parallel extension, based on the MPI parallel method, through parallel I / O reading data blocks and CPU parallel rendering algorithm, large-scale parallel rendering is realized. The problem of insufficient computing power of a single computing node and slow rendering is solved, and the powerful computing power of the supercomputing CPU cluster can be maximally utilized to realize real-time interactive rendering of super large-scale data.

[0102] It should be noted that, unless otherwise specified, all operations in the server module are parallel to improve efficiency.

[0103] Figure 4 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. As shown in Figure 4 The electronic device 500 includes one or more processors 501 and a memory 502.

[0104] The processor 501 can be a central processing unit (CPU) or other forms of processing units with data processing and / or instruction execution capabilities, and can control other components in the electronic device 500 to perform desired functions.

[0105] The memory 502 can include one or more computer program products that can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory, for example, can include random access memory (RAM), and / or a cache, etc. The non-volatile memory, for example, can include read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions can be stored on the computer-readable storage media, and the processor 501 can execute the program instructions to implement the supercomputer-oriented remote collaborative post-processing method of any embodiment of the present application described above and / or other desired functions. Various contents such as initial external parameters, threshold values, etc. can also be stored in the computer-readable storage media.

[0106] In one example, the electronic device 500 can further include an input device 503 and an output device 504, which are interconnected through a bus system and / or other forms of connection mechanism (not shown). The input device 503 can include, for example, a keyboard, a mouse, etc. The output device 504 can output various information to the outside, including pre-warning prompt information, braking force, etc. The output device 504 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0107] Of course, in order to simplify, Figure 4 Only some of the components in the electronic device 500 related to the present application are shown in the figure, and components such as buses, input / output interfaces, etc. are omitted. In addition, the electronic device 500 can also include any other appropriate components according to specific application cases.

[0108] In addition to the above method and device, the embodiments of the present application can also be a computer program product, which includes computer program instructions that make the processor execute the steps of the supercomputer-oriented remote collaborative post-processing method provided by any embodiment of the present application when the processor runs.

[0109] The computer program product can be written in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0110] In addition, the embodiments of the present application can also be a computer readable storage medium, which stores computer program instructions, and the computer program instructions make the processor execute the steps of the supercomputer-oriented remote collaborative post-processing method provided by any embodiment of the present application when the processor runs.

[0111] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0112] It should be noted that the terms used in the present application are only for describing specific embodiments, and are not intended to limit the scope of the present application. As shown in the specification and claims of the present application, unless the context clearly indicates otherwise, "one", "a", "an", and / or "the" do not refer to the singular, but also include the plural. The terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method or device including the element.

[0113] It should also be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", and the like indicate the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. Unless otherwise explicitly specified and limited, the terms "mount", "connect", "connect" and the like should be broadly understood, for example, it can be a fixed connection, or a detachable connection, or an integral connection; it can be a mechanical connection, or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or it can be the communication between two elements inside. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0114] The principles and implementation manners of the present application are described herein by using specific examples, and the above example descriptions are only used to help understand the method and core idea of the present application. The above is only the preferred implementation manner of the present application. It should be noted that, due to the limited nature of the language expression, there are objectively infinite specific structures, and for ordinary skilled persons in the technical field, some improvements, refinements or changes can be made without departing from the principles of the present application, or the above technical features can be combined in an appropriate manner; the improvements, refinements, changes or combinations, or the direct application of the inventive concept and technical solution to other occasions without improvement, shall be regarded as the protection scope of the present application.

Claims

1. A remote collaborative post-processing method for supercomputing, applied to a client side, characterized in that: include: Determine the target quantity, post-processing type, and associated configuration information for participation in post-processing; The target quantity, the post-processing type, and the associated configuration information are sent to the supercomputing cluster, so that the supercomputing cluster reads the corresponding simulation data in parallel from the matching simulation data file according to the target quantity, compresses the read simulation data, and obtains the data to be rendered. If the data volume of the data to be rendered is less than a threshold, the data to be rendered is sent to the client. If the data volume of the data to be rendered is greater than or equal to the threshold, the data to be rendered is rendered according to the post-processing type and the associated configuration information to obtain graphics rendering data, and the graphics rendering data is sent to the client. The system receives the data to be rendered sent by the supercomputing cluster and renders and displays the data according to the post-processing type and the associated configuration information; or, it receives the graphics rendering data sent by the supercomputing cluster and displays the graphics rendering data.

2. The method according to claim 1, characterized in that, The determination of the target quantity to participate in post-processing includes: The supercomputing cluster receives header metadata sent by the supercomputing cluster and displays the header metadata, wherein the header metadata is read by the supercomputing cluster from a matching simulation data file, and the matching simulation data file is determined by the supercomputing cluster based on the identifier sent by the client. Receive the target quantity selected by the user from the header metadata to participate in post-processing.

3. The method according to claim 2, characterized in that, Before determining the target quantity to participate in post-processing, the process also includes: Receive target information set by the user, wherein the target information is information related to the resources of the supercomputing cluster; The target information is sent to the supercomputing cluster; The system receives and displays the storage path information of the simulation data file returned by the supercomputing cluster, wherein the supercomputing cluster determines the storage path information based on the target information when it receives the target information. Receive a selection instruction for the matching simulation data file identifier triggered based on the storage path information, and send the selected identifier to the supercomputing cluster.

4. The method according to claim 2, characterized in that, Before determining the target quantity to participate in post-processing, the process also includes: The system receives a selection instruction for the matched simulation data file identifier triggered by virtual storage path information, and sends the selected identifier to the supercomputing cluster; wherein the virtual storage path information is the same as the real storage path information, but there is no real file data under the virtual storage path information.

5. A remote collaborative post-processing method for supercomputing, applied to a supercomputing cluster, executed in conjunction with the method described in any one of claims 1-4, characterized in that, include: The system receives a simulation data file identifier sent by the client, reads header metadata from the matching simulation data file based on the identifier, and sends the read header metadata to the client so that the client can determine the target quantity to participate in post-processing based on the header metadata. The system receives the target quantity, post-processing type, and associated configuration information sent by the client, and reads the corresponding simulation data from the matching simulation data file in parallel according to the target quantity through the calculation task. The read simulation data is compressed to obtain the data to be rendered; If the amount of data to be rendered is less than the threshold, the data to be rendered is sent to the client. If the amount of data to be rendered is greater than or equal to the threshold, the data to be rendered is rendered according to the post-processing type and the associated configuration information to obtain graphics rendering data, and the graphics rendering data is sent to the client.

6. The method according to claim 5, characterized in that, Before the simulation data file identifier sent by the receiving client, it also includes: In response to receiving the target information sent by the client, the storage path information of the simulation data file is determined according to the target information, and a matching calculation task is started; The storage path information is sent to the client so that the user can select the simulation data file identifier based on the storage path information.

7. A remote collaborative post-processing device for supercomputing, integrated into a client, characterized in that: include: The first determination module is used to determine the target quantity, post-processing type, and associated configuration information for participating in post-processing. The first sending module is used to send the target quantity, the post-processing type, and the associated configuration information to the supercomputing cluster, so that the supercomputing cluster can read the corresponding simulation data in parallel from the matching simulation data file according to the target quantity, compress the read simulation data to obtain the data to be rendered, if the data volume of the data to be rendered is less than a threshold, the data to be rendered is sent to the client, if the data volume of the data to be rendered is greater than or equal to the threshold, the data to be rendered is rendered according to the post-processing type and the associated configuration information to obtain graphics rendering data, and the graphics rendering data is sent to the client. The first receiving module is used to receive the data to be rendered sent by the supercomputing cluster, and render and display the data to be rendered according to the post-processing type and associated configuration information; or, to receive the graphics rendering data sent by the supercomputing cluster and display the graphics rendering data.

8. A remote collaborative post-processing device for supercomputing, integrated into a supercomputing cluster, characterized in that, include: The second receiving module is used to receive the simulation data file identifier sent by the client, read the header metadata from the matching simulation data file according to the identifier, and send the read header metadata to the client so that the client can determine the target quantity to participate in post-processing based on the header metadata. The third receiving module is used to receive the target quantity, post-processing type and associated configuration information sent by the client, and to read the corresponding simulation data in parallel from the matching simulation data file according to the target quantity through the calculation task. The compression processing module is used to compress the read simulation data to obtain the data to be rendered; The second sending module is used to send the data to be rendered to the client if the amount of data to be rendered is less than a threshold, and to render the data to be rendered according to the post-processing type and associated configuration information to obtain graphics rendering data, and send the graphics rendering data to the client if the amount of data to be rendered is greater than or equal to the threshold.

9. An electronic device, characterized in that, The electronic device includes: Processor and memory; The processor executes the steps of the remote collaborative post-processing method for supercomputing as described in any one of claims 1 to 6 by calling the program or instructions stored in the memory.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program or instructions that cause a computer to perform the steps of the remote collaborative post-processing method for supercomputing as described in any one of claims 1 to 6.

Images