Novel cross-physical cluster visualization method

Abstract

The invention discloses a novel cross-physical cluster visualization method, and relates to the field of high-performance computing, and the method comprises the steps: S1, enabling a local cluster management node to be in communication connection with a remote cluster management node through an independent network line; s2, initiating a visual job application at a local end; s3, the remote cluster management node allocates nodes after receiving the application, and a visualization function module executes visualization operation; s4, the protocol forwarding module converts a graphic access protocol used by the operation into a protocol suitable for browser access; and S5, integrating the job information by the local cluster, generating a link, opening the link, and accessing the visual job by using a browser. According to the novel cross-physical cluster visualization method provided by the invention, visual operation graphic display is realized through a graphic processing tool of a remote cluster computing node, a remote cluster management node converts a graphic access protocol into a browser applicable protocol, and a user can conveniently access through a browser through reverse proxy.

Description

Technical Field

[0001] The present invention relates to the field of high performance computing, and in particular to a novel visualization method across physical clusters. Background Art

[0002] In the field of high-performance computing (HPC), cluster scheduling plays a core role in resource optimization and task allocation. At present, the existing HPC cluster scheduling system has many defects in visual operation. In terms of operation mode, most of them rely on command line operation, which requires users to have professional knowledge and experience. For engineers who are accustomed to graphical user interfaces (GUIs), the learning cost is high and the operation is difficult, which seriously limits the expansion of the user group. At the same time, command line operation does not conform to the daily usage habits of most engineers, resulting in reduced work efficiency and poor user experience. In addition, the existing system generally lacks effective GUI support, making it difficult to give full play to the computing power of the cluster. In terms of new cluster access, new clusters often have physical isolation, and job submission mainly relies on command lines, which greatly increases the complexity of cluster management. In addition, if the GUI function is to be implemented, customers need to configure the environment separately, which not only brings additional workload, but also easily causes configuration errors, resulting in system instability. The problem is more prominent in the cross-physical isolation cluster environment. For visualization methods, existing methods are usually only applicable to personal configuration, lack effective integration and matching with cluster scheduling systems, have poor practicality in large-scale cluster job submission and management scenarios, cannot meet actual needs, and cause many inconveniences to user operations. In addition, in cluster environments with different hardware architectures, operating systems, and applications, existing visualization methods are difficult to adapt to diversity, further increasing the difficulty of implementing cross-cluster visualization management. Summary of the invention

[0003] In order to solve the problems of complex cross-physical cluster operations and poor stability in the prior art, the present invention provides a novel cross-physical cluster visualization method that can effectively improve the convenience of cross-cluster management.

[0004] To achieve the above object, the technical solution adopted by the present invention is:

[0005] The present invention provides a novel cross-physical cluster visualization method, which includes: S1: connecting a local cluster management node to a remote cluster management node through an independent network line, setting a visualization function module on the remote cluster computing node, setting a monitoring module for recording computing node information in the remote cluster environment, and setting a protocol forwarding module on the remote cluster management node for converting a graphic access protocol into a protocol suitable for browser access; S2: initiating a visualization job application to the remote cluster management node using a visualization interface at the local end, and sending the visualization job application to the remote cluster management node through a remote call service tool; S3: after receiving the application, the remote cluster management node allocates resources according to various job parameters and the actual resource status of each computing node. Node, the visualization function module executes the visualization job. After the job is started, the display number is specified through the graphic display protocol to be displayed on the corresponding remote display service; S4: The protocol forwarding module of the remote cluster management node is started by the remote cluster computing node through the remote call service tool. The protocol forwarding module converts the graphic access protocol used by the job into a protocol suitable for browser access, and registers the converted job information to the monitoring module; S5: The monitoring module of the remote cluster is monitored by the local cluster. When the local cluster detects the forwarded job information, it integrates the job information according to the preset rules and generates a link. When the local cluster opens the link, it forwards the access request to the remote cluster computing node according to the set rules, and then uses the browser to access the visualization job on the remote cluster computing node.

[0006] The novel cross-physical cluster visualization method provided by the present invention preferably comprises: a graphics presentation module for converting data into an intuitive graphics display according to system settings and operation requirements; a graphics projection module for projecting the graphical interface to a designated display device; and a graphics management module for optimizing and adjusting the resolution and layout of the graphics according to different application scenarios and user preferences.

[0007] The novel cross-physical cluster visualization method provided by the present invention preferably comprises the computing node information recorded by the monitoring module including the basic attributes, operation status and resource allocation status of the node, and the monitoring module performs real-time monitoring on the recorded computing node information.

[0008] The novel cross-physical cluster visualization method provided by the present invention preferably further includes a data synchronization module in step S1, which collects data of the local cluster and data of the remote cluster and synchronizes the data of the local cluster with the data of the remote cluster.

[0009] In the novel cross-physical cluster visualization method provided by the present invention, preferably, the visualization job application in step S2 includes detailed information of computing resources required for the job and a specific type of visualization application.

[0010] The above technical scheme has the following advantages or beneficial effects: A new type of cross-physical cluster visualization method provided by the present invention relates to the field of high-performance computing, and the method includes: S1: connecting the local cluster management node to the remote cluster management node through an independent network line, setting a visualization function module on the remote cluster computing node, setting a monitoring module for recording computing node information in the remote cluster environment, and setting a protocol forwarding module on the remote cluster management node for converting the graphic access protocol into a protocol suitable for browser access; S2: using the visualization interface on the local side to initiate a visualization job application to the remote cluster management node, and the visualization job application is sent to the remote cluster management node through a remote call service tool; S3: after receiving the application, the remote cluster management node The actual resource status allocation node is used, and the visualization function module executes the visualization job. After the job is started, the display number is specified through the graphic display protocol to display on the corresponding remote display service; S4: The protocol forwarding module of the remote cluster management node is started by the remote cluster computing node through the remote call service tool. The protocol forwarding module converts the graphic access protocol used by the job into a protocol suitable for browser access, and registers the converted job information to the monitoring module; S5: The monitoring module of the remote cluster is monitored by the local cluster. When the local cluster detects the forwarded job information, the job information is integrated and a link is generated according to the preset rules. When the local cluster opens the link, the access request is forwarded to the remote cluster computing node according to the set rules, and then the browser is used to access the visualization job on the remote cluster computing node. A new type of cross-physical cluster visualization method provided by the present invention realizes the visualization job graphic display through the graphic processing tool of the remote cluster computing node. The remote cluster management node converts the graphic access protocol into a browser-suitable protocol, and through the reverse proxy, the user can conveniently access it through the browser. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0012] Figure 1 This is a flow chart of a novel cross-physical cluster visualization method provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0013] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0014] Embodiment 1:

[0015] In the field of high-performance computing (HPC), cluster scheduling plays a core role in resource optimization and task allocation. At present, the existing HPC cluster scheduling system has many defects in visual operation. In terms of operation mode, most of them rely on command line operation, which requires users to have professional knowledge and experience. For engineers who are accustomed to graphical user interfaces (GUIs), the learning cost is high and the operation is difficult, which seriously limits the expansion of the user group. At the same time, command line operation does not conform to the daily usage habits of most engineers, resulting in reduced work efficiency and poor user experience. In addition, the existing system generally lacks effective GUI support, making it difficult to give full play to the computing power of the cluster. In terms of new cluster access, new clusters often have physical isolation, and job submission mainly relies on command lines, which greatly increases the complexity of cluster management. In addition, if the GUI function is to be implemented, customers need to configure the environment separately, which not only brings additional workload, but also easily causes configuration errors, resulting in system instability. The problem is more prominent in the cross-physical isolation cluster environment. For visualization methods, existing methods are usually only applicable to personal configuration, lack effective integration and matching with cluster scheduling systems, have poor practicality in large-scale cluster job submission and management scenarios, cannot meet actual needs, and cause many inconveniences to user operations. In addition, in cluster environments with different hardware architectures, operating systems, and applications, existing visualization methods are difficult to adapt to diversity, further increasing the difficulty of implementing cross-cluster visualization management.

[0016] In order to solve the problems of complex cross-physical cluster operations and poor stability in the prior art, the present invention provides a novel cross-physical cluster visualization method that can effectively improve the convenience of cross-cluster management.

[0017] like Figure 1 As shown:

[0018] The present invention provides a novel cross-physical cluster visualization method, which includes: S1: connecting a local cluster management node to a remote cluster management node through an independent network line, setting a visualization function module on the remote cluster computing node, setting a monitoring module for recording computing node information in the remote cluster environment, and setting a protocol forwarding module on the remote cluster management node for converting a graphic access protocol into a protocol suitable for browser access; S2: initiating a visualization job application to the remote cluster management node using a visualization interface at the local end, and sending the visualization job application to the remote cluster management node through a remote call service tool; S3: after receiving the application, the remote cluster management node allocates resources according to various job parameters and the actual resource status of each computing node. Node, the visualization function module executes the visualization job. After the job is started, the display number is specified through the graphic display protocol to be displayed on the corresponding remote display service; S4: The protocol forwarding module of the remote cluster management node is started by the remote cluster computing node through the remote call service tool. The protocol forwarding module converts the graphic access protocol used by the job into a protocol suitable for browser access, and registers the converted job information to the monitoring module; S5: The monitoring module of the remote cluster is monitored by the local cluster. When the local cluster detects the forwarded job information, it integrates the job information according to the preset rules and generates a link. When the local cluster opens the link, it forwards the access request to the remote cluster computing node according to the set rules, and then uses the browser to access the visualization job on the remote cluster computing node.

[0019] The specific process of the visualization method across physical clusters is as follows:

[0020] Set up system initialization, and use dedicated network lines or encrypted tunnel technology to establish a secure network communication connection between the local cluster management node and the remote cluster management node to ensure the stability and security of data transmission. Then deploy visualization function modules on the remote cluster computing node. The visualization function modules include graphics rendering module, graphics projection module and graphics management module; the graphics rendering module is used to realize the graphics rendering function of the visualization application, which can convert data into intuitive graphics display according to system settings and job requirements; the graphics projection module is used to project the graphical interface to the set display device task to ensure that the graphics can be accurately presented at the appropriate output end; the graphics management module is used to manage the display effect of the graphics, and can optimize and adjust the resolution, layout and other aspects of the graphics according to different application scenarios and user preferences. A monitoring module for recording computing node information is set in the remote cluster environment. The monitoring module records various key information of the computing node, such as the basic attributes, operating status, resource allocation of the node, etc., and monitors it in real time. In order to detect anomalies and potential problems in a timely manner. The core function of the protocol forwarding module is to convert a specific graphics access protocol into a protocol suitable for browser access, so that users can easily access remote visualization jobs with the help of common browsers.

[0021] Next, the process of visual job processing is to initiate a job application. The user initiates a visual job application on the local side using the visual interface. The visual job application includes detailed information on the computing resources required for the job and the specific type of visualization application. The visual job application sends a request to the remote cluster management node by using a remote call service tool (such as a tool based on the principle of network hooks). The application content covers detailed information on the computing resources required for the job (such as the number of CPU cores, memory capacity, storage size, etc.) and the specific type of visualization application (such as data visualization, scientific computing visualization, etc.). Then the job is scheduled and started. After receiving the application, the remote cluster management node allocates nodes to execute the visualization job based on the various parameters of the job and the actual resource status of each computing node (including idle computing resources, network bandwidth, etc.). After the job is started, the display number is specified with the help of the graphic display protocol to display on the corresponding remote display service, and the graphic display is optimized and managed through the desktop management tool. After the job scheduling and startup are completed, information registration begins. The remote computing node uses the remote call service tool to start the protocol forwarding module located on the remote management node. The protocol forwarding module converts the graphical interface access protocol used by the job into a protocol suitable for browser access, and registers the converted job-related information (including job number, network identifier, port number, and process number, etc.) into the monitoring module, which is a service registration monitoring tool. Finally, information acquisition and URL generation, as well as reverse proxy and access, the local cluster continuously monitors the service registration monitoring tool of the remote cluster. Once the forwarding information of the job (such as network identifier, port, job number, etc.) is obtained, the system will integrate and process this information according to the preset rules, thereby generating a unique specific link (URL), which contains all the key parameters required to access the remote visualization job. Reverse proxy and access is to forward the generated URL with the help of the reverse proxy tool. When the user clicks the URL in the local cluster, the reverse proxy tool will forward the user's access request to the corresponding remote computing node according to the established rules. Since the graphical interface access protocol has been converted into a protocol suitable for browser access through a protocol conversion tool, users do not need to use additional proxy software or perform complex configurations. They can directly use common browsers to conveniently access visualization jobs on remote computing nodes, thereby achieving efficient remote visualization operations.

[0022] Remote call service tools can be flexibly replaced by message queues, and use message queues (such as RabbitMQ, Kafka, etc.) to transmit remote call information. Message queues can realize asynchronous communication. The management node of the local cluster sends job application information to the message queue, and the management node of the remote cluster obtains and processes this information from the message queue. Graphic service tools can be implemented using commonly used remote desktops, such as VNC (Virtual Network Computing) and other RFB (Remote Frame Buffer), RDP (Remote Desktop Protocol) and other technologies.

[0023] The present invention uses a remote calling tool to send local job applications to a remote management node, and the management node reasonably allocates jobs based on the calling information and the computing node resource status. Visual job graphics display and optimization are realized by using a graphics processing tool through a remote computing node, and the protocol conversion tool on the management node converts the graphics access protocol into a browser applicable protocol, and through a reverse proxy, users can conveniently access it through a browser. Use a service registration monitoring tool to record computing node and job information in real time, providing a key basis for data synchronization. Data synchronization is based on the analysis of cluster data characteristics, and the selection of adaptation tools and strategies ensures cross-cluster data consistency and ensures stable operation of the system. From the perspective of systematic visualization, the present invention realizes unified visualization management of multiple local and remote physically isolated clusters. By constructing a set of operating systems that run through local and remote clusters, it covers the full process visualization from job application, scheduling, display to data synchronization, breaking the physical limitations between clusters, enabling users to conveniently manage and monitor cluster resources and job operation status distributed in different physical locations under a unified graphical interface, greatly improving the efficiency and convenience of cross-cluster management. Through the service registration monitoring tool, the present invention enables managers to grasp various key information of computing nodes in real time, such as node operation status, resource allocation, etc., and realize all-round monitoring of the entire cluster. At the same time, in a cross-physical cluster environment, the data synchronization mechanism ensures the consistency of data between clusters and avoids management confusion caused by data differences. This systematic visual management allows managers to discover and solve potential problems in a timely manner, optimize resource allocation, improve the overall operation efficiency and stability of the cluster, and effectively reduce management costs and risks.

[0024] In a preferred solution of this embodiment, step S1 also includes a data synchronization module, which collects and analyzes the data of the local cluster and the data of the remote cluster, and synchronizes the data of the local cluster with the data of the remote cluster; for high-frequency real-time updated data, it is configured to trigger synchronization using the mining function of the database transaction log, and compress and transmit the data through the high-speed network protocol and transmission framework; for low-frequency updated data, parameters and rules are set. In the systematic visualization operation across physical clusters, the data synchronization mechanism is very critical. First, the data synchronization module is used to analyze the data of the local and remote clusters in detail. In terms of data volume distribution, the proportion and quantity of various types of data in the cluster are counted; in terms of update frequency rules, the update cycle characteristics are determined through long-term monitoring; in terms of data structure complexity, the organizational form of the data is analyzed; and the internal connection between the data is mined using correlation analysis. Select an appropriate data synchronization tool based on the analysis results and customize the configuration. For high-frequency real-time updated data, it is configured to trigger synchronization using the database transaction log mining function, adopt high-speed network protocols and efficient transmission frameworks, and combine data compression technology to ensure transmission. For a large amount of low-frequency updated data, parameters and rules are set reasonably for each link of extraction, cleaning, conversion, and loading according to data characteristics and target cluster requirements. When building a two-way data synchronization architecture, timed task synchronization is adopted, and a flexible execution cycle is set in the scheduling system according to the frequency of data updates. In the synchronization script, a hash algorithm is used to compare data differences, and breakpoint resumption and error retransmission mechanisms are used to ensure data consistency and integrity, thereby improving system performance and reliability. In the present invention, a data synchronization tool is used for cross-cluster data transmission, which can also be replaced by cloud storage mounting. The remote cluster and the local cluster mount a shared cloud storage disk, which can save the data synchronization process.

[0025] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the application herein applied. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that the present application does not apply for. The specification and embodiments are considered to be exemplary only, and the true scope and spirit of the present application are indicated by the appended claims.

[0026] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A new visualization method across physical clusters, characterized in that: The novel visualization method across physical clusters includes: S1: connecting the local cluster management node to the remote cluster management node through an independent network line, setting a visualization function module on the remote cluster computing node, setting a monitoring module for recording computing node information in the remote cluster environment, and setting a protocol forwarding module for converting a graphic access protocol into a protocol suitable for browser access on the remote cluster management node; S2: Initiate a visualization job application to the remote cluster management node using a visualization interface at the local end, and the visualization job application is sent to the remote cluster management node through a remote call service tool; S3: After receiving the application, the remote cluster management node allocates nodes according to various parameters of the job and the actual resource status of each computing node, and the visualization function module executes the visualization job. After the job is started, the display number is specified through the graphic display protocol to display on the corresponding remote display service; S4: the protocol forwarding module of the remote cluster management node is started by the remote cluster computing node through a remote call service tool, the protocol forwarding module converts the graph access protocol used by the job into a protocol suitable for browser access, and registers the converted job information into the monitoring module; S5: The monitoring module of the remote cluster is monitored through the local cluster. When the local cluster detects the forwarded job information, the job information is integrated and a link is generated according to preset rules. When the local cluster opens the link, the access request is forwarded to the remote cluster computing node according to the set rules, and then the browser is used to access the visualization job on the remote cluster computing node.

2. The novel cross-physical cluster visualization method according to claim 1, characterized in that: The visualization function module includes: Graphics presentation module, used to convert data into intuitive graphics presentation according to system settings and operation requirements; A graphics projection module, used to project a graphical interface to a specified display device; The graphics management module is used to optimize the resolution and layout of graphics according to different application scenarios and user preferences.

3. The novel cross-physical cluster visualization method according to claim 1, characterized in that: The computing node information recorded by the monitoring module includes basic attributes, operating status and resource allocation of the node, and the monitoring module monitors the recorded computing node information in real time.

4. The novel cross-physical cluster visualization method according to claim 1, characterized in that: The S1 step also includes a data synchronization module, which collects data of the local cluster and data of the remote cluster, and synchronizes the data of the local cluster with the data of the remote cluster.

5. The novel cross-physical cluster visualization method according to claim 1, characterized in that: The visualization job application in step S2 includes detailed information on computing resources required for the job and a specific type of visualization application.

Images