Data processing method and device of node, electronic equipment and storage medium

Abstract

This application provides a data processing method, apparatus, electronic device, and storage medium for nodes. The method includes: automatically parsing a configuration file using a first script to accurately extract the user access list of each target node; integrating node resource information with the user access list; generating a visual table according to preset rules; and centrally displaying the table in an interactive interface. This achieves real-time, unified, and visual monitoring of node resource usage information and user access permissions in a heterogeneous HPC cluster environment without requiring unified node configuration. This reduces the configuration complexity and operation and maintenance costs of heterogeneous HPC clusters and improves cluster management efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, electronic device and storage medium for nodes. Background Technology

[0002] In the field of high performance computing (HPC), due to the large number of nodes, intensive computing power scheduling, and frequent concurrent access by multiple users in high performance computing clusters, the node resource load fluctuates drastically. Users need to keep track of the resource usage information of target nodes in the cluster and the user access list in real time in order to ensure efficient task scheduling and cluster security.

[0003] In related technologies, the Simple Linux Utility for Resource Management (Slurm) scheduling and management system obtains resource usage information and user access lists for each target node periodically through a unified node information collection method.

[0004] However, this unified collection method requires that the configurations of all nodes must be strictly consistent. In actual deployments, since high-performance computing clusters are usually composed of heterogeneous servers of various models and architectures, ensuring that the configurations of these differentiated target nodes are completely consistent is extremely difficult, resulting in low efficiency in cluster configuration management. Summary of the Invention

[0005] This application provides a data processing method, apparatus, electronic device, and storage medium for nodes to improve the efficiency of cluster configuration management.

[0006] Firstly, this application provides a data processing method for nodes, including:

[0007] In response to the first query request initiated by the user through the preset interactive interface, based on the first script deployed on each target node, the first resource usage information and configuration file of each target node are obtained. The first resource usage information is the resource usage information of the target node that was last collected.

[0008] Based on the first script on each target node, the configuration file of each target node is parsed to obtain the user access list of each target node, which includes users who can access the target node.

[0009] Based on the user access list and the first resource usage information of multiple target nodes, a visualization table is generated according to the preset visualization display rules, and the visualization table is displayed in the interactive interface.

[0010] In this solution, the user access list of each target node can be accurately extracted through the automatic parsing of the configuration file by the first script. The node resource information and the user access list are integrated and a visualization table is generated according to preset rules and centrally displayed in the interactive interface. This enables real-time, unified, and visualized monitoring of node resource usage information and user access permissions in a heterogeneous HPC cluster environment without the need for unified node configuration. This reduces the configuration complexity and operation and maintenance cost of heterogeneous HPC clusters and improves the efficiency of cluster management.

[0011] In some embodiments, for any one target node; based on the first script deployed on each target node, the first resource usage information and configuration file of each target node are obtained, including:

[0012] Based on the first script on the target node, obtain the first resource usage information of the target node from the preset storage module in the target node;

[0013] Based on the path preset in the first script on the target node, obtain the configuration file of the target node.

[0014] In this solution, the local acquisition logic for first resource usage information and the path-based acquisition logic for configuration files are uniformly encapsulated within the first script deployed on each target node, ensuring tight coupling between the data processing logic and the node itself. This design enables each target node to independently and autonomously complete its own data collection and permission information extraction without relying on centralized configuration or globally consistent data collection services. This effectively solves the problem of centralized collection failure due to differences in node configurations in heterogeneous clusters, enhancing the system's robustness and adaptability.

[0015] In some embodiments, before obtaining the first resource usage information of the target node from a preset storage module in the target node based on the first script on the target node, the method further includes:

[0016] Based on the second script deployed on the target node, the second resource usage information of the target node is collected periodically;

[0017] For any second resource usage information, the second resource usage information and the time information when the second resource usage information was collected are associated and stored in the storage module.

[0018] In this scheme, a second script is deployed on the target node to continuously collect resource usage information at preset intervals (e.g., 15 seconds) and store it locally, establishing a node-independent, continuously updated real-time and historical resource dataset. This approach ensures that the first script can retrieve the "most recent" resource snapshot (i.e., the first resource usage information) from local storage at any time, providing underlying data support for real-time queries and accumulating time-series data for subsequent historical trend analysis. By decoupling data collection from data processing (query response), data freshness is ensured, and the real-time impact of query operations on node performance is avoided, thus improving the performance and stability of the node's data processing system.

[0019] In some embodiments, obtaining first resource usage information of the target node from a preset storage module in the target node includes:

[0020] Obtain multiple time information corresponding to the target node from the storage module;

[0021] Sort multiple time information items in chronological order, and determine the last time information item after sorting as the target time information;

[0022] The second resource usage information corresponding to the target time information is determined as the first resource usage information.

[0023] In this solution, by extracting all historical collection timestamps from the storage module and sorting them to select the latest timestamp, the corresponding resource usage information is obtained. This ensures that the first resource usage information returned by each query is always the latest resource status that the target node can provide when responding to the query. This automatic filtering mechanism based on timestamp sorting guarantees the real-time performance and accuracy of resource monitoring data. It eliminates the need for administrators or users to manually determine the age of the data, simplifies the operation process, and avoids data inconsistency caused by data synchronization delays or transmission errors.

[0024] In some embodiments, based on user access lists of multiple target nodes and first resource usage information of multiple target nodes, a visualization table is generated according to preset visualization display rules, including:

[0025] Based on the first resource usage information of each target node, determine the idle / busy status of each target node;

[0026] Based on the user access lists of multiple target nodes, the first resource usage information of multiple target nodes, and the idle / busy status of multiple target nodes, a visualization table is generated according to the preset visualization display rules.

[0027] In this solution, the idle / busy status of each target node is automatically determined based on its first resource usage information. This idle / busy status is then integrated with resource information and user access lists and displayed in the interactive interface according to preset rules. This allows the resource status information of nodes and user access permissions to be presented to users more intuitively. Users do not need to master script or command line operations to intuitively and quickly locate the nodes they are authorized to access and that are idle through simple filtering operations. This achieves barrier-free visual interaction of resource status information and user permission information, greatly reducing the usage threshold for non-professional users.

[0028] In some embodiments, the method is applied to any single target node; the method further includes:

[0029] In response to a second query request initiated by a user through a preset interactive interface, based on a first script on the target node, the system retrieves multiple third resource usage information within a historical time period of the target node and the time information at which each third resource usage information was collected from the storage module.

[0030] Based on multiple third-party resource usage information and the time information when collecting each third-party resource usage information, a resource usage trend chart is generated.

[0031] The interactive interface displays a trend chart of resource usage.

[0032] In this solution, by enabling the tracing and visualization trend analysis of the historical resource usage information of individual nodes on a unified interactive interface, users can intuitively observe the fluctuation patterns and load changes of node resources (such as CPU, memory, temperature, etc.) within a specific historical period. This provides visualized data support for node performance analysis, anomaly troubleshooting, capacity planning, and task scheduling strategy optimization, enhancing the depth of cluster operation and maintenance and the scientific nature of decision-making.

[0033] Secondly, this application provides a data processing apparatus for a node, comprising:

[0034] The first processing module is used to respond to the first query request initiated by the user through the preset interactive interface, and obtain the first resource usage information and configuration file of each target node based on the first script deployed on each target node. The first resource usage information is the resource usage information of the target node that was collected most recently.

[0035] The second processing module is used to parse the configuration files of each target node based on the first script on each target node, and obtain the user access list of each target node, which includes users who can access the target node.

[0036] The third processing module is used to generate a visualization table based on the user access list of multiple target nodes and the first resource usage information of multiple target nodes, according to the preset visualization display rules, and display the visualization table in the interactive interface.

[0037] The data processing device for nodes provided in this application embodiment can execute the technical solutions shown in the above method embodiments. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0038] In some embodiments, for any target node; the first processing module is specifically used for:

[0039] Based on the first script on the target node, obtain the first resource usage information of the target node from the preset storage module in the target node;

[0040] Based on the path preset in the first script on the target node, obtain the configuration file of the target node.

[0041] In some embodiments, before obtaining the first resource usage information of the target node from a preset storage module in the target node based on the first script on the target node, the first processing module is further configured to:

[0042] Based on the second script deployed on the target node, the second resource usage information of the target node is collected periodically;

[0043] For any second resource usage information, the second resource usage information and the time information when the second resource usage information was collected are associated and stored in the storage module.

[0044] In some embodiments, the first processing module is specifically used for:

[0045] Obtain multiple time information corresponding to the target node from the storage module;

[0046] Sort multiple time information items in chronological order, and determine the last time information item after sorting as the target time information;

[0047] The second resource usage information corresponding to the target time information is determined as the first resource usage information.

[0048] In some embodiments, the third processing module is specifically used for:

[0049] Based on the first resource usage information of each target node, determine the idle / busy status of each target node;

[0050] Based on the user access lists of multiple target nodes, the first resource usage information of multiple target nodes, and the idle / busy status of multiple target nodes, a visualization table is generated according to the preset visualization display rules.

[0051] In some embodiments, for any one target node; the third processing module 503 is further configured to:

[0052] In response to a second query request initiated by a user through a preset interactive interface, based on a first script on the target node, the system retrieves multiple third resource usage information within a historical time period of the target node and the time information at which each third resource usage information was collected from the storage module.

[0053] Based on multiple third-party resource usage information and the time information when collecting each third-party resource usage information, a resource usage trend chart is generated.

[0054] The interactive interface displays a trend chart of resource usage.

[0055] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0056] The memory stores the instructions that the computer executes;

[0057] The processor executes computer-executable instructions stored in memory to implement the method as described in the first aspect.

[0058] The electronic device provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be repeated here.

[0059] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method as described in the first aspect.

[0060] When the computer-executable instructions in the computer-readable storage medium provided in this application are executed by a processor, the technical solutions shown in the above method embodiments can be implemented. The implementation principle and beneficial effects are similar, and will not be repeated here.

[0061] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method of the first aspect.

[0062] When the computer program in the computer program product provided in this application is executed by a processor, it can implement the technical solution shown in the above method embodiments. The implementation principle and beneficial effects are similar, and will not be repeated here.

[0063] This application provides a node data processing method, apparatus, electronic device, and storage medium. The method includes responding to a first query request initiated by a user through a preset interactive interface; obtaining first resource usage information and configuration files for each target node based on a first script deployed on each target node, wherein the first resource usage information is the most recently collected resource usage information of the target node; parsing the configuration files of each target node based on the first script on each target node to obtain a user access list for each target node, the user access list including users who can access the target node; and generating a visualization table based on the user access lists and first resource usage information of multiple target nodes according to preset visualization display rules, and displaying the visualization table in the interactive interface. In the above method, through automated parsing of configuration files, the user access lists of each target node can be accurately extracted, node resource information and user access lists can be integrated, and a visualization table can be generated according to preset rules and centrally displayed in the interactive interface. This achieves real-time, unified, and visualized monitoring of node resource usage information and user access permissions in a heterogeneous HPC cluster environment without requiring unified node configuration, reducing the configuration complexity and operation and maintenance costs of heterogeneous HPC clusters, and improving the efficiency of cluster management. Attached Figure Description

[0064] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0065] Figure 1 This application provides a schematic diagram of the architecture of a node data processing system.

[0066] Figure 2 A flowchart illustrating a node data processing method provided in an embodiment of this application;

[0067] Figure 3 A flowchart illustrating a method for generating a visual table as provided in an embodiment of this application;

[0068] Figure 4 A flowchart illustrating a method for generating a resource usage trend chart for a target node, provided in an embodiment of this application;

[0069] Figure 5 This is a schematic diagram of the structure of a node data processing device provided in an embodiment of this application;

[0070] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0071] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0072] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0073] In the field of high performance computing (HPC), due to the large number of nodes, intensive computing power scheduling, and frequent concurrent access by multiple users in high performance computing clusters, the node resource load fluctuates drastically. Users need to keep track of the resource usage information of target nodes in the cluster and the user access list in real time in order to ensure efficient task scheduling and cluster security.

[0074] In related technologies, the Simple Linux Utility for Resource Management (Slurm) scheduling and management system obtains resource usage information and user access lists for each target node periodically through a unified node information collection method.

[0075] However, this unified collection method requires that the configurations of all nodes must be strictly consistent. In actual deployments, since high-performance computing clusters are usually composed of heterogeneous servers of various models and architectures, ensuring that the configurations of these differentiated target nodes are completely consistent is extremely difficult, increasing the cost of cluster configuration management.

[0076] Based on the above-mentioned technical problems, the technical concept of the embodiments of this application is as follows:

[0077] In response to the first query request initiated by the user through a preset interactive interface, the system obtains the first resource usage information and configuration file of each target node based on the first script deployed on each target node. The first resource usage information is the most recently collected resource usage information of the target node. Based on the first script on each target node, the system parses the configuration file of each target node to obtain the user access list of each target node, which includes users who can access the target node. Based on the user access lists and first resource usage information of multiple target nodes, a visualization table is generated according to preset visualization display rules and displayed in the interactive interface. In the above method, through the automated parsing of the configuration file, the user access list of each target node can be accurately extracted, the node resource information and the user access list can be integrated, and a visualization table can be generated according to preset rules and centrally displayed in the interactive interface. This achieves real-time, unified, and visualized monitoring of node resource usage information and user access permissions in a heterogeneous HPC cluster environment without the need for unified node configuration, reducing the configuration complexity and operation and maintenance costs of heterogeneous HPC clusters and improving the efficiency of cluster management.

[0078] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0079] Figure 1 This is a schematic diagram of the architecture of a node data processing system provided in an embodiment of this application, such as... Figure 1 As shown, the data processing system includes: multiple target nodes, a visualization processing module, and an interactive interface. The multiple target nodes include target node 1, target node 2, ..., target node N, where N is an integer greater than or equal to 1.

[0080] Multiple target nodes are connected to the visualization processing module, which in turn is connected to the interactive interface.

[0081] Each target node includes a first script, a second script, a configuration file, and a storage module. For example, target node 1 includes first script 1, second script 1, configuration file 1, and storage module 1.

[0082] A cluster is a distributed computing / storage system composed of multiple target nodes, which work together through network communication.

[0083] The target node is a node in the cluster that needs to obtain real-time information on resource usage status and user access permissions.

[0084] The configuration file is a text file on the target node used to store user permissions for accessing that target node.

[0085] The first script is used to respond to the first query request initiated by the user through the preset interactive interface, and to obtain the first resource usage information and configuration file of the target node. The first resource usage information is the resource usage information of the target node in the most recent collection.

[0086] The first script is used to parse the configuration file of the target node to obtain the user access list of each target node, which includes users who can access the target node.

[0087] The first script is also used to send the user access list and first resource usage information of the corresponding target node to the visualization processing module.

[0088] The visualization processing module is used to generate visualization tables based on user access lists and first resource usage information of multiple target nodes, according to preset visualization display rules.

[0089] The visualization module is also used to display visualization tables in the interactive interface.

[0090] As is understandable, a visualization platform includes a visualization processing module and an interactive interface. Grafana is an example of such a platform.

[0091] Figure 2 A flowchart illustrating a node data processing method provided in an embodiment of this application is shown below. Figure 2 As shown, the method includes:

[0092] S201. In response to the first query request initiated by the user through a preset interactive interface, based on the first script deployed on each target node, obtain the first resource usage information and configuration file of each target node, wherein the first resource usage information is the resource usage information most recently collected by the target node.

[0093] In response to the first query request initiated by the user through the preset interactive interface, it can be understood that the visualization processing module sends the first query request to each target node.

[0094] Understandably, the first query request instructs the query to retrieve the most recent resource usage information (i.e., the first resource usage information) and the corresponding user access permission configuration (i.e., the user access list) for each target node, and requests a visual display of the query results.

[0095] Understandably, the most recent is the historical moment closest to the present.

[0096] In some embodiments, for any target node, based on the first script deployed on each target node, the first resource usage information and configuration file of each target node are obtained, including:

[0097] Based on the first script on the target node, obtain the first resource usage information of the target node from the preset storage module in the target node;

[0098] Based on the path preset in the first script on the target node, obtain the configuration file of the target node.

[0099] Understandably, the first script can be a Shell script or Python script deployed on each target node. It is used to respond to query requests, complete the core logic of resource data reading and configuration file parsing, and is the core carrier for interaction between the node side and the visualization processing module.

[0100] The first script may include a data acquisition module and an interface module.

[0101] The data acquisition module is used to obtain the configuration file of the target node based on a preset path.

[0102] The interface module is used to obtain the first resource usage information of the target node from the preset storage module in the target node.

[0103] The preset path is the path where the cluster administrator stores the configuration file path of each target node in the first script of each target node during the development phase. This allows the first script to retrieve the configuration file of the target node based on the preset path in the first script on that target node when the user needs to query the configuration file of the target node.

[0104] The default path is an absolute path. During the development phase, the administrator uses a cluster deployment tool to write the path in batches into the first script of each target node to ensure the accuracy and consistency of the path.

[0105] The configuration files include a list of configurations such as sshd_config, slurm_partition.conf, and / or allowed_users.

[0106] The first resource usage information may include one or more of the following: the usage of the node's Central Processing Unit (CPU) / Data Center Unit (DCU), the storage status of the node's storage modules, the node's driver version information, the node's power consumption, and / or the node's temperature.

[0107] CPU / DCU usage information may include utilization rate (in %) and core usage; storage module storage information may include used storage capacity, remaining storage capacity (in GB / MB), and storage utilization rate (in %); driver version information may include graphics processing unit (GPU) driver version and DCU driver version; power consumption is in W (watts), and temperature is in ℃ (degrees Celsius).

[0108] In some embodiments, before obtaining the first resource usage information of the target node from a preset storage module in the target node based on the first script on the target node, the method may further include:

[0109] Based on the second script deployed on the target node, the second resource usage information of the target node is collected periodically;

[0110] For any second resource usage information, the second resource usage information and the time information when the second resource usage information was collected are associated and stored in the storage module.

[0111] Optionally, the period can be 15 seconds (s).

[0112] The second script includes, but is not limited to, the DCU Exporter acquisition program and the GPU Exporter acquisition program.

[0113] For example, when the target node is a computing node equipped with a DCU, the second script deployed on the target node is a DCU Exporter collection program, which is used to collect DCU-related resource usage information in a targeted manner; when the target node is a computing node equipped with a GPU, the second script deployed on the target node is a GPU Exporter collection program, which is used to collect GPU-related resource usage information in a targeted manner.

[0114] The second script is a dedicated collection script deployed by the cluster administrator during the development phase on each target node according to the hardware configuration type of each target node (such as a computing node equipped with a DCU or a computing node equipped with a GPU) and resource monitoring requirements, to ensure the targeting and accuracy of resource data collection.

[0115] When the period is 15s, the second resource usage information of the target node is collected periodically, which can be understood as: the second resource usage information of the target node is collected every 15s.

[0116] It is understandable that the second resource usage information and the first resource usage information are essentially the same type of resource data, the only difference being the timing and purpose of acquisition: the second resource usage information is the raw data collected periodically and stored in the storage module of the target node; the first resource usage information is the most recently collected resource usage information filtered from the second resource usage information and is used for visualization.

[0117] Time information can be a timestamp.

[0118] The storage module associates the second resource usage information with the time information when the second resource usage information is collected. This can be understood as storing the second resource usage information, the time information when the second resource usage information is collected, and the correspondence between the second resource usage information and the time information when the second resource usage information is collected in the storage module. The storage format can be structured data (such as CSV files or SQLite database tables). For example, the storage format is "timestamp, CPU usage, memory usage, temperature, driver version". The timestamp is accurate to the second (e.g., 1710000000 corresponds to 2024-03-09 10:00:00), ensuring that the corresponding second resource usage information can be quickly located through the timestamp.

[0119] Optionally, the format of the second resource usage information can be metrics that conform to the Prometheus specification.

[0120] It should be noted that the times mentioned in this application are illustrative examples and are used for illustration only, and do not constitute a limitation on the specific implementation of this application.

[0121] In some embodiments, obtaining first resource usage information of the target node from a preset storage module in the target node includes:

[0122] Obtain multiple time information corresponding to the target node from the storage module;

[0123] Sort multiple time information items in chronological order, and determine the last time information item after sorting as the target time information;

[0124] The second resource usage information corresponding to the target time information is determined as the first resource usage information.

[0125] Understandably, the resource usage information most recently collected by the target node is the second resource usage information corresponding to the target time information.

[0126] To sort multiple time information in chronological order, one method is to sort them based on the numerical value of the timestamps. In this case, the last time information after sorting will be the one with the largest timestamp.

[0127] Storage modules, for example, are Prometheus.

[0128] S202. Based on the first script on each target node, parse the configuration file of each target node to obtain the user access list of each target node, wherein the user access list includes users who can access the target node.

[0129] For example, based on the first script on each target node, the sshd_config, slurm_partition.conf, and / or allowed_users of each target node are parsed to obtain the user access list of each target node.

[0130] Specifically, the first script reads the permission configuration items in the configuration file, extracts the user identifiers, usernames, or user group information that are allowed to log in, access, or use the target node, and organizes them into a user access list.

[0131] The target node's user access list includes one or more users who can access the target node.

[0132] S203. Based on the user access list of multiple target nodes and the first resource usage information of multiple target nodes, generate a visualization table according to the preset visualization display rules, and display the visualization table in the interactive interface.

[0133] The following, in conjunction with Table 1, provides an example of generating a visualization table based on the user access list and the first resource usage information of multiple target nodes, according to preset visualization display rules:

[0134] Table 1

[0135]

[0136] Table 1 includes multiple target nodes, multiple accessible users, multiple first resource usage information, and the correspondence between multiple target nodes, multiple user access lists, and multiple first resource usage information. For example, the accessible users of target node 1 include user 1 and user 2, and the first resource usage information of target node 1 includes: CPU utilization: utilization 1, memory usage: memory usage 1, temperature: temperature 1, driver version: version 1.

[0137] Understandably, the preset visualization display rules can be to organize the target node, accessible users, and primary resource usage information into a table in rows and columns, and sort them according to the time of the user access list and primary resource usage information of the target node received by the visualization processing module.

[0138] Understandably, the user access list of the target node includes the users who can access the target node.

[0139] The beneficial effects of this application's embodiments are as follows: In this application's embodiments, in response to a first query request initiated by a user through a preset interactive interface, based on a first script deployed on each target node, the first resource usage information and configuration file of each target node are obtained. The first resource usage information is the resource usage information most recently collected by the target node. Based on the first script on each target node, the configuration file of each target node is parsed to obtain a user access list for each target node. The user access list includes users who can access the target node. Based on the user access lists of multiple target nodes and the first resource usage information of multiple target nodes, a visualization table is generated according to preset visualization display rules and displayed in the interactive interface. In the above method, through automated parsing of the configuration file, the user access list of each target node can be accurately extracted. The node resource information and user access list are integrated, and a visualization table is generated according to preset rules and centrally displayed in the interactive interface. This achieves real-time, unified, and visualized monitoring of node resource usage information and user access permissions in a heterogeneous HPC cluster environment without the need for unified node configuration. This reduces the configuration complexity and operation and maintenance costs of heterogeneous HPC clusters and improves the efficiency of cluster management.

[0140] exist Figure 2 Based on the embodiments, the following is combined Figure 3 The document provides a detailed explanation of "generating a visualization table based on the user access list of multiple target nodes and the first resource usage information of multiple target nodes, according to preset visualization display rules".

[0141] Figure 3 A flowchart illustrating a method for generating a visual table, as provided in an embodiment of this application, is shown below. Figure 3 As shown, the method includes:

[0142] S301. Based on the first resource usage information of each target node, determine the idle / busy status of each target node.

[0143] The following example illustrates how to determine the idle / busy status of a target node, using CPU utilization, memory usage, and temperature as examples:

[0144] For example, the preset threshold ranges for CPU utilization, memory usage, and temperature are as follows: CPU utilization threshold range is [0%, 30%], memory usage threshold range is [0%, 40%], and temperature threshold range is [0℃, 60℃].

[0145] If the target node's first resource usage information shows that CPU utilization is ≤30%, memory usage is ≤40%, and temperature is ≤60℃, then the target node's idle / busy status is determined to be idle; if any one of the following conditions is met: CPU utilization >30%, memory usage >40%, or temperature >60℃, then the target node's idle / busy status is determined to be busy.

[0146] For example, if the CPU utilization of target node 1 is 25% (≤30%), the memory usage is 35% (≤40%), and the temperature is 55℃ (≤60℃), then the idle / busy state of target node 1 is determined to be idle; if the CPU utilization of target node 3 is 80% (>30%), the memory usage is 50% (>40%), and the temperature is 65℃ (>60℃), then the idle / busy state of target node 3 is determined to be busy.

[0147] It should be noted that the above content is merely an exemplary implementation of determining the idle / busy status of each target node based on its first resource usage information, and is not intended to limit the scope of protection of this application. In practical applications, DPU utilization, memory usage, and temperature can also be used as core judgment indicators. The judgment logic is consistent with the judgment logic related to CPU utilization (i.e., by setting a threshold range for each indicator, the idle / busy status is determined based on whether the actual indicator data of the target node meets the threshold conditions). The specific judgment process is similar to the principle of the aforementioned example and will not be elaborated here.

[0148] S302. Based on the user access list of multiple target nodes, the first resource usage information of multiple target nodes, and the idle / busy status of multiple target nodes, generate a visualization table according to the preset visualization display rules.

[0149] The following, in conjunction with Table 2, provides an example of generating a visualization table based on the user access list, the first resource usage information of multiple target nodes, and the idle / busy status of multiple target nodes, according to preset visualization display rules:

[0150] Table 2

[0151]

[0152] It should be noted that the specific values ​​of the first resource usage information in Table 2 are only for illustrative purposes. The actual values ​​are determined by the second script periodically collecting data and filtering it by the first script. The idle / busy status is derived based on the judgment rule of S201, which intuitively reflects the resource occupancy of each target node.

[0153] Understandably, the preset visualization display rules can be to organize the target node, accessible users, first resource usage information, and idle / busy status into a table in rows and columns, and sort them according to the time of the user access list and first resource usage information of the target node received by the visualization processing module. For example, if target node 1 is in the first row in Table 2, it means that the visualization processing module was the first to receive the user access list and first resource usage information of target node 1.

[0154] Furthermore, after displaying the visualization table in the interactive interface, users can also perform precise filtering based on their own access permissions and the node's idle / busy status to quickly locate target nodes that meet their needs. Specifically, users can use the filtering function provided by the interactive interface to input their own user identifier and the filter condition that the idle / busy status is "free". The system will automatically match target nodes in the visualization table that have access to users who include that user and whose idle / busy status is "free", and output the filtering results. For example, when user 1 initiates the first query request through the preset interactive interface, they can use the above filtering function to quickly filter out the target nodes that they have access to and are in an idle state (such as target node 1 in Table 2).

[0155] The beneficial effects of this application's embodiments are as follows: In this application's embodiments, the idle / busy status of each target node is determined based on the first resource usage information of each target node; based on the user access lists of multiple target nodes, the first resource usage information of multiple target nodes, and the idle / busy status of multiple target nodes, a visualization table is generated according to preset visualization display rules. In the above method, the idle / busy status of each target node is automatically determined based on the first resource usage information of each target node, and this idle / busy status is integrated with resource information and user access lists, and displayed in the interactive interface according to preset rules. This allows the resource status information of nodes and user access permissions to be presented to users more intuitively, enabling users to intuitively and quickly locate the nodes they are authorized to access and that are idle without needing to master script or command line operations, through simple filtering operations. This achieves barrier-free visual interaction of resource status information and user permission information, greatly reducing the usage threshold for non-professional users.

[0156] Based on any of the above embodiments, the following combination Figure 4 For any given target node, a detailed explanation of the method for generating a resource usage trend chart for that target node is provided.

[0157] Figure 4 A flowchart illustrating a method for generating a resource usage trend chart for a target node, as provided in this application embodiment, is shown below. Figure 4 As shown, the method includes:

[0158] S401, In response to the second query request initiated by the user through a preset interactive interface, based on the first script on the target node, obtain multiple third resource usage information within the historical time period of the target node and the time information when collecting each third resource usage information from the storage module.

[0159] Understandably, the second query request indicates that the user wants to query the resource usage changes of the target node over a historical period and display the query results in the form of a trend chart.

[0160] The historical time period can be a preset duration from the current moment to the past (such as the past 1 hour, 6 hours, 24 hours, etc.), or a time range that the user can customize through the interactive interface when initiating a second query request (such as "2024-03-09 08:00:00 to 2024-03-09 18:00:00").

[0161] The following details the process of retrieving multiple third-party resource usage information from the storage module within a historical time period of the target node, and the time information used to collect each third-party resource usage information:

[0162] Based on historical time periods, multiple candidate time information corresponding to the target node is obtained from the storage module, wherein the multiple candidate time information is within the historical time period; second resource usage information corresponding to the multiple candidate time information is obtained from the storage module.

[0163] Understandably, multiple third resource usage information within a historical period are equivalent to multiple candidate time information corresponding to second resource usage information, and the time information when collecting each third resource usage information is equivalent to multiple candidate time information.

[0164] It should be noted that the third resource usage information, the second resource usage information, and the first resource usage information are essentially the same type of resource data. The third resource usage information is the second resource usage information within a historical period.

[0165] S402. Based on the usage information of multiple secondary resources within a historical period and the time information when each secondary resource usage information was collected, generate a resource usage trend chart.

[0166] In some embodiments, the first script sends multiple third resource usage information within a historical period of the target node and the time information when each third resource usage information is collected to the visualization processing module. The visualization processing module generates a resource usage trend chart based on the multiple second resource usage information within the historical period and the time information when each second resource usage information is collected.

[0167] Specifically, the visualization module uses time information as the horizontal axis (arranged in chronological order) and various indicators in the third resource usage information (such as CPU utilization, memory usage, and temperature) as the vertical axis. It uses line charts, bar charts, or combined charts to draw the curves of each indicator changing over time, forming a resource usage trend chart. The trend chart supports functions such as displaying indicator numerical labels, threshold lines (such as thresholds used to determine idle / busy status for each indicator), and data range scaling, making it easy for users to intuitively observe the patterns of resource changes.

[0168] S403. Display a resource usage trend chart in the interactive interface.

[0169] In some embodiments, the visualization module displays a resource usage trend chart in the interactive interface.

[0170] Specifically, the visualization module transmits the generated resource usage trend chart to the interactive interface in the form of an image or an interactive chart. The interactive interface allows users to switch between viewing trend curves of different indicators (such as viewing CPU usage trends separately, or viewing memory usage and temperature trends simultaneously), adjust the time axis range (such as zooming in on detailed changes within a 1-hour period), and download trend chart data or image files. At the same time, the interactive interface synchronously displays the basic information of the target node (such as node ID and accessible users), ensuring that users can associate the core information of the node while viewing the trend.

[0171] The beneficial effects of this application's embodiments are as follows: In this application's embodiments, for any target node; in response to a second query request initiated by a user through a preset interactive interface, based on a first script on the target node, multiple third resource usage information within a historical time period of the target node and the time information when collecting each third resource usage information are obtained from the storage module; based on the multiple third resource usage information and the time information when collecting each third resource usage information, a resource usage trend chart is generated; and the resource usage trend chart is displayed in the interactive interface. In the above method, by realizing the tracing and visualization trend analysis of the historical resource usage information of a single node on a unified interactive interface, users can intuitively observe the fluctuation patterns and load changes of the node's resources (such as CPU, memory, temperature, etc.) within a specific historical time period, providing visualized data support for node performance analysis, anomaly troubleshooting, capacity planning, and task scheduling strategy optimization, thereby enhancing the depth of cluster operation and maintenance and the scientific nature of decision-making.

[0172] Figure 5 This is a schematic diagram of the structure of a node data processing device provided in an embodiment of this application. Figure 5 As shown, the node's data processing device 50 includes a first processing module 501, a second processing module 502, and a third processing module 503.

[0173] The first processing module 501 is used to respond to the first query request initiated by the user through a preset interactive interface, and obtain the first resource usage information and configuration file of each target node based on the first script deployed on each target node. The first resource usage information is the resource usage information most recently collected by the target node.

[0174] The second processing module 502 is used to parse the configuration file of each target node based on the first script on each target node, and obtain the user access list of each target node, which includes users who can access the target node.

[0175] The third processing module 503 is used to generate a visualization table based on the user access list of multiple target nodes and the first resource usage information of multiple target nodes, according to the preset visualization display rules, and display the visualization table in the interactive interface.

[0176] The data processing device 50 for nodes provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0177] In some embodiments, for any target node; the first processing module 501 is specifically used for:

[0178] Based on the first script on the target node, obtain the first resource usage information of the target node from the preset storage module in the target node;

[0179] Based on the path preset in the first script on the target node, obtain the configuration file of the target node.

[0180] In some embodiments, before obtaining the first resource usage information of the target node from a preset storage module in the target node based on the first script on the target node, the first processing module 501 is further configured to:

[0181] Based on the second script deployed on the target node, the second resource usage information of the target node is collected periodically;

[0182] For any second resource usage information, the second resource usage information and the time information when the second resource usage information was collected are associated and stored in the storage module.

[0183] In some embodiments, the first processing module 501 is specifically used for:

[0184] Obtain multiple time information corresponding to the target node from the storage module;

[0185] Sort multiple time information items in chronological order, and determine the last time information item after sorting as the target time information;

[0186] The second resource usage information corresponding to the target time information is determined as the first resource usage information.

[0187] In some embodiments, the third processing module 503 is specifically used for:

[0188] Based on the first resource usage information of each target node, determine the idle / busy status of each target node;

[0189] Based on the user access lists of multiple target nodes, the first resource usage information of multiple target nodes, and the idle / busy status of multiple target nodes, a visualization table is generated according to the preset visualization display rules.

[0190] In some embodiments, for any one target node; the third processing module 503 is further configured to:

[0191] In response to a second query request initiated by a user through a preset interactive interface, based on a first script on the target node, the system retrieves multiple third resource usage information within a historical time period of the target node and the time information at which each third resource usage information was collected from the storage module.

[0192] Based on multiple third-party resource usage information and the time information when collecting each third-party resource usage information, a resource usage trend chart is generated.

[0193] The interactive interface displays a trend chart of resource usage.

[0194] The data processing device 50 for nodes provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0195] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device 60 includes a processor 601 and a memory 602. The processor 601 is communicatively connected to the memory 602, which stores computer execution instructions. The processor 601 is configured to execute the technical solutions in any of the aforementioned method embodiments by executing the computer execution instructions stored in the memory 602.

[0196] Optionally, the memory 602 can be either standalone or integrated with the processor 601. Optionally, when the memory 602 is a device independent of the processor 601, the electronic device 60 may further include a bus 603 for connecting the aforementioned devices.

[0197] The electronic device is used to execute the technical solutions in any of the foregoing method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0198] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the technical solutions provided in any of the foregoing method embodiments.

[0199] This application also provides a computer program product, including a computer program, which, when executed by a processor, is used to implement the technical solutions provided in the foregoing method embodiments.

[0200] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0201] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0202] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0203] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0204] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0205] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0206] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0207] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0208] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data processing method of a node, characterized by, The method includes: In response to a first query request initiated by a user through a preset interactive interface, based on a first script deployed on each target node, the first resource usage information and configuration file of each target node are obtained, wherein the first resource usage information is the resource usage information of the target node most recently collected. Based on the first script on each target node, the configuration file of each target node is parsed to obtain the user access list of each target node, and the user access list includes users who can access the target node; Based on the user access list of multiple target nodes and the first resource usage information of multiple target nodes, a visualization table is generated according to the preset visualization display rules, and the visualization table is displayed in the interactive interface.

2. The method of claim 1, wherein, For any given target node; the step of obtaining the first resource usage information and configuration file of each target node based on the first script deployed on each target node includes: Based on the first script on the target node, the first resource usage information of the target node is obtained from the preset storage module in the target node; Based on the preset path in the first script on the target node, obtain the configuration file of the target node.

3. The method of claim 2, wherein, Before obtaining the first resource usage information of the target node from a preset storage module on the target node based on the first script on the target node, the method further includes: Based on the second script deployed on the target node, the second resource usage information of the target node is periodically collected; For any second resource usage information, the second resource usage information and the time information when the second resource usage information was collected are associated and stored in the storage module.

4. The method of claim 3, wherein, The step of obtaining the first resource usage information of the target node from the preset storage module in the target node includes: Obtain multiple time information corresponding to the target node from the storage module; The multiple time information items are sorted in chronological order, and the last time information item after sorting is determined as the target time information. The second resource usage information corresponding to the target time information is determined as the first resource usage information.

5. The method of claim 1, wherein, The user access list based on multiple target nodes and the first resource usage information of multiple target nodes are used to generate a visualization table according to preset visualization display rules, including: Based on the first resource usage information of each target node, determine the idle / busy status of each target node; Based on the user access lists of the multiple target nodes, the first resource usage information of the multiple target nodes, and the idle / busy status of the multiple target nodes, a visualization table is generated according to preset visualization display rules.

6. The method of claim 3, wherein, For any target node; the method further includes: In response to a second query request initiated by a user through a preset interactive interface, based on a first script on the target node, multiple third resource usage information within a historical period of the target node and the time information when collecting each third resource usage information are obtained from the storage module. Based on the usage information of the multiple third resources and the time information when each third resource usage information was collected, a resource usage trend chart is generated. The interactive interface displays a trend chart of resource usage.

7. A data processing apparatus of a node, characterized by The device includes: The first processing module is used to respond to the first query request initiated by the user through a preset interactive interface, and obtain the first resource usage information and configuration file of each target node based on the first script deployed on each target node. The first resource usage information is the resource usage information of the target node that was last collected. The second processing module is used to parse the configuration files of each target node based on the first script on each target node, and obtain the user access list of each target node, wherein the user access list includes users who can access the target node; The third processing module is used to generate a visualization table based on the user access list of multiple target nodes and the first resource usage information of multiple target nodes, according to preset visualization display rules, and display the visualization table in the interactive interface.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.

Images