An interactive distributed visualization management system and method

Abstract

The present invention discloses an interactive distributed visualization management system and method, and relates to the field of distributed technology. The technical key points of the scheme are as follows: by creating a virtual variable for each category of nodes, each node type in a data set is converted into a corresponding virtual variable representation, for each feature in the data set, the correlation coefficient between it and each virtual variable is calculated respectively, and features significantly related to the node type are screened out. Based on the screened out significantly related features, all nodes are classified, and the load situation measurement value of the node is calculated, and the load situation measurement value is compared with a load threshold value. The load balancing strategy is adjusted according to the comparison result. After the load balancing strategy is executed, the total load measurement value of all nodes of the same type is calculated, and the total load measurement value is compared with the total load threshold value. Corresponding measures are taken according to the comparison result, thereby realizing maximum utilization of system resources and avoiding resource waste.

Description

Technical Field

[0001] The present invention relates to the field of distributed technology, and in particular to an interactive distributed visualization management system and method. Background Art

[0002] With the rapid development of information technology, especially driven by technologies such as big data, cloud computing, and the Internet of Things, the complexity and data volume of various information systems have increased dramatically. In order to more effectively manage and utilize this information and achieve optimal allocation and efficient utilization of resources, interactive distributed visualization management methods and systems have emerged.

[0003] In a Chinese invention application with application publication number CN116562050A, a visual modeling method for a distributed simulation platform and a distributed simulation platform are disclosed, which relate to the field of distributed simulation technology, including: creating a distributed simulation platform and establishing a simulation model; configuring the parameters of the simulation model and obtaining input data, and inputting the simulation model to obtain simulation data; defining the main node and sub-node of the simulation platform according to the actual business logic, and establishing a mapping table of the main node and sub-node of the simulation platform through different simulation tasks; loading the component resources and simulation data of the simulation model through the mapping table to obtain the scheduling information of different simulation tasks to parse the simulation model; generating a page on the front end of the simulation platform based on the parsing results and the user's instructions to display the results of the simulation task, and completing the visual modeling of the data.

[0004] In view of the above invention, the prior art has the following deficiencies:

[0005] 1. In current distributed systems, although load balancing strategies have been implemented to balance the load of each node, the inherent configuration performance differences between different types of nodes are often ignored during the optimization process. This "one-size-fits-all" load balancing method is difficult to accurately match the unique performance advantages of each type of node, and cannot maximize the unique performance advantages of each type of node, and even causes performance bottlenecks and errors due to configuration mismatch;

[0006] 2. With the continuous growth of business needs, existing distributed systems often take a simple and crude approach, blindly increasing the scale of the entire system in order to meet the needs by piling up numbers. However, this approach may lead to a huge waste of resources because the newly added nodes may not be fully utilized. On the other hand, it may also introduce new performance bottlenecks, such as network delays, data synchronization problems, etc., thereby affecting the stability and reliability of the system. Summary of the invention

[0007] 1. Technical issues to be resolved

[0008] In view of the deficiencies in the prior art, the present invention provides an interactive distributed visualization management system and method, which creates a virtual variable for each category of nodes, converts each node type in a data set into a corresponding virtual variable representation, calculates the correlation coefficient between each feature in the data set and each virtual variable, screens out features that are significantly correlated with the node type, and classifies all nodes based on the screened out significantly correlated features, calculates the load condition measurement value of the node, compares the load condition measurement value with the load threshold, adjusts the load balancing strategy according to the comparison result, and after executing the load balancing strategy, calculates the total load measurement value of all nodes of the same type for each type of node, compares the total load measurement value with the total load threshold, and takes corresponding measures according to the comparison result, thereby solving the problems mentioned in the background technology.

[0009] (II) Technical solution

[0010] To achieve the above objectives, the present invention is implemented through the following technical solutions: an interactive distributed visualization management method, comprising the following steps:

[0011] Collect a dataset containing multiple classification node types and multiple features, set a set of node type categories, create a dummy variable for each category, and convert each node type in the dataset into a corresponding dummy variable representation, where each node is identified by a set of dummy variable values;

[0012] For each feature in the data set, calculate the correlation coefficient between it and each virtual variable, record and save all calculated correlation coefficients, remove features whose correlation with the target variable is lower than the correlation threshold, and screen out features that are significantly correlated with the node type;

[0013] Based on the significantly relevant features screened out, all nodes are classified, and the load situation measurement value of the node is calculated through the load pressure data of the node. The load situation measurement value is compared with the load threshold, and the load balancing strategy is adjusted according to the comparison result;

[0014] After executing the load balancing strategy, for each type of node, the total load measurement value of all nodes of the same type is calculated through the load pressure data of the node, the total load measurement value is compared with the total load threshold, and corresponding measures are taken according to the comparison result;

[0015] Build a dynamic visualization interface to display the key indicators of different types of nodes in real time. Select a specific node or node type through the interface to view its detailed load information and history. Provide an adjustment interface for load balancing strategies and adjust load balancing parameters.

[0016] Furthermore, a set of categories of node types is set, a dummy variable is created for each category, and each node type in the data set is converted into a corresponding dummy variable representation, where each node is identified by a set of dummy variable values.

[0017] Furthermore, a virtual variable RT is created. If the node is real-time processing, RT=1, otherwise RT=0; a virtual variable BT is created. If the node is batch processing, BT=1, otherwise BT=0; a virtual variable IoT is created. Nodes of the IoT category will be implicitly represented when RT=0 and BT=0, that is, IoT=1.

[0018] Furthermore, for each feature in the data set, the correlation coefficient between it and each virtual variable is calculated respectively, and all calculated correlation coefficients are recorded and saved. The calculation formula of the correlation coefficient is as follows:

[0019]

[0020] Among them, r X,RT represents the correlation coefficient between feature X and dummy variable RT, x i represents the observed value of the i-th feature X, RT i represents the observed value of the i-th dummy variable RT, i = 1, 2, ..., n, n represents the number of observed values, represents the mean of feature X, represents the mean of the dummy variable RT.

[0021] Furthermore, the correlation coefficient between each feature and different node types is analyzed to determine whether it is positively correlated, negatively correlated, or has no significant correlation, including:

[0022] When the correlation coefficient is greater than zero, it indicates a positive correlation; when the correlation coefficient is less than zero, it indicates a negative correlation; and when the correlation coefficient is equal to zero, it indicates no significant correlation.

[0023] Furthermore, the load pressure data of the node, including CPU utilization, memory utilization, and network utilization, is used to calculate the load measurement value of each node. The calculation formula is as follows:

[0024] FS j =k1*Cs j +k2*Ms j +k3*Ls j

[0025] Among them, FS j represents the load measurement value of the jth node, Cs j represents the CPU utilization of the jth node, Ms jIndicates the memory utilization of the jth node, Ls j represents the network utilization of the jth node, k1, k2 and k3 represent weights, 0 <k1<1,0<k2<1,0<k3<1。

[0026] Furthermore, a load threshold is preset, the load condition measurement value is compared with the load threshold, and the load balancing strategy is adjusted according to the comparison result, specifically including:

[0027] When the load measurement value is less than the load threshold, the load balancing strategy is adjusted to allocate requests or tasks of the same type of nodes to the current node to achieve balanced utilization of resources;

[0028] When the load condition measurement value is greater than or equal to the load threshold, the emergency load balancing strategy is implemented to automatically redirect some requests on the node in proportion to the preset ratio to the same type of nodes with a load condition measurement value less than the load threshold until the load condition measurement value of the current node is less than the load threshold.

[0029] Furthermore, after executing the load balancing strategy, the total load measurement value of all nodes of the same type is calculated through the load pressure data of the node, including CPU utilization, memory utilization, and network utilization. The calculation formula is as follows:

[0030]

[0031] Among them, FS represents the total load measurement value of all nodes of the same type, Cs j represents the CPU utilization of the jth node, Ms j Indicates the memory utilization of the jth node, Ls j represents the network utilization of the jth node, j = 1, 2, ..., M, M represents the number of nodes, k1, k2 and k3 represent weights, 0 <k1<1,0<k2<1,0<k3<1。

[0032] Furthermore, a total load threshold is preset, the total load measurement value is compared with the total load threshold, and corresponding measures are taken according to the comparison result, specifically including:

[0033] When the total load measurement value is less than the total load threshold, it means that the current resources are in excess. The load lower limit threshold is set in advance, and the services on the nodes whose load situation measurement value is lower than the load lower limit threshold are migrated to the nodes whose load situation measurement value is higher than the load lower limit threshold. The offline services are completely migrated to the nodes, and the resources are recovered;

[0034] When the total load measurement value is greater than or equal to the total load threshold, it means that a large number of nodes are currently under high load. The load upper limit threshold is set in advance, the number of nodes is increased or the node performance is improved until the load measurement values ​​of all nodes are lower than the load upper limit threshold.

[0035] An interactive distributed visualization management system, comprising: a feature engineering module, a feature selection module, a single-node load balancing module, a multi-node load balancing module and a visualization and interaction module; wherein,

[0036] The feature engineering module collects a data set containing multiple classification node types and multiple features, sets a set of categories for the node types, creates a dummy variable for each category, and converts each node type in the data set into a corresponding dummy variable representation, where each node is identified by a set of dummy variable values;

[0037] Feature selection module, for each feature in the data set, calculates the correlation coefficient between it and each virtual variable, records and saves all calculated correlation coefficients, removes features whose correlation with the target variable is lower than the correlation threshold, and screens out features that are significantly correlated with the node type;

[0038] The single-node load balancing module classifies all nodes based on the significantly relevant features screened out, calculates the load condition measurement value of the node through the load pressure data of the node, compares the load condition measurement value with the load threshold, and adjusts the load balancing strategy according to the comparison result;

[0039] The multi-node load balancing module, after executing the load balancing strategy, calculates the total load measurement value of all nodes of the same type through the load pressure data of the node, compares the total load measurement value with the total load threshold, and takes corresponding measures according to the comparison result;

[0040] The visualization and interaction module builds a dynamic visualization interface to display the key indicators of different types of nodes in real time. You can select a specific node or node type through the interface to view its detailed load information and historical records. It also provides an adjustment interface for the load balancing strategy and adjusts the load balancing parameters.

[0041] (III) Beneficial effects

[0042] The present invention provides an interactive distributed visualization management system and method, which has the following beneficial effects:

[0043] (1) By converting multi-classification node types into virtual variable representations, data sets containing multiple categories can be easily processed, making the data processing process more flexible. Each node is identified by a set of virtual variable values, which simplifies the complexity of the data and makes the subsequent data analysis and processing process clearer.

[0044] (2) By calculating the correlation coefficient between features and dummy variables, we can clearly understand which features have an important impact on the classification results, enhance the interpretability of the data, and screen out features that are significantly correlated with the target variable, which helps to process data more accurately and effectively and improve the accuracy of tasks such as classification and prediction.

[0045] (3) By assigning loads to the node type that is most suitable for processing that type of load, the performance advantages of each node can be maximized. For example, computing-intensive tasks are assigned to high-performance CPU nodes, while storage-intensive tasks are assigned to large-capacity storage nodes, thereby reducing performance waste caused by resource mismatch. Different types of nodes may have different hardware and software configurations. Centralizing the processing of the same type of load can reduce compatibility issues caused by configuration differences. This helps reduce system error rates and improve overall stability.

[0046] (4) Classified nodes can be allocated resources based on their specific load requirements. For example, for nodes with computationally intensive tasks, CPU and memory resources can be increased, while for nodes with large data transmission volumes, more emphasis may be placed on the allocation of network bandwidth. This targeted resource allocation can maximize the use of system resources and avoid resource waste. Classified nodes make system expansion more flexible and targeted. When business needs increase, corresponding types of nodes can be selectively added based on the load conditions of different types of nodes, rather than blindly increasing the scale of the entire system. This targeted expansion can more effectively meet business needs while reducing expansion costs.

[0047] (5) By displaying the key indicators of different types of nodes in real time, you can quickly understand the current load status and performance bottlenecks of the system. It provides an adjustment interface for the load balancing strategy, allowing users to adjust the load balancing parameters according to real-time data and business needs, so that the load balancing strategy can better adapt to changes in the system load and ensure efficient use of resources and stable operation of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of the steps of the interactive distributed visualization management method of the present invention;

[0049] Figure 2 It is a schematic diagram of the structure of the interactive distributed visualization management system of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] See also Figure 1 The present invention provides an interactive distributed visualization management method, comprising the following steps:

[0052] Step 1: Collect a data set containing multiple classification node types and multiple features, set a set of node type categories, create a dummy variable for each category, and convert each node type in the data set into a corresponding dummy variable representation, where each node is identified by a set of dummy variable values;

[0053] The step 1 includes the following contents:

[0054] Step 101: Collect a data set of multiple classification node types and multiple features through an API interface, log files, or by directly communicating with each node;

[0055] Step 102: Remove duplicate, invalid or abnormal data entries, process missing values, and use methods such as filling (such as mean, median, mode, etc.), interpolation or deletion to convert data from different sources and dimensions into a unified format and dimension for subsequent processing and analysis;

[0056] Step 103: extracting features related to the node type from the preprocessed data, including performance parameters, task characteristics, connection relationships, and hardware attributes;

[0057] It should be noted that performance parameters include CPU usage (directly recording or calculating average / peak CPU usage), memory usage (recording the node's memory usage or memory usage), disk I / O (monitoring disk read and write rates or latency), and network bandwidth (monitoring the node's inbound and outbound network traffic).

[0058] Task characteristics include processing time (recording or estimating the time required for a task from submission to completion), task type (marking whether the task is a real-time, batch or periodic task), and task load (evaluating the current task queue length or load of the node);

[0059] The connection relationship includes the communication frequency between nodes (recording the number or frequency of communication between nodes and other nodes), the data exchange volume (calculating the amount of data exchanged between nodes); hardware attributes include node type (such as server, embedded device, etc.), CPU model and number of cores (hardware specifications may affect processing capabilities), and memory capacity and type (affecting the execution efficiency of memory-intensive tasks);

[0060] Step 104: Set a category set of node types, such as {real-time processing, batch processing, Internet of Things}, create a dummy variable for each category (for three categories, two dummy variables are usually created to avoid multicollinearity), and convert each node type in the data set into a corresponding dummy variable representation, where each node is identified by a set of dummy variable values ​​(0 or 1);

[0061] It should be noted that when using correlation coefficients to evaluate the correlation between features and node types, node types are usually categorical variables (such as real-time processing, batch processing, and the Internet of Things), while features are continuous or ordered variables (such as CPU usage, memory usage, etc.). However, directly calculating the correlation between these features and node types may not be directly feasible because node types are categorical rather than continuous;

[0062] If the node type is multi-classified, first convert it into a dummy variable (also called a dummy variable or indicator variable), and then calculate the correlation between each dummy variable and the feature separately, focusing on the correlation between the feature and "whether it is a certain type of node". This is achieved by simplifying the multi-classification problem into multiple binary classification problems, or by calculating the correlation with the node type label encoding, and performing correlation analysis on the feature with the binary form of the target variable (real-time processing = 1, non-real-time processing = 0);

[0063] Assuming there are three node types: real-time processing (RT), batch processing (BT), and Internet of Things (IoT), each node type will be converted into one or more dummy variables. For the case of three categories, two dummy variables are created (because any two of the three categories can determine the third, so three dummy variables are not needed to avoid multicollinearity);

[0064] Real-time processing (RT): Create a dummy variable RT. If the node is real-time processing, RT=1, otherwise RT=0; Batch processing (BT): Similarly, create a dummy variable BT. If the node is batch processing, BT=1, otherwise BT=0; Internet of Things (IoT): Since only two dummy variables are created, there is no need to create a separate one for IoT. Nodes in the IoT category will be implicitly represented when RT=0 and BT=0, that is, IoT=1;

[0065] When using, combine the contents of step 101 to step 104:

[0066] By converting multi-classification node types into virtual variable representations, data sets containing multiple categories can be easily processed, making the data processing process more flexible. Each node is identified by a set of virtual variable values, which simplifies the complexity of the data and makes subsequent data analysis and processing clearer.

[0067] Step 2: For each feature in the data set, calculate the correlation coefficient between it and each virtual variable, record and save all calculated correlation coefficients, remove features whose correlation with the target variable is lower than the correlation threshold, and screen out features that are significantly correlated with the node type;

[0068] The step 2 includes the following contents:

[0069] Step 201: For each feature in the data set, calculate the correlation coefficient between it and each virtual variable, and record and save all calculated correlation coefficients;

[0070] For feature X and dummy variable RT, the calculation formula of the correlation coefficient is as follows:

[0071]

[0072] Among them, r X,RT represents the correlation coefficient between feature X and dummy variable RT, x i represents the observed value of the i-th feature X, RT i represents the observed value (0 or 1) of the i-th dummy variable RT, i = 1, 2, ..., n, n represents the number of observed values, represents the mean of feature X, represents the mean of the dummy variable RT;

[0073] The same calculation is performed for the BT dummy variable;

[0074] It should be noted that for each feature, you will get two (or more, if you have more categories) correlation coefficients, which represent the correlation between the feature and each node type. A positive correlation indicates that as the feature value increases, the probability of the node belonging to that category also increases. A negative correlation is the opposite. The larger the absolute value of the correlation coefficient, the stronger the relationship.

[0075] Step 202: Analyze the correlation coefficient between each feature and different node types to determine whether it is positively correlated, negatively correlated, or has no significant correlation, specifically including:

[0076] When the correlation coefficient is greater than zero, it indicates a positive correlation; when the correlation coefficient is less than zero, it indicates a negative correlation; and when the correlation coefficient is equal to zero, it indicates no significant correlation.

[0077] The value range of the correlation coefficient is between -1 and 1. The closer the correlation coefficient is to 1 or -1, the stronger the correlation is, and the closer it is to 0, the weaker the correlation is.

[0078] Step 203: pre-set a correlation threshold, remove features whose correlation with the target variable is lower than the correlation threshold, and screen out features that are significantly correlated with the node type;

[0079] When used, combine the contents of step 201 to step 203:

[0080] By calculating the correlation coefficient between features and virtual variables, we can clearly understand which features have an important impact on the classification results, enhance the interpretability of the data, and screen out features that are significantly correlated with the target variable, which helps to process data more accurately and effectively and improve the accuracy of tasks such as classification and prediction.

[0081] Step 3: Based on the significantly relevant features screened out, all nodes are classified, and the load situation measurement value of the node is calculated through the load pressure data of the node. The load situation measurement value is compared with the load threshold, and the load balancing strategy is adjusted according to the comparison result;

[0082] The step three includes the following contents:

[0083] Step 301: Based on the screened significant related features, all nodes are classified, and for each type of node, its load pressure (such as CPU usage, memory occupancy, network bandwidth, etc.) data is collected;

[0084] It should be noted that, through the significantly correlated features screened out, all nodes can be classified by selecting appropriate classification algorithms (such as decision trees, random forests, K-nearest neighbors, support vector machines, neural networks, etc.), using the significantly correlated features screened out as input features and the node type as the target variable for training. After the training is completed, the trained classification model is used to classify new data and predict the type of each node;

[0085] You can also manually define classification rules based on business knowledge and data characteristics. For example, based on the correlation between features and node types, you can pre-set thresholds to filter out strongly correlated nodes. The node type corresponding to the feature is the type of the node. Then, traverse each node in the data set, compare its features with the classification rules, and assign the node to the corresponding type based on the matching results.

[0086] Step 302: Calculate the load condition measurement value of each node through the load pressure data of the node, including CPU utilization, memory utilization and network utilization. The calculation formula is as follows:

[0087] FS j =k1*Cs j +k2*Ms j +k3*Ls j

[0088] Among them, FS j represents the load measurement value of the jth node, Cs j represents the CPU utilization of the jth node, Ms j Indicates the memory utilization of the jth node, Ls j represents the network utilization of the jth node, k1, k2 and k3 represent weights, 0 <k1<1,0<k2<1,0<k3<1;

[0089] Step 303: presetting a load threshold, comparing the load condition measurement value with the load threshold, and adjusting the load balancing strategy according to the comparison result, specifically including:

[0090] When the load condition measurement value is less than the load threshold, it means that the current node load is at a relatively low level and the system resources are sufficient. The load balancing strategy is adjusted to allocate requests or tasks of the same type of nodes to the current node to achieve balanced utilization of resources.

[0091] When the load measurement value is greater than or equal to the load threshold, it means that the current node has reached or exceeded its capacity and may face performance bottlenecks or failure risks. The emergency load balancing strategy is implemented to automatically redirect some requests on the node in proportion to the preset ratio (such as 20%, 30%, etc.) to the same type of nodes with a load measurement value less than the load threshold until the load measurement value of the current node is less than the load threshold.

[0092] It should be noted that in the process of optimizing the load balancing strategy, in view of the inherent configuration performance differences between various types of nodes, in order to ensure that system resources are fully utilized and effectively avoid potential compatibility issues, a key strategy is to allocate the main load to the same type of nodes in a targeted manner. This approach can not only maximize the unique performance advantages of each type of node, such as allocating computing-intensive tasks to high-performance processor nodes and storage-intensive tasks to large-capacity storage nodes, but also significantly reduce performance bottlenecks and errors caused by configuration mismatches;

[0093] When used, combine the contents of step 301 to step 303:

[0094] By assigning loads to the node type that is most suitable for processing that type of load, the performance advantages of each node can be maximized. For example, computing-intensive tasks are assigned to high-performance CPU nodes, while storage-intensive tasks are assigned to large-capacity storage nodes, thereby reducing performance waste caused by resource mismatch. Different types of nodes may have different hardware and software configurations. Centralizing the processing of the same type of load can reduce compatibility issues caused by configuration differences. This helps reduce system error rates and improve overall stability.

[0095] Step 4: After executing the load balancing strategy, for each type of node, calculate the total load measurement value of all nodes of the same type through the load pressure data of the node, compare the total load measurement value with the total load threshold, and take corresponding measures according to the comparison result;

[0096] Step 401: After executing the load balancing strategy, the total load measurement value of all nodes of the same type is calculated through the load pressure data of the node, including CPU utilization, memory utilization and network utilization. The calculation formula is as follows:

[0097]

[0098] Among them, FS represents the total load measurement value of all nodes of the same type, Cs j represents the CPU utilization of the jth node, Ms j Indicates the memory utilization of the jth node, Ls j represents the network utilization of the jth node, j = 1, 2, ..., M, M represents the number of nodes, k1, k2 and k3 represent weights, 0 <k1<1,0<k2<1,0<k3<1;

[0099] Step 402: pre-set a total load threshold, compare the total load measurement value with the total load threshold, and take corresponding measures according to the comparison result, specifically including:

[0100] When the total load measurement value is less than the total load threshold, it means that the current resources are in excess. The load lower limit threshold is set in advance, and the services on the nodes whose load situation measurement value is lower than the load lower limit threshold are migrated to the nodes whose load situation measurement value is higher than the load lower limit threshold. The offline services are completely migrated to the nodes, and the resources are recovered;

[0101] When the total load measurement value is greater than or equal to the total load threshold, it means that a large number of nodes are currently under high load. The load upper limit threshold is set in advance, the number of nodes is increased or the node performance is improved until the load measurement values ​​of all nodes are lower than the load upper limit threshold.

[0102] It should be noted that in a distributed system, each type of node carries different workloads according to its role and responsibility (such as database server, application server, cache server, etc.), and therefore has specific requirements for system resources (CPU, memory, storage, network bandwidth, etc.). Therefore, it is necessary to make corresponding adjustments and optimizations based on the load requirements of each type of node;

[0103] When used, combine the contents of step 401 to step 402:

[0104] Classified nodes can be allocated resources based on their specific load requirements. For example, for nodes with compute-intensive tasks, CPU and memory resources can be increased, while for nodes with large data transmission volumes, more emphasis may be placed on the allocation of network bandwidth. This targeted resource allocation can maximize the use of system resources and avoid resource waste. Classified nodes make system expansion more flexible and targeted. When business needs increase, corresponding types of nodes can be selectively added based on the load conditions of different types of nodes, rather than blindly increasing the scale of the entire system. This targeted expansion can more effectively meet business needs while reducing expansion costs.

[0105] Step 5: Build a dynamic visualization interface to display the key indicators of different types of nodes in real time. Select a specific node or node type through the interface to view its detailed load information and history. Provide an adjustment interface for the load balancing strategy and adjust the load balancing parameters.

[0106] The step five includes the following contents:

[0107] Step 501: construct a dynamic visualization interface, using a variety of chart forms such as heat maps, bar charts, pie charts, line charts, etc., to display key indicators such as CPU, memory, and network utilization of different types of nodes in real time, and intuitively display the load distribution of each type of node;

[0108] Step 502: Setting a scheduled task or using an event-driven mechanism to periodically obtain the latest node status information from the data source and update it to the visualization interface;

[0109] Step 503: Select a specific node or node type through the interface to view its detailed load information and history records, provide an interface for adjusting the load balancing strategy, adjust the load balancing parameters according to the actual situation, including load thresholds, weights, etc., and support users to manually intervene in the load balancing process, such as manually migrating tasks, adjusting resource allocation, etc.;

[0110] When used, combine the contents of step 501 to step 503:

[0111] By displaying key indicators of different types of nodes in real time, you can quickly understand the current load status and performance bottlenecks of the system. It provides an adjustment interface for the load balancing strategy, allowing users to adjust load balancing parameters based on real-time data and business needs, so that the load balancing strategy can better adapt to changes in system load and ensure efficient use of resources and stable operation of the system.

[0112] See also Figure 2 The present invention also provides an interactive distributed visualization management system, including: a feature engineering module, a feature selection module, a single-node load balancing module, a multi-node load balancing module, and a visualization and interaction module; wherein,

[0113] The feature engineering module collects a data set containing multiple classification node types and multiple features, sets a set of categories for the node types, creates a dummy variable for each category, and converts each node type in the data set into a corresponding dummy variable representation, where each node is identified by a set of dummy variable values;

[0114] Feature selection module, for each feature in the data set, calculates the correlation coefficient between it and each virtual variable, records and saves all calculated correlation coefficients, removes features whose correlation with the target variable is lower than the correlation threshold, and screens out features that are significantly correlated with the node type;

[0115] The single-node load balancing module classifies all nodes based on the significantly relevant features screened out, calculates the load condition measurement value of the node through the load pressure data of the node, compares the load condition measurement value with the load threshold, and adjusts the load balancing strategy according to the comparison result;

[0116] The multi-node load balancing module, after executing the load balancing strategy, calculates the total load measurement value of all nodes of the same type through the load pressure data of the node, compares the total load measurement value with the total load threshold, and takes corresponding measures according to the comparison result;

[0117] The visualization and interaction module builds a dynamic visualization interface to display the key indicators of different types of nodes in real time. You can select a specific node or node type through the interface to view its detailed load information and historical records. It also provides an adjustment interface for the load balancing strategy and adjusts the load balancing parameters.

[0118] In the application, the several formulas involved are all calculated by removing dimensions and taking their numerical values. The formula is a formula of the most recent real situation obtained by collecting a large amount of data and performing software simulation. The coefficients in the formula are set by technicians in this field according to actual conditions.

[0119] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product. A person of ordinary skill in the art may appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented in combination with electronic hardware, computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution.

[0120] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0121] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. An interactive distributed visualization management method, characterized in that: The following steps are involved: Collect a dataset containing multiple classification node types and multiple features, set a set of node type categories, create a dummy variable for each category, and convert each node type in the dataset into a corresponding dummy variable representation, where each node is identified by a set of dummy variable values; For each feature in the data set, calculate the correlation coefficient between it and each virtual variable, record and save all calculated correlation coefficients, remove features whose correlation with the target variable is lower than the correlation threshold, and screen out features that are significantly correlated with the node type; Based on the significantly relevant features screened out, all nodes are classified, and the load situation measurement value of the node is calculated through the load pressure data of the node. The load situation measurement value is compared with the load threshold, and the load balancing strategy is adjusted according to the comparison result; After executing the load balancing strategy, for each type of node, the total load measurement value of all nodes of the same type is calculated through the load pressure data of the node, the total load measurement value is compared with the total load threshold, and corresponding measures are taken according to the comparison result; Build a dynamic visualization interface to display the key indicators of different types of nodes in real time. Select a node or node type through the interface to view its detailed load information and historical records. Provide an adjustment interface for the load balancing strategy and adjust the load balancing parameters.

2. The interactive distributed visualization management method according to claim 1, characterized in that: Create a virtual variable RT. If the node is real-time processing, RT=1, otherwise RT=0. Create a virtual variable BT. If the node is batch processing, BT=1, otherwise BT=0. Create a virtual variable IoT. Nodes of the IoT category will be implicitly represented when RT=0 and BT=0, that is, IoT=1.

3. The interactive distributed visualization management method according to claim 1, characterized in that: For each feature in the data set, calculate the correlation coefficient between it and each virtual variable, record and save all calculated correlation coefficients. The calculation formula of the correlation coefficient is as follows: Among them, r X,RT represents the correlation coefficient between feature X and dummy variable RT, x i represents the observed value of the i-th feature X, RT i represents the observed value of the i-th dummy variable RT, i = 1, 2, ..., n, n represents the number of observed values, represents the mean of feature X, represents the mean of the dummy variable RT.

4. The interactive distributed visualization management method according to claim 3, characterized in that: Analyze the correlation coefficient between each feature and different node types to determine whether it is positively correlated, negatively correlated, or not significantly correlated, including: When the correlation coefficient is greater than zero, it indicates a positive correlation; when the correlation coefficient is less than zero, it indicates a negative correlation; and when the correlation coefficient is equal to zero, it indicates no significant correlation.

5. The interactive distributed visualization management method according to claim 1, characterized in that: The load pressure data of the node, including CPU utilization, memory utilization, and network utilization, is used to calculate the load measurement value of each node. The calculation formula is as follows: FS j =k1*Cs j +k2*Ms j +k3*Ls j Among them, FS j represents the load measurement value of the jth node, Cs j represents the CPU utilization of the jth node, Ms j Indicates the memory utilization of the jth node, Ls j represents the network utilization of the jth node, k1, k2 and k3 represent weights, 0 <k1<1,0<k2<1,0<k3<1。 6. The interactive distributed visualization management method according to claim 5, characterized in that: Preset the load threshold, compare the load measurement value with the load threshold, and adjust the load balancing strategy according to the comparison result, including: When the load measurement value is less than the load threshold, the load balancing strategy is adjusted to assign requests or tasks of the same type of nodes to the current node; When the load condition measurement value is greater than or equal to the load threshold, the emergency load balancing strategy is implemented to automatically redirect some requests on the node in proportion to the preset ratio to the same type of nodes with a load condition measurement value less than the load threshold until the load condition measurement value of the current node is less than the load threshold.

7. The interactive distributed visualization management method according to claim 1, characterized in that: After executing the load balancing strategy, the total load measurement value of all nodes of the same type is calculated through the node load pressure data, including CPU utilization, memory utilization, and network utilization. The calculation formula is as follows: Among them, FS represents the total load measurement value of all nodes of the same type, Cs j represents the CPU utilization of the jth node, Ms j Indicates the memory utilization of the jth node, Ls j represents the network utilization of the jth node, j = 1, 2, ..., M, M represents the number of nodes, k1, k2 and k3 represent weights, 0 <k1<1,0<k2<1,0<k3<1。 8. The interactive distributed visualization management method according to claim 7, characterized in that: The total load threshold is preset, the total load measurement value is compared with the total load threshold, and corresponding measures are taken according to the comparison result, including: When the total load measurement value is less than the total load threshold, a load lower limit threshold is set in advance, and services on nodes whose load situation measurement value is lower than the load lower limit threshold are migrated to nodes whose load situation measurement value is higher than the load lower limit threshold, and the offline services are completely migrated to the nodes, and resources are recovered; When the total load measurement value is greater than or equal to the total load threshold, the load upper limit threshold is set in advance, the number of nodes is increased or the node performance is improved until the load measurement values ​​of all nodes are lower than the load upper limit threshold.

9. An interactive distributed visualization management system, used to implement the method according to any one of claims 1 to 8, characterized in that: include: The feature engineering module collects a data set containing multiple classification node types and multiple features, sets a set of categories for the node types, creates a dummy variable for each category, and converts each node type in the data set into a corresponding dummy variable representation, where each node is identified by a set of dummy variable values; Feature selection module, for each feature in the data set, calculates the correlation coefficient between it and each virtual variable, records and saves all calculated correlation coefficients, removes features whose correlation with the target variable is lower than the correlation threshold, and screens out features that are significantly correlated with the node type; The single-node load balancing module classifies all nodes based on the significantly relevant features screened out, calculates the load condition measurement value of the node through the load pressure data of the node, compares the load condition measurement value with the load threshold, and adjusts the load balancing strategy according to the comparison result; The multi-node load balancing module, after executing the load balancing strategy, calculates the total load measurement value of all nodes of the same type through the load pressure data of the node, compares the total load measurement value with the total load threshold, and takes corresponding measures according to the comparison result; The visualization and interaction module builds a dynamic visualization interface to display the key indicators of different types of nodes in real time. You can select a node or node type through the interface to view its detailed load information and historical records. It also provides an interface for adjusting the load balancing strategy and adjusting the load balancing parameters.

Images