System performance optimization platform construction method and system based on big data

By analyzing the distributed computing cluster of cloud service systems, identifying virtual machines and logical links, calculating transmission capacity and load balancing deviations, setting scheduling priorities, and building performance optimization solutions, the low performance optimization efficiency of big data systems in the existing technology is solved. The problem of the system improves the processing capacity and resource utilization rate.

CN119025289AActive Publication Date: 2024-11-26TIANJIN DONGJIANG STARLINK DIGITAL TECHNOLOGY CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411510086.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-28
Publication Date
2024-11-26
Estimated Expiration
2044-10-28

AI Technical Summary

Technical Problem

The existing system performance optimization platform construction method is inefficient when processing massive data, resulting in reuse of computing resources and affecting the system performance optimization effect.

Method used

By obtaining the distributed computing cluster of cloud service system, identifying service virtual machines, analyzing interactive virtual machines and logical links, calculating link transmission capacity and load balancing deviations, combining cluster logs and operation data, setting scheduling priorities, and building performance optimization plan.

Benefits of technology

The performance optimization efficiency of big data systems is improved, and by understanding the allocable bandwidth and load balancing of the logical link, evaluating cluster performance, optimizing resource allocation, and improving system response speed and throughput.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025289A_ABST
    Figure CN119025289A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of big data processing, and provides a system performance optimization platform construction method and system based on big data, and the method comprises the steps: extracting a distributed computing cluster of a cloud service system, recognizing a service virtual machine in the distributed computing cluster, querying a service function and a configuration parameter corresponding to the service virtual machine, and constructing a system performance optimization platform. Analyzing an interactive virtual machine in the service virtual machine, and determining a logic link corresponding to the interactive virtual machine; the link transmission capacity corresponding to the logic link is calculated, and the cluster transmission capacity corresponding to the distributed computing cluster is calculated; calculating load balance deviation between the logic links, performing normalization processing on the load balance deviation to obtain a target balance degree, collecting cluster operation data corresponding to the distributed computing cluster, and analyzing a cluster state corresponding to the distributed computing cluster; and calculating a cluster center load corresponding to the distributed computing cluster, and constructing a performance optimization scheme of the distributed computing cluster. The method aims at improving the system performance optimization efficiency of the big data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of big data processing technology, and in particular to a method and system for constructing a system performance optimization platform based on big data. Background Art

[0002] Big data system performance optimization refers to improving the performance of big data processing systems through various technical means and optimization strategies while ensuring data processing quality and efficiency. It can help the system process massive data faster and provide higher concurrency, lower latency, better scalability and higher throughput.

[0003] The existing method for building a system performance optimization platform is to adopt a batch processing method, which is suitable for the scenario of static data mining of massive data. The mode is to store first and then calculate. The data may be reused or processed, which increases the reuse of computing resources and thus reduces the efficiency of system performance optimization. Summary of the invention

[0004] The present invention provides a method and system for constructing a system performance optimization platform based on big data, the main purpose of which is to improve the efficiency of system performance optimization of big data.

[0005] To achieve the above-mentioned purpose, the present invention provides a method for constructing a system performance optimization platform based on big data, comprising: Acquire a cloud service system to be optimized, extract a distributed computing cluster of the cloud service system, identify a service virtual machine in the distributed computing cluster, query service functions and configuration parameters corresponding to the service virtual machine, analyze the interactive virtual machines in the service virtual machines according to the service functions, and determine the logical links corresponding to the interactive virtual machines; According to the configuration parameters, the link transmission capacity corresponding to the logical link is calculated, and according to the link transmission capacity, the cluster transmission capacity corresponding to the distributed computing cluster is calculated; According to the configuration parameters, the load balancing deviation between the logical links is calculated, the load balancing deviation is normalized to obtain a target balance degree, cluster operation data corresponding to the distributed computing cluster is collected, and the cluster state corresponding to the distributed computing cluster is analyzed according to the cluster operation data; Schedule the cluster log corresponding to the distributed computing cluster, calculate the cluster center load corresponding to the distributed computing cluster based on the cluster log, set the scheduling priority corresponding to the distributed computing cluster in combination with the cluster transmission capacity, the target balance and the cluster center load, and construct a performance optimization plan for the distributed computing cluster based on the scheduling priority and the cluster status.

[0006] Optionally, analyzing the interactive virtual machine in the service virtual machine according to the service function includes: Identify function information corresponding to the service function, and determine function modules and module components corresponding to the service virtual machine according to the function information; Scheduling the module data flow corresponding to the functional module, and analyzing the data endpoints corresponding to the module data flow; Access the component codes corresponding to the module components and analyze the dependency relationships between the component codes; The interactive virtual machine in the service virtual machine is analyzed according to the data endpoint and the dependency relationship.

[0007] Optionally, the determining the logical link corresponding to the interactive virtual machine includes: Obtain virtual network configuration information corresponding to the interactive virtual machine, and query communication equipment and equipment configuration information corresponding to the interactive virtual machine; Analyzing a device topology structure corresponding to the communication device according to the device configuration information; Determine the network topology structure corresponding to the interactive virtual machine in combination with the device topology structure and the network configuration information; According to the network topology, a logical link corresponding to the interactive virtual machine is determined.

[0008] Optionally, calculating the link transmission capacity corresponding to the logical link according to the configuration parameter includes: Calculating the idle bandwidth corresponding to the logical link according to the configuration parameters; Performing path tracing on each link in the logical link to obtain a tracing path; According to the tracking path, counting the number of passes corresponding to each link in the logical link; According to the number of passes and the idle bandwidth, the link transmission capacity corresponding to the logical link is calculated using the following formula: Where A represents the link transmission capacity corresponding to the logical link, represents the idle bandwidth of the ith logical link, Indicates the number of times the i-th logical link passes, and i represents the sequence number corresponding to the logical link Optionally, calculating the idle bandwidth corresponding to the logical link according to the configuration parameters includes: Extracting the network bandwidth corresponding to the interactive virtual machine from the configuration parameters; Calculating the link bandwidth corresponding to the logical link according to the network bandwidth; Counting the number of upstream and downstream virtual machines corresponding to the logical link to obtain a first number of virtual machines and a second number of virtual machines; According to the number of virtual machines 1 and the number of virtual machines 2, the global bandwidth corresponding to the logical link is calculated by the following formula: Wherein, D represents the global bandwidth corresponding to the logical link, F1 and F2 represent the number of virtual machines 1 and 2 of the Fth logical link, respectively. represents the total bandwidth corresponding to the a-th distributed computing cluster, a represents the sequence number corresponding to the distributed computing cluster, and R represents the number of distributed computing clusters; The idle bandwidth corresponding to the logical link is calculated according to the global bandwidth and the link bandwidth.

[0009] Optionally, calculating the cluster transmission capacity corresponding to the distributed computing cluster according to the link transmission capacity includes: Counting cluster failures corresponding to the distributed computing cluster, and calculating a cluster failure rate corresponding to the distributed computing cluster according to the cluster failures; Allocating a cluster weight corresponding to the distributed computing cluster according to the cluster failure rate; Combining the cluster weight and the link transmission capacity, the cluster transmission capacity corresponding to the distributed computing cluster is calculated by the following formula: Among them, G represents the cluster transmission capacity corresponding to the distributed computing cluster, represents the cluster weight, a is the serial number of the distributed computing cluster, represents the bth link transmission capacity, b is the sequence number corresponding to the link transmission capacity, and q is the total number corresponding to the link transmission capacity.

[0010] Optionally, calculating the load balancing deviation between the logical links according to the configuration parameter includes: Monitoring a link flow value corresponding to the logical link, and recording a monitoring period corresponding to the link flow value; Abnormal elimination is performed on the link flow value to obtain a target flow value; Calculate the average flow rate corresponding to the logical link by combining the target flow rate value and the monitoring period; The load balancing deviation corresponding to the logical link is calculated in combination with the configuration parameters and the average flow.

[0011] Optionally, the calculating the load balancing deviation corresponding to the logical link by combining the configuration parameter and the average flow includes: Extracting bandwidth parameters and port parameters from the configuration parameters; Calculating a flow extreme value corresponding to the logical link according to the bandwidth parameter and the port parameter; Calculating a load balancing deviation corresponding to the logical link according to the traffic extreme value and the average traffic; According to the load balancing deviation, calculating a coefficient of variation corresponding to the load balancing deviation; The load balancing deviation and the coefficient of variation are combined to calculate the load balancing deviation corresponding to the logical link.

[0012] Optionally, calculating the cluster center load corresponding to the distributed computing cluster according to the cluster log includes: According to the cluster log, searching for the cluster center node corresponding to the distributed computing cluster; Analyze the node performance indicators corresponding to the cluster center nodes using a preset performance analysis tool; Extracting the index log information corresponding to the node performance index from the cluster log; Calculate the index load corresponding to the node performance index according to the index log information; The cluster center load corresponding to the distributed computing cluster is calculated according to the indicator load.

[0013] A system for building a system performance optimization platform based on big data, characterized in that the system includes: A logical link determination module is used to obtain a cloud service system to be optimized, extract a distributed computing cluster of the cloud service system, identify a service virtual machine in the distributed computing cluster, query a service function and configuration parameters corresponding to the service virtual machine, analyze an interactive virtual machine in the service virtual machine according to the service function, and determine a logical link corresponding to the interactive virtual machine; A transmission capacity calculation module, used to calculate the link transmission capacity corresponding to the logical link according to the configuration parameters, and calculate the cluster transmission capacity corresponding to the distributed computing cluster according to the link transmission capacity; A state analysis module, configured to calculate the load balancing deviation between the logical links according to the configuration parameters, normalize the load balancing deviation to obtain a target balance degree, collect cluster operation data corresponding to the distributed computing cluster, and analyze the cluster state corresponding to the distributed computing cluster according to the cluster operation data; A performance optimization module is used to schedule the cluster log corresponding to the distributed computing cluster, calculate the cluster center load corresponding to the distributed computing cluster based on the cluster log, set the scheduling priority corresponding to the distributed computing cluster in combination with the cluster transmission capacity, the target balance and the cluster center load, and construct a performance optimization plan for the distributed computing cluster based on the scheduling priority and the cluster status.

[0014] The present invention queries the service function and configuration parameters corresponding to the service virtual machine, thereby understanding the technical service and corresponding attribute information of the service virtual machine, and analyzes the interactive virtual machine in the service virtual machine according to the service function, so as to obtain the virtual machines that interact with each other in the service virtual machine. The present invention calculates the link transmission capacity corresponding to the logical link according to the configuration parameters, and can understand the allocable bandwidth value corresponding to the logical link, thereby facilitating the subsequent calculation and processing of the cluster transmission capacity. The present invention calculates the load balancing deviation between the logical links according to the configuration parameters, and can understand the load balancing degree between the logical links, and then analyzes the stability between the logical links. The present invention calculates the cluster center load corresponding to the distributed computing cluster according to the cluster log, and can understand the resource usage of the central node in the distributed computing cluster, thereby facilitating the evaluation of the overall performance of the distributed computing cluster, providing a basis for the subsequent scheduling priority setting, and facilitating the improvement of the performance optimization efficiency of the distributed computing cluster. Therefore, the method and system for constructing a system performance optimization platform based on big data provided in the embodiment of the present invention can improve the system performance optimization efficiency of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flowchart of a method for constructing a system performance optimization platform based on big data provided by one embodiment of the present invention; Figure 2 A functional module diagram of a system performance optimization platform construction system based on big data provided by one embodiment of the present invention.

[0016] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0017] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0018] The embodiment of the present application provides a method for constructing a system performance optimization platform based on big data. In the embodiment of the present application, the execution subject of the method for constructing a system performance optimization platform based on big data includes but is not limited to at least one of the electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided by the embodiment of the present application. In other words, the method for constructing a system performance optimization platform based on big data can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.

[0019] Reference Figure 1 FIG. 1 is a flow chart of a method for constructing a system performance optimization platform based on big data provided by an embodiment of the present invention. In this embodiment, the method for constructing a system performance optimization platform based on big data includes steps S1-S4.

[0020] S1. Obtain a cloud service system to be optimized, extract a distributed computing cluster of the cloud service system, identify service virtual machines in the distributed computing cluster, query service functions and configuration parameters corresponding to the service virtual machines, analyze interactive virtual machines in the service virtual machines according to the service functions, and determine logical links corresponding to the interactive virtual machines.

[0021] The present invention queries the service functions and configuration parameters corresponding to the service virtual machine, thereby understanding the technical services and corresponding attribute information of the service virtual machine, and analyzing the interactive virtual machines in the service virtual machine according to the service functions, so as to obtain the virtual machines that interact with each other in the service virtual machines, wherein the cloud service system is a system that provides computing, storage, network, application and other services through cloud computing technology, the distributed computing cluster is a cluster of computing resources composed of multiple servers in the cloud service system, the service virtual machine is a virtual computing environment in a physical computer in the distributed computing cluster, the service functions and the configuration parameters are respectively the technical services and related attribute information corresponding to the service virtual machine, the interactive virtual machine is the virtual machine that interacts with each other in the service virtual machine, and optionally, the service virtual machine in the distributed computing cluster can be obtained by identifying the IP address or host name in the distributed computing cluster; the service functions and configuration parameters corresponding to the service virtual machine can be obtained by querying the official documents of the service machine, and the official documents contain introductions and instructions to the service functions and configuration parameters.

[0022] As an embodiment of the present invention, analyzing the interactive virtual machine in the service virtual machine according to the service function includes: identifying functional information corresponding to the service function, determining functional modules and module components corresponding to the service virtual machine according to the functional information, scheduling module data streams corresponding to the functional modules, analyzing data endpoints corresponding to the module data streams, accessing component codes corresponding to the module components, analyzing dependencies between the component codes, and analyzing the interactive virtual machine in the service virtual machine according to the data endpoints and the dependencies.

[0023] Among them, the functional information is the description information in the service function, the functional module is the business logic component of the service virtual machine, which implements specific functions or services and provides an external interface for other components or systems to call, the module component is the component in the service virtual machine used to manage and organize functional modules, which includes module loading, initialization, runtime management, communication and other functions, the module data flow is the description of data input, processing and output in the functional module, the data endpoint is the data source and data destination corresponding to the module data flow, the component code is the composition code of the module component, such as the code corresponding to the composition function of the module component, and the dependency relationship represents the relationship between the component codes that need to rely on the functions, data or interfaces of other modules when performing tasks.

[0024] Optionally, the functional information corresponding to the service function can be identified by OCR recognition technology; the functional modules and module components corresponding to the service virtual machine can be determined according to the information semantics corresponding to the functional information; the module data flow corresponding to the functional module can be implemented by a scheduling algorithm, such as a high response ratio priority scheduling algorithm; the data endpoint corresponding to the module data flow can be obtained by analyzing the format and type of the data of the module data flow, and the data format and type obtained after different modules process the data will be different; the component code corresponding to the module component can be accessed by IDE tools; the dependency relationship between the component codes can be implemented by analyzing the reference relationship between the component codes, such as calling functions of other components; the interactive virtual machine in the service virtual machine can determine the module of the data source and the module of the data output according to the input and output in the data endpoint, and analyze the virtual machine with interaction in combination with the dependency relationship.

[0025] The present invention can obtain the data transmission path between the interactive virtual machines by determining the logical link corresponding to the interactive virtual machines, providing a basis for calculating the subsequent link transmission capacity, wherein the logical link is the data transmission path between the interactive virtual machines.

[0026] As an embodiment of the present invention, determining the logical link corresponding to the interactive virtual machine includes: obtaining virtual network configuration information corresponding to the interactive virtual machine, querying the communication equipment and equipment configuration information corresponding to the interactive virtual machine, analyzing the equipment topology structure corresponding to the communication equipment according to the equipment configuration information, determining the network topology structure corresponding to the interactive virtual machine in combination with the equipment topology structure and the network configuration information, and determining the logical link corresponding to the interactive virtual machine according to the network topology structure.

[0027] Among them, the virtual network configuration information is the IP address and subnet information corresponding to the interactive virtual machine, the communication device is a network device for communication between the interactive virtual machines, such as a virtual interactive machine and a physical interactive machine, the device configuration information is the port connection and VLAN configuration information corresponding to the communication device, the device topology structure is the connection method and relationship between the communication devices, and the network topology structure is the network communication connection method and relationship between the interactive virtual machines.

[0028] Optionally, the virtual network configuration information corresponding to the interactive virtual machine can be obtained by using a network management tool, and the network management tool includes an SNMP (Simple Network Management Protocol) management tool; the communication device and device configuration information corresponding to the interactive virtual machine can be queried through a virtualization management platform, such as VMware vSphere; the device topology structure corresponding to the communication device can be obtained by analyzing the device configuration information, and the device configuration information usually includes the IP address, subnet mask, gateway, MAC address and other information of each device, which can help determine the connection method between devices, such as through a router connection, a switch connection or a direct connection, etc. In addition, the physical connection between devices can be understood by viewing the port information in the device configuration information. For example, the switch configuration will specify the connection object of each port, such as which server, virtual machine or other switch is connected to; the network topology structure corresponding to the interactive virtual machine can be obtained by combining the device topology structure and the network configuration information, and the subnet where the interactive virtual machine is located and the network connection method between other virtual machines or physical devices can be determined through the device topology structure and the network configuration information; the logical link corresponding to the interactive virtual machine can be obtained according to the connection method in the network topology structure.

[0029] S2. Calculate the link transmission capacity corresponding to the logical link according to the configuration parameters, and calculate the cluster transmission capacity corresponding to the distributed computing cluster according to the link transmission capacity.

[0030] The present invention calculates the link transmission capacity corresponding to the logical link according to the configuration parameters, and can understand the allocatable bandwidth value corresponding to the logical link, thereby facilitating the subsequent calculation and processing of the cluster transmission capacity, wherein the link transmission capacity represents the data transmission rate corresponding to the logical link.

[0031] As an embodiment of the present invention, the calculating the link transmission capacity corresponding to the logical link according to the configuration parameters includes: calculating the idle bandwidth corresponding to the logical link according to the configuration parameters, performing path tracing on each link in the logical link to obtain a tracing path, counting the number of passes corresponding to each link in the logical link according to the tracing path, and calculating the link transmission capacity corresponding to the logical link according to the number of passes and the idle bandwidth using the following formula: Where A represents the link transmission capacity corresponding to the logical link, represents the idle bandwidth of the ith logical link, It indicates the number of times the i-th logical link is passed, and i indicates the sequence number corresponding to the logical link.

[0032] The idle bandwidth represents the remaining transmission volume corresponding to the logical link, the path tracking is the path tracking through the logical link when the distributed computing cluster is processing, and the number of passes is the number of times each link in the logical link is passed when the distributed computing cluster is processing.

[0033] Optionally, the path tracing of each link in the logical link may be implemented by a path tracing algorithm, such as a depth-first search; the number of passes corresponding to each link in the logical link may be obtained by statistically analyzing the tracing path.

[0034] Further, as an optional embodiment of the present invention, the calculating the idle bandwidth corresponding to the logical link according to the configuration parameters includes: extracting the network bandwidth corresponding to the interactive virtual machine from the configuration parameters, calculating the link bandwidth corresponding to the logical link according to the network bandwidth, counting the number of upstream and downstream virtual machines corresponding to the logical link to obtain a first number of virtual machines and a second number of virtual machines, and calculating the global bandwidth corresponding to the logical link according to the first number of virtual machines and the second number of virtual machines by the following formula: Wherein, D represents the global bandwidth corresponding to the logical link, F1 and F2 represent the number of virtual machines 1 and 2 of the Fth logical link, respectively. represents the total bandwidth corresponding to the a-th distributed computing cluster, a represents the sequence number corresponding to the distributed computing cluster, and R represents the number of distributed computing clusters; The idle bandwidth corresponding to the logical link is calculated according to the global bandwidth and the link bandwidth.

[0035] Among them, the network bandwidth is the data transmission volume corresponding to the interactive virtual machine, the link bandwidth represents the total transmission volume corresponding to the logical link, the number of virtual machines one and the number of virtual machines two respectively represent the number of virtual machines placed at the front and back positions corresponding to the logical link, and the global bandwidth represents the minimum guaranteed bandwidth reserved in advance for the distributed computing cluster by the logical link in order to ensure the normal operation of the task.

[0036] Optionally, the link bandwidth corresponding to the logical link can be obtained by summing the network bandwidth corresponding to the logical link; the number of upstream and downstream virtual machines corresponding to the logical link can be obtained by counting according to the corresponding link topology structure diagram; the idle bandwidth corresponding to the logical link can be obtained by subtracting the global bandwidth from the link bandwidth.

[0037] The present invention calculates the cluster transmission capacity corresponding to the distributed computing cluster based on the link transmission capacity, and then obtains the allocatable bandwidth value corresponding to the distributed computing cluster, providing a basis for subsequent performance optimization processing, wherein the cluster transmission capacity represents the allocatable bandwidth value corresponding to the distributed computing cluster.

[0038] As an embodiment of the present invention, the cluster transmission capacity corresponding to the distributed computing cluster is calculated according to the link transmission capacity, including: counting cluster failures corresponding to the distributed computing cluster, calculating the cluster failure rate corresponding to the distributed computing cluster according to the cluster failures, allocating a cluster weight corresponding to the distributed computing cluster according to the cluster failure rate, and combining the cluster weight and the link transmission capacity to calculate the cluster transmission capacity corresponding to the distributed computing cluster by the following formula: Among them, G represents the cluster transmission capacity corresponding to the distributed computing cluster, represents the cluster weight, a is the serial number of the distributed computing cluster, represents the bth link transmission capacity, b is the sequence number corresponding to the link transmission capacity, and q is the total number corresponding to the link transmission capacity.

[0039] Among them, the cluster failure refers to all failures that occur in the distributed computing cluster, such as node failure, network failure, storage failure, etc. The cluster failure rate represents the probability of failure of the distributed computing cluster within a specific time, and the cluster weight represents the importance assigned to the distributed computing cluster when processing tasks.

[0040] Optionally, the cluster failure corresponding to the distributed computing cluster can be obtained by counting historical failure data; the cluster failure rate corresponding to the distributed computing cluster can be obtained by calculating the ratio of the frequency corresponding to the cluster failure to the total number of failures; the cluster weight corresponding to the distributed computing cluster can be set according to the numerical value of the cluster failure rate, for example, if the failure rate is 10, the cluster weight can be set to 0.9.

[0041] S3. According to the configuration parameters, calculate the load balancing deviation between the logical links, normalize the load balancing deviation to obtain the target balance degree, collect the cluster operation data corresponding to the distributed computing cluster, and analyze the cluster status corresponding to the distributed computing cluster according to the cluster operation data.

[0042] The present invention calculates the load balancing deviation between the logical links based on the configuration parameters, so as to understand the load balancing degree between the logical links and further analyze the stability between the logical links, wherein the load balancing deviation represents the degree of load distribution deviation between the logical links.

[0043] As an embodiment of the present invention, the load balancing deviation between the logical links is calculated according to the configuration parameters, including: monitoring the link flow value corresponding to the logical link, and recording the monitoring period corresponding to the link flow value, eliminating abnormalities in the link flow value to obtain a target flow value, combining the target flow value and the monitoring period to calculate the average flow corresponding to the logical link, and combining the configuration parameters and the average flow to calculate the load balancing deviation corresponding to the logical link.

[0044] Among them, the link flow value is the data transmission volume corresponding to the logical link during the monitoring period, and the target flow value is the flow value obtained after removing the abnormal values ​​in the link flow value. Optionally, the link flow value corresponding to the logical link can be obtained by monitoring a monitoring system, such as a Nagios system; the average flow corresponding to the logical link can be obtained by calculating the ratio of the link flow value and the monitoring period.

[0045] Further, as an optional embodiment of the present invention, the combining of the configuration parameters and the average traffic to calculate the load balancing deviation corresponding to the logical link includes: extracting the bandwidth parameters and port parameters in the configuration parameters, calculating the traffic extreme value corresponding to the logical link according to the bandwidth parameters and the port parameters, calculating the load balancing deviation corresponding to the logical link according to the traffic extreme value and the average traffic, calculating the coefficient of variation corresponding to the load balancing deviation according to the load balancing deviation, and calculating the load balancing deviation corresponding to the logical link in combination with the load balancing deviation and the coefficient of variation.

[0046] Among them, the traffic extreme value represents the maximum transmission traffic and the minimum transmission traffic corresponding to the logical link, the minimum transmission traffic is set to 0, the load balancing deviation represents the load distribution balance corresponding to the logical link, and the coefficient of variation represents the distribution volatility corresponding to the load balancing deviation.

[0047] Optionally, the bandwidth parameters and port parameters in the configuration parameters can be extracted by an extraction function, and the extraction function is compiled by a scripting language, such as JAVA language; the calculation method of the traffic extreme value corresponding to the logical link is: traffic maximum value = bandwidth in the bandwidth parameter * port rate in the port parameter; the calculation formula of the load balancing deviation corresponding to the logical link is: load balancing deviation = (maximum traffic - minimum traffic) / average traffic; the coefficient of variation corresponding to the load balancing deviation can be obtained by calculating the standard deviation of the load balancing deviation; the load balancing deviation corresponding to the logical link can be obtained by calculating the coefficient of variation / the load balancing deviation.

[0048] The present invention can eliminate the influence between the load balancing deviations by normalizing the load balancing deviations, so as to facilitate comparison between the load balancing deviations. According to the cluster operation data, the cluster state corresponding to the distributed computing cluster is analyzed, and then the health state and load distribution of the distributed computing cluster are understood, so as to facilitate fault self-checking or determine the load that needs to be optimized. The target balance degree is the balance degree obtained after the load balancing deviations are unified to the same standard. The cluster operation data is the data generated during the operation of the distributed computing cluster, such as component status, performance indicators and event record data. The cluster state is a description of the operation state of the distributed computing cluster. Optionally, the normalization of the load balancing deviation can be achieved through a maximum and minimum normalization method. The cluster operation data corresponding to the distributed computing cluster can be achieved through sensors, such as energy consumption sensors and load sensors. The cluster state corresponding to the distributed computing cluster can be analyzed based on the cluster operation data, such as whether there is abnormal data in the operation data, and the abnormal state of the cluster is judged based on the abnormal data. The completion state corresponding to the operation data can analyze whether the corresponding cluster is idle.

[0049] S4. Schedule the cluster log corresponding to the distributed computing cluster, calculate the cluster center load corresponding to the distributed computing cluster based on the cluster log, set the scheduling priority corresponding to the distributed computing cluster in combination with the cluster transmission capacity, the target balance and the cluster center load, and construct a performance optimization plan for the distributed computing cluster based on the scheduling priority and the cluster status.

[0050] The present invention calculates the cluster center load corresponding to the distributed computing cluster based on the cluster log, so as to understand the resource usage of the central node in the distributed computing cluster, thereby facilitating the evaluation of the overall performance of the distributed computing cluster, providing a basis for the subsequent setting of scheduling priorities, and facilitating the improvement of the performance optimization efficiency of the distributed computing cluster, wherein the cluster log is information generated and recorded by each node in the distributed computing cluster, and the cluster center load refers to the load of the central node in the distributed computing cluster.

[0051] As an embodiment of the present invention, calculating the cluster center load corresponding to the distributed computing cluster based on the cluster log includes: searching for the cluster center node corresponding to the distributed computing cluster based on the cluster log, analyzing the node performance index corresponding to the cluster center node using a preset performance analysis tool, extracting the index log information corresponding to the node performance index from the cluster log, calculating the index load corresponding to the node performance index based on the index log information, and calculating the cluster center load corresponding to the distributed computing cluster based on the index load.

[0052] Among them, the cluster center node is the control node or management node corresponding to the distributed computing cluster, which is responsible for coordinating and scheduling tasks and resource allocation in the cluster. The performance analysis tool is a tool used to analyze various indicators in the cluster center node to determine the corresponding node performance indicators, such as the Perf tool. The node performance indicators are standards for evaluating the performance and efficiency of the cluster center node, such as CPU utilization, response time, memory utilization and throughput.

[0053] Optionally, the cluster center node corresponding to the distributed computing cluster can be obtained by searching the cluster configuration file; the indicator load corresponding to the node performance indicator can be calculated based on the indicator log information. For example, based on the extracted indicator log information, the average value of the performance indicator of each node in a given time period is calculated. For example, for CPU utilization, the average value of the CPU utilization of all nodes can be calculated, and the indicator load is obtained based on the average value; the cluster center load corresponding to the distributed computing cluster can be obtained by summing the indicator loads using a load balancing algorithm.

[0054] The present invention sets the scheduling priority corresponding to the distributed computing cluster by combining the cluster transmission capacity, the target balance and the cluster center load, so as to optimize the allocation of resources so that the cluster can get a faster response, wherein the scheduling priority represents the priority of resource scheduling corresponding to the distributed computing cluster. Optionally, the scheduling priority corresponding to the distributed computing cluster can be set by combining the cluster transmission capacity, the target balance and the corresponding values ​​of the cluster center load. For example, a high cluster transmission capacity indicates that transmission resources such as network bandwidth and storage bandwidth are sufficient, data transmission can be completed faster, and data transmission between tasks will not affect the entire cluster. It has a greater impact and the target balance is high, then the scheduling priority should give priority to allocating tasks to nodes with more idle resources to balance the load of each node in the cluster. The cluster center load is high, that is, there are more tasks waiting to be executed in the task queue, and these tasks may need to be completed more urgently. In this case, the scheduling priority can be set according to factors such as the importance, urgency and data transmission requirements of the task, and these high-priority tasks are scheduled first; the performance optimization plan of the distributed computing cluster can be constructed according to the scheduling priority and the cluster status. For example, if the cluster status is normal and idle, and the scheduling priority is high, the cluster can be scheduled as a load optimization cluster.

[0055] The present invention queries the service function and configuration parameters corresponding to the service virtual machine, thereby understanding the technical service and corresponding attribute information of the service virtual machine, and analyzes the interactive virtual machines in the service virtual machine according to the service function, so as to obtain the virtual machines that interact with each other in the service virtual machine. The present invention calculates the link transmission capacity corresponding to the logical link according to the configuration parameters, and can understand the allocable bandwidth value corresponding to the logical link, thereby facilitating the subsequent calculation and processing of the cluster transmission capacity. The present invention calculates the load balancing deviation between the logical links according to the configuration parameters, and can understand the load balancing degree between the logical links, and then analyzes the stability between the logical links. The present invention calculates the cluster center load corresponding to the distributed computing cluster according to the cluster log, and can understand the resource usage of the central node in the distributed computing cluster, thereby facilitating the evaluation of the overall performance of the distributed computing cluster, providing a basis for the subsequent scheduling priority setting, and facilitating the improvement of the performance optimization efficiency of the distributed computing cluster. Therefore, the method for constructing a system performance optimization platform based on big data provided in an embodiment of the present invention can improve the system performance optimization efficiency of data.

[0056] like Figure 2 As shown, it is a functional module diagram of a system performance optimization platform construction system based on big data provided by one embodiment of the present invention.

[0057] The system 100 for constructing a system performance optimization platform based on big data described in the present invention can be installed in an electronic device. According to the functions implemented, the system 100 for constructing a system performance optimization platform based on big data can include a logical link determination module 101, a transmission capacity calculation module 102, a state analysis module 103 and a performance optimization module 104. The module described in the present invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0058] In this embodiment, the functions of each module / unit are as follows: The logical link determination module 101 is used to obtain a cloud service system to be optimized, extract a distributed computing cluster of the cloud service system, identify a service virtual machine in the distributed computing cluster, query a service function and configuration parameter corresponding to the service virtual machine, analyze an interactive virtual machine in the service virtual machine according to the service function, and determine a logical link corresponding to the interactive virtual machine; The transmission capacity calculation module 102 is used to calculate the link transmission capacity corresponding to the logical link according to the configuration parameters, and calculate the cluster transmission capacity corresponding to the distributed computing cluster according to the link transmission capacity; The state analysis module 103 is used to calculate the load balancing deviation between the logical links according to the configuration parameters, normalize the load balancing deviation to obtain a target balance degree, collect cluster operation data corresponding to the distributed computing cluster, and analyze the cluster state corresponding to the distributed computing cluster according to the cluster operation data; The performance optimization module 104 is used to schedule the cluster log corresponding to the distributed computing cluster, calculate the cluster center load corresponding to the distributed computing cluster based on the cluster log, set the scheduling priority corresponding to the distributed computing cluster in combination with the cluster transmission capacity, the target balance and the cluster center load, and construct a performance optimization plan for the distributed computing cluster based on the scheduling priority and the cluster status.

[0059] In detail, each module described in the system 100 for constructing a system performance optimization platform based on big data described in the embodiment of the present application is used in the same manner as described above. Figure 1 The technical means described in the method for building a system performance optimization platform based on big data are the same and can produce the same technical effects, so they will not be repeated here.

[0060] In the several embodiments provided by the present invention, it should be understood that the provided methods and systems can be implemented in other ways. For example, the method embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for constructing a system performance optimization platform based on big data, characterized in that: The method comprises: Acquire a cloud service system to be optimized, extract a distributed computing cluster of the cloud service system, identify a service virtual machine in the distributed computing cluster, query service functions and configuration parameters corresponding to the service virtual machine, analyze the interactive virtual machines in the service virtual machines according to the service functions, and determine the logical links corresponding to the interactive virtual machines; According to the configuration parameters, the link transmission capacity corresponding to the logical link is calculated, and according to the link transmission capacity, the cluster transmission capacity corresponding to the distributed computing cluster is calculated; According to the configuration parameters, the load balancing deviation between the logical links is calculated, the load balancing deviation is normalized to obtain a target balance degree, cluster operation data corresponding to the distributed computing cluster is collected, and the cluster state corresponding to the distributed computing cluster is analyzed according to the cluster operation data; Schedule the cluster log corresponding to the distributed computing cluster, calculate the cluster center load corresponding to the distributed computing cluster based on the cluster log, set the scheduling priority corresponding to the distributed computing cluster in combination with the cluster transmission capacity, the target balance and the cluster center load, and construct a performance optimization plan for the distributed computing cluster based on the scheduling priority and the cluster status.

2. The method for constructing a system performance optimization platform based on big data according to claim 1, characterized in that: The step of analyzing the interactive virtual machine in the service virtual machine according to the service function includes: Identify function information corresponding to the service function, and determine function modules and module components corresponding to the service virtual machine according to the function information; Scheduling the module data flow corresponding to the functional module, and analyzing the data endpoints corresponding to the module data flow; Access the component codes corresponding to the module components and analyze the dependency relationships between the component codes; The interactive virtual machine in the service virtual machine is analyzed according to the data endpoint and the dependency relationship.

3. The method for constructing a system performance optimization platform based on big data according to claim 1, characterized in that: The determining the logical link corresponding to the interactive virtual machine includes: Obtain virtual network configuration information corresponding to the interactive virtual machine, and query communication equipment and equipment configuration information corresponding to the interactive virtual machine; Analyzing a device topology structure corresponding to the communication device according to the device configuration information; Determine the network topology structure corresponding to the interactive virtual machine in combination with the device topology structure and the network configuration information; According to the network topology, a logical link corresponding to the interactive virtual machine is determined.

4. The method for constructing a system performance optimization platform based on big data according to claim 1, characterized in that: The calculating, according to the configuration parameters, a link transmission capacity corresponding to the logical link comprises: Calculating the idle bandwidth corresponding to the logical link according to the configuration parameters; Performing path tracing on each link in the logical link to obtain a tracing path; According to the tracking path, counting the number of passes corresponding to each link in the logical link; According to the number of passes and the idle bandwidth, the link transmission capacity corresponding to the logical link is calculated using the following formula: Where A represents the link transmission capacity corresponding to the logical link, represents the idle bandwidth of the ith logical link, It indicates the number of times the i-th logical link is passed, and i indicates the sequence number corresponding to the logical link.

5. The method for constructing a system performance optimization platform based on big data according to claim 4, characterized in that: The calculating, according to the configuration parameters, the idle bandwidth corresponding to the logical link comprises: Extracting the network bandwidth corresponding to the interactive virtual machine from the configuration parameters; Calculating the link bandwidth corresponding to the logical link according to the network bandwidth; Counting the number of upstream and downstream virtual machines corresponding to the logical link to obtain a first number of virtual machines and a second number of virtual machines; According to the number of virtual machines 1 and the number of virtual machines 2, the global bandwidth corresponding to the logical link is calculated by the following formula: Wherein, D represents the global bandwidth corresponding to the logical link, F1 and F2 represent the number of virtual machines 1 and 2 of the Fth logical link, respectively. represents the total bandwidth corresponding to the a-th distributed computing cluster, a represents the sequence number corresponding to the distributed computing cluster, and R represents the number of distributed computing clusters; The idle bandwidth corresponding to the logical link is calculated according to the global bandwidth and the link bandwidth.

6. The method for constructing a system performance optimization platform based on big data according to claim 1, characterized in that: The calculating, according to the link transmission capacity, the cluster transmission capacity corresponding to the distributed computing cluster includes: Counting cluster failures corresponding to the distributed computing cluster, and calculating a cluster failure rate corresponding to the distributed computing cluster according to the cluster failures; Allocating a cluster weight corresponding to the distributed computing cluster according to the cluster failure rate; Combining the cluster weight and the link transmission capacity, the cluster transmission capacity corresponding to the distributed computing cluster is calculated by the following formula: Among them, G represents the cluster transmission capacity corresponding to the distributed computing cluster, represents the cluster weight, represents the bth link transmission capacity, b is the sequence number corresponding to the link transmission capacity, and q is the total number corresponding to the link transmission capacity.

7. The method for constructing a system performance optimization platform based on big data according to claim 1, characterized in that: The calculating, according to the configuration parameters, the load balancing deviation between the logical links comprises: Monitoring a link flow value corresponding to the logical link, and recording a monitoring period corresponding to the link flow value; Abnormal elimination is performed on the link flow value to obtain a target flow value; Calculate the average flow rate corresponding to the logical link by combining the target flow rate value and the monitoring period; The load balancing deviation corresponding to the logical link is calculated in combination with the configuration parameters and the average flow.

8. The method for constructing a system performance optimization platform based on big data according to claim 7, characterized in that: The calculating the load balancing deviation corresponding to the logical link by combining the configuration parameter and the average flow includes: Extracting bandwidth parameters and port parameters from the configuration parameters; Calculating a flow extreme value corresponding to the logical link according to the bandwidth parameter and the port parameter; Calculating a load balancing deviation corresponding to the logical link according to the traffic extreme value and the average traffic; According to the load balancing deviation, calculating a coefficient of variation corresponding to the load balancing deviation; The load balancing deviation and the coefficient of variation are combined to calculate the load balancing deviation corresponding to the logical link.

9. The method for constructing a system performance optimization platform based on big data according to claim 1, characterized in that: The calculating, according to the cluster log, the cluster center load corresponding to the distributed computing cluster includes: According to the cluster log, searching for the cluster center node corresponding to the distributed computing cluster; Analyze the node performance indicators corresponding to the cluster center nodes using a preset performance analysis tool; Extracting the index log information corresponding to the node performance index from the cluster log; Calculate the index load corresponding to the node performance index according to the index log information; The cluster center load corresponding to the distributed computing cluster is calculated according to the indicator load.

10. A system performance optimization platform construction system based on big data, characterized in that: The system comprises: A logical link determination module is used to obtain a cloud service system to be optimized, extract a distributed computing cluster of the cloud service system, identify a service virtual machine in the distributed computing cluster, query a service function and configuration parameters corresponding to the service virtual machine, analyze an interactive virtual machine in the service virtual machine according to the service function, and determine a logical link corresponding to the interactive virtual machine; A transmission capacity calculation module, used to calculate the link transmission capacity corresponding to the logical link according to the configuration parameters, and calculate the cluster transmission capacity corresponding to the distributed computing cluster according to the link transmission capacity; A state analysis module, configured to calculate the load balancing deviation between the logical links according to the configuration parameters, normalize the load balancing deviation to obtain a target balance degree, collect cluster operation data corresponding to the distributed computing cluster, and analyze the cluster state corresponding to the distributed computing cluster according to the cluster operation data; A performance optimization module is used to schedule the cluster log corresponding to the distributed computing cluster, calculate the cluster center load corresponding to the distributed computing cluster based on the cluster log, set the scheduling priority corresponding to the distributed computing cluster in combination with the cluster transmission capacity, the target balance and the cluster center load, and construct a performance optimization plan for the distributed computing cluster based on the scheduling priority and the cluster status.

Citation Information

Patent Citations

  • Method for realizing dynamic scheduling of server resources in virtualized cluster environment

    CN108519919A

  • Device and method for testing load balancing function and performance of virtualization platform

    CN112559129A

  • Distributed power consumption information data acquisition system considering network load

    CN113422842A

  • Cluster resource load balancing method and device, electronic equipment and medium

    CN114443284A

  • Service node scheduling method and device, computer equipment, storage medium and program product

    CN118784727A