Performance monitoring system and method for expansion and upgrading of big data platform

Abstract

The invention discloses a performance monitoring system and method for expansion and upgrading of a big data platform, and relates to the technical field of big data analysis. According to the performance monitoring system and method, key information of newly-added Web servers is collected to form a standardized configuration data set, and node deployment normalization and cluster compatibility in the expansion stage are improved; operating load logs before capacity expansion are directionally pulled and screened, and a historical cluster server operating load set is constructed through structural analysis; first dynamic weight configuration is completed by combining two types of load sets, the cache types of the Web servers are distinguished according to cache configuration identifiers, and secondary weight adjustment is performed on the shared distributed cache Web servers based on the hit rate, so that the defects of existing weight configuration are overcome, accurate weight adaptation is realized, the cache hit rate is increased, the frequency of requesting to penetrate through a database is reduced, and the service life of the Web servers is prolonged. The database load is reduced, the response delay is shortened, resource imbalance and service instability are avoided, and the performance of platform expansion upgrading operation is guaranteed.

Description

Technical Field

[0001] This invention relates to the field of big data analytics, specifically a performance monitoring system and method for expanding and upgrading big data platforms. Background Technology

[0002] As data scale continues to expand and business complexity continues to increase, the number of access requests and data processing pressure faced by big data platforms are surging simultaneously. Expansion and upgrading have become the core support for ensuring the platform's data processing efficiency, service response timeliness and operational stability. They are also necessary measures to avoid system overload and ensure the continuous and reliable operation of business. Expansion methods mainly include horizontal expansion, which is to add server nodes to expand the cluster size, and vertical expansion, which is to upgrade the existing server hardware configuration.

[0003] During the current expansion and upgrade process of big data platforms, the performance monitoring mechanism fails to fully integrate historical load data before expansion with real-time load data after expansion for dynamic calibration of load balancing weights. It also lacks differentiated adaptation for local caching and shared distributed caching types on web servers and fails to consider the correlation between shared distributed cache hit rates. This results in inaccurate load balancing weight configurations and a lack of targeted and effective performance monitoring during the expansion phase. Consequently, after expansion and upgrades, issues such as unstable service response, unbalanced resource utilization, and low cache access efficiency are prone to occur. These problems are often misjudged as inherent defects in the expansion and upgrade process itself, when in fact they are caused by the lack of performance monitoring and the imperfect weight adjustment mechanism during the expansion and upgrade process. This severely restricts the implementation effect and subsequent stable operation of big data platform expansion and upgrades. Therefore, a performance monitoring system and method for big data platform expansion and upgrades are urgently needed. Summary of the Invention

[0004] The purpose of this invention is to provide a performance monitoring system and method for expanding and upgrading big data platforms, so as to solve the problems raised in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a performance monitoring method for expanding and upgrading a big data platform, the performance monitoring method comprising the following steps: Step S1: After completing the deployment and configuration of the new Web server during the expansion phase of the big data platform, enter the configuration information of the new Web server into the big data platform cluster registry for registration. After the registration verification is passed and the new node is connected to the cluster communication link, establish a traffic forwarding link between the new node and the existing cluster through the load balancing service, simultaneously enable the service instance of the new Web server to complete the cluster collaborative parameter calibration, and configure the load balancing weight of each Web server according to the load balancing service API. Step S1-1: During the expansion phase of the big data platform, after completing the hardware deployment, network configuration, and service operation environment deployment of the new Web server, collect the node unique identifier, hardware parameters, network access address, service listening port, and cluster access credentials of the new Web server to form a standardized configuration dataset. Step S1-2: Enter the standardized configuration dataset into the big data platform cluster registry, and start the registry configuration information integrity verification, node identity authentication and cluster routing association process; Steps S1-3: After verification, authentication and association are passed, establish a bidirectional communication link between the new node and the existing cluster according to the cluster communication protocol, configure the traffic forwarding rules of the new node based on the load balancing service, start the service instance of the new Web server, calibrate the session sharing parameters, data transmission protocol parameters and service health check parameters required for cluster collaborative operation, and call the load balancing service API to enter the initial load balancing weight parameters of each Web server. By collecting key information such as the unique node identifier and hardware parameters of newly added web servers, a standardized configuration dataset is formed. After verifying the integrity of the configuration information, authenticating the node identity, and associating with the cluster routing, a bidirectional communication link is established between the new node and the existing cluster. Traffic forwarding rules are configured and cluster collaboration parameters such as session sharing parameters are calibrated. Initial load balancing weight parameters are entered to ensure that the new node can compliantly access the cluster and has the prerequisites for load sharing.

[0006] Step S2: Through the big data platform cluster log caching business API, based on the unique node identifier of the original Web server before the big data platform expansion, retrieve the running load log data within the preset time window before the expansion operation is executed, and determine the valid running load log data by verifying the time sequence relationship between the log timestamp and the expansion trigger timestamp. Step S2-1: Call the big data platform cluster log caching business API, inject the big data platform expansion trigger timestamp and the unique node identifier of the original web server before the big data platform expansion, initiate a targeted log retrieval request, and retrieve historical running load log data according to the unique node identifier of the original web server. Step S2-2: Verify the log timestamp of each item in the historical running load log data pulled, and filter out log items whose log timestamps are earlier than the big data platform expansion trigger timetamp to form valid running load log data. By calling the big data platform cluster log caching business API, injecting the expansion trigger timestamp and the original web server's unique node identifier to initiate a targeted request, historical operating load log data within a preset time window before expansion is retrieved. The log timestamps are verified one by one to filter out valid data earlier than the expansion trigger timestamp, providing accurate and time-series-valid historical data support for subsequent load analysis.

[0007] Step S3: Parse the valid runtime load log data to obtain the historical runtime load data of each Web server before the expansion of the big data platform, and record it as the historical cluster server runtime load set; obtain the runtime load data of each Web server after the expansion of the big data platform, and record it as the real-time cluster server runtime load set. The effective runtime load log data generated in step S2 is structured and parsed to extract CPU utilization, memory usage, and access request processing volume recorded in the logs. The access request processing volume is represented as the ratio of the number of active connections to the maximum number of connections processed by the service instance of the Web server in historical time. Based on the unique node identifiers of the original Web servers before the big data platform expansion, the data is categorized and aggregated, using the unique node identifiers of the Web servers as keys and the extracted CPU utilization, memory usage, and access request processing volume recorded in the logs as values, to construct a historical cluster server runtime load set. Through the big data platform cluster monitoring business API interface, the CPU utilization, memory usage, and access request processing volume of all Web servers after the big data platform expansion are collected in real time. All Web servers include newly added Web servers and existing Web servers. Based on the unique node identifiers of each Web server, the data is synchronously stored using the storage method of the historical cluster server runtime load set to construct a real-time cluster server runtime load set. By structured parsing of effective operational load log data, CPU utilization, memory usage, and access request processing volume are extracted and categorized according to the original Web server's unique node identifier to construct a historical cluster server operational load set. At the same time, similar load data of all Web servers after expansion are collected in real time and associated with the same storage method to construct a real-time cluster server operational load set, achieving comprehensive coverage and standardized storage of load data before and after expansion.

[0008] Step S4: After configuring the load balancing weight through the load balancing business API, the deployment configuration information of each Web server is parsed by traversing the registry of the big data platform cluster. The load balancing weight of each Web server is dynamically configured for the first time by combining the real-time cluster server running load set and the historical cluster server running load set. Step S4-1: Extract the CPU utilization, memory usage, and access request processing volume of each Web server in the historical cluster server load set. Using the unique node identifier of the Web server as the lookup index, sort the CPU utilization, memory usage, and access request processing volume of the Web server in log timestamp order. Normalize each parameter after sorting so that the values ​​of CPU utilization, memory usage, and access request processing volume of each Web server are in the range of 0 to 1. Step S4-2: Calculate the load value by calling the load calculation weight parameters configured in the big data platform cluster registry. The load calculation weight parameters include CPU utilization weight coefficient, memory usage weight coefficient, and access request processing weight coefficient. The sum of the load calculation weight parameters is 1. The load calculation weight parameters are set based on the initial weights of the Web server hardware configuration standards and service business types during the cluster deployment phase. The load calculation weight parameters can be updated as needed through the cluster management interface. Step S4-3: Calculate the load value based on the load calculation weight parameters. Perform weighted fusion calculation on the normalized CPU utilization, memory usage and access request processing volume of each Web server to obtain the historical load value corresponding to different access request volumes in the historical period. Select the access request volume as the horizontal axis and the historical load value as the vertical axis to build an independent historical access request volume load curve for each Web server. Step S4-4: Obtain the current access request volume of each Web server in the real-time cluster server running load set. Based on the current access request volume, match all historical load values ​​corresponding to the same access request volume in the historical access request volume load curve of the corresponding Web server. Use the three Sigma principle to remove outliers in the group of historical load values. Calculate the mean of the remaining historical load values ​​as the standard running load. Calculate the standard deviation of the remaining historical load values ​​to determine the fluctuation range. Combine the total number of Web servers before and after the expansion of the big data platform, multiply the standard running load by the total number of Web servers before the expansion, and then divide by the total number of Web servers after the expansion to obtain the theoretical load value of each Web server. Step S4-5: Based on the real-time cluster server running load set, synchronously calculate the current actual load value of each Web server; compare the current actual load value with the theoretical load value, and perform the first dynamic configuration of the load balancing weight of the Web servers as follows: Step S4-5-1: When the actual load value is higher than the theoretical load value and exceeds the fluctuation range, the load balancing weight of the Web server is reduced through the load balancing business API based on the unique node identifier of the Web server. Step S4-5-2: When the actual load value is lower than the theoretical load value and exceeds the fluctuation range, adjust the corresponding load balancing weight through the load balancing business API according to the unique node identifier of the Web server. Step S4-5-3: When the actual load value is within the theoretical load value and fluctuation range, keep the current load balancing weight of the Web server unchanged and complete the first dynamic configuration. By extracting parameters from the historical cluster server load set and sorting and normalizing them, the load calculation weight parameters are called to weight and fuse the historical load values ​​to obtain the historical access request volume load curve of a single server. After matching the current access request volume, the three Sigma principle is used to remove outliers and calculate the standard operating load and theoretical load value. The load balancing weight is dynamically adjusted by comparing the actual load value and the theoretical load value, thereby improving the data support strength and accurate adaptability of the weight configuration.

[0009] Step S5: Obtain and parse the running logs of each Web server to obtain the shared distributed cache hit rate of each Web server; perform a second dynamic configuration of the load balancing weight based on the shared distributed cache hit rate of each Web server. Step S5-1: Call the big data platform cluster registry to extract the cache configuration identifier field of each web server. Use this field to distinguish between local cache web servers and shared distributed cache web servers. For local cache web servers, keep the access path configuration of their service instances unchanged, and do not allocate new traffic to them in the load balancing business, maintaining the load balancing weight after the first dynamic configuration. For shared distributed cache web servers, mark them as objects for secondary weight adjustment. Step S5-2: Obtain the running log of the weight secondary adjustment object through the big data platform cluster log parsing interface, and extract the total number of cache query requests and the number of cache hit requests recorded in the running log. Based on the correspondence between the number of cache hit requests and the total number of cache query requests, calculate the shared distributed cache hit rate of each shared distributed cache web server; synchronously associate the load balancing weight and shared distributed cache hit rate of each shared distributed cache web server after the first dynamic configuration. Step S5-3: For the shared distributed cache web server, perform secondary configuration of load balancing weights based on its shared distributed cache hit rate. The specific process is as follows: Step S5-3-1: When the hit rate of the shared distributed cache is higher than the associated baseline hit rate during the first dynamic configuration, its load balancing weight is increased through the load balancing business API. Step S5-3-2: When the hit rate of the shared distributed cache is lower than the associated baseline hit rate at the time of the first dynamic configuration, the load balancing weight is reduced through the load balancing business API. Step S5-3-3: When the shared distributed cache hit rate is consistent with the associated baseline hit rate during the first dynamic configuration, keep its load balancing weight unchanged after the first dynamic configuration and complete the second dynamic configuration. During the secondary configuration of load weights, the local cache web server maintains the load balancing weight unchanged after the first dynamic configuration, while the shared distributed cache web server adjusts its weight up or down based on the cache hit rate. The total weight of all shared distributed cache web servers after the secondary adjustment remains consistent with the total weight before the adjustment, ensuring the overall stability of the cluster load balancing. By extracting the cache configuration identifier field of the web server, the local cache and the shared distributed cache web server are distinguished. The access path and the original weight of the local cache web server are kept unchanged. For the shared distributed cache web server, the total number of cache query requests and the number of hits in the operation log are extracted to calculate the hit rate. The weight after the first dynamic configuration is associated with the hit rate and then adjusted a second time. This realizes the weight optimization of cache type differentiation and hit rate association, which makes up for the lack of such consideration in the original mechanism.

[0010] Furthermore, a performance monitoring system for expanding and upgrading a big data platform includes a server configuration registration module, a load log processing module, a running load set construction module, a load weight primary configuration module, and a load weight secondary configuration module. The server configuration registration module is used for inputting and verifying configuration information of newly added Web servers during the expansion phase of the big data platform, as well as cluster access and initial weight configuration. The load log processing module is used for selectively pulling historical operating load log data of the original Web servers before the expansion of the big data platform and filtering valid data. The operating load set construction module is used for parsing valid historical load log data and collecting real-time load data after expansion to construct historical and real-time cluster server operating load sets respectively. The load weight primary configuration module is used to perform the first dynamic configuration of the load balancing weight of each Web server by combining the historical and real-time cluster server operating load sets. The load weight secondary configuration module is used to perform the second dynamic configuration of the load balancing weight based on the shared distributed cache hit rate of the Web servers. The server configuration registration module includes a configuration data acquisition unit and a registration verification and association unit. The configuration data acquisition unit is used to collect the node unique identifier, hardware parameters, network access address, service listening port, and cluster access credentials of the newly added Web server and form a standardized configuration dataset. The registration verification and association unit is used to enter the standardized configuration dataset into the cluster registry to complete the configuration information integrity verification, node identity authentication, and cluster routing association. The load log processing module includes a targeted log retrieval unit and a valid log filtering unit. The targeted log retrieval unit is used to call the cluster log caching business API, inject the expansion trigger timestamp and the original Web server's unique node identifier to initiate a log retrieval request. The valid log filtering unit is used to perform timestamp verification on the retrieved historical running load log data and filter out log entries that are earlier than the expansion trigger timestamp. The runtime load collection construction module includes a historical load parsing unit and a real-time load acquisition unit. The historical load parsing unit is used to structure and parse valid runtime load log data, extract CPU utilization, memory usage, and access request processing volume, and classify and collect them to construct a historical cluster server runtime load collection. The real-time load acquisition unit is used to collect the CPU utilization, memory usage, and access request processing volume of all Web servers after expansion through the cluster monitoring business API interface, and associate and store them to construct a real-time cluster server runtime load collection. The load weight configuration module includes a load parameter calculation unit and a weight dynamic adjustment unit. The load parameter calculation unit is used to normalize historical load parameters, call the load calculation weight parameters to calculate historical load values ​​and construct a historical access request volume load curve, and synchronously calculate the real-time actual load value and the theoretical load value. The weight dynamic adjustment unit is used to compare the current actual load value of the Web server with the theoretical load value, and complete the first load balancing weight dynamic configuration through the load balancing business API. The load weight secondary configuration module includes a cache type differentiation unit and a hit rate correlation adjustment unit. The cache type differentiation unit is used to call the cluster registry to extract the cache configuration identifier field, distinguish between local cache and shared distributed cache web servers, and mark the weight secondary adjustment objects. The hit rate correlation adjustment unit is used to calculate the cache hit rate of the shared distributed cache web server, correlate the weight after the first dynamic configuration, and complete the second load balancing weight dynamic configuration.

[0011] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention forms a standardized configuration dataset by collecting key information of newly added Web servers. After integrity verification, identity authentication and cluster routing association, a two-way communication link is established and cluster collaboration parameters are calibrated. Initial load balancing weight parameters are entered, which solves the problems of non-standard access and poor collaboration of newly added nodes in the existing expansion process. It ensures that newly added nodes are compliantly integrated into the cluster, lays the foundation for load sharing, and improves the standardization of node deployment and cluster compatibility during the expansion phase.

[0012] 2. This invention retrieves historical load logs of the original Web servers before expansion, filters valid data after timestamp verification, and then constructs a set of historical cluster server loads through structured parsing. At the same time, it collects load data of all Web servers after expansion in real time to construct a real-time set. This makes up for the lack of accurate historical data support and comprehensive real-time data coverage in existing technologies, providing complete and effective data basis for load balancing weight configuration and enhancing the targeting of performance monitoring.

[0013] 3. This invention performs the first dynamic weight configuration by combining historical and real-time load data, and then distinguishes the web server cache type based on the cache configuration identifier. A second weight adjustment is then performed on the shared distributed cache web server based on the hit rate. This solves the problems of existing weight configurations lacking data support and not considering cache differences and hit rates. It achieves accurate adaptation of load balancing weights, ensures a stable increase in the hit rate of the shared distributed cache, reduces the frequency of access requests directly penetrating to the backend database, reduces the query load and data processing pressure of the backend database, shortens data query response latency, achieves efficient matching of cache resources and access requests, avoids resource imbalance and unstable service response, and ensures the platform's operating performance after expansion. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating a performance monitoring method for expanding and upgrading a big data platform according to the present invention. Figure 2 This is a schematic diagram of the structure of a performance monitoring system for expanding and upgrading a big data platform according to the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] Example 1: As Figure 1 As shown, this invention provides a technical solution: a performance monitoring method for the expansion and upgrade of a big data platform. The performance monitoring method includes the following steps: Step S1: After completing the deployment and configuration of the new Web server during the expansion phase of the big data platform, enter the configuration information of the new Web server into the big data platform cluster registry for registration. After the registration verification is passed and the new node is connected to the cluster communication link, establish a traffic forwarding link between the new node and the existing cluster through the load balancing service, simultaneously enable the service instance of the new Web server to complete the cluster collaborative parameter calibration, and configure the load balancing weight of each Web server according to the load balancing service API. Step S1-1: During the expansion phase of the big data platform, after completing the hardware deployment, network configuration, and service operation environment deployment of the new Web server, collect the node unique identifier, hardware parameters, network access address, service listening port, and cluster access credentials of the new Web server to form a standardized configuration dataset. Step S1-2: Enter the standardized configuration dataset into the big data platform cluster registry, and start the registry configuration information integrity verification, node identity authentication and cluster routing association process; Steps S1-3: After verification, authentication and association are passed, establish a bidirectional communication link between the new node and the existing cluster according to the cluster communication protocol, configure the traffic forwarding rules of the new node based on the load balancing service, start the service instance of the new Web server, calibrate the session sharing parameters, data transmission protocol parameters and service health check parameters required for cluster collaborative operation, and call the load balancing service API to enter the initial load balancing weight parameters of each Web server. In practice, key configuration information of newly added web servers is collected in a standardized manner to ensure that the cluster registry can accurately identify node attributes. The node access qualification is verified through a triple mechanism of configuration integrity verification, node identity authentication, and cluster routing association. Then, a two-way communication link is built according to the cluster communication protocol. By calibrating collaborative parameters such as session sharing parameters, the consistency between the operation of new nodes and the existing cluster is ensured. The initial load balancing weight parameters must be adapted to the server hardware parameters and business type. Special attention should be paid to the authenticity and completeness of the configuration information to avoid node access failure or cluster collaboration abnormalities due to missing or incorrect information.

[0017] Step S2: Through the big data platform cluster log caching business API, based on the unique node identifier of the original Web server before the big data platform expansion, retrieve the running load log data within the preset time window before the expansion operation is executed, and determine the valid running load log data by verifying the time sequence relationship between the log timestamp and the expansion trigger timestamp. Step S2-1: Call the big data platform cluster log caching business API, inject the big data platform expansion trigger timestamp and the unique node identifier of the original web server before the big data platform expansion, initiate a targeted log retrieval request, and retrieve historical running load log data according to the unique node identifier of the original web server. Step S2-2: Verify the log timestamp of each item in the historical running load log data pulled, and filter out log items whose log timestamps are earlier than the big data platform expansion trigger timetamp to form valid running load log data. In practical implementation, relying on the targeted query capabilities of the big data platform's cluster log caching business API, the range of historical operational load log data before expansion is locked by injecting the expansion trigger timestamp and the original Web server's unique node identifier. This ensures that the pulled data is highly relevant to the expansion analysis scenario. The core of log timestamp verification is to define the temporal validity of the data, retaining only log entries before the expansion operation to exclude interfering data. During implementation, attention must be paid to the accuracy of parameter matching in the log pull request to avoid deviations in the pulled data due to incorrect node identifiers or timestamps. At the same time, it is necessary to ensure that log verification is performed line by line, without omitting any invalid log entries.

[0018] Step S3: Parse the valid runtime load log data to obtain the historical runtime load data of each Web server before the expansion of the big data platform, and record it as the historical cluster server runtime load set; obtain the runtime load data of each Web server after the expansion of the big data platform, and record it as the real-time cluster server runtime load set. The effective runtime load log data generated in step S2 is structured and parsed to extract CPU utilization, memory usage, and access request processing volume recorded in the logs. The access request processing volume is represented as the ratio of the number of active connections to the maximum number of connections processed by the service instance of the Web server in historical time. Based on the unique node identifiers of the original Web servers before the big data platform expansion, the data is categorized and aggregated, using the unique node identifiers of the Web servers as keys and the extracted CPU utilization, memory usage, and access request processing volume recorded in the logs as values, to construct a historical cluster server runtime load set. Through the big data platform cluster monitoring business API interface, the CPU utilization, memory usage, and access request processing volume of all Web servers after the big data platform expansion are collected in real time. All Web servers include newly added Web servers and existing Web servers. Based on the unique node identifiers of each Web server, the data is synchronously stored using the storage method of the historical cluster server runtime load set to construct a real-time cluster server runtime load set. In practical implementation, the core of constructing the historical cluster server load set is the structured parsing and classification of valid logs. The extracted CPU utilization, memory usage, and access request processing volume are key indicators reflecting server load. Associating and storing data by unique node identifiers ensures the traceability of historical data. The construction of the real-time cluster server load set needs to cover all web servers, without omitting any newly added or existing nodes. During implementation, the storage method must be consistent with that of the historical set to ensure the comparability of the two types of data. At the same time, attention should be paid to the accuracy of load parameter extraction to avoid data distortion due to parsing logic deviations.

[0019] Step S4: After configuring the load balancing weight through the load balancing business API, the deployment configuration information of each Web server is parsed by traversing the registry of the big data platform cluster. The load balancing weight of each Web server is dynamically configured for the first time by combining the real-time cluster server running load set and the historical cluster server running load set. Step S4-1: Extract the CPU utilization, memory usage, and access request processing volume of each Web server in the historical cluster server load set. Using the unique node identifier of the Web server as the lookup index, sort the CPU utilization, memory usage, and access request processing volume of the Web server in log timestamp order. Normalize each parameter after sorting so that the values ​​of CPU utilization, memory usage, and access request processing volume of each Web server are in the range of 0 to 1. Step S4-2: Calculate the load value by calling the load calculation weight parameters configured in the big data platform cluster registry. The load calculation weight parameters include CPU utilization weight coefficient, memory usage weight coefficient, and access request processing weight coefficient. The sum of the load calculation weight parameters is 1. The load calculation weight parameters are set based on the initial weights of the Web server hardware configuration standards and service business types during the cluster deployment phase. The load calculation weight parameters can be updated as needed through the cluster management interface. Step S4-3: Calculate the load value based on the load calculation weight parameters. Perform weighted fusion calculation on the normalized CPU utilization, memory usage and access request processing volume of each Web server to obtain the historical load value corresponding to different access request volumes in the historical period. Select the access request volume as the horizontal axis and the historical load value as the vertical axis to build an independent historical access request volume load curve for each Web server. Step S4-4: Obtain the current access request volume of each Web server in the real-time cluster server running load set. Based on the current access request volume, match all historical load values ​​corresponding to the same access request volume in the historical access request volume load curve of the corresponding Web server. Use the three Sigma principle to remove outliers in the group of historical load values. Calculate the mean of the remaining historical load values ​​as the standard running load. Calculate the standard deviation of the remaining historical load values ​​to determine the fluctuation range. Combine the total number of Web servers before and after the expansion of the big data platform, multiply the standard running load by the total number of Web servers before the expansion, and then divide by the total number of Web servers after the expansion to obtain the theoretical load value of each Web server. Step S4-5: Based on the real-time cluster server running load set, synchronously calculate the current actual load value of each Web server; compare the current actual load value with the theoretical load value, and perform the first dynamic configuration of the load balancing weight of the Web servers as follows: Step S4-5-1: When the actual load value is higher than the theoretical load value and exceeds the fluctuation range, the load balancing weight of the Web server is reduced through the load balancing business API based on the unique node identifier of the Web server. Step S4-5-2: When the actual load value is lower than the theoretical load value and exceeds the fluctuation range, adjust the corresponding load balancing weight through the load balancing business API according to the unique node identifier of the Web server. Step S4-5-3: When the actual load value is within the theoretical load value and fluctuation range, keep the current load balancing weight of the Web server unchanged and complete the first dynamic configuration. In practical implementation, the first dynamic configuration combines historical and real-time load data to achieve precise weight adaptation. Parameter normalization can eliminate the differences in the dimensions of different indicators. The setting of load calculation weight parameters must be in line with the Web server hardware configuration and business needs. The historical access request volume load curve can intuitively reflect the server load change pattern with the request volume. The application of the three Sigma principle can eliminate the interference of abnormal historical data on the calculation of theoretical load value. During implementation, it is necessary to ensure that the calculation of theoretical load value is strictly related to the total number of servers before and after expansion. When comparing actual and theoretical load values, the fluctuation range must be accurately determined to ensure that the triggering conditions for weight adjustment are rigorous and accurate.

[0020] Step S5: Obtain and parse the running logs of each Web server to obtain the shared distributed cache hit rate of each Web server; perform a second dynamic configuration of the load balancing weight based on the shared distributed cache hit rate of each Web server. Step S5-1: Call the big data platform cluster registry to extract the cache configuration identifier field of each web server. Use this field to distinguish between local cache web servers and shared distributed cache web servers. For local cache web servers, keep the access path configuration of their service instances unchanged, and do not allocate new traffic to them in the load balancing business, maintaining the load balancing weight after the first dynamic configuration. For shared distributed cache web servers, mark them as objects for secondary weight adjustment. Step S5-2: Obtain the running log of the weight secondary adjustment object through the big data platform cluster log parsing interface, and extract the total number of cache query requests and the number of cache hit requests recorded in the running log. Based on the correspondence between the number of cache hit requests and the total number of cache query requests, calculate the shared distributed cache hit rate of each shared distributed cache web server; synchronously associate the load balancing weight and shared distributed cache hit rate of each shared distributed cache web server after the first dynamic configuration. Step S5-3: For the shared distributed cache web server, perform secondary configuration of load balancing weights based on its shared distributed cache hit rate. The specific process is as follows: Step S5-3-1: When the hit rate of the shared distributed cache is higher than the associated baseline hit rate during the first dynamic configuration, its load balancing weight is increased through the load balancing business API. Step S5-3-2: When the hit rate of the shared distributed cache is lower than the associated baseline hit rate at the time of the first dynamic configuration, the load balancing weight is reduced through the load balancing business API. Step S5-3-3: When the shared distributed cache hit rate is consistent with the associated baseline hit rate during the first dynamic configuration, keep its load balancing weight unchanged after the first dynamic configuration and complete the second dynamic configuration. In practice, secondary configuration is based on the caching characteristics of web servers to achieve differentiated weight optimization. The cache configuration identifier field is the key basis for distinguishing between local cache and shared distributed cache web servers. Local cache web servers maintain their original configuration because their access paths are not affected by expansion. The hit rate of shared distributed cache web servers directly reflects their data response efficiency. Adjusting the weights in conjunction with the first configuration can achieve a match between load and cache efficiency. During implementation, it is necessary to ensure the accuracy of cache type differentiation to avoid confusion between the two types of servers and configuration errors. At the same time, the hit rate calculation must be based on complete operation logs to ensure the reliability of the adjustment basis.

[0021] Example 2, as Figure 2As shown, the present invention provides a performance monitoring system for the expansion and upgrade of a big data platform. The performance monitoring system includes a server configuration registration module, a load log processing module, a running load set construction module, a load weight primary configuration module, and a load weight secondary configuration module. The server configuration registration module is used for inputting and verifying configuration information of newly added Web servers during the expansion phase of the big data platform, as well as cluster access and initial weight configuration. The load log processing module is used for selectively pulling historical operating load log data of the original Web servers before the expansion of the big data platform and filtering valid data. The operating load set construction module is used for parsing valid historical load log data and collecting real-time load data after expansion to construct historical and real-time cluster server operating load sets respectively. The load weight primary configuration module is used to perform the first dynamic configuration of the load balancing weight of each Web server by combining the historical and real-time cluster server operating load sets. The load weight secondary configuration module is used to perform the second dynamic configuration of the load balancing weight based on the shared distributed cache hit rate of the Web servers. The server configuration registration module includes a configuration data acquisition unit and a registration verification and association unit. The configuration data acquisition unit is used to collect the node unique identifier, hardware parameters, network access address, service listening port, and cluster access credentials of the newly added Web server and form a standardized configuration dataset. The registration verification and association unit is used to enter the standardized configuration dataset into the cluster registry to complete the configuration information integrity verification, node identity authentication, and cluster routing association. The load log processing module includes a targeted log retrieval unit and a valid log filtering unit. The targeted log retrieval unit is used to call the cluster log caching business API, inject the expansion trigger timestamp and the original Web server's unique node identifier to initiate a log retrieval request. The valid log filtering unit is used to perform timestamp verification on the retrieved historical running load log data and filter out log entries that are earlier than the expansion trigger timestamp. The runtime load collection construction module includes a historical load parsing unit and a real-time load acquisition unit. The historical load parsing unit is used to structure and parse valid runtime load log data, extract CPU utilization, memory usage, and access request processing volume, and classify and collect them to construct a historical cluster server runtime load collection. The real-time load acquisition unit is used to collect the CPU utilization, memory usage, and access request processing volume of all Web servers after expansion through the cluster monitoring business API interface, and associate and store them to construct a real-time cluster server runtime load collection. The load weight configuration module includes a load parameter calculation unit and a weight dynamic adjustment unit. The load parameter calculation unit is used to normalize historical load parameters, call the load calculation weight parameters to calculate historical load values ​​and construct a historical access request volume load curve, and synchronously calculate the real-time actual load value and the theoretical load value. The weight dynamic adjustment unit is used to compare the current actual load value of the Web server with the theoretical load value, and complete the first load balancing weight dynamic configuration through the load balancing business API. The load weight secondary configuration module includes a cache type differentiation unit and a hit rate correlation adjustment unit. The cache type differentiation unit is used to call the cluster registry to extract the cache configuration identifier field, distinguish between local cache and shared distributed cache web servers, and mark the weight secondary adjustment objects. The hit rate correlation adjustment unit is used to calculate the cache hit rate of the shared distributed cache web server, correlate the weight after the first dynamic configuration, and complete the second load balancing weight dynamic configuration.

[0022] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, it is intended that all variations falling within the meaning and scope of equivalents of the claims be included within the present invention.

Claims

1. A performance monitoring method for scaling up and upgrading a big data platform, characterized in that: The performance monitoring method includes the following steps: Step S1: After completing the deployment and configuration of the new Web server during the expansion phase of the big data platform, enter the configuration information of the new Web server into the big data platform cluster registry for registration. After the registration verification is passed and the new node is connected to the cluster communication link, establish a traffic forwarding link between the new node and the existing cluster through the load balancing service, simultaneously enable the service instance of the new Web server to complete the cluster collaborative parameter calibration, and configure the load balancing weight of each Web server according to the load balancing service API. Step S2: Through the big data platform cluster log caching business API, based on the unique node identifier of the original Web server before the big data platform expansion, retrieve the running load log data within the preset time window before the expansion operation is executed, and determine the valid running load log data by verifying the time sequence relationship between the log timestamp and the expansion trigger timestamp. Step S3: Parse the valid runtime load log data to obtain the historical runtime load data of each Web server before the expansion of the big data platform, and record it as the historical cluster server runtime load set; obtain the runtime load data of each Web server after the expansion of the big data platform, and record it as the real-time cluster server runtime load set. Step S4: After configuring the load balancing weight through the load balancing business API, the deployment configuration information of each Web server is parsed by traversing the registry of the big data platform cluster. The load balancing weight of each Web server is dynamically configured for the first time by combining the real-time cluster server running load set and the historical cluster server running load set. Step S5: Obtain and parse the running logs of each Web server to obtain the shared distributed cache hit rate of each Web server; perform a second dynamic configuration of the load balancing weight based on the shared distributed cache hit rate of each Web server.

2. The performance monitoring method for big data platform expansion and upgrade according to claim 1, characterized in that: The specific steps of step S1 are as follows: Step S1-1: During the expansion phase of the big data platform, after completing the hardware deployment, network configuration, and service operation environment deployment of the new Web server, collect the node unique identifier, hardware parameters, network access address, service listening port, and cluster access credentials of the new Web server to form a standardized configuration dataset. Step S1-2: Enter the standardized configuration dataset into the big data platform cluster registry, and start the registry configuration information integrity verification, node identity authentication and cluster routing association process; Steps S1-3: After verification, authentication, and association are successful, establish a bidirectional communication link between the new node and the existing cluster according to the cluster communication protocol, configure the traffic forwarding rules of the new node based on the load balancing service, start the service instance of the new Web server, calibrate the session sharing parameters, data transmission protocol parameters, and service health check parameters required for cluster collaborative operation, and call the load balancing service API to enter the initial load balancing weight parameters of each Web server.

3. The performance monitoring method for big data platform expansion and upgrade according to claim 2, characterized in that: The specific steps of step S2 are as follows: Step S2-1: Call the big data platform cluster log caching business API, inject the big data platform expansion trigger timestamp and the unique node identifier of the original web server before the big data platform expansion, initiate a targeted log retrieval request, and retrieve historical running load log data according to the unique node identifier of the original web server. Step S2-2: Verify the log timestamps of each item in the retrieved historical operational load log data, and filter out log items whose log timestamps are earlier than the big data platform expansion trigger timestamp to form valid operational load log data.

4. The performance monitoring method for big data platform expansion and upgrade according to claim 3, characterized in that: In step S3, the effective runtime load log data generated in step S2 is structured and parsed to extract the CPU utilization, memory usage, and access request processing volume recorded in the logs. The access request processing volume is represented as the ratio of the number of active connections to the maximum number of connections processed by the service instance of the Web server in historical time. Based on the unique node identifiers of the original Web servers before the big data platform expansion, the data is categorized and aggregated, using the unique node identifiers of the Web servers as keys and the extracted CPU utilization, memory usage, and access request processing volume recorded in the logs as values, to construct a historical cluster server runtime load set. Through the big data platform cluster monitoring business API interface, the CPU utilization, memory usage, and access request processing volume of all Web servers after the big data platform expansion are collected in real time. All Web servers include newly added Web servers and existing Web servers. Based on the unique node identifiers of each Web server, the data is synchronously stored using the storage method of the historical cluster server runtime load set to construct a real-time cluster server runtime load set.

5. The performance monitoring method for big data platform expansion and upgrade according to claim 4, characterized in that: The specific steps of step S4 are as follows: Step S4-1: Extract the CPU utilization, memory usage, and access request processing volume of each Web server in the historical cluster server running load set. Use the unique node identifier of the Web server as the lookup index and sort the CPU utilization, memory usage, and access request processing volume of the Web server in the order of log timestamp. Normalize each parameter after sorting so that the CPU utilization, memory usage, and access request processing volume of each web server are in the range of 0 to 1. Step S4-2: Calculate the load value by calling the load calculation weight parameters configured in the big data platform cluster registry. The load calculation weight parameters include CPU utilization weight coefficient, memory usage weight coefficient, and access request processing weight coefficient. The sum of the load calculation weight parameters is 1. The load calculation weight parameters are set based on the initial weights of the Web server hardware configuration standards and service business types during the cluster deployment phase. The load calculation weight parameters can be updated as needed through the cluster management interface. Step S4-3: Calculate the load value based on the load calculation weight parameters. Perform weighted fusion calculation on the normalized CPU utilization, memory usage and access request processing volume of each Web server to obtain the historical load value corresponding to different access request volumes in the historical period. Select the access request volume as the horizontal axis and the historical load value as the vertical axis to construct an independent historical access request volume load curve for each Web server.

6. A performance monitoring method for expanding and upgrading a big data platform according to claim 5, characterized in that: Step S4 also includes: Step S4-4: Obtain the current access request volume of each Web server in the real-time cluster server running load set. Based on the current access request volume, match all historical load values ​​corresponding to the same access request volume in the historical access request volume load curve of the corresponding Web server. Use the three Sigma principle to remove outliers in the group of historical load values. Calculate the mean of the remaining historical load values ​​as the standard running load. Calculate the standard deviation of the remaining historical load values ​​to determine the fluctuation range. Combine the total number of Web servers before and after the expansion of the big data platform, multiply the standard running load by the total number of Web servers before the expansion, and then divide by the total number of Web servers after the expansion to obtain the theoretical load value of each Web server. Step S4-5: Based on the real-time cluster server running load set, synchronously calculate the current actual load value of each Web server; compare the current actual load value with the theoretical load value, and perform the first dynamic configuration of the load balancing weight of the Web servers as follows: Step S4-5-1: When the actual load value is higher than the theoretical load value and exceeds the fluctuation range, the load balancing weight of the Web server is reduced through the load balancing business API based on the unique node identifier of the Web server. Step S4-5-2: When the actual load value is lower than the theoretical load value and exceeds the fluctuation range, adjust the corresponding load balancing weight through the load balancing business API according to the unique node identifier of the Web server. Step S4-5-3: When the actual load value is within the theoretical load value and fluctuation range, keep the current load balancing weight of the Web server unchanged and complete the first dynamic configuration.

7. A performance monitoring method for expanding and upgrading a big data platform according to claim 6, characterized in that: The specific steps of step S5 are as follows: Step S5-1: Call the big data platform cluster registry to extract the cache configuration identifier field of each web server. Use this field to distinguish between local cache web servers and shared distributed cache web servers. For local cache web servers, keep the access path configuration of their service instances unchanged, and do not allocate new traffic to them in the load balancing business, maintaining the load balancing weight after the first dynamic configuration. For shared distributed cache web servers, mark them as objects for secondary weight adjustment. Step S5-2: Obtain the running log of the weight secondary adjustment object through the big data platform cluster log parsing interface, and extract the total number of cache query requests and the number of cache hit requests recorded in the running log. Based on the correspondence between the number of cache hit requests and the total number of cache query requests, calculate the shared distributed cache hit rate of each shared distributed cache web server; synchronously associate the load balancing weight and shared distributed cache hit rate of each shared distributed cache web server after the first dynamic configuration. Step S5-3: For the shared distributed cache web server, perform secondary configuration of load balancing weights based on its shared distributed cache hit rate. The specific process is as follows: Step S5-3-1: When the hit rate of the shared distributed cache is higher than the associated baseline hit rate during the first dynamic configuration, its load balancing weight is increased through the load balancing business API. Step S5-3-2: When the hit rate of the shared distributed cache is lower than the associated baseline hit rate at the time of the first dynamic configuration, the load balancing weight is reduced through the load balancing business API. Step S5-3-3: When the shared distributed cache hit rate is consistent with the associated baseline hit rate during the first dynamic configuration, keep the load balancing weight after the first dynamic configuration unchanged and complete the second dynamic configuration.

8. A performance monitoring system for expanding and upgrading a big data platform, which is applied to the performance monitoring method for expanding and upgrading a big data platform as described in any one of claims 1-7, characterized in that: The performance monitoring system includes a server configuration registration module, a load log processing module, a running load set construction module, a load weight primary configuration module, and a load weight secondary configuration module. The server configuration registration module is used for inputting and verifying configuration information of newly added Web servers during the expansion phase of the big data platform, as well as cluster access and initial weight configuration. The load log processing module is used for selectively pulling historical operating load log data of the original Web servers before the expansion of the big data platform and filtering valid data. The operating load set construction module is used for parsing valid historical load log data and collecting real-time load data after expansion to construct historical and real-time cluster server operating load sets respectively. The load weight primary configuration module is used to perform the first dynamic configuration of the load balancing weight of each Web server by combining the historical and real-time cluster server operating load sets. The load weight secondary configuration module is used to perform the second dynamic configuration of the load balancing weight based on the shared distributed cache hit rate of the Web servers.

9. A performance monitoring system for expanding and upgrading a big data platform according to claim 8, characterized in that: The server configuration registration module includes a configuration data acquisition unit and a registration verification and association unit. The configuration data acquisition unit is used to collect the node unique identifier, hardware parameters, network access address, service listening port, and cluster access credentials of the newly added Web server and form a standardized configuration dataset. The registration verification and association unit is used to enter the standardized configuration dataset into the cluster registry to complete the configuration information integrity verification, node identity authentication, and cluster routing association. The load log processing module includes a targeted log retrieval unit and a valid log filtering unit; The targeted log retrieval unit is used to call the cluster log caching service API, inject the expansion trigger timestamp and the original Web server's unique node identifier to initiate a log retrieval request; the effective log filtering unit is used to perform timestamp verification on the retrieved historical runtime load log data and filter out log entries that are earlier than the expansion trigger timestamp. The runtime load collection construction module includes a historical load parsing unit and a real-time load acquisition unit. The historical load parsing unit is used to structure and parse valid runtime load log data, extract CPU utilization, memory usage, and access request processing volume, and classify and collect them to construct a historical cluster server runtime load collection. The real-time load acquisition unit is used to collect the CPU utilization, memory usage, and access request processing volume of all Web servers after expansion through the cluster monitoring business API interface, and associate and store them to construct a real-time cluster server runtime load collection.

10. A performance monitoring system for expanding and upgrading a big data platform according to claim 8, characterized in that: The load weight configuration module includes a load parameter calculation unit and a weight dynamic adjustment unit. The load parameter calculation unit is used to normalize historical load parameters, call the load calculation weight parameters to calculate historical load values ​​and construct a historical access request volume load curve, and synchronously calculate the real-time actual load value and the theoretical load value. The weight dynamic adjustment unit is used to compare the current actual load value of the Web server with the theoretical load value, and complete the first load balancing weight dynamic configuration through the load balancing business API. The load weight secondary configuration module includes a cache type differentiation unit and a hit rate correlation adjustment unit. The cache type differentiation unit is used to call the cluster registry to extract the cache configuration identifier field, distinguish between local cache and shared distributed cache web servers, and mark the weight secondary adjustment objects. The hit rate correlation adjustment unit is used to calculate the cache hit rate of the shared distributed cache web server, correlate the weight after the first dynamic configuration, and complete the second load balancing weight dynamic configuration.

Images