Micro-service dynamic management method and system for distributed computing power network call chain

Abstract

The invention provides a micro-service dynamic governance method and system for a distributed computing power network call chain, and the method comprises the steps: initializing, namely configuring a service guarantee level and setting a service governance triggering condition; collecting data, and preprocessing the data; performing risk prediction according to the preprocessed data; performing service management according to a risk prediction result; after the service management is completed, service recovery is carried out; the system adopts the method to realize micro-service dynamic management of the distributed computing power network call chain. According to the invention, the real data of the system is fully used to carry out positive feedback correction on the algorithm, so that the accuracy of service quality is improved; a data analysis algorithm is adopted for accurate calculation, and step-by-step and gradient governance services are provided within a period of time. In other words, system misjudgment caused by uncertain factors of business scenes and client actions is compatible, and the target of smooth transition is achieved.

Description

Technical Field

[0001] The present invention relates to a microservice dynamic governance method and system, and in particular to a microservice dynamic governance method and system for a distributed computing power network call chain. Background Art

[0002] This section merely provides background information related to the present disclosure and is not necessarily prior art.

[0003] In recent years, microservice architecture has further decoupled the system architecture with its decentralization, high autonomy, and automation, and is more in line with agile iteration and DevOps culture. It can use lower costs to meet the development needs of enterprise software R&D architecture and improve R&D efficiency. Especially with the rise of Kubernetes native cloud technology, it is popular in the construction of computing power network virtualization. At present, all software systems are gradually becoming microservices and containerized. The number of containers is increasing. How to manage containers, continuous integration and orchestration has become a problem. In order to solve the containerization and orchestration problems, Kubernetes was introduced to uniformly schedule and maintain containers and resources in the cluster. The entire status of the cluster and the corresponding monitoring of services, but there is a problem between the unlimited elastic expansion of business and the balance between resources. How to ensure the normal operation of microservices under the entire computing power network system while being able to evict services and balance service scheduling based on the weight coefficients of microservices when insufficient resources occur is very important. In this way, when cluster node pods are evicted, services with high weight coefficients can operate normally, thereby ensuring that the core services of the system are not affected, such as: payment services, commodity SPU services, system monitoring services, and log services. When cluster eviction occurs, it is expected that the log service and system monitoring service can be evicted while retaining the core payment, SPU and other core services.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0005] Purpose of the invention: The technical problem to be solved by the present invention is to address the deficiencies of the existing technology and provide a method and system for dynamic governance of microservices in a distributed computing power network call chain.

[0006] In order to solve the above technical problems, the present invention discloses a method and system for dynamic microservice governance of a distributed computing network call chain, wherein the system includes the following steps:

[0007] Step 1: Initialization, which is to configure the service assurance level and set the service governance trigger conditions;

[0008] Step 2: data collection and preprocessing;

[0009] Step 3: perform risk prediction based on the preprocessed data;

[0010] Step 4: Conduct service governance based on risk prediction results;

[0011] Step 5: After completing the service governance, perform service recovery to implement dynamic microservice governance of the distributed computing power network call chain.

[0012] Furthermore, the configuration of the service assurance level described in step 1 includes:

[0013] The assurance level of the microservices in the distributed computing network call chain is divided into three levels, and a label is configured for each microservice to obtain label data of the service assurance level; the label data includes: level type, operating unit name, number of replicas, namespace and business name; among them, the level type includes: low level, medium level and high level.

[0014] Furthermore, the service governance triggering conditions described in step 1 include:

[0015] The maximum number of call failures of the microservice that meets the preset level is greater than the failure number threshold;

[0016] The average delay of the interface call is greater than the time threshold;

[0017] The CPU usage and memory usage of the computing power node are greater than the preset thresholds.

[0018] Furthermore, the data collection described in step 2 includes:

[0019] Using a service operation tool, every preset time period, collects service performance data of all microservices in the distributed computing network, including: service call operation status, call failure number and interface call delay time;

[0020] A resource monitoring tool is used to collect resource performance data of all nodes in the distributed computing network at the same preset time period, including: CPU utilization, memory utilization and storage utilization.

[0021] Furthermore, the pre-processing described in step 2 includes:

[0022] The service performance data and resource performance data are merged and processed as follows:

[0023] Read the service performance data and resource performance data within the preset time period respectively; according to the principle of time proximity, merge the resource performance data of the node within the preset time period with the same start and end time with the service performance data of each microservice in the node;

[0024] Perform box processing on the merged data time column to generate time serial number;

[0025] The combined data includes:

[0026] Service execution time, sequence number, running unit name, service status type, number of failures, waiting time, CPU usage, memory usage, and storage usage.

[0027] Furthermore, the risk prediction described in step 3 includes:

[0028] In step 3-1, when the maximum number of service call failures in the merged data exceeds the failure number threshold, the average time of the interface call delay exceeds the time threshold, and the CPU usage of the node exceeds the threshold or the memory usage exceeds the threshold, execute step 3-2 to perform risk prediction;

[0029] Step 3-2: Read the merged data of the node for the last hour and write it to a temporary file;

[0030] Step 3-3: extract the m data with the smallest time sequence numbers from the temporary file, and use the time sequence numbers of the remaining data, the corresponding service performance data, and the resource performance data within each time sequence number as input data for the prediction model to predict risks;

[0031] Step 3-4, the prediction model scores the input data;

[0032] Step 3-5: Filter out data with scores less than the preset score and save it as the final risk prediction result.

[0033] Furthermore, the prediction model described in steps 3-4 scores the input data, including:

[0034] Step 3-4-1: parse the input data and find the time sequence number corresponding to the median of CPU usage, memory usage, and storage usage, as well as the time sequence number corresponding to the maximum value of CPU usage, memory usage, and storage usage;

[0035] In step 3-4-2, if the medians of CPU usage, memory usage, and storage usage are not in the same time sequence, then continue to determine whether the medians of CPU usage and memory usage are in the same time sequence. If so, take this time sequence and the next median array, for a total of two sets of data; if the medians of CPU usage and memory usage are not in the same time sequence, then take the median sequence of CPU usage, memory usage, and storage usage and the CPU usage, memory usage, and storage usage values ​​closest to the median, respectively, store the above data in a medium data table, and reversely map the time sequence number to the time value, giving all data corresponding timestamps;

[0036] Step 3-4-3, using the same method as step 3-4-2, create a high-level data table based on the time sequence number corresponding to the maximum value of CPU usage, memory usage, and storage usage;

[0037] In step 3-4-4, the mean square error between the time in the medium data table and the time in the high-level data table is calculated and compared with the preset risk value λ. If it is less than λ, the timing time is set to the minimum time in the high-level data table, and the microservice treatment is triggered at this time.

[0038] Furthermore, the service governance described in step 4 includes:

[0039] Step 4-1: Read the resource performance data and microservice list used by the current node;

[0040] Step 4-2: Calculate the mean square error (Vmse) of CPU usage and memory usage using the resource performance data and the data corresponding to the minimum time sequence number in the medium data table. If the mean square error (Vmse) is greater than the first flow control threshold, stop service management. If the mean square error (Vmse) is less than the first flow control threshold, execute step 4-3.

[0041] Step 4-3: Perform first-level service flow control, as follows:

[0042] Use the resource performance data and the data corresponding to the minimum time sequence number in the medium data table to calculate the mean square error (Vmse) of CPU usage and memory usage. If the mean square error (Vmse) is greater than or equal to the second flow control threshold, stop service governance. Otherwise, continue collecting data for one cycle and use the method in step 4-2 to calculate the mean square error (Vmse) of CPU usage and memory usage. If the mean square error (Vmse) is greater than or equal to the first flow control threshold and less than the third flow control threshold, perform flow control on the microservice with a level type of low. Otherwise, execute step 4-4.

[0043] Step 4-4, second-level service flow control, is as follows:

[0044] Collect resource performance data for more than two preset periods, and use the resource performance data and the data corresponding to the minimum time sequence number in the medium data table to calculate the mean square error (Vmse) of CPU usage and memory usage. If the mean square error (Vmse) is greater than or equal to the second flow control threshold and less than the third flow control threshold, perform flow control on the microservice with a level type of medium. Otherwise, execute steps 4-5.

[0045] Steps 4-5, the first level of service circuit breaker, are as follows:

[0046] Wait for more than 2 preset cycles, then collect resource performance data, use the resource performance data and the data corresponding to the minimum time sequence number in the medium data table to calculate the mean square error Vmse of CPU usage and memory usage. If the mean square error Vmse is greater than or equal to the second flow control threshold and less than the third flow control threshold, continue to maintain the second-level service flow control; if the mean square error Vmse is less than the second flow control threshold, use the resource performance data and the high-level data table to continue calculating the mean square error Vmse. If the mean square error Vmse is greater than or equal to the second flow control threshold and less than the third flow control threshold, continue to save the second-level service flow control; if the mean square error Vmse is less than the second flow control threshold, perform circuit breaker processing on the microservice with the low level type and go to steps 4-6;

[0047] Steps 4-6, the second level of service circuit breaker, are as follows:

[0048] Collect resource performance data for two preset periods. Use the resource performance data and the data corresponding to the minimum time sequence number in the advanced data table to calculate the mean square error (Vmse) of CPU usage and memory usage. If the mean square error (Vmse) is greater than or equal to the fourth flow control threshold and less than the third flow control threshold, continue with the second-level service flow control.

[0049] Wait for 2 preset cycles, collect resource performance data, use the resource performance data and the data corresponding to the minimum time sequence number in the advanced data table to calculate the mean square error Vmse of CPU utilization and memory utilization. If the mean square error Vmse is greater than or equal to the fourth flow control threshold and less than the third flow control threshold, keep the current service governance policy unchanged; if the mean square error Vmse is less than the fourth flow control threshold, perform circuit breaker processing on the microservice with a level type of medium.

[0050] Furthermore, the service recovery described in step 5 includes:

[0051] Perform service recovery in the reverse order of service governance, which is:

[0052] Second-level service circuit breaker recovery, first-level service circuit breaker recovery, second-level service flow control, and first-level service flow control.

[0053] The present invention also proposes a microservice dynamic governance system for a distributed computing power network call chain, which is used to implement the aforementioned method, including:

[0054] Configuration module, used to configure service assurance levels and set service governance trigger conditions;

[0055] Data acquisition module, used for data acquisition and preprocessing;

[0056] The risk prediction module is used to perform risk prediction based on the pre-processed data output by the data acquisition module;

[0057] The service governance module conducts service governance based on risk prediction results and performs service recovery after completing service governance.

[0058] Beneficial effects:

[0059] 1. The present invention links computing power node services with business systems to ensure that users' key business activities can operate normally when computing power resources are insufficient.

[0060] 2. The present invention makes full use of the system's own real data to perform positive feedback correction on the algorithm, thereby improving the accuracy of service quality.

[0061] 3. This invention uses data analysis algorithms to perform precise calculations and provide a gradual and graded governance service over a period of time. This allows for the misjudgment of the system caused by uncertainties in business scenarios and customer actions, achieving a smooth transition. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more apparent.

[0063] Figure 1 It is a schematic diagram of the overall process of the present invention.

[0064] Figure 2 It is a schematic diagram of the service governance process in the present invention.

[0065] Figure 3 This is a diagram of a service level label.

[0066] Figure 4 This is a schematic diagram of the data structure after the merger. DETAILED DESCRIPTION

[0067] The present invention provides a distributed computing power network business application microservice dynamic management method and system. The overall idea is to manage microservices based on computing power clusters, such as Figure 1 As shown, it is divided into five steps, which are: manually classifying application services; monitoring the running performance of microservices in the computing system in the computing network cluster, and collecting performance service running status, call failure times, interface call delay time data, and computing node resource usage performance data; cleaning and organizing performance data; using the collected data as input parameters, detecting high-risk services through AI models; and downgrading or fusing high-risk, non-core, and low-security business services to ensure the normal operation of business-critical services, and issuing abnormal alarms.

[0068] In this method, the data collection module regularly collects and stores performance data. Service operation performance data comes from the Elastic Search APM performance monitoring tool, and edge computing node resource operation performance data comes from Prometheus Node Export. The microservice operation risk discovery model includes two algorithms: the "Random Forest" algorithm for operation data prediction and the "Standard Anomaly Score" algorithm for risk monitoring anomaly detection.

[0069] The technical solutions of the present invention are as follows:

[0070] The present invention provides a distributed computing power network business application microservice dynamic governance method and system serving a distributed computing power network cluster.

[0071] The service performance data collection tools and computing node resource monitoring data collection tools involved in this patented method are not covered in this patented method. The underlying algorithms involved in the microservice risk discovery AI model are not covered in this patented method. This patented method focuses on 1) hierarchical management of application systems; 2) cleansing and conversion of service operation data and resource monitoring; and 3) governance and alerting of computing cluster application services.

[0072] The computing network involved in this method consists of multiple sub-clusters and a main cluster management center node. Data collection and data cleaning and conversion, AI risk automatic identification, and service governance are deployed and run as public services in the cluster. Sub-clusters uniformly send alerts to the main cluster management center node, and service risk alerts and service governance results are notified to the main cluster management center node. The central management node completes the service governance alerts and notifications to application customers.

[0073] The main process of the method of the present invention is as follows:

[0074] The execution steps are:

[0075] S1: Application service assurance level label configuration and service governance trigger condition setting

[0076] S1-1: Service assurance level configuration, for example, the service assurance level configuration is 3 levels, namely, level 1 core business system, level 2 key business system service, and level 3 general business system;

[0077] S1-2: Label the service level. Because a function provided by the application system may be completed by multiple microservices in collaboration, there will be multiple services in each level. When configuring, label these services with level tags and record the data in the database. The database data structure includes {service level, service corresponding business system name, service name, service required resource information, minimum number of concurrent services} information; Figure 3 As shown, the storage data format is id, which indicates the primary key; level, which indicates the level type; service_name, which indicates the name of the operating unit; replicas, which indicates the number of replicas; namespace, which indicates the namespace; business_name, which indicates the business name;

[0078] S1-3: Set the service governance conditions to meet the average maximum number of service call failures and average interface call delay time of the high-level application system;

[0079] S2: Regular collection of service operation performance data and resource performance monitoring, data cleaning and data merging processing

[0080] S2-1: The service operation tool collects all service operation data in the cluster every m minutes. Performance indicator parameters include service call operation status, call failure count, and interface call delay time.

[0081] S2-2: The resource monitoring tool collects resource usage data of all nodes every m minutes, including real-time data of CPU usage, memory usage, and storage usage;

[0082] S2-3: Data merging and processing: Read the real-time data of the last hour of the two types of data from the database respectively. Then, according to the principle of time proximity, calculate the resource statistics of the nodes within the same m-minute start and end time, and merge them with each service performance data in the node. The length of the merged data is equal to the service performance data, which is longer than the resource performance data length, because there are multiple services running at each time point. Time is binned according to the step size m. After processing, all services in the same m-minute time period have the same execution sequence number.

[0083] like Figure 4 As shown, the merged data structure is:

[0084] id, indicating the primary key; timestamp, indicating the service execution time; bin_number, indicating the sequence number; pod_name, indicating the name of the running unit; unit_status, indicating the service status type; failed_account, indicating the number of failures; wait_time, indicating the waiting time; node_cpu_used_rate, indicating the CPU usage rate; node_mempry_used_rate, indicating the memory usage rate; node_storage_used_rate, indicating the storage usage rate;

[0085] When the monitoring service finds that the number of service calls exceeds the threshold or the interface call time exceeds the threshold, and the CPU usage of the preset node exceeds cpu_rate, and the memory usage exceeds memory_rate, risk prediction is triggered.

[0086] S3: Risk Prediction

[0087] S3-1: Read the merged data of the last hour and write it to a temporary file

[0088] S3-2: intercept the m data with the smallest sequence number in the temporary file, use the time sequence number of the remaining data as the input data for the risk prediction module, start risk prediction, and write the predicted data into the prediction.csv file; the amount of data in the budget result is equal to the sequence number n * the number of service data running per minute. The predicted data includes the time sequence number and the service operation performance and resource usage within the corresponding time sequence number

[0089] S3-3: Scoring of prediction data: The algorithm model will automatically take the output prediction.csv as input and use the scoring model to score the data.

[0090] S3-4: Filter out data with a score less than 0.7 and output data with a score greater than 0.7 as the final risk prediction result prediction_scoring.csv

[0091] S4: Predictive Data Analysis

[0092] Parse the data in prediction_scoring.csv and find the median number of cpu, memory, and storage; the maximum number of cpu, memory, and storage;

[0093] S4-1: If the median of cpu, memory, and storage are not in the same time sequence, continue to check whether cpu and memory are in the same time sequence. If so, take this time sequence and the next median array, a total of 2 sets of data; if the cpu and memory medians are not in the same time interval, take the median sequence number of cpu, memory, and storage and the cpu, memory, and storage values ​​near the median, store these data as median_dataset table, and reversely map the time sequence number to the time value to timestamp all data

[0094] S4-2: Using a similar method, find the service data within time sequence n and time sequence n-1 when the CPU, memory, and storage all reach their maximum values ​​and record the data into the high_dataset table.

[0095] S4-3: Calculate the mean square error between the time in median_dataset and the time in high_dataset, and compare it with the preset risk value λ (Lambda); if the mean square error is less than λ, it means that a scheduled task needs to be started, and the scheduled time is the minimum time in high_dataset. The scheduled task triggers the microservice governance action.

[0096] S5: Service governance, e.g. Figure 2 shown.

[0097] The following example illustrates the service governance process using a user-configured three-level service model. The three levels are ranked high, medium, and low, respectively. The reference value for each level of service governance is the mean squared error (Vmse) (V: value, mse: Mean Squared Error). Vmse is calculated using the implementation data and the corresponding CPU / memory data in the media_dataset or high_dataset datasets.

[0098] In the service governance service, four Vmse thresholds and trigger conditions need to be preset or configured.

[0099] failed_account: number of failed calls

[0100] wait_time: interface call time

[0101] cpu_rate: cpu usage

[0102] memory_rate: memory usage

[0103] Vmse_low_fc: low level service flow control threshold

[0104] Vmse_medium_fc: Service flow control threshold for medium level

[0105] Vmse_low_cb: Service fuse for low level

[0106] Vmse_medium_cb: Service fuse for medium level

[0107] Vmse_low_fc > Vmse_medium_fc > Vmse_medium_cb > Vmse_medium_cb

[0108] Recovery waiting interval wait_peroid; at least greater than 1 data collection cycle

[0109] S5-1: Read the current system resource usage data and service list

[0110] S5-2: Use the most recent real-time resource data and the resource data corresponding to the minimum time serial number in median_dataset to calculate the mean square deviation of CPU and memory. If the mean square deviation Vmse is greater than Vmse_low_fc, stop service governance; if the mean square deviation is less than Vmse_low_fc, enter S5-2-1

[0111] S5-2-1: Low-level service flow control. Calculate the mean square deviation of CPU and memory using the real-time data and the resource data corresponding to the maximum time serial number in median_dataset. If the final Vmse >= Vmse_low_cb, stop service governance; if Vmse < Vmse_low_cb, enter S5-2-2

[0112] S5-2-2: Continue to collect data for 1 cycle, and use the method in 5-2 to calculate the mean square deviation of CPU and memory. If Vmse_low_fc <= Vmse < Vmse_medium_fc, then perform flow control on the services with low service level

[0113] S5-3: Medium-level service flow control. At least 2 data collection cycles, use the most recent real-time data to execute S5-2-2. If Vmse_low_cb <= Vmse < Vmse_medium_fc, perform flow control on the services with medium service level;

[0114] S5-4: Low-level service fuse. Wait for real-time data for at least 2 data cycles, and continue to calculate the mean square error using the recently collected real-time data and median_dataset. If Vmse_low_cb <= Vmse < Vmse_medium_fc, continue to maintain the medium data flow control. If Vmse < Vmse_low_cb, then use this real-time data and high_dataset to continue calculating the mean square error. If Vmse_low_cb <= Vmse < Vmse_medium_fc, continue to perform flow control on the medium-level service. If Vmse < Vmse_low_cb, then perform fuse processing on the low-level service; enter S5-4-1

[0115] S5-5: Medium-level service fuse. Collect resource performance data twice, and calculate the mean square error using the data with a smaller serial number and high_dataset data. If Vmse_medium_cb <= Vmse < Vmse_medium_fc, continue to perform flow control on the meidum service. Wait for at least 2 data collection cycles, and calculate the mean square error using the real-time resource performance data and high_dataset data. If Vmse_medium_cb <= Vmse < Vmse_medium_fc, keep the service policy governance policy unchanged. If Vmse < Vmse_medium_cb, perform fuse processing on medium;

[0116] S6: Service recovery

[0117] Service recovery needs to be triggered after the fuse service runs for the wait_period cycle and meets the cpu and memory usage conditions. Then calculate the Vmse value, and gradually recover the services that have been flow-controlled or fused according to the Vmse value. The recovery order is medium service -> low service -> medium service flow control -> low service flow control.

[0118] S6-1: Medium-level service fuse recovery. When Vmse >= Vmse_medium_cb, resume the operation of the medium-level service, and at the same time perform flow control on the medium-level service. After waiting for the wait_period time, continue to calculate the Vmse value

[0119] S6-2: Low-level service fuse recovery. When Vmse_low_cb <= Vmse < Vmse_medium_fc, resume the operation of the low-level service, and at the same time perform flow control on the low-level service. After waiting for the wait_period time, continue to calculate the Vmse value

[0120] S6-3: Medium-level service flow control recovery. When Vmse_medium_fc = <Vmse < Vmse_low_fc, cancel the medium-level service flow control, and calculate the Vmse value again after waiting for the wait_period time.

[0121] S6-4: Medium-level service flow control recovery. When Vmse_low_fc = <Vmse, cancel the low-level service flow control.

[0122] Example:

[0123] Taking a specific implementation scenario as an example: In the property information digital management platform of an industrial park, there are various digital services, such as parking lot service system, elevator monitoring system, charging pile management system, property office OA system, property charging system, large screen service in the fire control room, access control monitoring system, video monitoring system, alarm system, and platform service management system. The above service systems run in a container cluster, and there are multiple computing power nodes in this cluster.

[0124] To verify the effectiveness of this method, do not deploy the resource consumption service system on each computing power node. This service calls each other and frequently writes disk data, occupying a large amount of network bandwidth, cpu, and memory; add services with increased call response delay and random call failures to the high-level application system services, and start this service as multiple replicas;

[0125] The implementation process is as follows:

[0126] Step S001: Application system level configuration

[0127] High: Parking lot service system, alarm system, video monitoring system, resource consumption system

[0128] Medium: Large screen service in the fire control room, elevator monitoring system, access control monitoring system

[0129] Low: Charging pile management system, property office OA system, property charging system

[0130] Step S011: Configure governance thresholds

[0131] failed_account = 1000;

[0132] wait_time = 2s

[0133] Low-level service flow control threshold, Vmse_low_fc = 0.55

[0134] Low-level service fuse, Vmse_low_cb = 0.375

[0135] Medium-level service flow control threshold, Vmse_medium_fc = 0.125

[0136] Medium-level service circuit breaker, Vmse_medium_cb = 0.1

[0137] Governance recovery waiting period, wait_peroid=7

[0138] Cpu_rate:50%,memory_=70%

[0139] S012: Data collection. Data collection is performed periodically and continuously after the service starts. Service monitoring data collects service operation performance parameters every 3 minutes and stores the data; resource monitoring data collects system resource usage performance parameters every 3 minutes and stores the data in the Hbase database.

[0140] If the total number of business system service failures collected for five consecutive times exceeds 1,000 times, the average delay exceeds 2 seconds, the CPU usage exceeds 50%, and the memory usage exceeds 70%, the data service governance process will be triggered;

[0141] S002: raw data processing;

[0142] Read the service monitoring indicator data and resource monitoring indicator data for the last hour separately, as both real-time data are stored as unstructured data. After reading the data, merge the data. Since the two collection methods have the same interval, the results are the same. Find the time point where the two data overlap or the time difference is less than 1 minute to start merging the data. For the specific data structure, see the solution section.

[0143] The merged data is binned with a step size of 3 minutes, and the data within 3 minutes are marked in reverse chronological order; the maximum time mark is n

[0144] S003: retain all data marked as 1, use all service data of the service systems marked 2 to n as input data, call the prediction model, and obtain the model output result;

[0145] S004: prediction result analysis;

[0146] The prediction results show that the median number of calls to all application system services is the total number, and the average interface call latency is 1.08 seconds. The median container CPU usage is 58%, and the number of memory locations is 69%. The corresponding time series are 18 and 19, and the mapping time is from 18:30:00 to 19:30:00. This data is recorded as medium_dataset. Similarly, the mapping time period for high_dataset is between 8:30 and 9:30.

[0147] S005: Service governance;

[0148] S051: The first-order service governance collects real-time data of two performance parameters, set as the 0th minute, calculates the mean square deviation of each performance parameter with the same parameter in medium_dataset, and finally calculates the mean square deviation results of all parameters, and calculates the number of service failures and service delay time, and the Vmse values ​​of memory usage respectively.

[0149] Vmse_callfaild=0.373,

[0150] Vmse_delaytime=0.451,

[0151] vmse_cpu=0.350

[0152] vmse_memory=0.344

[0153] Continue to increase resource consumption of abnormal system. Use the data of the 3rd minute medium_dataset to calculate Vmse, and calculate

[0154] Vmse_callfaild=0.373,

[0155] Vmse_delaytime=0.331,

[0156] vmse_cpu=0.316

[0157] vmse_memory=0.344

[0158] All values ​​are less than 0.375, and flow control is performed on low-level applications such as charging pile management systems, property office OA systems, and property charging systems; OA system access delays are severe;

[0159] S052 Second-stage service governance: Continue to increase the number of service outages and resource consumption frequency of the resource consumption abnormal system; use the data at the 6th and 9th minutes and the data in the medium_dataset to calculate their Vmse values ​​respectively:

[0160] Vmse_callfaild=0.224,

[0161] Vmse_delaytime=0.256,

[0162] vmse_cpu=0.259

[0163] vmse_memory=0.277

[0164] Wait_period=3min

[0165] The system continues to perform flow control on medium-level application systems.

[0166] S053 continues to obtain data at the 12th and 15th minutes, and calculates its Vms value with high_dataset. Both are less than 0.375, and the low-level service is stopped.

[0167] Continue to increase the number of abnormal system service operations;

[0168] S054: Continue to obtain data from the 18th and 21st minutes, and calculate their Vms values ​​with high_dataset. If both are less than 0.125, downgrade the mediun service and reduce the number of medium service replicas.

[0169] S055: Continue to obtain data from the 24th and 27th minutes, and calculate their Vms values ​​with high_dataset. If both are less than 0.1, the mediun service is throttled.

[0170] Reduce the number of abnormal system services running and wait for a while

[0171] S055: Continue to obtain data from the 33rd and 36th minutes, calculate the Vmse value of the data and high_dataset, 0.125>Vmse>0.1, and gradually restore the service data of the medium-level application system; restore one application system at a time, and it takes 9 minutes to restore the services of 3 application systems.

[0172] Reduce the number and frequency of abnormal system service operations, and wait for a while

[0173] S056: Continue to obtain monitoring indicator data at the 48th and 52nd minutes, calculate its Vmse value with high_dataset, and if Vmse>0.125, gradually restore low-level application system services; a total of 9 minutes

[0174] Stop the interference of abnormal service system, wait for a while, and then resume flow control

[0175] S057: Continue to obtain monitoring indicator data at the 64th and 67th minutes, calculate its Vmse value with that of high_medium, and if Vmse is greater than 0.45, gradually restore the number of replicas of the medium-level application system service three times, and cancel the flow control of the medium-level service.

[0176] Stop the interference of abnormal service system, wait for a while, and then resume flow control

[0177] S058: Continue to obtain monitoring indicator data at the 79th and 82nd minutes, calculate its Vmse value with that of high_medium, and if Vmse is greater than 0.55, gradually restore the number of replicas of the low-level application system service at intervals of three times.

[0178] At this point, it took a total of 82 minutes from the discovery of the anomaly to the final service restoration, and the system completed the process from service governance to service recovery.

[0179] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium is capable of storing a computer program that, when executed by the data processing unit, can run the invention content of the method and system for dynamic microservice governance of a distributed computing network call chain provided by the present invention, as well as some or all of the steps in each embodiment. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0180] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, in essence or in other words, the part that contributes to the prior art, can be embodied in the form of a computer program, i.e., a software product. The computer program software product can be stored in a storage medium and includes a number of instructions for enabling a device including a data processing unit (which can be a personal computer, server, single-chip microcomputer, MCU, or network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0181] The present invention provides a method and system for dynamically managing microservices in a distributed computing network call chain. There are many methods and approaches for implementing this technical solution. The above is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications should also be considered within the scope of protection of the present invention. Any components not specified in this embodiment may be implemented using existing technologies.

Claims

1. A method for dynamic microservice governance of a distributed computing network call chain, characterized in that: The following steps are involved: Step 1: Initialization, which is to configure the service guarantee level and set the service governance trigger conditions; Step 2: data collection and preprocessing; Step 3: perform risk prediction based on the preprocessed data; Step 4: Conduct service governance based on risk prediction results; Step 5: After completing the service governance, perform service recovery to implement dynamic microservice governance of the distributed computing power network call chain.

2. According to claim 1, a method for dynamic microservice governance of a distributed computing network call chain is characterized in that: Configure the service guarantee level described in step 1, including: The assurance level of the microservices in the distributed computing network call chain is divided into three levels, and a label is configured for each microservice to obtain label data of the service assurance level; the label data includes: level type, operating unit name, number of replicas, namespace and business name; among which, the level type includes: low level, medium level and high level.

3. According to claim 2, a method for dynamic microservice governance of a distributed computing network call chain is characterized in that: The service governance trigger conditions described in step 1 include: The maximum number of call failures of the microservice that meets the preset level is greater than the failure number threshold; The average delay of the interface call is greater than the time threshold; The CPU usage and memory usage of the computing power node are greater than the preset thresholds.

4. According to claim 3, a method for dynamic microservice governance of a distributed computing network call chain is characterized in that: Data collection as described in step 2, including: Using a service operation tool, every preset time period, collect service performance data of all microservices in the distributed computing network, including: service call operation status, call failure number and interface call delay time; A resource monitoring tool is used to collect resource performance data of all nodes in the distributed computing network at the same preset time period, including: CPU utilization, memory utilization and storage utilization.

5. According to claim 4, a method for dynamic microservice governance of a distributed computing network call chain is characterized in that: The pre-processing described in step 2 includes: The service performance data and resource performance data are merged and processed as follows: The service performance data and resource performance data within the preset time period are read respectively; according to the principle of time proximity, the resource performance data of the node within the preset time period with the same start and end time is merged with the service performance data of each microservice in the node; Perform box processing on the merged data time column to generate a time sequence number; The combined data includes: Service execution time, sequence number, running unit name, service status type, number of failures, waiting time, CPU usage, memory usage, and storage usage.

6. According to claim 5, a method for dynamic microservice governance of a distributed computing network call chain is characterized in that: Conduct risk profiling as described in step 3, including: Step 3-1: When the maximum number of service call failures in the merged data is greater than the failure number threshold, and the average time of the interface call delay is greater than the time threshold, and the CPU usage of the node exceeds the threshold or the memory usage exceeds the threshold, execute step 3-2 to perform risk prediction; Step 3-2, read the merged data of the node in the last hour and write it to a temporary file; Step 3-3, intercepting m data with the smallest time sequence number in the temporary file, and using the time sequence number of the remaining data, the service performance data and resource performance data corresponding to each time sequence number as input data of the prediction model for predicting risks; Step 3-4, the prediction model scores the input data; Step 3-5, filter out data with scores less than the preset score and save it as the final risk prediction result.

7. According to claim 6, a method for dynamic microservice governance of a distributed computing network call chain is characterized in that: The prediction model described in steps 3-4 scores the input data, including: Step 3-4-1, parse the input data, find the time sequence number corresponding to the median of CPU usage, memory usage, and storage usage, and the time sequence number corresponding to the maximum value of CPU usage, memory usage, and storage usage; Step 3-4-2, if the medians of CPU usage, memory usage, and storage usage are not in the same time sequence number, then continue to determine whether the medians of CPU usage and memory usage are in the same time sequence number. If so, take this time sequence number and the next median array, a total of 2 groups of data; if the medians of CPU usage and memory usage are not in the same time sequence number, then take the median sequence number of CPU usage, memory usage, and storage usage and the values ​​of CPU usage, memory usage, and storage usage closest to the median, respectively, store the above data as a medium data table, and reversely map the time sequence number to the time value, and stamp all data with corresponding timestamps; Step 3-4-3, using the same method as step 3-4-2, create a high-level data table based on the time sequence number corresponding to the maximum value of CPU usage, memory usage, and storage usage; Step 3-4-4, calculate the mean square error between the time in the medium data table and the time in the high-level data table, and compare it with the preset risk value λ. If it is less than, set the timing time to the minimum time in the high-level data table, and use this timing to trigger the microservice governance.

8. According to claim 7, a method for dynamic microservice governance of a distributed computing network call chain is characterized in that: Perform service governance as described in step 4, including: Step 4-1, read the resource performance data and microservice list used by the current node; Step 4-2, using the resource performance data and the data corresponding to the minimum time sequence number in the medium data table, calculate the mean square error Vmse of the CPU usage rate and the memory usage rate. If the mean square error Vmse is greater than the first flow control threshold, stop service management. If the mean square error Vmse is less than the first flow control threshold, execute step 4-3; Step 4-3, perform the first level service flow control, as follows: Use the resource performance data and the data corresponding to the minimum time sequence number in the medium data table to calculate the mean square error Vmse of the CPU usage rate and the memory usage rate. If the mean square error Vmse is greater than or equal to the second flow control threshold, stop service governance. Otherwise, continue to collect data for one cycle and use the method in step 4-2 to calculate the mean square error Vmse of the CPU usage rate and the memory usage rate. If the mean square error Vmse is greater than or equal to the first flow control threshold and less than the third flow control threshold, perform flow control on the microservice with a level type of low. Otherwise, execute step 4-4. Step 4-4, second level service flow control, is as follows: Collect resource performance data of more than two preset cycles, use the resource performance data and the data corresponding to the minimum time sequence number in the medium data table to calculate the mean square error Vmse of CPU usage and memory usage. If the mean square error Vmse is greater than or equal to the second flow control threshold and less than the third flow control threshold, perform flow control on the microservice with the level type of medium, otherwise execute steps 4-5; Step 4-5, the first level service fuse is as follows: Wait for more than 2 preset cycles, collect resource performance data again, use the resource performance data and the data corresponding to the minimum value of the time sequence number in the medium data table to calculate the mean square error Vmse of the CPU usage and memory usage. If the mean square error Vmse is greater than or equal to the second flow control threshold and less than the third flow control threshold, continue to maintain the second-level service flow control; if the mean square error Vmse is less than the second flow control threshold, use the resource performance data and the high-level data table to continue to calculate the mean square error Vmse. If the mean square error Vmse is greater than or equal to the second flow control threshold and less than the third flow control threshold, continue to save the second-level service flow control; if the mean square error Vmse is less than the second flow control threshold, perform fuse processing on the microservice with the level type of low, and enter steps 4-6; Steps 4-6, the second level service fuse, are as follows: Collect resource performance data of two preset cycles, use the resource performance data and the data corresponding to the minimum time sequence number in the advanced data table to calculate the mean square error Vmse of CPU usage and memory usage. If the mean square error Vmse is greater than or equal to the fourth flow control threshold and less than the third flow control threshold, continue to perform the second level service flow control; Wait for 2 preset cycles, collect resource performance data, use the resource performance data and the data corresponding to the minimum time serial number in the advanced data table to calculate the mean square error Vmse of CPU utilization and memory utilization. If the mean square error Vmse is greater than or equal to the fourth flow control threshold and less than the third flow control threshold, keep the current service governance strategy unchanged; if the mean square error Vmse is less than the fourth flow control threshold, perform circuit breaker processing on the microservice with a level type of medium.

9. A method for dynamic microservice governance of a distributed computing network call chain according to claim 8, characterized in that: Perform service recovery as described in step 5, including: Perform service recovery in the reverse order of service governance, in the following order: Second-level service circuit breaker recovery, first-level service circuit breaker recovery, second-level service flow control, and first-level service flow control.

10. A microservice dynamic governance system for a distributed computing network call chain, characterized in that: Used to implement the method according to any one of claims 1 to 9, comprising: Configuration module, used to configure service assurance levels and set service governance trigger conditions; A data acquisition module, used for data acquisition and preprocessing; The risk prediction module is used to perform risk prediction based on the pre-processed data output by the data acquisition module; The service governance module conducts service governance based on risk prediction results and performs service recovery after completing service governance.

Images