Intelligent operation and maintenance service method and system

Abstract

The invention discloses an intelligent operation and maintenance service method and system, which are combined with an intelligent algorithm, make comprehensive judgment on the current overall health of the system based on existing monitoring data, make relatively accurate pre-judgment on the health of the system in a period of time in the future, automatically dispatch related processing measures, and automatically dispatch the related processing measures before a problem occurs or when the problem just occurs. By immediately implementing the treatment measures, the system problem frequency and the problem duration can be obviously reduced. According to the scheme, quantitative evaluation inspection can be performed on each container / server node within a fixed time range (such as 1 hour), if the problem container / server node is exposed through the health score, grading processing can be performed through the health score, and automatic operation and maintenance can be performed through intelligent operation and maintenance scheduling of system resources subsequently, so that the efficiency is improved. And the operation and maintenance efficiency can be obviously improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer software and network technology, and specifically relates to an intelligent operation and maintenance service method and system. Background Art

[0002] During the operation of a computer system, regular or real-time monitoring is required to obtain the current system status, so that problems can be promptly handled when they occur or are about to occur, and the system can be maintained stable. Most companies' IT operation and maintenance teams can obtain detailed data on system operation monitoring in real time by collecting system operation logs, inserting system probes, etc., and visualize various monitoring data through front-end visualization technology, thus realizing automated operation and maintenance and facilitating the real-time operation status of the monitoring system. However, this monitoring method only achieves online visualization of operation and maintenance data. Although operation and maintenance staff can see various monitoring detailed data records in real time, they generally rely on operation and maintenance staff to monitor the normal operation of the system by staring at the monitoring tools or setting fixed thresholds.

[0003] In traditional automated operation and maintenance work, operation and maintenance personnel will face various performance indicators of a large number of container / server nodes in actual work. When faced with a large number of container / service nodes, it is difficult to quantitatively evaluate the comprehensive service capabilities of each container / server node. Moreover, during implementation, we found that in the process of IT operation and maintenance, automated operation and maintenance with monitoring alone is not enough. Since the threshold is fixed, if there is a problem, no matter how big or small, an alarm or error will be issued. Sometimes, when dealing with normal data fluctuations, an alarm will also be triggered (such as the large traffic and large resource usage caused by big promotions and flash sales). In the past, after the operation and maintenance staff manually judged the severity of the problem, they would arrange for manual handling of the error problem, which was inefficient. In addition, existing monitoring tools can only faithfully record data and do not provide any suggestions on future trends of the data. Summary of the invention

[0004] To solve the above problems, the present invention discloses an intelligent operation and maintenance service method and system, which, in combination with an intelligent algorithm, makes a comprehensive judgment on the current overall health of the system based on the existing monitoring data, and makes a relatively accurate prediction of the system health in the future, and automatically schedules relevant processing measures. Processing measures are implemented immediately before or when problems just occur, which can significantly reduce the number of system problems and the duration of problems.

[0005] To achieve the above object, the technical solution of the present invention is as follows:

[0006] Intelligent operation and maintenance service methods, including:

[0007] Step 1, data collection:

[0008] Collect the data that needs to be monitored for each container / server node and report it to the system;

[0009] Step 2, data integration:

[0010] Store the collected data, and clean and integrate the data by object and time sequence;

[0011] Step 3: Calculate the health of performance indicators:

[0012] The health of the indicators is graded, and different judgment criteria for different grades are set for each indicator. The data processed in step 2 is converted into various health data according to the pre-set health judgment criteria for each indicator;

[0013] Step 4: Calculate the health of the container / server node:

[0014] Each container / server calculates the overall health of the container / server node based on the health data calculated in step 3 and the overall health calculation model;

[0015] Step 5: System warning and troubleshooting:

[0016] Regularly inspect all monitored containers / server nodes, set warning ranges based on the health of each container / server node, and remind operation and maintenance personnel to pay attention to the containers / server nodes in this range; set operation and maintenance intervention ranges to remind operation and maintenance personnel to manually intervene to eliminate various abnormal problems found in monitoring; set operation and maintenance fault ranges to remind operation and maintenance personnel to treat them as operation and maintenance accidents, eliminate accidents as soon as possible, and restore normal use of container / server nodes;

[0017] Configure automatic operation and maintenance processing rules. When the node status or overall health meets the rules, the automatic processing plan configured in the rules is executed.

[0018] Furthermore, the data collected in step 1 include: server host CPU, memory, storage usage, server CPU waiting time for disk writing to complete, database response time, TPS ratio, slow transaction processing speed, application business error log, Apdex index, access error rate, port response time, web availability.

[0019] Furthermore, in step 3, 12 health indicators are set, and the health classification includes: health is set to 1 point; attention is set to 0.7 points, risk is set to 0.5 points, and danger is set to 0 points.

[0020] Furthermore, the overall health calculation model in step 4 is: f(x)=a1x1+a2x2+a3x3…+a12x12, wherein the preset values ​​of the parameters a1-a12 are: a1=4; a2=6; a3=6; a4=6; a5=6; a6=4; a7=4; a8=4; a9=10; a10=15; a11=15; a12=20.

[0021] Furthermore, the automatic operation and maintenance processing rules in step 5 include simple rules and complex rules;

[0022] The simple rules include: when system resources are insufficient, increase CPU computing power, memory or hard disk storage; or shut down slow SQL queries that affect performance;

[0023] The complex rules include:

[0024] When the overall health of the node is within the range of 70-80 points, the automatic scheduling solution corresponding to the lowest single health score will be automatically started;

[0025] When the overall health of the node is within the range of 60-70 points, the automatic scheduling solution corresponding to the lowest two individual health scores will be automatically started;

[0026] When the overall health of the node is within the range of <60 points, the automatic scheduling solution corresponding to the lowest three individual health scores will be automatically started. If the overall health score is <60 points for two observation cycles, the operation and maintenance staff will be alerted and an operation and maintenance work order will be generated.

[0027] The present invention also provides an intelligent operation and maintenance service system, including a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the aforementioned intelligent operation and maintenance service method.

[0028] Furthermore, the computer program includes a data acquisition module, a data integration module, a performance indicator health calculation module, a node health calculation module, and a system early warning and fault handling module. The data acquisition module is used to implement step 1 of the intelligent operation and maintenance service method, the data integration module is used to implement step 2 of the intelligent operation and maintenance service method, the performance indicator health calculation module is used to implement step 3 of the intelligent operation and maintenance service method, the node health calculation module is used to implement step 4 of the intelligent operation and maintenance service method, and the system early warning and fault handling module is used to implement step 5.

[0029] The present invention also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps in the intelligent operation and maintenance service method are implemented.

[0030] The beneficial effects of the present invention are:

[0031] 1. The solution of the present invention can conduct quantitative assessment and inspection of each container / server node within a fixed time range (such as 1 hour). If the problematic container / server node is exposed through the health score, it can be graded and processed according to the health score. It is clear at a glance whether manual intervention is needed in the future, which can significantly improve the operation and maintenance efficiency. The results of health classification and grading can be used to spread warnings through the interface external message channel, and the warning function of problems that cannot be automatically handled by the intelligent operation and maintenance system can be connected with Email, DingTalk, SMS, etc., and synchronized to the core operation and maintenance personnel in a timely manner.

[0032] 2. Through the intelligent operation and maintenance service background function provided by the present invention, the system resource scheduling management tool can be opened to realize the intelligent operation and maintenance scheduling of system resources. Through the system monitoring points and the scores of each sub-item, we can easily judge the crux of the problem points in the system, and solve the temporary shortage of the system by increasing the CPU computing power, memory, hard disk storage and other methods based on various pre-set rules when there is a bottleneck in resources; for example, when there is a slow SQL query, the slow query is actively disconnected; when there are too many application errors, such as when the system has a downgrade plan, the downgrade plan is automatically started, etc., to keep the basic smooth and stable application services. At the same time, after connecting with the online work order system, the online work order is automatically generated and automatically assigned to the operation and maintenance staff for processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of the flow of the intelligent operation and maintenance service method provided by the present invention.

[0034] Figure 2 Schematic diagram of the classification of indicators for container / server monitoring indicators. DETAILED DESCRIPTION

[0035] The technical solution provided by the present invention will be described in detail below in conjunction with specific embodiments. It should be understood that the following specific implementation methods are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0036] The intelligent operation and maintenance service method provided by the present invention comprises the following steps:

[0037] Step 1, data collection: There are many ways to collect data, including log point collection (reporting and parsing through Logstash, Filebeat, etc.), JMX standard interface output monitoring indicators, the monitored object provides RESTAPI for data collection (such as Hadoop, ES), and through the system command line, unified SDK for intrusive point collection and reporting. The objects collected include each container / server CPU, memory, storage, port, database, application, etc. The collected data includes but is not limited to: server host CPU, memory, storage usage, server CPU waiting time for disk write completion, database response time, TPS ratio, slow transaction processing speed, application business error log, Apdex index, access error rate, port response time, web availability. The collected data is reported to the monitoring system through TCP, UDP or HTTP protocol.

[0038] Step 2, data integration: store the collected data in MySQL, Oracle or even HBase, and clean and integrate the data by object and time series. Data cleaning can use conventional methods to remove obviously abnormal data and integrate it into a standardized format.

[0039] Step 3: Calculate the health of performance indicators: divide the monitoring indicators of a single container / server into three categories: server performance monitoring indicators, database performance monitoring indicators, and application monitoring indicators, and define the health of indicators for each monitoring value.

[0040] In such Figure 2 In the table shown, the server performance monitoring indicators include CPU indicators, memory indicators, storage indicators, which are used to indicate the usage of CPU, memory, and storage respectively, and CPUIowait indicators, which are used to indicate how fast the server CPU waits for disk writing. Database performance monitoring indicators include response time indicators, which are used to indicate the response speed of http requests; TPS ratio indicators, which are used to indicate the ratio between the actual system call and the preset call; slow transaction indicators, which are used to indicate the processing speed of slower requests. Application monitoring indicators include ERROR log indicators, which are used to indicate the number of error logs obtained within a certain period of time; Apdex indicators, which are used to indicate the Apdex value within a certain period of time, and access error rate index, which is used to indicate the error rate within a certain period of time; port response indicators, which are used to indicate whether the port is available; web availability indicators, which are used to indicate whether the web application is available.

[0041] Figure 2 The example of health rating of indicators is shown, where healthy is set to 1 point, needing attention is set to 0.7 points, risk is set to 0.5 points, and danger is set to 0 points. Figure 2As shown, the judgment criteria corresponding to different levels of various indicators are pre-set (such as data interval range, judgment mark, etc.). According to the various data collected in real time in step 1 and Figure 2 The preset grading judgment standard can convert the indicator-related data collected at each time point into the health data of a single indicator at a single time node, and then calculate the health of the indicator within this time range through the batch health data collected within a time range. Figure 2 In the "Time Range", 5 minutes is used as an example. Assuming that basic monitoring indicator data is collected every minute and the health of the indicator is calculated every minute, to avoid abnormal interference caused by a single outlier indicator, the average data per hour is used as an example for the indicator calculation value. The health of each indicator of a single container / server is defined as shown in the following table:

[0042]

[0043]

[0044]

[0045] The above indicator health calculation model is a business experience model and is only used as an example. After accumulating data to a certain extent, machine learning methods such as logistic regression models can be used based on actual data to achieve better system monitoring and automatic operation and maintenance effects.

[0046] A regression model is a mathematical model that quantitatively describes a statistical relationship. For example, the mathematical model of multiple linear regression can be expressed as y=β0+β1*x+εi, where β0, β1, …, βp are p+1 parameters to be estimated, εi is a random variable that is independent of each other and obeys the same normal distribution N(0,σ2), y is a random variable; x can be a random variable or a non-random variable, and βi is called a regression coefficient, which represents the degree of influence of the independent variable on the dependent variable.

[0047] Logistic regression is used to calculate the probability of "event = Success" and "event = Failure". When the dependent variable is of binary (1 / 0, true / false, yes / no) type, we should use logistic regression. Here, the value of Y ranges from 0 to 1, which can be expressed by the following equation:

[0048] odd=p / (1-p)

[0049] In the above formula, p represents the probability of having a certain feature.

[0050] Although neural networks and even current large model methods may achieve better model results, since operation and maintenance services need to locate specific problems, it is important to use the logistic regression method to confirm the occurrence of problems and quickly locate the problem points through the high and low scores of specific items. Therefore, in this scenario, it is more appropriate to use the logistic regression method to establish a corresponding relationship model between the health value of each indicator and the collected indicator data.

[0051] Step 4: Calculate the health of the container / server node:

[0052] Each container / server calculates the overall health of the container / server node based on the health sub-indicators calculated in step 3 and after data mining and analysis. Taking the 12 health sub-indicators x1, x2...x12 in step 3 as an example, the overall health calculation model is:

[0053] f(x)=a1x1+a2x2+a3x3……+a12x12,

[0054] Among them, the preset values ​​of the parameters a1-a12 in the model are: a1=4; a2=6; a3=6; a4=6; a5=6; a6=4; a7=4; a8=4; a9=10; a10=15; a11=15; a12=20.

[0055] Therefore, in this example, the health value of a single container / server node ranges from 0 to 100.

[0056] The results of health classification and grading can be used to spread warnings through external message channels through interfaces. The warning function for problems that cannot be automatically handled by the intelligent operation and maintenance system can be connected through Email, DingTalk, SMS, etc., and synchronized to core operation and maintenance personnel in a timely manner.

[0057] Step 5: System warning and troubleshooting:

[0058] System warning part: On the basis of normal operation and maintenance monitoring warning and fault handling, increase the inspection of all monitoring containers / server nodes within a certain range. Take one hour as an example. The time range is: inspect all monitoring containers / server nodes every hour, calculate the health scores of each item and the overall health score of the node in step 3 and step 4 according to the collected monitoring indicators, and set the range of 80 to 90 points as the warning range according to the health score of each container / server node, prompting the operation and maintenance personnel to pay attention to the container / server node in this range; set the range of 60 to 80 points as the operation and maintenance intervention range, prompting the operation and maintenance personnel to intervene manually to eliminate various abnormal problems found in monitoring; set the range of <60 points as the operation and maintenance fault range, prompting the operation and maintenance personnel to treat it as an operation and maintenance accident, eliminate the accident as soon as possible, and restore the normal use of the container / server node. The above score ranges for different treatments need to be adjusted according to the historical data of the target system.

[0059] Automatic fault handling part: In this step, in addition to the calculation rules of system monitoring points, more operation and maintenance automatic processing rules can be configured. The automatic fault handling rules are relatively complex. The system comes with some rules, but whether to enable them requires secondary confirmation by the operation and maintenance staff based on the current system operation status of their organization.

[0060] In actual processing, according to the operation and maintenance requirements, after configuring the operation and maintenance processing rules, configure the corresponding problem processing scheduling process. Once the rules are triggered, the scheduling program will automatically handle the problem immediately to eliminate the impact of some simple problems on the system. Simple rules, such as judging when the temporary increase in application access volume causes insufficient system resources, temporarily solve it by increasing CPU computing power, memory or hard disk storage; shut down and interrupt slow SQL queries that affect performance, etc.

[0061] However, simple rules cannot completely solve complex system problems. Built-in rules can also include some complex rules, such as:

[0062] Based on the score range of 60-80, since each individual health score corresponds to a corresponding automatic scheduling solution, in the score range of 70-80, the automatic scheduling solution corresponding to the lowest individual health score is automatically activated; in the score range of 60-70, the automatic scheduling solution corresponding to the lowest 2 individual health scores is automatically activated.

[0063] If the score is less than 60, the automatic scheduling solution corresponding to the lowest three individual health scores will be automatically started. If the overall health score is less than 60 for two observation cycles, the operation and maintenance staff will be alerted and an operation and maintenance work order will be generated.

[0064] In addition, the hourly health score calculation of the container / server node and the operation and maintenance response operation records are synchronized through the data in the system, collected by the operation and maintenance database, and recorded in the operation and maintenance data warehouse to form time series data records for further analysis and data mining in the future.

[0065] Based on the above-mentioned intelligent operation and maintenance service method, the present invention also provides an intelligent operation and maintenance service system, including a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the above-mentioned intelligent operation and maintenance service method. Specifically, the intelligent operation and maintenance service system includes a data acquisition module, a data integration module, a performance indicator health calculation module, a node health calculation module, and a system early warning and fault handling module, wherein the data acquisition module is used to implement the data acquisition function of step 1, the data integration module is used to implement the data integration function of step 2, the performance indicator health calculation module is used to implement the function of calculating each monitoring indicator in step 3, the node health calculation module is used to implement the function of calculating the health of each container / server node in step 4, and the system early warning and fault handling module is used to implement the function of issuing an early warning based on the node health and automatically handling the fault according to the preset processing rules in step 5.

[0066] The present invention also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps in the intelligent operation and maintenance service method are implemented.

[0067] It should be noted that the above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. An intelligent operation and maintenance service method, characterized in that: include: Step 1, data collection: Collect the data that needs to be monitored for each container / server node and report it to the system; Step 2, data integration: Store the collected data, and clean and integrate the data by object and time sequence; Step 3: Calculate the health of performance indicators: The health of the indicators is graded, and different judgment criteria for different grades are set for each indicator. The data processed in step 2 is converted into various health data according to the pre-set health judgment criteria for each indicator; Step 4: Calculate the health of the container / server node: Each container / server calculates the overall health of the container / server node based on the health data calculated in step 3 and the overall health calculation model; Step 5: System warning and troubleshooting: Regularly inspect all monitored containers / server nodes, set warning ranges based on the health of each container / server node, and remind operation and maintenance personnel to pay attention to the containers / server nodes in this range; set operation and maintenance intervention ranges to remind operation and maintenance personnel to manually intervene to eliminate various abnormal problems found in monitoring; set operation and maintenance fault ranges to remind operation and maintenance personnel to treat them as operation and maintenance accidents, eliminate accidents as soon as possible, and restore normal use of container / server nodes; Configure automatic operation and maintenance processing rules. When the node status or overall health meets the rules, the automatic processing plan configured in the rules is executed.

2. The intelligent operation and maintenance service method according to claim 1, characterized in that: The data collected in step 1 include: server host CPU, memory, storage usage, server CPU waiting time for disk write completion, database response time, TPS ratio, slow transaction processing speed, application business error log, Apdex index, access error rate, port response time, web availability.

3. The intelligent operation and maintenance service method according to claim 1, characterized in that: In the step 3, 12 health indicators are set, and the health classification includes: health is set to 1 point; attention is set to 0.7 points, risk is set to 0.5 points, and danger is set to 0 points.

4. The intelligent operation and maintenance service method according to claim 1, characterized in that: The overall health calculation model in step 4 is: f(x)=a1x1+a2x2+a3x3...+a12x12, wherein the preset values ​​of the parameters a1-a12 are: a1=4; a2=6; a3=6; a4=6; a5=6; a6=4; a7=4; a8=4; a9=10; a10=15; a11=15; a12=20.

5. The intelligent operation and maintenance service method according to claim 1, characterized in that: The automatic operation and maintenance processing rules in step 5 include simple rules and complex rules; The simple rules include: when system resources are insufficient, increase CPU computing power, memory or hard disk storage; or shut down slow SQL queries that affect performance; The complex rules include: When the overall health of the node is within the range of 70-80 points, the automatic scheduling solution corresponding to the lowest single health score will be automatically started; When the overall health of the node is within the range of 60-70 points, the automatic scheduling solution corresponding to the lowest two individual health scores will be automatically started; When the overall health of the node is within the range of <60 points, the automatic scheduling solution corresponding to the lowest three individual health scores will be automatically started. If the overall health score is <60 points for two observation cycles, the operation and maintenance staff will be alerted and an operation and maintenance work order will be generated.

6. An intelligent operation and maintenance service system, comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the intelligent operation and maintenance service method described in any one of claims 1-5.

7. The intelligent operation and maintenance service system according to claim 6, characterized in that: The computer program includes a data acquisition module, a data integration module, a performance indicator health calculation module, a node health calculation module, and a system early warning and fault handling module. The data acquisition module is used to implement step 1 of the intelligent operation and maintenance service method, the data integration module is used to implement step 2 of the intelligent operation and maintenance service method, the performance indicator health calculation module is used to implement step 3 of the intelligent operation and maintenance service method, the node health calculation module is used to implement step 4 of the intelligent operation and maintenance service method, and the system early warning and fault handling module is used to implement step 5.

8. The present invention also provides a computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps in the intelligent operation and maintenance service method described in any one of claims 1 to 5 are implemented.

Images