Intelligent inspection method for electric power information system

Abstract

The invention relates to the field of monitoring and maintenance of an electric power information system, in particular to an intelligent inspection method of the electric power information system. The invention provides a method for carrying out intelligent inspection on an electric power information system, which is characterized in that a Python script is utilized to carry out regular analysis and diagnosis on to-be-diagnosed information, feedback information is automatically generated according to a diagnosed abnormal rating result and a data type of the to-be-diagnosed information, and the feedback information is sent to mobile equipment of operation and maintenance personnel through an internal and external network isolation device. The objective of the invention is to improve the problems of insufficient automation degree, long fault response time, insufficient preventability and private network limitation in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of monitoring and maintenance of electric power information systems, and in particular to an intelligent inspection method for electric power information systems. Background Art

[0002] As the State Grid's business continues to deepen and expand, the digital support system architecture is becoming increasingly complex, and cloud-based systems have become a mainstream trend. The number of State Grid power information systems deployed on the cloud has increased dramatically, and the business network architecture has become larger and more complex. In order to ensure the normal operation of the State Grid's core business, the requirements for the stability and reliability of the information system are increasing.

[0003] Since the scale of business clusters deployed on the cloud varies, each cluster has a set of Prometheus monitoring components. In the existing technology, operators have to switch back and forth between multiple clusters to observe complete monitoring information during inspections, which increases the complexity of extracting monitoring information, which is not only inefficient but also prone to errors. In addition, the existing technology often uses data visualization for real-time monitoring, which requires operators to pay attention all the time. Only when the monitoring information reaches the alarm threshold will an information prompt be issued, and the inspection data cannot be sent uniformly. In addition, the power system itself is subject to the limitations of the private network, which makes it impossible to directly send alarms and inspection data to personal terminals. In summary, there is an urgent need for a method for monitoring and intelligent inspection of the server cluster of the power information system to improve the problems of insufficient automation, long fault response time, insufficient prevention and limitations of the private network in the existing technology.

[0004] Kubernetes, as the main tool for container orchestration, makes the deployment of containerized applications simple and efficient; Prometheus is an open source data collection and monitoring framework. The basic principle is to periodically capture the status of monitored components through the HTTP protocol. Any component can be connected to the monitoring as long as it provides the corresponding HTTP interface, without the need for any SDK or other integration process; Grafana is a dashboard and graphic editor that customizes the configuration of multiple display dashboards to form a large visual screen to display the collected data in real time. Summary of the invention

[0005] The purpose of the present invention is to regularly analyze and diagnose the indicator information pulled from the storage module through Python scripts, and send the feedback information text file to the personal terminal through shell scripts, internal and external network isolation device databases, and Python technology to achieve intelligent inspections and improve the problems of insufficient automation, long fault response time, insufficient preventiveness, and private network limitations in the existing technology.

[0006] The method for carrying out intelligent inspection of the power information system provided by the present invention comprises the following steps:

[0007] Periodically acquiring monitoring information from multiple service clusters, and filtering information to be diagnosed based on the monitoring information;

[0008] Performing an abnormality rating on the information to be diagnosed based on a preset range, and generating a feedback information data set based on the rating result of the information to be diagnosed and the data type of the information to be diagnosed;

[0009] Write the feedback information data set into a text file;

[0010] The text file is sent to the mobile device of the operation and maintenance personnel by using an interface call.

[0011] The intelligent inspection method provided by the present invention has the technical effects of automation, fast response time and strong preventiveness.

[0012] Optionally, the business cluster is a cloud cluster system deployed by K8S adopted by the power information system.

[0013] Optionally, the business cluster provides monitoring information through a federated cluster approach.

[0014] Optionally, the monitoring information includes:

[0015] Host operation data: server IP, number of connections, CPU usage, memory usage, disk usage, IOut il usage, average load in 5 groups;

[0016] Data: namespace, physical node, container, network service, configuration file;

[0017] Temporary parameters: kubernetes-cadvisor, kubernetes-service-endpoints, kubernetes-services, kubernetes-ingresses, kubernetes-pods.

[0018] Optionally, the monitoring information is average data during operation.

[0019] Optionally, a time series database is used to store monitoring information.

[0020] Optionally, a time range is preset based on a monitoring density requirement of an operation and maintenance personnel, and information to be diagnosed is filtered from the monitoring information based on the preset time range.

[0021] Optionally, SQL is used to filter and rate the information to be diagnosed based on a preset range, and SQL is used to search for predefined response measures based on the rating results of the diagnostic information and the data type of the information to be diagnosed to generate a feedback information data set.

[0022] Optionally, the step of sending the text file to the mobile device of the operation and maintenance personnel includes:

[0023] Transmit the text file of feedback information to the intranet database of the power information system;

[0024] The program automatically obtains the field data in the specified intranet database table;

[0025] Provide the above data to external calls in the form of interfaces;

[0026] Transfer the above interface to the mobile device of the operation and maintenance personnel.

[0027] Optionally, the channels for sending the text file to the mobile device of the operation and maintenance personnel include: email, SMS, and WeChat. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a structural block diagram of the management cluster of the present invention;

[0029] Figure 2 It is a structural block diagram of the service cluster of the present invention;

[0030] Figure 3 It is a deployment structure diagram and principle block diagram of the present invention;

[0031] Figure 4 It is a schematic diagram of the execution flow of the Python inspection script of the present invention;

[0032] Figure 5 It is a flow chart of two processing methods of monitoring information of the present invention;

[0033] Figure 6 It is a schematic diagram of the inspection result sending process of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be understood by people with general skills in the field to which the present invention belongs. "Including" and similar words used in this article mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0035] An embodiment of the present invention provides a method for carrying out intelligent inspection in an electric power information system, comprising the following steps:

[0036] The first step is to periodically obtain monitoring information from multiple service clusters, and filter out information to be diagnosed based on the monitoring information;

[0037] Step 2: performing abnormal rating on the information to be diagnosed based on a preset range, and generating a feedback information data set based on the rating result of the information to be diagnosed and the data type of the information to be diagnosed;

[0038] Step 3: Write the feedback information data set into a text file;

[0039] Step 4: Send the text file to the mobile device of the operation and maintenance personnel by using an interface call.

[0040] The example of the present invention implements the method through an inspection system including a management cluster and multiple service clusters.

[0041] The inspection system deployment structure diagram and principle block diagram are as follows Figure 3 As shown in the figure, it includes a management cluster and multiple business clusters. The business cluster captures monitoring information through multiple nodes, and the management cluster stores, visualizes, and regularly inspects the monitoring information.

[0042] The management cluster structure diagram is as follows Figure 1 As shown, it includes S101 monitoring module, S102 alarm module, S103 UI visualization module, S104 message notification module, S105 inspection function component, and S106 storage module.

[0043] The S101 monitoring module includes:

[0044] The indicator crawling rule maintenance unit is used to configure the crawling rules of all set resources in the Prometheus.yml configuration file of Prometheus;

[0045] Monitoring component deployment unit, used to deploy the file prometheus-all.yaml through the cluster

[0046] A monitoring component is deployed for at least one service cluster to be monitored, and the monitoring component periodically captures average data of each resource operation in the service cluster according to a preset capture rule.

[0047] The S102 alarm module includes:

[0048] The alarm rule maintenance unit is used to configure the alarm rules of all set resources in the Prometheus.yml configuration file of Prometheus;

[0049] A receiving unit, configured to receive the alarm information sent by the S101 monitoring module when the S101 monitoring module determines that the average indicator data captured on the service cluster to be monitored triggers the alarm rule, and push the alarm information to the alarm management component Alertmanager;

[0050] The sending unit is used to send the alarm information in the alarm management component Alertmanager to the S104 message notification module.

[0051] The S103UI visualization module includes:

[0052] A data storage unit, used to store captured business cluster data;

[0053] The data interface unit is used to further process the business cluster data captured by Prometheus;

[0054] The data display unit is used to display the business cluster data captured by Prometheus through specific applications. Grafana is commonly used for display. Multiple display dashboards can be customized to form a large visual screen to display text files in real time. It has excellent compatibility.

[0055] The S104 message notification module is used to provide a channel for notifying technicians of alarm information, configure the account and password of the message sending channel, and manage different alarm information to be sent to the mobile devices of operation and maintenance personnel through various channels, such as email, SMS, WeChat group, etc.

[0056] The S105 inspection function module uses a Python inspection script to regularly analyze and diagnose the data collected by Prometheus from the storage module and collects them into a text file so that the S104 message notification module can send out the inspection results.

[0057] The S106 storage module is used to store the collected data in a time series database (TSDB). Time is its main axis, and the data arrives in sequence. It is provided to the S103 UI visualization module, S105 inspection function module and other modules through the Prometheus API interface for further processing of the data.

[0058] The business cluster structure diagram is as follows Figure 2 As shown, it includes the Prometheus component and kube-state-metrics component of the container orchestration system and data monitoring system.

[0059] The S201 and S204 container orchestration systems include:

[0060] Cluster control unit: apiserver server (kube-apiserver), scheduler (kube-scheduler) and control manager (kube-controller-manager);

[0061] Cluster worker node unit: worker node proxy component (kubelet), worker node network proxy component (kube-proxy);

[0062] Cluster storage unit: storage system (etcd).

[0063] The S102, S105 Prometheus components and the S103, S106 kube-state-metrics components include:

[0064] Image unit: kube-state-metrics2-7-0.tar image package and prometheus.tar image package;

[0065] Configuration file units: kube-state-metrics.yaml and prometheus-all.yaml

[0066] In each cluster to be monitored, start it in Deployment mode to obtain resources in the K8S cluster including Namespace / Node / Pod / Service / ConfigMap in real time.

[0067] The above is the structure of the inspection system. Through the inspection system, the method provided by the example of the present invention can be well implemented. When executing the method, it is also necessary to pay attention to:

[0068] When executing the first step, the Prometheus component only needs to start the corresponding version of the image prometheus. It does not need to deploy Alertmanager and Grafana at one time like the official website. The business cluster minimizes components without affecting data collection.

[0069] When executing the first step, each business cluster node deploys a host indicator collector Node-expoter collection component, which only pulls the CPU / memory / network / storage disk indicators that storage users need and care about the most, and filters out a large number of indicators that are useless to users.

[0070] When executing the first step, the Prometheus microservice started by the Deployment was restarted by the K8S cluster, resulting in the loss of cluster collection data. Therefore, the K8S cluster collection data storage time was adjusted to 48 hours in the prometheus-all.yaml configuration file.

[0071] (storage.tsdb.retention=48h), configure apiVersion to expose the application port of Ingress, and pull monitoring information to the management cluster.

[0072] When executing the first step, a time range is preset based on the monitoring density requirement of the operation and maintenance personnel, and the information to be diagnosed is filtered from the monitoring information based on the preset time range.

[0073] When executing the first step, the monitoring information obtained can be processed in two ways: Figure 5 As shown, S502-S505 are steps for converting monitoring information into a display text file, and then displaying the text file through a visualization tool such as Grafana, and S506 and S507 are steps for regularly diagnosing the monitoring information through a Python script to generate a feedback information text file.

[0074] When executing the second step, the feedback information data set generated by the diagnostic information is implemented by the Python inspection script. The execution process of the Python inspection script is as follows: Figure 4 As shown, including:

[0075] S401, obtaining monitoring information, Prometheus master and slave K8S data source of the business cluster regularly collect monitoring information;

[0076] S402, analyze the above monitoring information, process and analyze it to ensure the accuracy of the data. When the Python inspection script calls the PromeQL query interface, it is generally a timely query, which can only represent the inspection situation at that time. By adding "start = initial time format & end = end time format & step = interval seconds" to perform a range query on the indicator item, the result is more accurate.

[0077] S403, the Python inspection script diagnostic information, defines the normal operating range and warning level, detects abnormal modes, predicts potential failures, etc. The Python script is used to filter the data that enters the database after the inspection script is executed. For example, if the memory usage rate exceeds 89%, the column name of the memory range value will be checked. If it is greater than 85% and less than 90%, it will be placed in the general abnormal table. If it exceeds 90%, it will be placed in the severe abnormal table.

[0078]

[0079] S404, the Python inspection script performance report generates a report based on the analysis results, triggers an alarm, or automatically performs certain maintenance tasks. For the tables of various types of abnormal data collected, a report is generated from the table data and sent to a designated WeChat group through a scheduled task. For example, if the disk space of a certain IP in the abnormal table exceeds 90%, the script executes "traverse the IP address and send a remote deletion command remote_command to delete some specified invalid logs

[0080] for(ip,)in ips:

[0081] try:

[0082] ssh = paramiko.SSHClient()

[0083] ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

[0084] ssh.connect(ip,username='username',password='password',port=port)

[0085] stdin,stdout,stderr=ssh.exec_command(remote_command)"

[0086] S405, the Python inspection script predicts and recommends corresponding measures for maintenance personnel, or recommends the execution of predefined response measures. The recommended speech is preset in the recommendation table tables2. For example, if the memory of a certain IP exceeds 90%, it is recommended to execute commands such as free-h and top to view abnormal programs and execute restart and other speech;

[0087]

[0088]

[0089] To help understand the above steps, here is an example of a Python inspection script to query the IP health value:

[0090] Query all IP health values ​​(indicator data) from the data collected by prometheus, and define the query interface function as: http: / / prometheus interface IP: port / api / v1 / query? query = (PromQL statement).

[0091] The format of the obtained indicator items is:

[0092]

[0093] The health of the system is determined based on the health value of the IP (0-100, the larger the value, the healthier the system). If the system health value is lower than 85, the query interface function will be executed multiple times to query the corresponding IP's five-minute load, CPU usage, memory usage, disk usage, number of connections, TCP_tw and other data and store them in the dictionary.

[0094]

[0095]

[0096] When executing step 4, if Figure 6 As shown, the steps of sending a text file to a mobile device of an operation and maintenance personnel include:

[0097] Use Shell script to transfer the text file located in the Promethues server to the intranet database of the intranet isolation device;

[0098] The intranet end of the intranet isolation program automatically obtains the field data in the specified intranet database table;

[0099] The external network end of the internal and external network isolation program regularly provides the above data to external calls in the form of interfaces;

[0100] Transfer the above interface to the specified terminal through Python technology.

[0101] The channels for sending text files to the mobile devices of operation and maintenance personnel include: email, SMS, and WeChat.

[0102] Although the embodiments of the present invention are described in detail above, it is obvious to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations are within the scope and spirit of the present invention as described in the claims. Moreover, the present invention described herein may have other embodiments and may be implemented or realized in a variety of ways.

Claims

1. An intelligent inspection method for a power information system, characterized in that: The following steps are involved: Periodically acquiring monitoring information from multiple service clusters, and filtering information to be diagnosed based on the monitoring information; Performing an abnormality rating on the information to be diagnosed based on a preset range, and generating a feedback information data set based on the rating result of the information to be diagnosed and the data type of the information to be diagnosed; Write the feedback information data set into a text file; The text file is sent to the mobile device of the operation and maintenance personnel by using an interface call.

2. The intelligent inspection method according to claim 1, characterized in that: The business cluster is a cloud cluster system deployed by K8S adopted by the power information system.

3. The intelligent inspection method according to claim 1, characterized in that: The business cluster provides monitoring information through a federated cluster approach.

4. The intelligent inspection method according to claim 1, characterized in that: The monitoring information includes: Host operation data: server IP, number of connections, CPU usage, memory usage, disk usage, IOutil usage, average load in 5 groups; Data: namespace, physical node, container, network service, configuration file; Temporary parameters: kubernetes-cadvisor, kubernetes-service-endpoints, kubernetes-services, kubernetes-ingresses, kubernetes-pods.

5. The intelligent inspection method according to claim 1, characterized in that: The monitoring information is all average data during the operation.

6. The intelligent inspection method according to claim 1, characterized in that: Use a time series database to store monitoring information.

7. The intelligent inspection method according to claim 1, characterized in that: A time range is preset based on the monitoring density requirement of the operation and maintenance personnel, and the information to be diagnosed is filtered from the monitoring information based on the preset time range.

8. The intelligent inspection method according to claim 1, characterized in that: SQL is used to filter and rate the information to be diagnosed based on a preset range, and SQL is used to find predefined response measures based on the rating results of the diagnostic information and the data type of the information to be diagnosed to generate a feedback information data set.

9. The intelligent inspection method according to claim 1, characterized in that: The step of sending the text file to the mobile device of the operation and maintenance personnel includes: Transmit the text file of feedback information to the intranet database of the power information system; The program automatically obtains the field data in the specified intranet database table; Provide the above data to external calls in the form of interfaces; Transfer the above interface to the mobile device of the operation and maintenance personnel.

10. The intelligent inspection method according to claim 1, characterized in that: The channels for sending the text file to the mobile device of the operation and maintenance personnel include: email, text message, and WeChat.

Images