Power grid data detection method and system, electronic equipment and storage medium

Abstract

The invention discloses a power grid data detection method and system, electronic equipment and a storage medium. The method comprises the steps that power grid data are obtained, and the power grid data are from databases or log files of a source end, a source pasting layer and a sharing layer; processing the power grid data according to a pre-stored anomaly detection algorithm to obtain an anomaly detection result; according to the multi-index anomaly detection method, the anomaly detection result is displayed, the corresponding alarm information is determined according to the anomaly detection result, and the multi-index anomaly detection can integrate the information of the multiple indexes for analysis, so that the anomaly condition is detected more comprehensively and more accurately. Compared with traditional anomaly detection, multi-index anomaly detection can better reflect the overall situation of the system, the situation of missing report and false report is reduced, and the multi-index anomaly detection can be flexibly configured and select proper indexes and algorithms according to different service scenes and requirements so as to achieve the optimal detection effect.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically, to a detection method, system, electronic device and storage medium for power grid data. Background Art

[0002] Anomaly detection has always been a very important branch of machine learning. In various artificial intelligence applications such as computer vision, data mining, and natural language processing, anomaly detection algorithms are a hot research direction. Especially in the era of big data, the speed of manual data processing lags far behind that of machines.

[0003] As the power industry's reliance on big data platforms grows, more and more system data is efficiently integrated into big data platforms, greatly enriching data analysis and application scenarios, and providing a solid data foundation for business decision-making and application scenarios. However, in the process of data flow from the source system to the final application scenario, it needs to cross multiple data links. Excessively long data links increase the complexity and latency of data transmission, and put forward higher requirements for the timeliness and accuracy of data operation and maintenance.

[0004] At present, big data platforms still have significant deficiencies in link monitoring, especially the lack of a mature and global link monitoring mechanism. Currently, they mainly rely on manual and scattered, non-real-time monitoring work between systems. This method is not only inefficient and difficult to ensure real-time acquisition of data link status, but also greatly limits fault response speed and problem-solving capabilities, posing a potential threat to the stable operation of data application scenarios.

[0005] The big data platform lacks real-time monitoring methods for processes from data access to data use. Specifically, in the scenario where OGG (Oracle GoldenGate) synchronizes database data to the platform, the synchronization process can generate detailed log files, but the platform lacks the ability to parse these logs in real time, resulting in the inability to immediately present synchronization status information. When the table structure changes on the source side, the OGG process will be interrupted immediately, affecting data access. More importantly, the platform has failed to establish an effective real-time early warning system, so that when an abnormality occurs in data synchronization, it is impossible to notify the relevant operation and maintenance personnel in the first place, thereby delaying the golden opportunity for abnormality handling. Therefore, how to provide a detection method, system, electronic equipment and storage medium for power grid data has become a technical problem that needs to be solved urgently in this field. Summary of the invention

[0006] The purpose of the present invention is to provide a method, system, electronic equipment and storage medium for detecting power grid data.

[0007] The first aspect of the present invention discloses a method for detecting power grid data; the method comprises:

[0008] Step S1, obtaining power grid data, wherein the power grid data comes from a database or log file of a source end, a source layer, or a shared layer;

[0009] Step S2: Processing the power grid data according to a pre-stored anomaly detection algorithm to obtain an anomaly detection result; wherein the pre-stored anomaly detection algorithm at least includes a multi-index anomaly detection algorithm based on a statistical model or a multi-index anomaly detection algorithm based on machine learning;

[0010] Step S3: display the abnormality detection result, and determine the corresponding alarm information according to the abnormality detection result.

[0011] According to the method of the first aspect of the present invention, in step S2, processing the power grid data according to a pre-stored anomaly detection algorithm to obtain an anomaly detection result includes:

[0012] The power grid data is subjected to multi-index anomaly detection based on a statistical method, and the relationship between the indicators is modeled and analyzed by calculating the statistical quantities of the mean, variance, and covariance of the multiple indicators to obtain the anomaly detection result.

[0013] According to the method of the first aspect of the present invention, in step S2, processing the power grid data according to a pre-stored anomaly detection algorithm to obtain an anomaly detection result includes:

[0014] Based on a multi-index anomaly detection model, multi-index anomaly detection is performed on the power grid data to obtain the anomaly detection result; the multi-index anomaly detection model is obtained by training a machine learning model with sample data to learn the relationship between indicators and anomaly patterns.

[0015] According to the method of the first aspect of the present invention, in step S3, displaying the abnormality detection result and determining corresponding alarm information according to the abnormality detection result includes:

[0016] When the task running time exceeds the preset threshold, an alarm message is sent;

[0017] If the task is not completed within the scheduled time, an alarm message is triggered;

[0018] If the task status changes from "executing" to "abnormal", an alarm will be issued immediately;

[0019] If the task fails multiple times in a row, an alarm message will be sent;

[0020] Compare the task running time within the preset time period. If a significant increase is found, a warning message will be displayed.

[0021] According to the method of the first aspect of the present invention, the method further comprises:

[0022] Use Kibana or Elasticsearch API services to query grid data at each layer within a preset time period;

[0023] Compare the consistency of power grid data at different layers at the same time point to obtain consistent comparison results.

[0024] According to the method of the first aspect of the present invention, the method further comprises:

[0025] Cleaning the power grid data, removing invalid data, and obtaining cleaned data;

[0026] The cleaned data is grouped and aggregated according to the grouping, name, and path dimensions of the API to obtain grouped data;

[0027] Based on the grouped data, the key indicators of average execution time, number of calls, and call success rate of each API service in a historical preset time period are calculated.

[0028] According to the method of the first aspect of the present invention, the method further comprises:

[0029] Based on the pre-trained ARIMA time series analysis model, the future performance indicators of the API service are predicted to obtain the prediction results.

[0030] The second aspect of the present invention discloses a detection system for power grid data; the system comprises:

[0031] The first processing module is configured to obtain power grid data, wherein the power grid data comes from a database or log file of a source end, a source layer, or a shared layer;

[0032] The second processing module is configured to process the power grid data according to a pre-stored anomaly detection algorithm to obtain an anomaly detection result; wherein the pre-stored anomaly detection algorithm at least includes a multi-index anomaly detection algorithm based on a statistical model or a multi-index anomaly detection algorithm based on machine learning;

[0033] The third processing module is configured to display the abnormality detection result and determine corresponding alarm information according to the abnormality detection result.

[0034] According to the system of the second aspect of the present invention, the second processing module is specifically configured to process the power grid data according to a pre-stored anomaly detection algorithm to obtain an anomaly detection result, including:

[0035] The power grid data is subjected to multi-index anomaly detection based on a statistical method, and the relationship between the indicators is modeled and analyzed by calculating the statistical quantities of the mean, variance, and covariance of the multiple indicators to obtain the anomaly detection result.

[0036] According to the system of the second aspect of the present invention, the second processing module is specifically configured to perform multi-indicator anomaly detection on the power grid data based on a multi-indicator anomaly detection model to obtain the anomaly detection result; the multi-indicator anomaly detection model is obtained by training a machine school model with sample data to learn the relationship between indicators and abnormal patterns.

[0037] According to the system of the second aspect of the present invention, the third processing module is specifically configured to send an alarm message when the task running time exceeds a preset threshold;

[0038] If the task is not completed within the scheduled time, an alarm message is triggered;

[0039] If the task status changes from "executing" to "abnormal", an alarm will be issued immediately;

[0040] If the task fails multiple times in a row, an alarm message will be sent;

[0041] Compare the task running time within the preset time period. If a significant increase is found, a warning message will be displayed.

[0042] According to the system of the second aspect of the present invention, the third processing module is specifically configured to use the API service of Kibana or Elasticsearch to query the power grid data of each layer within a preset time period;

[0043] Compare the consistency of power grid data at different layers at the same time point to obtain consistent comparison results.

[0044] According to the system of the second aspect of the present invention, the third processing module is specifically configured to clean the power grid data, remove invalid data, and obtain cleaned data;

[0045] The cleaned data is grouped and aggregated according to the grouping, name, and path dimensions of the API to obtain grouped data;

[0046] Based on the grouped data, the key indicators of average execution time, number of calls, and call success rate of each API service in a historical preset time period are calculated.

[0047] According to the system of the second aspect of the present invention, the third processing module is specifically configured to predict the future performance indicators of the API service according to a pre-trained ARIMA time series analysis model to obtain a prediction result.

[0048] The third aspect of the present invention discloses an electronic device. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in any one of the power grid data detection methods in the first aspect of the present disclosure are implemented.

[0049] The fourth aspect of the present invention discloses a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any one of the power grid data detection methods in the first aspect of the present disclosure are implemented.

[0050] According to the technical content disclosed in the present invention, the following beneficial effects are achieved: by acquiring power grid data, wherein the power grid data comes from a database or log file of a source end, a source layer, or a shared layer; processing the power grid data according to a pre-stored anomaly detection algorithm to obtain an anomaly detection result; wherein the pre-stored anomaly detection algorithm at least includes a multi-index anomaly detection algorithm based on a statistical model or a multi-index anomaly detection algorithm based on machine learning; displaying the anomaly detection result, and determining the corresponding alarm information according to the anomaly detection result, and multi-index anomaly detection can analyze information of multiple indicators comprehensively, thereby detecting abnormal situations more comprehensively and accurately. Compared with traditional anomaly detection, multi-index anomaly detection can better reflect the overall situation of the system, reduce the situation of missed reports and false reports, and multi-index anomaly detection can be flexibly configured according to different business scenarios and needs, and select appropriate indicators and algorithms to achieve the best detection effect.

[0051] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0053] Figure 1 A flowchart of a method for detecting power grid data according to an embodiment;

[0054] Figure 2 A schematic diagram of the structure of a power grid data detection platform provided according to an embodiment;

[0055] Figure 3 is a structural diagram of a power grid data detection system according to an embodiment of the present invention;

[0056] Figure 4 The figure is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0058] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0059] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0060] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0061] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0062] Embodiment 1:

[0063] The invention discloses a method for detecting power grid data. Figure 1 FIG. 1 is a flow chart of a method for detecting power grid data according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0064] Step S1, obtaining power grid data, wherein the power grid data comes from a database or log file of a source end, a source layer, or a shared layer;

[0065] Step S2: Processing the power grid data according to a pre-stored anomaly detection algorithm to obtain an anomaly detection result; wherein the pre-stored anomaly detection algorithm at least includes a multi-index anomaly detection algorithm based on a statistical model or a multi-index anomaly detection algorithm based on machine learning;

[0066] Specifically, the multi-indicator anomaly detection algorithm based on statistical models is a multi-indicator anomaly detection algorithm based on statistical methods. By calculating the mean, variance, covariance and other statistics of multiple indicators, the relationship between the indicators is modeled and analyzed to discover anomalies.

[0067] The multi-indicator anomaly detection algorithm based on machine learning is an anomaly detection algorithm for multiple indicators based on machine learning. By training the multi-indicator anomaly detection model, it automatically learns the relationship between indicators and abnormal patterns to achieve more accurate and reliable anomaly detection.

[0068] Step S3: display the abnormality detection result, and determine the corresponding alarm information according to the abnormality detection result.

[0069] Specifically, the embodiment of the present application can perform data visualization and alarm, wherein:

[0070] The data is presented to users in the form of charts through visualization tools. These visualization charts can intuitively reflect the operating status of the system and the execution of business processes, helping users quickly understand the system situation. The front end is implemented with the Vue framework, Vue.js is used to build the user interface, combined with Vuex for state management, and Vue Router for page routing management. The back end uses the Spring Boot framework to build a RESTful API, which has the advantages of rapid development, easy deployment and integration.

[0071] Set up an alarm mechanism based on preset alarm rules and thresholds. When the system is abnormal or the performance index exceeds the threshold, an alarm is automatically triggered to notify the user to handle it. The alarm notification can be sent to the user via email.

[0072] In step S2, the power grid data is processed according to a pre-stored anomaly detection algorithm to obtain an anomaly detection result, including:

[0073] Based on statistical methods, anomaly detection of multiple indicators is performed on power grid data. By calculating the mean, variance, and covariance statistics of multiple indicators, the relationship between the indicators is modeled and analyzed to obtain anomaly detection results.

[0074] In step S2, the power grid data is processed according to a pre-stored anomaly detection algorithm to obtain an anomaly detection result, including:

[0075] Based on the multi-indicator anomaly detection model, multi-indicator anomaly detection is performed on the power grid data to obtain anomaly detection results; the multi-indicator anomaly detection model is obtained by training the machine school model with sample data to learn the relationship between indicators and abnormal patterns.

[0076] In step S3, the abnormality detection result is displayed, and the corresponding alarm information is determined according to the abnormality detection result, including:

[0077] When the task running time exceeds the preset threshold, an alarm message is sent;

[0078] If the task is not completed within the scheduled time, an alarm message is triggered;

[0079] If the task status changes from "executing" to "abnormal", an alarm will be issued immediately;

[0080] If the task fails multiple times in a row, an alarm message will be sent;

[0081] Compare the task running time within the preset time period. If a significant increase is found, a warning message will be displayed.

[0082] Specifically, the alarm rules include: (1) sending an alarm when the task running time exceeds the preset threshold; (2) triggering an alarm if the task is not completed within the scheduled time; (3) immediately issuing an alarm when the task status changes from "executing" to "abnormal"; (4) sending an alarm after a task fails five times in a row; (5) issuing an alarm when a significant increase is found after comparing the running time of recent tasks; (6) immediately issuing an emergency alarm for serious abnormalities (such as system crashes); (7) sending a normal alarm for general problems (such as minor delays); (8) automatically retrying three times after a task fails, and issuing an alarm if it still fails; (9) generating a task execution report every day, marking slow tasks and abnormal tasks; (10) summarizing the slow tasks and abnormalities of the week before the weekend for special analysis.

[0083] Optionally, the method further comprises:

[0084] Use Kibana or Elasticsearch API services to query grid data at each layer within a preset time period;

[0085] Compare the consistency of power grid data at different layers at the same time point to obtain consistent comparison results.

[0086] Optionally, the method further comprises:

[0087] Clean the power grid data, remove invalid data, and obtain cleaned data;

[0088] Group and aggregate the cleaned data according to the API group, name, and path dimensions to obtain grouped data;

[0089] Based on the grouped data, calculate the key indicators of average execution time, number of calls, and call success rate of each API service within the historical preset time period.

[0090] Specifically, API service monitoring: clean the collected data, remove invalid or abnormal data, group and aggregate the data according to API grouping, name, path and other dimensions. Calculate key indicators such as the average execution time, number of calls, and call success rate of each API in the past 30 days.

[0091]

[0092] Where n is the number of calls to the API in the past 30 days, and execution duration i is the execution duration of each call.

[0093] Call success rate = total number of calls / number of successful calls × 100%

[0094] Select the ARIMA time series analysis model for prediction based on the data characteristics. Use historical data to train the model and optimize the model parameters to improve the prediction accuracy. Use the trained model to predict the performance indicators of future API services and verify the accuracy of the model by comparing the actual data with the predicted data. The formula is as follows:

[0095] yt=c+φ1yt-1+φ2yt-2+…+φpyt-p+θ1∈t-1+θ2∈t-2+…+θq∈tq

[0096] yt-i is the actual value at the past moment, ∈ti is the prediction error at the past moment, φi and θi are model parameters, and c is a constant term.

[0097] Optionally, the method further comprises:

[0098] Based on the pre-trained ARIMA time series analysis model, the future performance indicators of the API service are predicted to obtain the prediction results.

[0099] In summary, the scheme proposed in the present invention obtains power grid data, wherein the power grid data comes from the database or log file of the source end, the source layer, and the shared layer; processes the power grid data according to the pre-stored anomaly detection algorithm to obtain the anomaly detection result; wherein the pre-stored anomaly detection algorithm at least includes a multi-index anomaly detection algorithm based on a statistical model or a multi-index anomaly detection algorithm based on machine learning; displays the anomaly detection result, and determines the corresponding alarm information according to the anomaly detection result. The multi-index anomaly detection can analyze the information of multiple indicators comprehensively, so as to detect abnormal situations more comprehensively and accurately. Compared with traditional anomaly detection, multi-index anomaly detection can better reflect the overall situation of the system, reduce the situation of missed reports and false reports, and multi-index anomaly detection can be flexibly configured according to different business scenarios and needs, and select appropriate indicators and algorithms to achieve the best detection effect.

[0100] Embodiment 2:

[0101] The power grid big data intelligent anomaly detection platform implements functions such as table data fluctuation monitoring, data consistency monitoring, source table structure monitoring, periodic task monitoring, SQL statement monitoring, and API service monitoring. It ensures that the data between the big data platform and the business system remain consistent, and when an anomaly occurs, the relevant operation and maintenance personnel can be notified in time. The overall architecture diagram is as follows: Figure 2 shown.

[0102] Table data fluctuation monitoring:

[0103] This module mainly monitors the data fluctuation of the shared layer table, and displays information including table name, system to which the table belongs, T-1 data volume, T-2 data volume, T-3 data volume, volatility and solutions.

[0104] in:

[0105] For each row of data, only two of the three fields, T-1 data volume, T-2 data volume, and T-3 data volume, will have values.

[0106] The presence of values ​​in the T-1 and T-2 fields indicates that the table being monitored is of high timeliness.

[0107] The presence of values ​​in the T-2 and T-3 fields indicates that the table being monitored is of low timeliness.

[0108] 1) High timeliness table, the data volume of fields T-1 and T-2 in the page has values

[0109] The T-1 data volume statistics are the T-1 data volume of the more important tables.

[0110] The T-2 data volume statistics are the T-2 data volume of the more important tables.

[0111] Volatility calculation formula: (T1-T2) / T2*100%

[0112] If the data volume of partition T1 is 0, the volatility is -100%. If the data volume of partition T2 is 0, the volatility is recorded as 100%. If the volatility exceeds plus or minus 10%, an alarm is triggered. If the volatility exceeds plus or minus 50% during weekends and holidays, an alarm is triggered.

[0113] 2) Low timeliness table, the data volume of fields T-2 and T-3 in the page has values

[0114] Since the low-timeliness table has low timeliness and only the latest T-2 data can be obtained, the T-2 and T-3 data of the low-timeliness table are obtained for comparison.

[0115] Volatility calculation formula: (T2–T3) / T3*100%

[0116] If the data volume of the T2 partition is 0, the volatility is -100%. If the data volume of the T3 partition is 0, the volatility is recorded as 100%. An alarm is triggered when the volatility exceeds plus or minus 10%, and an alarm is triggered when the volatility exceeds plus or minus 50% during weekends and holidays.

[0117] The above two tables can be filtered and distinguished by the filter condition [Data volume].

[0118] Data consistency monitoring:

[0119] Use Logstash to collect data from the database or log files of the source end, source layer, and shared layer, and extract the number of records in each table or data set. Store the collected data entries in Elasticsearch for subsequent query and analysis. The data contains information such as timestamp, data source (source end, source layer, shared layer), table name, and number of records. Use Kibana or Elasticsearch's API to query the number of data entries in each layer within a specific time period, and compare the number of data entries in different layers at the same time point to determine data consistency.

[0120] Absolute difference formula:

[0121] Difference = |number of records at the source end - number of records at the source layer|

[0122] Percent Difference formula:

[0123]

[0124] Cumulative Difference:

[0125]

[0126] Combine multiple indicators (absolute difference, percentage difference, cumulative difference, etc.) to form a composite indicator, and comprehensively evaluate data consistency through methods such as weighted or principal component analysis.

[0127] Source-side table structure monitoring:

[0128] Query the metadata table to obtain the table structure information of the latest day, which usually includes table name, column name, data type, whether it is allowed to be empty, etc. Read the previously stored historical table structure information from the platform table. Traverse the table structure information obtained from the source end, for each table, find the corresponding historical record in the platform table, and compare the detailed information of the two table structures (such as column name, data type, etc.).

[0129] If any differences are found, the table structure is marked as changed. If it does not exist in the history, it is considered a newly added table; if a table exists in the history but not in the latest, it is considered a deleted table. Comparison formula:

[0130] Difference comparison (initial structure, latest structure) = (difference column, difference index, ...)

[0131] The difference column = initial column - latest column ∪ latest column - initial column

[0132] Difference index = initial index - latest index ∪ latest index - initial index

[0133] Periodic task monitoring:

[0134] Design database query or stream processing logic to calculate the task running time (end time - start time) in real time. Check the task status in real time, and identify slow tasks and abnormal task status (such as failure, timeout, suspension, etc.) by comparing the running time with the preset threshold. And push the abnormal information to the corresponding person in charge in real time via email.

[0135] Collect historical task execution data, including task ID, name, type, business date, node type, running time, execution status, etc. Extract features that may affect task execution time and status from historical data, such as task type, node type, execution time (such as the difference between weekends and weekdays), historical running time, etc.

[0136] SQL statement monitoring:

[0137] In order to display the list of SQL statement execution information with data in the current valid time period, it is necessary to define the calculation formula for each attribute. These calculations are based on a log table (sql_performance_log) containing SQL execution information, which records the key information of each SQL execution, such as execution time, whether there is an error, execution date, etc.

[0138] Monitor the complexity of query statements: The complexity of SQL statements directly affects their execution efficiency. Complex queries, such as those containing multiple subqueries, a large number of JOIN operations, complex aggregation and sorting, will increase execution time.

[0139] Monitor the use of indexes: Indexes are an important means of improving query efficiency. If the fields involved in the query are not indexed, or the index design is unreasonable (such as the index column contains a large number of duplicate values, the index column order is improper, etc.), the query efficiency will be low. At the same time, please note that NULL cannot be used as the value of the index column, because any column containing NULL values ​​will not be included in the index.

[0140] Monitor the use of LIKE statements: When using the LIKE statement for fuzzy search, if the wildcard character (%) appears at the beginning of the search term, the system will perform a full table scan and cannot use the index, thus reducing the query speed.

[0141] Monitor ORDER BY and GROUP BY statements: If the columns in these statements are not indexed or contain calculated expressions, they will slow down the query.

[0142] Monitor the use of IN and EXISTS: Using EXISTS in a subquery is usually more efficient than using IN because EXISTS stops searching when it finds the first record that meets the conditions.

[0143] Health = (1-error rate)*(1-slow rate / 100)*(average execution time / benchmark time), where the benchmark time is a preset average execution time that is considered "healthy".

[0144] Average execution time = SUM(execution_time) / COUNT(*), where execution_time is the time (in milliseconds) for each execution and COUNT(*) is the number of executions.

[0145] 90th percentile: 90% of SQL execution time is below this value. Use the quantile function (PERCENTILE_CONT) provided by the database, or estimate it by sorting and counting.

[0146] Calculation method: PERCENTILE_CONT(0.9)WITHIN GROUP(ORDER BY execution_time)OVER(PARTITION BY sql_id)

[0147] 99th percentile value: 99% of SQL execution time is below this value. Use the quantile function (PERCENTILE_CONT) provided by the database, or estimate it by sorting and counting.

[0148] Calculation method: PERCENTILE_CONT(0.99)WITHIN GROUP(ORDER BY execution_time)OVER(PARTITION BY sql_id)

[0149] Error rate = SUM(CASE WHEN error_flag THEN 1ELSE 0END) / COUNT(*)*100, where error_flag is a flag indicating whether an execution error occurs.

[0150] Slow ratio: The ratio of SQL statements with execution time exceeding a certain threshold (such as twice the average execution time) to the total execution times.

[0151] Slow percentage = SUM(CASE WHEN execution_time>(average execution time*2)THEN 1ELSE0END) / COUNT(*)*100

[0152] Throughput = total number of calls / (total call time / 1000), where the total call time is in milliseconds and is converted to seconds by dividing by 1000.

[0153] The embodiment of the present application is to solve the problem that the big data platform lacks real-time monitoring means from data access to data use. The anomaly detection algorithm can solve many problems in user operation and maintenance, such as:

[0154] Table data fluctuation monitoring: monitor the data fluctuation of key tables in the shared layer, and issue an alarm when the data volume on the current day fluctuates more than the threshold compared to the data volume yesterday.

[0155] Data consistency monitoring: Based on ELK monitoring data, the number of data entries in the source end, source layer, and shared layer of the tables of the relevant systems is displayed to determine whether the data is consistent.

[0156] Source table structure monitoring: Displays information about the source database, source system, and source table, and determines whether the table structure has changed by comparing the table structure of the source table. The table structure of the source table is obtained from the metadata table of the source database system, stored in the platform table, and then compared with the table structure of the latest day to determine whether there is any change.

[0157] Periodic task monitoring: Displays the task ID, name, type, business date, node type, task time, running time, execution status, person in charge, etc. of each project space. Filter out the slow tasks and abnormal task status every day and feedback them to the relevant person in charge for timely disposal.

[0158] SQL statement monitoring: displays a list of SQL statement execution information with data within the current effective time period. The following attributes are displayed by default: health, SQL statement, average execution time, 90th percentile, 99th percentile, error rate, slow percentage, throughput rate, total number of calls, and total call time.

[0159] API service monitoring: Displays the average execution time, number of calls, call success rate and call time in the past 30 days from the dimensions of API group, name, path, request method, description, release time, service status, etc. You can intuitively see the basic information of API services; timely discover abnormal services in the past 30 days for quick disposal, and predict the API monitoring data in the next 5 days.

[0160] Anomaly detection can bring many benefits in operation and maintenance scenarios, mainly including the following aspects:

[0161] Fault prediction and prevention: Through comprehensive analysis of multiple indicators, multi-indicator anomaly detection can timely discover potential fault risks, and predict and prevent faults before they occur, thereby reducing the probability of faults and improving system reliability and stability.

[0162] Fault diagnosis and location: When a system failure occurs, multi-indicator anomaly detection can help operation and maintenance personnel quickly locate the cause and location of the failure, shorten troubleshooting time, and improve fault handling efficiency.

[0163] Performance optimization and capacity planning: Multi-indicator anomaly detection can help operation and maintenance personnel understand the actual operation and resource utilization of the system, thereby optimizing system performance and improving resource utilization efficiency. At the same time, multi-indicator anomaly detection can also predict future system capacity requirements and help operation and maintenance personnel perform capacity planning, thereby avoiding system failures and performance degradation caused by insufficient capacity.

[0164] Automated operation and maintenance and intelligent management: Multi-indicator anomaly detection can be combined with automated operation and maintenance and intelligent management to achieve automated monitoring and management of the system, reduce manual intervention, and improve operation and maintenance efficiency and reliability.

[0165] Specifically, comprehensiveness: Multi-indicator anomaly detection can analyze information from multiple indicators, thereby detecting anomalies more comprehensively and accurately. Compared with traditional anomaly detection, multi-indicator anomaly detection can better reflect the overall situation of the system and reduce missed reports and false positives.

[0166] Flexibility: Multi-indicator anomaly detection can be flexibly configured according to different business scenarios and requirements, and appropriate indicators and algorithms can be selected to achieve the best detection effect. In addition, multi-indicator anomaly detection can also be dynamically adjusted according to real-time data to ensure the real-time and accuracy of the detection results.

[0167] Scalability: Multi-indicator anomaly detection can process and analyze different types of indicators, including system performance indicators, business indicators, security indicators, etc., and has strong scalability and adaptability.

[0168] Adaptability: The design of multi-indicator anomaly detection fully considers the complexity of the environment and the diversity of data. It achieves high adaptability through deep integration of machine learning and related technologies. It can continuously learn and optimize from actual operation data, automatically identify and adapt to various abnormal patterns. The system uses a variety of machine learning methods such as unsupervised learning, semi-supervised learning or reinforcement learning, combined with data mining and feature engineering technology, to automatically extract key features and patterns in the data. Through continuous model training and iterative optimization, it can continuously update its anomaly detection model to more accurately reflect the changing trends of the current environment and data. It can adapt to different application scenarios and data characteristics and improve the accuracy and reliability of detection. This adaptability not only improves the intelligence level of the system, but also provides users with a more flexible and efficient monitoring solution.

[0169] Real-time: The system has intelligent anomaly monitoring capabilities and can scan any subtle changes in the data link around the clock. Once a link anomaly is detected, such as a surge in traffic, a sudden drop in response time, a surge in error rate, etc., the system will immediately trigger the early warning mechanism and accurately send the alarm information to the preset relevant responsible person via email. This instant communication mechanism ensures that problems are discovered and responded to in the first time, allowing the team to quickly intervene to troubleshoot and resolve the problem. In addition, the system also supports custom thresholds and alarm rules, allowing users to flexibly adjust the monitoring sensitivity according to actual needs, further improving the pertinence and efficiency of exception handling.

[0170] Intuitiveness: By seamlessly integrating the six core detection data on the same visualization platform, the deep connection and integration of data is achieved. The platform uses advanced charting technology, such as dynamic line charts, pie charts, heat maps and scatter plots, to present data in a multi-dimensional and multi-level manner. This intuitive display method not only makes complex data links clear at a glance, but also greatly enhances users' keen insight into data change trends, abnormal fluctuations and potential problems. Both senior decision makers and front-line operation and maintenance personnel can quickly grasp the overall information and provide solid data support for accurate decision-making and efficient optimization.

[0171] Embodiment 3:

[0172] The invention discloses a detection system for power grid data. Figure 3 FIG. 1 is a structural diagram of a power grid data detection system according to an embodiment of the present invention; Figure 3 As shown, the system 100 includes:

[0173] The first processing module 101 is configured to obtain power grid data, wherein the power grid data comes from a database or log file of a source end, a source layer, or a shared layer;

[0174] The second processing module 102 is configured to process the power grid data according to a pre-stored anomaly detection algorithm to obtain an anomaly detection result; wherein the pre-stored anomaly detection algorithm at least includes a multi-index anomaly detection algorithm based on a statistical model or a multi-index anomaly detection algorithm based on machine learning;

[0175] The third processing module 103 is configured to display the abnormality detection result and determine corresponding alarm information according to the abnormality detection result.

[0177] According to the system of the second aspect of the present invention, the second processing module is specifically configured to process the power grid data according to a pre-stored anomaly detection algorithm to obtain an anomaly detection result, including:

[0178] Based on statistical methods, anomaly detection of multiple indicators is performed on power grid data. By calculating the mean, variance, and covariance statistics of multiple indicators, the relationship between the indicators is modeled and analyzed to obtain anomaly detection results.

[0179] According to the system of the second aspect of the present invention, the second processing module is specifically configured to perform multi-indicator anomaly detection on the power grid data based on a multi-indicator anomaly detection model to obtain anomaly detection results; the multi-indicator anomaly detection model is obtained by training the machine school model with sample data to learn the relationship between indicators and abnormal patterns.

[0180] According to the system of the second aspect of the present invention, the third processing module is specifically configured to send an alarm message when the task running time exceeds a preset threshold;

[0181] If the task is not completed within the scheduled time, an alarm message is triggered;

[0182] If the task status changes from "executing" to "abnormal", an alarm will be issued immediately;

[0183] If the task fails multiple times in a row, an alarm message will be sent;

[0184] Compare the task running time within the preset time period. If a significant increase is found, a warning message will be displayed.

[0185] According to the system of the second aspect of the present invention, the third processing module is specifically configured to use the API service of Kibana or Elasticsearch to query the power grid data of each layer within a preset time period;

[0186] Compare the consistency of power grid data at different layers at the same time point to obtain consistent comparison results.

[0187] According to the system of the second aspect of the present invention, the third processing module is specifically configured to clean the power grid data, remove invalid data, and obtain cleaned data;

[0188] Group and aggregate the cleaned data according to the API group, name, and path dimensions to obtain grouped data;

[0189] Based on the grouped data, calculate the key indicators of average execution time, number of calls, and call success rate of each API service within the historical preset time period.

[0190] According to the system of the second aspect of the present invention, the third processing module is specifically configured to predict the future performance indicators of the API service based on the pre-trained ARIMA time series analysis model to obtain a prediction result.

[0191] Embodiment 4:

[0192] The present invention discloses an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in a method for detecting power grid data in any one of the embodiments 1 disclosed in the present invention are implemented.

[0193] Figure 4 is a structural diagram of an electronic device according to an embodiment of the present invention, such as Figure 4 As shown, the electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, near field communication (NFC) or other technologies. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a button, a trackball or a touch pad set on the housing of the electronic device, or an external keyboard, touch pad or mouse, etc.

[0194] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a structural diagram of the part related to the technical solution of the present disclosure, and does not constitute a limitation on the electronic device to which the technical solution of the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0195] Embodiment 5:

[0196] The present invention discloses a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the method for detecting power grid data in any one of the embodiments 1 of the present invention are implemented.

[0197] Please note that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above embodiments only express several implementation methods of the present application, and their descriptions are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present application, several variations and improvements can be made, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the patent in this application shall be based on the attached claims.

[0198] The embodiments of the subject matter and functional operations described in this specification may be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules in computer program instructions encoded on a tangible non-temporary program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0199] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits), and the apparatus can also be implemented as special purpose logic circuits.

[0200] Computers suitable for executing computer programs include, for example, general and / or special microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, the computer will also include one or more large-capacity storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer will be operably coupled to this large-capacity storage device to receive data from it or to transmit data to it, or both. However, the computer does not necessarily have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0201] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0202] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of the specific embodiments of specific inventions. Certain features described in multiple embodiments in this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claim protection, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of a sub-combination.

[0203] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0204] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0205] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0206] Although some specific embodiments of the present invention have been described in detail by way of example, it will be appreciated by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It will be appreciated by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. A method for detecting power grid data, characterized in that: The method comprises: Step S1, obtaining power grid data, wherein the power grid data comes from a database or log file of a source end, a source layer, or a shared layer; Step S2: Processing the power grid data according to a pre-stored anomaly detection algorithm to obtain an anomaly detection result; wherein the pre-stored anomaly detection algorithm at least includes a multi-index anomaly detection algorithm based on a statistical model or a multi-index anomaly detection algorithm based on machine learning; Step S3: display the abnormality detection result, and determine the corresponding alarm information according to the abnormality detection result.

2. A method for detecting power grid data according to claim 1, characterized in that: In step S2, the power grid data is processed according to a pre-stored anomaly detection algorithm to obtain an anomaly detection result, including: The power grid data is subjected to multi-index anomaly detection based on a statistical method, and the relationship between the indicators is modeled and analyzed by calculating the statistical quantities of the mean, variance, and covariance of the multiple indicators to obtain the anomaly detection result.

3. A method for detecting power grid data according to claim 1, characterized in that: In step S2, the power grid data is processed according to a pre-stored anomaly detection algorithm to obtain an anomaly detection result, including: Based on a multi-index anomaly detection model, multi-index anomaly detection is performed on the power grid data to obtain the anomaly detection result; the multi-index anomaly detection model is obtained by training a machine learning model with sample data to learn the relationship between indicators and anomaly patterns.

4. A method for detecting power grid data according to claim 1, characterized in that: In the step S3, the displaying of the abnormality detection result and determining the corresponding alarm information according to the abnormality detection result include: When the task running time exceeds the preset threshold, an alarm message is sent; If the task is not completed within the scheduled time, an alarm message is triggered; If the task status changes from "executing" to "abnormal", an alarm will be issued immediately; If the task fails multiple times in a row, an alarm message will be sent; Compare the task running time within the preset time period. If a significant increase is found, a warning message will be displayed.

5. A method for detecting power grid data according to claim 1, characterized in that: The method further comprises: Use Kibana or Elasticsearch API services to query grid data at each layer within a preset time period; Compare the consistency of power grid data at different layers at the same time point to obtain consistent comparison results.

6. A method for detecting power grid data according to claim 1, characterized in that: The method further comprises: Cleaning the power grid data, removing invalid data, and obtaining cleaned data; The cleaned data is grouped and aggregated according to the grouping, name, and path dimensions of the API to obtain grouped data; Based on the grouped data, the key indicators of average execution time, number of calls, and call success rate of each API service in a historical preset time period are calculated.

7. A method for detecting power grid data according to claim 6, characterized in that: The method further comprises: Based on the pre-trained ARIMA time series analysis model, the future performance indicators of the API service are predicted to obtain the prediction results.

8. A detection system for power grid data, characterized in that: The system comprises: The first processing module is configured to obtain power grid data, wherein the power grid data comes from a database or log file of a source end, a source layer, or a shared layer; The second processing module is configured to process the power grid data according to a pre-stored anomaly detection algorithm to obtain an anomaly detection result; wherein the pre-stored anomaly detection algorithm at least includes a multi-index anomaly detection algorithm based on a statistical model or a multi-index anomaly detection algorithm based on machine learning; The third processing module is configured to display the abnormality detection result and determine corresponding alarm information according to the abnormality detection result.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps in the method for detecting power grid data according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the method for detecting power grid data according to any one of claims 1 to 7 are implemented.

Images