Method and system for collecting massive heterogeneous logs of electric power automation system

Abstract

The invention relates to the technical field of electric power system automation, and provides an electric power automation system heterogeneous log collection method which comprises the following steps: S1, obtaining a data head: obtaining a field name and a data sample of to-be-transmitted log data from an electric power system through a network; s2, judging a difference item; s3, identifying a data format; s4, classifying data types; and S5, data storage. The invention further provides an electric power automation system heterogeneous log collection system. According to the method, the log data newly accessed to the power system can be automatically matched with the existing log data format, and the conversion rule is automatically generated, so that the labor cost is greatly reduced, and the time and period of the heterogeneous log acquisition process of the power automation system are greatly shortened; and the conversion rules are integrated and run, so that environmental dependence is avoided, bugs are effectively reduced, and the stability and reliability of the system are greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of power system automation, and in particular to a method and system for collecting massive heterogeneous logs of a power automation system. Background Art

[0002] The prior art includes methods and systems for integrating and analyzing massive power system log data using big data technology, such as the power system log integration and analysis method and system disclosed in Chinese invention patent application number 201910124737.7.

[0003] However, such methods are often based on the premise that log data is neat, while the logs of power systems actually have the problem of data integration of multiple systems. For example, the systems used by wind power plants and hydropower plants are different. Generally speaking, there are wind power, photovoltaic (solar energy), hydropower, thermal power, etc. In theory, when the systems are coordinated, their data can be easily integrated and analyzed, which is convenient for deployment and helps to detect abnormal situations in a timely manner. However, in practice, there are often newly built power plants or power plants newly added to the system, and the systems they use are completely different from the previous systems. As a result, the formats of log data are very different, which leads to great troubles in connecting log data.

[0004] At present, in the case of large differences in log data formats in the log collection process, manual analysis is generally performed, and then conversion rules are written according to the different log data formats. The conversion rules are written as scripts and the scripts are called to convert the data format each time the logs are collected.

[0005] There are at least the following problems in the existing technology: conversion rules are cumbersome to write, the labor cost is extremely high, it is difficult to automate, and the low efficiency often leads to an excessively prolonged integration process; the independent operation of scripts is dependent on the environment and often has unpredictable bugs, which seriously affect the stability and reliability of the system. Summary of the invention

[0006] Purpose of the invention: The purpose of the embodiments of the present invention is to provide a method and system for collecting heterogeneous logs in an electric power automation system, so that the log data newly connected to the electric power system can automatically match the existing log data format and automatically generate conversion rules, thereby greatly reducing labor costs and effectively reducing bugs, thereby greatly improving the stability and reliability of the system.

[0007] In order to solve the above technical problems, the present invention provides a method for collecting heterogeneous logs in an electric power automation system, comprising the following steps:

[0008] S1. Get data header: Get the field name and data sample of the log data to be transmitted from the power system through the network;

[0009] S2. Determine the difference item: Determine whether the field names and data formats of the log data to be transmitted and the log data stored in the database are consistent. If consistent, proceed to step S5. If inconsistent, proceed to step S3.

[0010] S3. Identify data formats: Identify the data formats of inconsistent fields in the data sample, determine the fields in the database whose data formats have matching possibilities, and obtain a list to be matched;

[0011] S4. Data type classification: Take data examples according to the list to be matched, calculate the similarity between the data sample and the data example, take the field name in the list to be matched with the highest similarity as the field storage name of the log data to be transmitted, and establish a conversion rule for converting the field name of the log data to be transmitted into the field storage name;

[0012] S5. Data storage: The acquired log data to be transmitted is stored. If there is a conversion rule, the conversion rule is applied first and then stored.

[0013] In this way, the heterogeneous log data newly connected to the power automation system can be automatically matched with the existing log data format, and the conversion rules can be automatically generated with extremely high calculation efficiency.

[0014] Furthermore, to ensure the convenience of subsequent verification, after step S4, the following steps are also included:

[0015] S4.10. Data marking: Fields that match the conversion rules will be marked with conversion marks when stored, and inconsistent fields that do not match the conversion rules will be marked with warning marks.

[0016] In step S4, the similarity between the data sample and the data example is calculated, which specifically includes the following steps:

[0017] S4.1. For each field, determine whether it is a numeric value based on the corresponding data in the data sample, and if so, convert it to a numeric type;

[0018] S4.2. For numerical data, calculate the difference between the data sample and the corresponding field data in the data example one by one, and then calculate the similarity based on the difference;

[0019] S4.3. For text type data, calculate the text length of the corresponding field data in the data sample and the data example respectively, and then calculate the similarity based on the text length.

[0020] In step S4.2, the similarity is calculated based on the difference, using the following formula:

[0021]

[0022] Where d is the similarity calculation result, xi is the calculated difference in the i-th data sample, and yi is the calculated difference in the i-th data example.

[0023] In step S4, the number of data examples taken is consistent with the number of data samples.

[0024] The number of data samples is 50 to 100 items.

[0025] The present invention also provides a heterogeneous log collection system for an electric power automation system, comprising:

[0026] A communication module, used to obtain log data to be transmitted;

[0027] An identification and classification module is used to implement the above-mentioned heterogeneous log collection method for the electric power automation system;

[0028] Data sample module, used to cache data samples;

[0029] Database, used to store data.

[0030] It also includes a cache module for caching the log data to be transmitted that has not been processed by the identification and classification module, and sending the log data to be transmitted that has been processed by the identification and classification module to the database for storage.

[0031] The cache module is also used to apply conversion rules.

[0032] The data examples stored in the data example module have a number of 100 to 200 items and include all fields.

[0033] Beneficial effects: Compared with the prior art, the present invention can enable the log data newly connected to the power system to automatically match the existing log data format and automatically generate conversion rules, thereby greatly reducing labor costs and greatly shortening the time and cycle of the heterogeneous log collection process of the power automation system; the conversion rules are integrated and run to avoid environmental dependence and effectively reduce bugs, thereby greatly improving the stability and reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] One or more embodiments are exemplarily described by the figures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments.

[0035] Figure 1 It is a flowchart of at least one implementation method of a method for collecting heterogeneous logs of an electric power automation system according to the present invention;

[0036] Figure 2 It is a module connection diagram of an implementation scheme of a heterogeneous log collection system for an electric power automation system of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the embodiments of the present invention, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is for the convenience of description and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined and referenced with each other without contradiction.

[0038] The first embodiment of the present invention relates to a method for collecting heterogeneous logs in a power automation system. The process is as follows: Figure 1 As shown, the specific steps include:

[0039] S1. Get data header: Get the field name and data sample of the log data to be transmitted from the power system through the network;

[0040] S2. Determine the difference item: Determine whether the field names and data formats of the log data to be transmitted and the log data stored in the database are consistent. If consistent, proceed to step S5. If inconsistent, proceed to step S3.

[0041] S3. Identify data formats: Identify the data formats of inconsistent fields in the data sample, determine the fields in the database whose data formats have matching possibilities, and obtain a list to be matched;

[0042] S4. Data type classification: Take data examples according to the list to be matched, calculate the similarity between the data sample and the data example, take the field name in the list to be matched with the highest similarity as the field storage name of the log data to be transmitted, and establish a conversion rule for converting the field name of the log data to be transmitted into the field storage name;

[0043] S5. Data storage: The acquired log data to be transmitted is stored. If there is a conversion rule, the conversion rule is applied first and then stored.

[0044] The step division of the above methods is only for the purpose of clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this patent.

[0045] In general, the logs of power systems may contain data in the form of short texts of less than 25 words, long texts of less than 300 words, extra-long texts of unlimited length, floating-point data, integers, positive integers, etc. The most likely cause conflicts is that, on the one hand, integers or floating-point data are often stored in text format, and data that is actually short text (such as a person's name) is often stored in extra-long texts of unlimited length. On the other hand, for the same data, the names used by different power plants may also be very different. For example, the term "electricity consumption" is stored as power consumption in some power systems, while it is stored as kilowatt-hours in other power systems. However, no matter what format is used for storage and transmission, the meaning of the data itself will not differ too much. Therefore, the above process, based on the method of calculating data similarity, can effectively achieve the unification of the final data format, thereby generating conversion rules.

[0046] The second implementation mode of the present invention is substantially the same as the first implementation mode, mainly in order to ensure further optimization for facilitating post-verification. After step S4, the following steps are further included:

[0047] S4.10. Data marking: Fields that match the conversion rules will be marked with conversion marks when stored, and inconsistent fields that do not match the conversion rules will be marked with warning marks.

[0048] In this way, the possible wrong conversion marks and warning marks that require manual intervention are distinguished, so that the staff can find them at a glance and deal with them in time when checking the log data. Generally speaking, conversion marks are marked with yellow exclamation marks and warning marks are marked with red question marks. They can be marked at the front of the line in the data display interface, or the marks can be set as a separate field.

[0049] The third embodiment of the present invention is substantially the same as the first embodiment, mainly in that the preferred scheme of the data type classification method is used. In step S4, the similarity between the data sample and the data example is calculated, which specifically includes the following steps:

[0050] S4.1. For each field, determine whether it is a numeric value based on the corresponding data in the data sample, and if so, convert it to a numeric type;

[0051] S4.2. For numerical data, calculate the difference between the data sample and the corresponding field data in the data example one by one, and then calculate the similarity based on the difference;

[0052] S4.3. For text type data, calculate the text length of the corresponding field data in the data sample and the data example respectively, and then calculate the similarity based on the text length.

[0053] Since the performance of equipment in different power systems varies greatly, the differences in the values ​​in the log data are also extremely large. Directly calculating the similarity using the values ​​in the log data is likely to result in erroneous results. However, based on the characteristics of the power system, the floating range of values ​​between similar equipment and similar data will not differ too much. Therefore, using the difference between the data to calculate the similarity can effectively ensure accuracy.

[0054] Furthermore, in step S4.2, the similarity is calculated based on the difference, using the following formula:

[0055]

[0056] Where d is the similarity calculation result, xi is the calculated difference in the i-th data sample, and yi is the calculated difference in the i-th data example.

[0057] The use of this formula can effectively shield the differences caused by numerical sorting, and at the same time achieve highly accurate similarity judgment corresponding to the characteristics of power system data with extremely small calculation amount.

[0058] Furthermore, in step S4, the number of data examples taken is consistent with the number of data samples.

[0059] Furthermore, the number of the data samples is 50 to 100 items.

[0060] A fourth embodiment of the present invention relates to a heterogeneous log collection system for a power automation system. Figure 2 As shown, including:

[0061] A communication module, used to obtain log data to be transmitted;

[0062] An identification and classification module, used to implement the first to third embodiments of the present invention;

[0063] Data sample module, used to cache data samples;

[0064] Database, used to store data.

[0065] It is not difficult to find that this embodiment is a system embodiment corresponding to the above embodiment, and this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the first embodiment.

[0066] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of the present invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other units in this embodiment.

[0067] Furthermore, it also includes a cache module for caching the log data to be transmitted that has not been processed by the identification and classification module, and sending the log data to be transmitted that has been processed by the identification and classification module to the database for storage.

[0068] Furthermore, the cache module is also used to apply conversion rules.

[0069] Furthermore, the data sample module stores 100 to 200 data samples, including all fields. Since the data samples are used to calculate similarity, including all fields can ensure the effectiveness of log collection and avoid omission of matching types.

[0070] Generally, data samples and data examples can be understood as row and column data represented in a tabular form, where rows are items of the above data and columns are fields of the above data. In actual transmission, tabular form is generally not used, but they can all be converted into tabular form. Therefore, the difference in conceptual expression lies only in the difference in data representation method, and there is no difference in the nature of the data.

[0071] Those skilled in the art will appreciate that the above-mentioned embodiments are specific embodiments for implementing the present invention, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present invention.

Claims

1. A method for collecting massive heterogeneous logs in an electric power automation system, characterized in that: The following steps are involved: S1. Get data header: Get the field name and data sample of the log data to be transmitted from the power system through the network; S2. Determine the difference: Determine whether the field names and data formats of the log data to be transmitted and the log data stored in the database are consistent. If they are consistent, the process proceeds to step S5. If they are inconsistent, the process proceeds to step S3. S3. Identify data formats: Identify the data formats of inconsistent fields in the data sample, determine the fields in the database whose data formats have matching possibilities, and obtain a list to be matched; S4. Data type classification: Take data examples according to the list to be matched, calculate the similarity between the data sample and the data example, take the field name in the list to be matched with the highest similarity as the field storage name of the log data to be transmitted, and establish a conversion rule for converting the field name of the log data to be transmitted into the field storage name; S5. Data storage: The acquired log data to be transmitted is stored. If there is a conversion rule, the conversion rule is applied first and then stored.

2. The method for collecting massive heterogeneous logs of a power automation system according to claim 1, characterized in that: After step S4, the following steps are also included: S4.

10. Data marking: Fields that match the conversion rules will be marked with conversion marks when stored, and inconsistent fields that do not match the conversion rules will be marked with warning marks.

3. The method for collecting massive heterogeneous logs of a power automation system according to claim 1, characterized in that: In step S4, the similarity between the data sample and the data example is calculated, which specifically includes the following steps: S4.

1. For each field, determine whether it is a numeric value based on the corresponding data in the data sample, and if so, convert it to a numeric type; S4.

2. For numerical data, calculate the difference between the data sample and the corresponding field data in the data example one by one, and then calculate the similarity based on the difference; S4.

3. For text type data, calculate the text length of the corresponding field data in the data sample and the data example respectively, and then calculate the similarity based on the text length.

4. The method for collecting massive heterogeneous logs of a power automation system according to claim 3, characterized in that: In step S4.2, the similarity is calculated based on the difference, using the following formula: In the formula, d is the similarity calculation result, x i is the calculated difference in the i-th data sample, y i is the calculated difference in the ith data example.

5. The method for collecting massive heterogeneous logs of a power automation system according to claim 1, characterized in that: In step S4, the number of data examples taken is consistent with the number of data samples.

6. The method for collecting massive heterogeneous logs of a power automation system according to claim 1, characterized in that: The number of data samples is 50 to 100 items.

7. A massive heterogeneous log collection system for an electric power automation system, characterized in that: include: A communication module, used to obtain log data to be transmitted; An identification and classification module, used to implement the method for collecting heterogeneous logs of a power automation system as claimed in any one of claims 1 to 6; Data sample module, used to cache data samples; Database, used to store data.

8. The massive heterogeneous log collection system for the electric power automation system according to claim 7, characterized in that: It also includes a cache module for caching the log data to be transmitted that has not been processed by the identification and classification module, and sending the log data to be transmitted that has been processed by the identification and classification module to the database for storage.

9. The massive heterogeneous log collection system for the electric power automation system according to claim 8, characterized in that: The cache module is also used to apply conversion rules.

10. The massive heterogeneous log collection system for electric power automation system according to claim 7, characterized in that: The data examples stored in the data example module have a number of 100 to 200 items and include all fields.

Images