Substation-oriented multi-source heterogeneous data checking method and system

Abstract

The invention discloses a transformer substation-oriented multi-source heterogeneous data checking method and system. The method comprises the following steps of obtaining various types of data sources, executing a preliminary data cleaning process and outputting standardized data; aiming at database systems of different manufacturers, designing a customized migration tool for each database; extracting specific attribute information contained by the special domain analysis module by constructing the special domain analysis module; sequentially verifying the source database and the target database by adopting a progressive consistency verification strategy, and recording data difference information detected in each stage; a change rationality verification module is configured to analyze an SCD file of the transformer substation, and change rationality judgment is performed on difference items from a source database to a target database; performing unified modeling and graphical display on the obtained difference information through a difference data visualization module; according to the method, the data consistency and correctness in the upgrading and transformation process of the transformer substation monitoring system database are comprehensively verified, so that the stability and the operation reliability of the system are improved.

Description

Technical Field

[0001] This invention relates to the field of power system automation and information technology, specifically to a method and system for verifying multi-source heterogeneous data in substations. Background Technology

[0002] During the upgrade and transformation of substation monitoring systems, numerous updates to backend database configuration items are involved, including adding and deleting devices and modifying parameters. Improper operation may introduce configuration conflicts or potential risks. During the download of remote control units (RTUs), incorrect setting of important parameters or incorrect downloading of RTU configuration parameters may cause conflicts between RTU units, interruption of dispatching channels, and abrupt changes in RTU data transmission. Therefore, it is necessary to check and compare the RTU parameters before and after modification. In the migration and adaptation of domestic databases, there is currently a lack of a complete and effective mechanism to verify data consistency before and after the migration, leading to potential data loss risks and system compatibility issues.

[0003] The current data verification of substations faces several key challenges: First, traditional manual data verification methods suffer from efficiency bottlenecks and quality defects, requiring manual item-by-item verification of all station data, which is time-consuming and has a high rate of missed detections. Second, existing automated tools still have limitations in terms of functional coverage and intelligence, lacking the ability to parse customized data types specific to power systems. Third, different devices at the same project site may use database systems from different vendors, and cross-vendor data migration carries risks of data inconsistency and system compatibility. Fourth, database consistency verification before and after renovation and expansion projects often outputs numerous detailed differences, making it difficult to detect potential conflicts in a timely manner. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to provide a method and system for verifying multi-source heterogeneous data in substations, which solves the problems of difficulty in integrating multi-source heterogeneous data, incomplete consistency verification, and low verification efficiency in the existing technology.

[0005] Technical solution: The present invention provides a method for verifying multi-source heterogeneous data in substations, comprising the following steps: (1) Obtain engineering backup files, online databases and database systems from different manufacturers of the substation monitoring system; collect multi-source heterogeneous data through data source identification and data extraction units, and perform preliminary data cleaning process; finally output standardized data through a unified interface. (2) For database systems from different vendors, design customized migration tools for each database to seamlessly convert from the database formats of various vendors to a unified standardized format; (3) By constructing a special domain parsing module, special parsing rules are applied to unstructured complex type fields detected in standardized output data to extract the specific attribute information contained therein; (4) A progressive consistency verification strategy is adopted, and the source database and the target database are verified in three stages: metadata, key domain and full data verification. The data difference information detected in each stage is recorded. (5) By configuring the change rationality verification module, the SCD file of the substation is parsed, its structural diagram model is constructed based on the node information at each level, and the change rationality is judged for the difference items from the source database to the target database by combining the diagram model differences between the source database and the target database corresponding to different versions of SCD files. (6) Through the difference data visualization module, the obtained difference information is uniformly modeled and graphically displayed, and standardized difference reports are generated. At the same time, the system records complete operation logs, introduces an access control mechanism, and performs access control and audit tracking on the report content.

[0006] Furthermore, in step (1), the collected data undergoes field format standardization conversion, synonym mapping, and numerical precision calibration to unify the format structure and measurement precision of the multi-source heterogeneous data.

[0007] Furthermore, in step (3), the special domain parsing module processes as follows: For the file type field and the virtual domain field used to represent a set of binary state information of an object, a preset special parsing rule is applied to extract the specific attribute information of the single object or list of objects contained therein, forming comparable basic data elements.

[0008] Furthermore, the pre-defined dedicated parsing rules are as follows: Based on the custom structure corresponding to the data type, the read binary stream data is written to the memory space of its object, thereby completing the parsing operation of specific attributes; for virtual field fields stored by bits, 32-bit integer variables are used to store 32 such virtual field fields; during processing, the integer field to which the virtual field belongs must be read first, and then the corresponding value is extracted according to its specific position in the integer field.

[0009] Furthermore, in step (4), the progressive consistency strategy is as follows: Metadata verification: compare the database metadata attributes such as table structure, field definition and primary key information in different data sources; Key domain verification: quickly compare the key parameter fields in the database based on the globally unique identifier field; Full data verification: perform a comprehensive verification of the data records of all tables in the database, one by one and field by field, and use a parallel computing mechanism to improve the verification efficiency; Among them, the key domain in the progressive consistency verification strategy is the key field for the operation of the substation monitoring system; the key fields include: dispatch number, important address, and associated index number, which are information items that have a key impact on the operation of the substation; the key domain is determined by the on-site engineering personnel based on the actual commissioning experience and is used as the key comparison field in the data consistency verification process; Furthermore, in step (5), when the configuration change rationality verification module parses the SCD file, it extracts each intelligent electronic device (IED), logical device (LD), and logical node (LN), and constructs connection edges according to parent-child or logical relationships to form a structural topology diagram reflecting the substation secondary system; by comparing the differences between the structural topology diagrams constructed based on two different versions of SCD files, it determines the corresponding database differences.

[0010] Furthermore, in step (6), the difference report generated by the difference data visualization module includes difference summary information and a detailed difference table; each difference item is grouped and displayed according to category and priority.

[0011] Furthermore, every data access and verification operation is fully logged; at the same time, based on the preset permission management policy, the access scope of the difference report is restricted to ensure the security and traceability of the verification results.

[0012] The present invention discloses a multi-source heterogeneous data verification system for substations, characterized in that it includes: Preprocessing module: Used to acquire various data sources such as engineering backup files, online databases, and database systems from different manufacturers of the substation monitoring system; collects multi-source heterogeneous data through data source identification and data extraction units, and performs a preliminary data cleaning process; finally, standardized data is output through a unified interface. Conversion module: Designed for different vendors' database systems, it provides customized migration tools for each database, enabling seamless conversion from various vendors' database formats to a unified standardized format; Parsing module: Used to apply special parsing rules to unstructured complex type fields detected in standardized output data by constructing special domain parsing modules, and extract the specific attribute information contained therein; Consistency verification module: Used to perform a progressive consistency verification strategy to verify the source database and target database in three stages: metadata verification, key domain verification, and full data verification, and record the data difference information detected at each stage; Reasonableness Verification Module: This module is used to parse the SCD file of the substation by configuring the reasonableness verification module, construct its structural diagram model based on the node information at each level, and combine the differences in the diagram model between the source database and the target database corresponding to different versions of SCD files to determine the reasonableness of the changes from the source database to the target database. Visualization module: This module is used to uniformly model and graphically display the acquired difference information through the difference data visualization module, and supports the generation of standardized difference reports. At the same time, the system records complete operation logs, introduces an access control mechanism, and performs access control and audit tracking on the report content.

[0013] An electronic device according to the present invention includes a memory and a processor. The memory stores a computer program, and the processor executes the program to implement the steps of the method.

[0014] The present invention provides a computer-readable storage medium, characterized in that it stores a computer program, which, when executed by a processor, implements the steps of the method.

[0015] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: The present invention achieves efficient verification of various data sources in the substation monitoring system by constructing a unified data access mechanism, customized migration tools, special domain resolution services, progressive consistency verification strategies, and change rationality verification mechanisms. Attached Figure Description

[0016] Figure 1 This is a flowchart of the present invention; Figure 2 This is a schematic diagram of the progressive consistency verification module of the present invention; Figure 3 This is a logical diagram of the configuration change rationality verification module of the present invention. Detailed Implementation

[0017] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0018] like Figure 1 As shown, this embodiment of the invention provides a method for verifying multi-source heterogeneous data in substations, including the following steps: Step 1, see Figure 1This paper constructs a multimodal data access mechanism adaptable to various heterogeneous data sources. Considering the actual needs of substation monitoring systems, this method supports multimodal data access, covering engineering backup file parsing, real-time online database extraction, and standardized adaptation across vendor databases. To achieve seamless access to different data formats, it integrates data source identification and data extraction units, and incorporates a preliminary data cleaning and standardization mechanism. This mechanism can automatically identify and verify the system of the connected device and its corresponding database type and version, and perform operations such as format compliance verification, synonym semantic alignment, and floating-point precision conversion on the raw data.

[0019] Step 2: Addressing the differences between databases from various vendors commonly used in substations, a customized migration tool is designed. This tool achieves the migration from different source database formats to a standardized, self-developed file library format through a four-step process: metadata parsing, semantic mapping, data conversion, and integrity verification. Finally, the data acquisition layer outputs standardized data through a unified interface, ensuring data quality meets subsequent processing requirements.

[0020] Preferably, in the early stages of design and development, the correctness of the migration tool's execution results is verified using the following method: migrating standardized self-developed document library data to a target domestic database, then migrating it back from the target domestic database to the standardized self-developed document library, and using this method to verify the data consistency of the document library before and after the migration. Subsequently, these verified migration tools are integrated into the multimodal data access mechanism of this invention, enabling this method to verify data across cross-vendor database systems.

[0021] Step 3: The substation database contains various complex data types, such as file type fields and virtual field fields representing a set of binary state information for objects. For such unstructured or semi-structured fields, relying solely on overall consistency verification methods for conventional fields such as integers and strings is insufficient to reveal the specific differences in their internal structure. Therefore, this invention designs a special field parsing module and a dedicated verification module. By configuring dedicated parsers for different complex field types, the originally unstructured data is parsed into structured object instances or lists of object instances, and then their attribute information is extracted to achieve fine-grained consistency verification at the attribute level for special field objects in substations.

[0022] Step 4: This method employs a three-level progressive data verification strategy. (See attached document.) Figure 3A three-tiered fusion mechanism—metadata verification, key domain consistency validation, and dynamic full-data validation—enables on-demand cross-node data verification. Metadata information is extracted from both the source and target node databases, obtaining table attributes such as table structure, field definitions, field lengths, primary key information, default values, and null constraints. By comparing the metadata, the consistency of table structure and field attributes between the source and target databases is verified, establishing a foundation for data comparability. Secondly, key domain comparison is performed, selectively comparing important parameters such as scheduling numbers, important addresses, and associated index numbers in the database. Finally, full-data validation is conducted, checking all records and fields one by one to find differences in data values. This layered validation process allows for the rapid identification of key differences affecting important functions based on the actual needs of on-site engineers, followed by a fine-grained comprehensive comparison.

[0023] Preferably, the list of key domains is compiled based on the extensive practical experience of field engineers and covers the core data required for system operation. During the key domain comparison phase, only the aforementioned key fields are verified.

[0024] Preferably, full data verification compares all contents of tables with the same name in both databases. The specific method for determining regular fields is as follows: A data record is uniquely identified by its primary key value, which generally corresponds to the ID number of that data record, but special cases involving composite primary keys need to be handled. Check whether the primary key values ​​of all data records in the source and target tables are the same. If a primary key value exists only in the source table, it is considered a missing record; if a primary key value exists only in the target table, it is considered a newly added record. For records with the same primary key value in both the source and target tables, compare the values ​​of other fields one by one. Records with the same primary key value but different values ​​in other fields are considered modified records.

[0025] Preferably, in the full data verification stage, to improve efficiency and reduce resource consumption, this invention adopts a parallel computing framework, dividing the total task into multiple sub-tasks, and calculating the optimal task sequence for each processing unit based on a dynamic allocation strategy. Specifically, firstly, based on the computing power of the computer processor, a corresponding number of data comparison calculation units are created; secondly, considering factors such as data volume and data complexity, the verification calculation workload of different tables or fields is evaluated, and a verification sub-task with an equivalent computational workload is allocated to each processing unit on an average basis.

[0026] Preferably, during the full data verification phase, a mode selection function is added, supporting both background mode and remote control mode. In background mode, only tables related to the background four-remote functions are compared; in remote control mode, only tables related to the remote control functions are compared, specifically including tables related to important parameters of the remote control unit.

[0027] Step 5: Based on the actual application scenario requirements of this method, select the configuration change rationality verification function to verify the rationality of the difference results obtained in Step 4. Specifically, parse the substation SCD file, extract information such as IED, LD, and LN, and construct a graphical model reflecting the entire substation secondary system structure. By comparing SCD file models from different time points or different versions, identify configuration differences, and use this as a basis to judge the rationality of database difference items to ensure the rationality of configuration changes.

[0028] Preferably, the process begins with preliminary identification of discrepancy nodes based on plain text matching. Then, a matching algorithm is executed on the discrepancy nodes in the old and new configuration graphs to calculate subgraph structural similarity. For nodes with similarity exceeding a threshold, a set of matching pairs is generated, and their path strings in the old and new graphs are extracted to form path mapping pairs, which are the migration nodes. Other nodes that do not match are treated as either added or deleted nodes. Finally, a difference structure graph is constructed as the semantic basis for subsequent database difference interpretation. For database field information not covered by the SCD file, manual verification is recommended.

[0029] Step 6: Classify and grade the discrepancy results obtained from the verification, and construct a visualization module to uniformly model and display the discrepancy information from multiple dimensions. To facilitate engineers' quick understanding and review of discrepancies, this method introduces a semantic mapping engine to convert the original field values ​​stored in the verification results into natural language descriptions with actual business meaning, thereby improving data readability. Furthermore, this method integrates a standardized discrepancy report generation module, constructing standardized report templates to record detailed discrepancy data, historical verification records, and other relevant information, helping engineers or operations personnel quickly locate the root cause of problems. The discrepancy report supports export in multiple formats for easy subsequent review and archiving. Simultaneously, the system sets up a multi-level access control mechanism to ensure that sensitive information in the discrepancy report is only accessible to authorized users. In addition, the system fully records the time, operator, and verification results of each verification operation, achieving full-process audit tracking.

[0030] Preferably, the semantic mapping engine performs semantic transformation processing on specific common type fields. For fields stored in the database in the form of index numbers, such as constant name type fields and alarm mode fields, the engine converts them into corresponding functional descriptions; for data stored in the form of encoded values, such as IP type and time type fields, it also performs corresponding formatting transformation output; for special field data such as complex file types, based on the output of the aforementioned special field parsing module, an independent display framework is designed for each type of special field, and the actual content differences of these special fields are displayed in formatted charts.

Claims

1. A substation-oriented multi-source heterogeneous data checking method, characterized in that, The method comprises the following steps: (1) obtaining engineering backup files of different device nodes of a substation monitoring system, online databases, and various types of data sources of database systems of different manufacturers; Through the data source identification and data extraction unit, multi-source heterogeneous data is collected, and a preliminary data cleaning process is performed; finally, standardized data is output through a unified interface; (2) For database systems of different manufacturers, a customized migration tool is designed for each database to seamlessly convert from the database format of each manufacturer to a unified standardized format; (3) By constructing a special domain analysis module, special analysis rules are applied to the unstructured complex type fields detected in the standardized output data to extract specific attribute information contained therein; (4) A progressive consistency verification strategy is adopted to verify the metadata, key fields, and full data of the source database and the target database in three stages, and record the data difference information detected in each stage; (5) By configuring a change rationality verification module, the SCD file of the substation is parsed, a structure diagram model is constructed based on the node information at each level, and the difference between the graph models corresponding to the different versions of the SCD files of the source database and the target database is combined to determine the change rationality of the difference items from the source database to the target database; (6) Through the difference data visualization module, the difference information obtained is uniformly modeled and graphically displayed, and a standardized difference report is generated; at the same time, the system records complete operation logs, introduces a permission control mechanism, and performs access control and audit tracking on the report content.

2. The substation-oriented multi-source heterogeneous data checking method according to claim 1, characterized in that, In step (1), the collected data is subjected to field format standardization conversion, synonymous item mapping, and numerical accuracy calibration processing to unify the multi-source heterogeneous data in terms of format structure and measurement accuracy.

3. The substation-oriented multi-source heterogeneous data checking method according to claim 1, characterized in that, In step (3), the special domain analysis module processing procedure is as follows: for file type fields and virtual domain fields representing a set of binary state information of an object, a preset special analysis rule is applied to extract the specific attribute information of a single object or object list contained therein to form comparable basic data elements.

4. The substation-oriented multi-source heterogeneous data checking method according to claim 1, characterized in that, The preset special analysis rule is as follows: according to the custom structure corresponding to the data type, the binary stream data obtained is written into the memory space of the object to complete the analysis operation of the specific attributes; for virtual domain fields stored by bits, 32 such virtual domain fields are stored using a 32-bit integer variable; when processing, the integer field to which the virtual domain field belongs is first read, and then the corresponding value is extracted according to its specific position in the integer field.

5. The substation-oriented multi-source heterogeneous data checking method according to claim 1, characterized in that, In step (4), the progressive consistency strategy is as follows: metadata verification: comparing database metadata attributes such as table structure, field definition, and primary key information in different data sources; key field verification: based on the globally unique identification field, quickly comparing the key parameter fields in the database; full data verification: comprehensively verifying all table data records in the database, and using parallel computing mechanism to improve verification efficiency; wherein the key field in the progressive consistency verification strategy is the key field of the substation monitoring system operation; the key field includes: dispatch number, important address, and associated index number, which are information items that have a key impact on substation operation; the key field is determined by field engineers based on actual operation experience and is the key comparison field in the data consistency verification process.

6. The substation-oriented multi-source heterogeneous data checking method according to claim 1, characterized in that, In step (5), the configuration change rationality verification module extracts IED, LD, and LN when parsing the SCD file, and constructs connection edges according to the parent-child or logical relationship to form a structure topology graph reflecting the secondary system of the substation. By comparing the differences between the structure topology graphs constructed based on two different versions of SCD files, the corresponding database difference items are determined.

7. The substation-oriented multi-source heterogeneous data checking method according to claim 1, characterized in that, In step (6), the difference data visualization module generates a difference report containing difference summary information and detailed difference tables; each difference item is grouped and displayed according to category and priority.

8. The substation-oriented multi-source heterogeneous data checking method according to claim 7, characterized in that, Complete log records are kept for each data access and verification operation; at the same time, based on the pre-set permission management strategy, the access range of the difference report is limited to ensure the security and traceability of the checking results.

9. A substation-oriented multi-source heterogeneous data checking system, characterized in that, It includes: Preprocessing module: used to obtain engineering backup files, online databases, and various types of data sources of database systems of different manufacturers of substation monitoring systems; Through data source identification and data extraction unit, collect multi-source heterogeneous data, and perform preliminary data cleaning process at the same time; finally, output standardized data through unified interface; Conversion module: used for each database to design customized migration tools for different manufacturers' database systems, and seamlessly convert from various manufacturers' database formats to standardized formats; Parsing module: used to apply special parsing rules to the unstructured complex type fields detected in the standardized output data through the construction of a special field parsing module, and extract the specific attribute information contained therein; Consistency verification module: used to use the progressive consistency verification strategy to verify the metadata, key field, and full data of the source database and target database in three stages, and record the data difference information detected in each stage; Rationality verification module: used to parse the SCD file of the substation by configuring the change rationality verification module, construct its structure graph model based on the node information at each level, and combine the differences between the graph models of the source database and the target database corresponding to different versions of SCD files to determine the change rationality of the difference items from the source database to the target database; The visualization module is used for unified modeling and graphical display of the acquired difference information through the difference data visualization module, and supports generation of a standardized difference report; meanwhile, the system records complete operation logs, introduces a permission control mechanism, and performs access control and audit tracking on the report content.

10. An electronic device, comprising: The computer program product comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method in claims 1-8 when executing the program.

Images