Metadata integration-based hydraulic power plant unit data defect analysis method

Abstract

The invention discloses a hydraulic power plant unit data defect analysis method based on metadata integration, and relates to the field of big data, the method comprises the following steps: establishing a mapping relation for a plurality of business system data recording unit operation data, and uniformly registering the mapping relation to a metadatabase to carry out metadata integration; performing various data defect analysis on the integrated metadatabase to determine data quality; wherein the data defect analysis comprises integrity analysis, legality analysis, consistency analysis, timeliness analysis, correlation analysis, uniqueness analysis, logic consistency analysis or trend anomaly analysis. Through the scheme of the invention, a physical model can be combined with data driving, the existing domain knowledge is fully utilized, the demand for a large amount of labeled data is reduced, the model interpretability is improved, and the understanding and discovery of data defect causes and engineering practical application are facilitated.

Description

Technical Field

[0001] The present application relates to the field of big data, and in particular to a method for analyzing data defects of hydropower plant units based on metadata integration. Background Art

[0002] As one of the most mature and environmentally friendly renewable energy sources, hydropower plays an indispensable role in the global energy supply. Hydropower plants convert water energy into electrical energy through hydroelectric generators, and their efficiency and reliability directly affect the stability and economy of the power system. However, various data defects may occur during the operation of hydropower plant units. These defects may be caused by sensor failures, data transmission errors, or inaccurate processing algorithms, and in severe cases may even lead to incorrect decision support and equipment failure. Therefore, accurate analysis methods are particularly important for monitoring and diagnosing these data defects, which can not only ensure the efficient operation of the equipment, but also significantly reduce maintenance costs and extend the life of the equipment.

[0003] At present, the analysis of data defects in hydropower plant units mainly relies on data quality monitoring (DQM) technology and machine learning algorithms. DQM technology identifies outliers or missing values ​​in the data by setting thresholds and performing statistical analysis. In addition, machine learning methods such as decision trees, random forests, and support vector machines have been applied to complex data defect diagnosis to identify and predict potential data anomalies by training models. These technologies can effectively process a certain amount of data and identify problems to a certain extent, but they have some problems that cannot be ignored. For example, traditional DQM technology is inefficient when processing large-scale and high-dimensional data, and has limited ability to recognize complex and subtle data patterns; in recent years, artificial intelligence technologies such as deep learning have been widely used in the field of data analysis. Compared with traditional methods, deep learning models have the advantage of automatically learning features from data and perform well in processing nonlinear, high-dimensional and complex data. However, deep learning models also have some defects, such as the need for a large amount of labeled data for supervised learning, more empirical factors in model structure and hyperparameter selection, and poor "black box" interpretability. Especially in the field of hydropower, the acquisition of labeled data is usually costly, which limits the application of deep learning methods.

[0004] Therefore, there is an urgent need for a technical solution that can combine physical models with data-driven approaches, make full use of existing domain knowledge, reduce the demand for large amounts of labeled data, and improve model interpretability to facilitate understanding of the causes of discovered data defects and their application in engineering practice. Summary of the invention

[0005] In order to solve the deficiencies of the prior art, the embodiment of the present application provides a method for analyzing the data defects of hydropower plant units based on metadata integration. The present application solves the technical problems of the prior art such as large demand for labeled data and poor interpretability.

[0006] An embodiment of the present application provides a method for analyzing data defects of hydropower plant units based on metadata integration, including: establishing a mapping relationship for multiple business system data that record unit operation data, and uniformly registering them to a metadata database for metadata integration; performing multiple data defect analyses on the integrated metadata database to determine data quality; wherein the data defect analysis includes integrity analysis, legality analysis, consistency analysis, timeliness analysis, correlation analysis, uniqueness analysis, logical consistency analysis or trend anomaly analysis.

[0007] In a possible implementation, a mapping relationship is established for business system data of multiple recording unit operation data, and the data is uniformly registered in a metadata database for metadata integration, including: randomly selecting two business systems, and extracting metadata information including table structure, field name, data type and constraints in the data table; identifying semantically equivalent metadata items to establish a mapping relationship; wherein the mapping relationship is established through a mapping function; and registering the extracted and mapped metadata in a unified metadata database.

[0008] In a possible implementation, the integrity analysis includes: checking the integrity of fields in the data record based on constraints in the metadata information; wherein in the non-null constraint: Among them, non_null_rate indicates the ratio of non-null values ​​in the field, count(*) indicates calculating the total number of all records, and count(field≠NULL) indicates calculating the number of records where a specific field is not NULL. If non_null_rate<1, the output has a data defect of missing fields.

[0009] In a possible implementation, the legality analysis includes: checking the legality of the data value according to the value range constraint of the metadata; wherein in the status code: Among them, invalid_rate represents the proportion of illegal data. It means calculating the number of records whose field values ​​are not in the set {0, 1, 2, ···, k}. count(*) means calculating the total number of all records. If invalid_rate>0, the output indicates a data defect with an illegal status code.

[0010] In a possible implementation, the consistency analysis includes: based on metadata mapping, checking the consistency of the same business data in different systems; wherein in the measurement value mapping: Among them, inconsistent_rate indicates the proportion of data inconsistency, count(deviation>0) indicates calculating the absolute value of the numerical difference between the corresponding fields of the two systems, and count(*) indicates calculating the total number of all records. If inconsistent_rate>0, the output values ​​between different systems are inconsistent.

[0011] In a possible implementation, the timeliness analysis includes: using the timestamp information of the metadata to evaluate the timeliness of data update; the evaluation method includes: delay = now()-max(timestamp), where delay represents the delay time of data update, now() represents the current time point, and max(timestamp) represents the maximum timestamp in the data; if delay>threshold, the output data is not updated in time and there is a lag defect.

[0012] In a possible implementation, the association analysis includes: evaluating the referential integrity of the data based on the association constraints including foreign keys between metadata; wherein in the field reference: Among them, ref_integrity_rate represents the reference integrity ratio, count(meter_data.meter_id∈meter.meter i d) Indicates that the meter_id in the meter_data table exists in the meter in the meter table. i d, count(meter_data) represents the number of meter_data; if ref_integrity_rate<1, there are isolated records in the table meter_data in the output, and meter_id cannot be found in the main table meter, which violates referential integrity.

[0013] In a possible implementation, the uniqueness analysis includes: evaluating data uniqueness through count aggregation; wherein in the field for the uniqueness constraint: Among them, duplicate_rate represents the ratio of duplicate values ​​in the data, count(distinctfield) represents the number of distinct values ​​of a specific field, and count(field) represents the total number of values ​​of a specific field. If duplicate_rate>0, duplicate values ​​exist in the output, violating the uniqueness requirement.

[0014] In a possible implementation, the logical consistency analysis includes: combining the business semantics of the metadata to construct a logical consistency rule for the data; wherein in the correspondence between the power consumption and the electricity fee: Among them, illogical_rate represents the proportion of illogical data, count(value < fee) represents the number of records where the power consumption is less than the electricity fee, and count(*) represents the total number of records; if illogical_rate > 0, then data violating the business logic is output.

[0015] In a possible implementation, the trend anomaly analysis includes: using the time dimension information of the metadata to analyze the change trend of the data; in the power generation: Among them, CV represents the coefficient of variation, represents the sum of the squares of the differences between all data points and their expected values, represents the average value of all data points; if CV exceeds the threshold, then it is output that the data trend is abnormal.

[0016] As can be seen from the above technical solution, in an analysis method for hydropower plant unit data defects based on metadata integration disclosed in the present application, through a multi-analysis method combining metadata and domain rules, the physical model and data-driven can be combined, making full use of the existing domain knowledge, reducing the demand for a large amount of labeled data, and improving the model interpretability, facilitating the understanding of the causes of the discovered data defects and engineering practice applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] Figure 1 It is a schematic flowchart of an analysis method for hydropower plant unit data defects based on metadata integration provided by an embodiment of the present application;

[0019] Figure 2 It is a schematic flowchart of a method for metadata integration provided by an embodiment of the present application. DETAILED DESCRIPTION

[0020] Now, various exemplary embodiments of the present application will be described in detail with reference to the drawings. It should be noted that: unless otherwise specifically stated, the relative arrangements, numerical expressions, and numerical values of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0021] Those skilled in the art will appreciate that the terms "first", "second" and the like in the embodiments of the present application are only used to distinguish different steps, devices or modules, etc., and do not represent any specific technical meaning, nor do they represent the necessary logical order between them. It should also be understood that in the embodiments of the present application, "multiple" may refer to two or more, and "at least one" may refer to one, two or more. It should also be understood that for any component, data or structure mentioned in the embodiments of the present application, in the absence of explicit limitation or contrary revelation given in the context, it can generally be understood as one or more. In addition, the term "and / or" in the present application is only a kind of association relationship describing the associated objects, indicating that there can be three kinds of relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present application generally indicates that the associated objects before and after are an "or" relationship. It should also be understood that the description of each embodiment in the present application emphasizes the differences between the embodiments, and the same or similar parts can refer to each other. For the sake of brevity, they will not be repeated one by one.

[0022] At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present application and its application or use. The techniques, methods and devices known to ordinary technicians in the relevant fields may not be discussed in detail, but where appropriate, the techniques, methods and devices should be considered as part of the specification. It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] Figure 1 The flowchart of the method for analyzing the data defects of hydropower plant units based on metadata integration provided in the embodiment of the present application is as follows. Figure 1 As shown, at step S101, a mapping relationship is established for the business system data of multiple record unit operation data, and they are uniformly registered in the metadata database for metadata integration (such as Figure 2 shown).

[0025] It should be noted that before integration, a unified metadata standard and protocol need to be established. Specifically, the metadata description standard should be defined first, including: basic information such as data item name, code, data type, unit, etc. Management information such as data source, update frequency, quality assessment, etc. Business information such as data semantics, associations, and application scenarios.

[0026] Next, we compiled a standard vocabulary in the field of hydropower plant units to standardize terminology. We also built a conceptual model of unit equipment and systems based on ontology to formalize the relationship between concepts. For example, turbine is-part-of unit, runner is-part-of turbine, turbine efficiency is-property-of turbine.

[0027] Then, develop metadata exchange protocols to achieve interoperability between different systems. Protocols may include: OAI-PMH (Open Archives Initiative Protocol for Metadata Harvesting), OGCCSW (Catalogue Service for the Web).

[0028] Furthermore, for the information integration of heterogeneous data sources, the source data of each business system is first collected, including: real-time and historical data of automation systems such as unit SCADA and MIS. Equipment ledgers, defect work orders, maintenance records, etc. of management systems such as EAM and ERP. Sensor measurement point data of professional systems such as condition monitoring and fault diagnosis.

[0029] Then, the heterogeneous data is mapped to a unified metadata standard to achieve information integration at the semantic level. Mapping methods include: manually configuring mapping rules, such as: EAM.EquipmentID→MetaModel.DeviceID. Automatic mapping based on ontology reasoning, such as: EAM.Turbine≡MetaModel.Turbine. And using large language models (such as GPT-4) for semantic understanding and conversion.

[0030] In dealing with data conflicts and ensuring the consistency of information after integration, the methods used include: rule-based conflict detection and resolution, priority = data quality score × data timeliness, and multi-source information fusion, such as DS evidence theory, Bayesian reasoning, etc.

[0031] In step S102, various data defect analyses are performed on the integrated metadata library to determine data quality; wherein the data defect analysis includes integrity analysis, legality analysis, consistency analysis, timeliness analysis, relevance analysis, uniqueness analysis, logical consistency analysis or trend anomaly analysis.

[0032] In one embodiment, the integrity analysis includes: checking the integrity of fields in the data record based on constraints in the metadata information; wherein in the non-null constraint: Among them, non_null_rate indicates the ratio of non-null values ​​in the field, count(*) indicates calculating the total number of all records, and count(field≠NULL) indicates calculating the number of records where a specific field is not NULL. If non_null_rate<1, the output has a data defect of missing fields.

[0033] In actual operation, you can traverse each field in the data set and calculate the non_null_rate for fields with non-null constraints. If the non_null_rate of any field is less than 1, a prompt will be output to indicate that there is a data defect with missing values ​​in the field. In addition to non-null constraints, other constraints can be defined in metadata, such as data type constraints, value range constraints, etc. Integrity analysis needs to check whether the data meets all these constraints. Data that meets all constraints can be considered complete.

[0034] The purpose of the legality analysis is to check whether the data complies with the expected business rules and constraints. In the embodiment provided, it describes the legality check for the value range of a specific field. Specifically, this embodiment uses the value range constraints defined in the metadata to check the legality of the data. The value range constraints specify the range or set of values ​​that a field is allowed to take, and any value that does not belong to the range or set is considered illegal data. In another embodiment, the legality analysis includes:

[0035] According to the value range constraints of metadata, the legitimacy of data values ​​is checked; in the status code: Among them, invalid_rate represents the proportion of illegal data. It means calculating the number of records whose field values ​​are not in the set {0, 1, 2, ···, k}. count(*) means calculating the total number of all records. If invalid_rate>0, the output indicates a data defect with an illegal status code.

[0036] In yet another embodiment, the consistency analysis includes: based on metadata mapping, checking the consistency of the same business data in different systems; wherein in the measurement value mapping: Among them, inconsistent_rate represents the proportion of data inconsistency, count(deviation>0) represents the calculation of the absolute value of the numerical difference between the corresponding fields of the two systems, and count(*) represents the calculation of the total number of all records; if inconsistent_rate>0, the output value between different systems has the defect of inconsistent values. In this embodiment, the core of consistency analysis is to perform consistency checks on the same business data in different systems based on metadata mapping. In the case of measurement value mapping, the inconsistent_rate indicator is used to measure the degree of data inconsistency.

[0037] Specifically, inconsistent_rate refers to the proportion of inconsistent data. It is calculated as follows: first calculate the absolute value of the difference in the values ​​of the corresponding fields between the two systems, and record the number of records with absolute values ​​greater than 0 as count(deviation>0); then calculate the total number of all records, recorded as count(*); finally, divide count(deviation>0) by count(*) to get inconsistent_rate.

[0038] If the value of inconsistent_rate is greater than 0, it indicates that there is data inconsistency. At this time, defect information that there are inconsistencies in the values ​​between different systems can be output. This consistency analysis method based on metadata mapping and inconsistent_rate indicators can effectively discover and quantify the numerical differences and inconsistencies of the same business data between different systems. By comparing the corresponding field values ​​under the mapping relationship, specific inconsistent records can be accurately located, thereby supporting subsequent analysis and repair work. In addition, the inconsistent_rate indicator provides a standardized measurement method that facilitates tracking and evaluating improvements in data consistency. The entire consistency analysis process can be executed automatically, supporting cross-system, large-scale data consistency checks, and helping to ensure the integrity and reliability of business data.

[0039] In another embodiment, the timeliness analysis includes: using the timestamp information of the metadata to evaluate the timeliness of data update; wherein the evaluation method includes: delay = now()-max(timestamp), wherein delay represents the delay time of data update, now() represents the current time point, and max(timestamp) represents the maximum timestamp in the data; if delay>threshold, the output data is not updated in time and there is a lag defect. Specifically, the evaluation method is to calculate the delay indicator, which reflects the time difference between the current time point and the latest timestamp in the data. By calculating the delay, it is possible to measure whether the data update is timely. If the delay value exceeds a preset threshold, it indicates that there is a lag in the data update, and the output data is not updated in time. Defect information.

[0040] This timeliness analysis method based on timestamp can quickly discover data update lags. Timely acquisition of the latest data is crucial for data-driven businesses, and delayed data may lead to wrong decisions. By setting reasonable thresholds, timeliness requirements can be adjusted and controlled according to business needs. The advantage of this method is that the idea is simple and direct, and only the difference between the current time and the latest timestamp needs to be compared. It does not rely on complex data processing logic and can be easily integrated into the existing data quality assessment process. At the same time, timestamp information is usually stored with the data as part of the metadata, so the implementation cost of this method is low. In general, using timestamps for timeliness analysis is an effective means of data quality assessment, which can effectively discover timeliness issues such as data update lags, thereby maintaining the freshness and reliability of the data and ensuring that the data can be used in a timely manner.

[0041] In yet another embodiment, the association analysis includes: evaluating the referential integrity of the data based on association constraints including foreign keys between metadata; wherein in the field reference: Among them, ref_integrity_rate represents the reference integrity ratio, count(meter_data.meter_id∈meter.meter i d) Indicates that the meter_id in the meter_data table exists in the meter in the meter table. id, count(meter_data) indicates the number of meter_data; if ref_integrity_rate<1, the output shows that there are isolated records in the meter_data table, and the meter_id cannot be found in the main table meter, violating the referential integrity. Furthermore, the value of ref_integrity_rate is between 0 and 1, reflecting the proportion of records in the meter_data table that meet the foreign key constraint. If ref_integrity_rate is less than 1, it indicates that the meter_id of the meter_data record cannot find the corresponding value in the meter main table, violating the referential integrity constraint, and the defect information is output.

[0042] This integrity analysis based on association constraints can effectively discover association problems between cross-table data. In real applications, data often comes from multiple heterogeneous data sources, which are associated together through foreign keys. If the association is broken, it will affect subsequent data query and use.

[0043] The advantage of this method is that it directly utilizes the foreign key constraint metadata of the database without the need for additional business rule definition. It is simple and efficient, and only requires statistics on table data to evaluate data quality. At the same time, this analysis can also be extended to other association constraints, such as check constraints, to fully ensure data consistency. It can also ensure that the data association relationship across data sources is correct, avoid data pollution and isolation, and help improve the utilization value of data assets.

[0044] Uniqueness means that certain field values ​​in the database must be unique within the same table, and duplicate values ​​are not allowed. It ensures the distinctiveness and accuracy of the data. In another embodiment, the uniqueness analysis includes: evaluating data uniqueness through count aggregation; wherein in the field for the uniqueness constraint: Among them, duplicate_rate represents the ratio of duplicate values ​​in the data, count(distinctfield) represents the number of distinct values ​​of a specific field, and count(field) represents the total number of values ​​of a specific field. If duplicate_rate>0, duplicate values ​​exist in the output, violating the uniqueness requirement.

[0045] The value of duplicate_rate ranges from 0 to 1, reflecting the proportion of duplicate field values. If duplicate_rate is greater than 0, it indicates the existence of duplicate values, violating the uniqueness constraint of the field, and the defect information is output. This uniqueness analysis based on statistical aggregation can quantitatively evaluate the uniqueness degree of field values. In practical applications, the uniqueness of many entity class data needs to be ensured, such as ID numbers, bank account numbers, phone numbers, etc. If duplicates occur, it will lead to data pollution and ambiguity. The advantage of this method is that it directly utilizes the unique constraint metadata of the database without the need for additional business rule definitions. It is simple and efficient, and only needs to perform aggregation statistics on field values to evaluate data quality.

[0046] Logical consistency analysis is a very important step in the field of data quality control. It aims to ensure that data conforms to expected business rules and constraints, thus guaranteeing data integrity and correctness. In this embodiment, the process of logical consistency analysis includes the following key steps:

[0047] Construct logical consistency rules: Combine the metadata and domain knowledge of the relevant business domain to construct logical rules that describe how the data should behave. These rules usually reflect common sense constraints in business logic, such as "electricity consumption should be less than electricity charges". The rules can be expressed in a formal language or a domain-specific language.

[0048] Apply logical consistency rules: Apply the constructed logical consistency rules to the actual data set to check whether the data violates these rules. This process often involves a large amount of data processing and calculation.

[0049] Quantitatively analyze violations: For the detected violated data, further statistical and analytical work is required to calculate the proportion, distribution, etc. of the violated data. This provides guidance for subsequent data repair work.

[0050] In this example, for the business scenario of "electricity consumption and electricity charges", this application proposes an indicator illogical_rate for quantifying the violation of the logical rule that "electricity consumption should be less than electricity charges". This indicator calculates the proportion of records where the electricity consumption value is less than the electricity charge value among all records. In another embodiment, the logical consistency analysis includes: combining the business semantics of the metadata to construct the logical consistency rules of the data; in the corresponding relationship between electricity consumption and electricity charges: Among them, illogical_rate represents the proportion of data that does not conform to logic, count(value<fee) represents the number of records where the electricity consumption is less than the electricity charge, and count(*) represents the total number of records; if illogical_rate>0, output that there is data violating the business logic.

[0051] In another embodiment, the trend anomaly analysis includes: In another embodiment, the steps of implementing the trend anomaly analysis are as follows: Using the time dimension information in the metadata, the data is arranged in chronological order to form time series data. Time series data refers to a series of data observations arranged in chronological order. Trend analysis is performed on the time series data to discover the potential laws or patterns of data changes. Common trends include upward trends, downward trends, cyclical trends, and seasonal trends. The coefficient of variation (CV) is calculated as an indicator to measure the degree of data fluctuation. The coefficient of variation is a form of relative standard deviation, which divides the standard deviation by the mean to eliminate the dimension effect.

[0052] Then set the threshold of the coefficient of variation. The threshold can be determined according to the specific application scenario and data characteristics. Determine whether the calculated coefficient of variation exceeds the set threshold: if the coefficient of variation exceeds the threshold, a warning is output that the data trend is abnormal; if the coefficient of variation does not exceed the threshold, the data change is considered to be within the normal range.

[0053] Furthermore, the time dimension information of metadata is used to analyze the change trend of data; in terms of power generation: Where CV represents the coefficient of variation, It means calculating the sum of the squares of the differences between all data points and their expected values. Represents the average value of all data points; if CV exceeds the threshold, the output data trend is abnormal.

[0054] Through the above steps, abnormal trend changes in time series data can be automatically detected, providing a basis for subsequent fault diagnosis and prevention. As a dimensionless statistic, the coefficient of variation can well measure the degree of discreteness of data. Setting reasonable thresholds according to different application scenarios can balance the sensitivity and robustness of detection.

[0055] Figure 2 A flowchart of a metadata integration method provided in an embodiment of the present application. Specifically, at step S201, two business systems are randomly selected, and metadata information including table structure, field name, data type and constraint in the data table is extracted. At step S202, semantically equivalent metadata items are identified to establish a mapping relationship; wherein the mapping relationship is established by a mapping function. At step S203, the extracted and mapped metadata is registered in a unified metadata database.

[0056] For example, a hydropower plant has two business systems A and B, which record the unit operation data respectively. Now it is necessary to integrate the metadata of the two systems.

[0057] First, perform metadata extraction:

[0058] System A records data table: meter_data(tstimestamp, meter_idint, valuefloat, statusint).

[0059] System B records the data table: device_data(timedatetime, device_namevarchar, readingdecimal(10, 2), statetinyint).

[0060] Extract metadata information such as table structure, field names, data types, constraints, etc.

[0061] Next, perform metadata mapping:

[0062] Identify semantically equivalent metadata items and establish mapping relationships:

[0063] timestamp<--->datetime represents timestamp;

[0064] meter_id<--->device_name indicates the device ID;

[0065] value<--->reading indicates the measured value;

[0066] status<--->state indicates the status code.

[0067] Specifically, the mapping function can be expressed as:

[0068]

[0069] Then, perform metadata registration: register the extracted and mapped metadata into the unified metadata database in the format of meta_repo (systemvarchar, table_namevarchar, field_namevarchar, data_typevarchar, constraintvarchar, mappingvarchar).

[0070] Example data such as: ('A', 'meter_data', 'ts', 'timestamp', 'primarykey', 'timestamp')('A', 'meter_data', 'value', 'float', 'notnull', 'reading')('B', 'device_data','time','datetime','primarykey','timestamp')('B','device_data','reading','decimal(10,2)','notnull','value')......

[0071] The metadata integration of the two systems is completed through the above method, which can lay the foundation for subsequent data defect analysis.

[0072] Furthermore, an embodiment of the present application also provides an analysis device for hydropower plant unit data defects based on metadata integration, including: a processor, a memory, and a system bus; the processor and the memory are connected via the system bus; the memory is used to store one or more programs, and the one or more programs include instructions, which, when executed by the processor, enable the processor to execute any of the above methods.

[0073] Furthermore, an embodiment of the present application also provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any one of the above-mentioned analysis methods.

[0074] It can be known from the description of the above implementation mode that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the analysis method described in each embodiment or some parts of the embodiment of the present application.

[0075] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0076] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0077] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for analyzing data defects of hydropower plant units based on metadata integration, characterized in that: include: Establish mapping relationships between business system data that record unit operation data, and uniformly register them in the metadata database for metadata integration; The integrated metadata database is subjected to various data defect analyses to determine data quality; wherein the data defect analyses include integrity analysis, legality analysis, consistency analysis, timeliness analysis, relevance analysis, uniqueness analysis, logical consistency analysis or trend anomaly analysis.

2. The analysis method according to claim 1, characterized in that The mapping relationship is established for multiple business system data that record unit operation data, and they are uniformly registered in the metadata database for metadata integration, including: Randomly select two business systems and extract metadata information including table structure, field name, data type and constraints from the data table; Identifying semantically equivalent metadata items to establish a mapping relationship; wherein the mapping relationship is established by a mapping function; Register the extracted and mapped metadata into the unified metadata database.

3. The analysis method according to claim 1, characterized in that The integrity analysis includes: Based on the constraints in the metadata information, check the integrity of the fields in the data record; in the non-null constraint: Among them, non_null_rate indicates the ratio of non-null values ​​in the field, count(*) indicates calculating the total number of all records, and count(≠NULL) indicates calculating the number of records in which a specific field is not NULL. If non_null_rate<1, the output has a data defect of missing fields.

4. The analysis method according to claim 1, characterized in that The legality analysis includes: According to the value range constraints of metadata, the legitimacy of data values ​​is checked; in the status code: Among them, invalid_rate represents the proportion of illegal data. It means calculating the number of records whose field values ​​are not in the set {0, 1, 2, ···, k}. count(*) means calculating the total number of all records. If invalid_rate>0, the output indicates a data defect with an illegal status code.

5. The analysis method according to claim 1, characterized in that The consistency analysis includes: Based on metadata mapping, the consistency of the same business data in different systems can be checked; in the measurement value mapping: Among them, inconsistent_rate indicates the proportion of data inconsistency, count(deviation>0) indicates calculating the absolute value of the numerical difference between the corresponding fields of the two systems, and count(*) indicates calculating the total number of all records. If inconsistent_rate>0, the output values ​​between different systems are inconsistent.

6. The analysis method according to claim 1, characterized in that The timeliness analysis includes: The metadata timestamp information is used to evaluate the timeliness of data updates. The evaluation method includes: delay = now() - max(timestamp), where delay represents the delay time of data updates, now() represents the current time point, and max(timestamp) represents the maximum timestamp in the data. If delay>threshold, the output data is not updated in a timely manner and there is a lag defect.

7. The analysis method according to claim 1, characterized in that The correlation analysis includes: Evaluate the referential integrity of data based on the association constraints between metadata, including foreign keys; in field references: Among them, ref_integrity_rate represents the reference integrity ratio, count(meter_data.meter_id∈meter.meter i d) Indicates that the meter_id in the meter_data table exists in the meter in the meter table. i d, count(meter_data) represents the number of meter_data; if ref_integrity_rate<1, there are isolated records in the table meter_data in the output, and meter_id cannot be found in the main table meter, which violates referential integrity.

8. The analysis method according to claim 1, characterized in that The uniqueness analysis includes: Data uniqueness is evaluated through count aggregation; in the fields for the uniqueness constraint: Among them, duplicate_rate represents the ratio of duplicate values ​​in the data, ciunt(distinctfield) represents the number of distinct values ​​of a specific field, and count(field) represents the total number of values ​​of a specific field. If duplicate_rate>0, duplicate values ​​exist in the output, which violates the uniqueness requirement.

9. The analysis method according to claim 1, characterized in that The logical consistency analysis includes: Construct the logical consistency rules of data in combination with the business semantics of metadata; among them, in the correspondence between electricity quantity and electricity charge: Among them, illogical_rate represents the proportion of data that does not conform to logic, count(value < fee) represents the number of records where the electricity quantity is less than the electricity charge, and count(*) represents the total number of records; if illogical_rate > 0, then output the data that violates the business logic.

10. The analysis method according to claim 1, characterized in that The trend anomaly analysis includes: Use the time dimension information of metadata to analyze the changing trend of data; in power generation: Where CV represents the coefficient of variation, It means calculating the sum of the squares of the differences between all data points and their expected values. Represents the average value of all data points; if CV exceeds the threshold, the output data trend is abnormal.

Images