Data quality rule intelligent generation optimization method, system, equipment and medium

Abstract

The invention discloses a data quality rule intelligent generation optimization method, system and device and a medium. The method comprises the steps of obtaining multi-source data of a power system; extracting different features from the multi-source data of the power system, and constructing a data feature map; based on the data characteristic spectrum, a business rule is generated through multi-level analysis of multi-source data, and a rule base is obtained; and iteratively updating the business rule through a learning and feedback mechanism according to data change and abnormity, and carrying out quality detection on the data by utilizing the updated rule base. According to the invention, through multi-source data feature extraction and map construction, automatic generation of business rules is realized, and manual intervention is reduced; in combination with real-time monitoring and deep reinforcement learning, automatic iterative updating of the rule is realized, and the rule lagging problem is solved; the generation efficiency, the execution accuracy and the dynamic adaptability of the data quality rule can be improved, the data governance cost is reduced, and the data quality is guaranteed.

Description

Technical Field

[0001] This invention relates to the field of data detection technology, and in particular to a method, system, device and medium for intelligent generation and optimization of data quality rules. Background Technology

[0002] Against the backdrop of digital transformation, enterprise data is experiencing explosive growth, posing a significant challenge to the governance of massive amounts of structured, semi-structured, and unstructured data for organizations in the power industry, such as China Southern Power Grid. Traditional methods for generating data quality rules are ill-suited to the complexity of data in new power systems, necessitating intelligent approaches to automate rule generation and dynamic optimization. Data quality, as a core element of data governance, directly impacts business decisions and system operation through the accuracy, adaptability, and efficiency of its rules. However, existing technologies have significant limitations when dealing with the fusion of multi-source heterogeneous data and dynamic changes in business scenarios, requiring technological breakthroughs to improve data governance efficiency.

[0003] Existing data quality rule generation technologies suffer from numerous shortcomings. They rely excessively on manual intervention, with the entire rule-building process dependent on domain expert experience. This ranges from traversing and analyzing data dictionaries and manually writing SQL validation statements during rule design, to insufficient sampling coverage during the validation phase, and finally, a mere 9.7% coverage rate for structured data rules during maintenance, with over 2000 rule failures per month. This results in exorbitant labor costs and difficulties in knowledge solidification. Rule updates lag significantly behind business changes, with script development taking an average of long times and failing to match the frequency of data structure changes in business systems. Delayed change propagation leads to a monthly backlog of problematic data, and cross-system rule dependencies further exacerbate cascading error rates. Furthermore, they lack adaptability to various scenarios. Traditional rules perform poorly in parsing hierarchical relationships, handling sliding window calculation timeouts, and improving keyword recall rates when dealing with multimodal data such as nested XML structures for equipment ledgers, time-series data for transmission lines, and customer service work order texts, failing to meet the data governance needs of modern power systems. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides an intelligent data quality rule generation optimization method to solve the problems of high reliance on manual intervention and lagging rule updates in existing data quality rule generation technologies.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a method for intelligent generation and optimization of data quality rules, comprising:

[0008] Acquire multi-source data from the power system;

[0009] Different features are extracted from the multi-source data of the power system to construct a data feature map;

[0010] Based on the data feature map, business rules are generated through multi-level analysis of multi-source data to obtain a rule base;

[0011] Based on data changes and anomalies, the business rules are iteratively updated through a learning and feedback mechanism, and the updated rule base is used to perform data quality checks.

[0012] As a preferred embodiment of the intelligent generation and optimization method for data quality rules described in this invention, the method involves: extracting different features from multi-source data of the power system to construct a data feature map, including:

[0013] Collect and acquire structured data, semi-structured data, and unstructured data;

[0014] Extract field-level statistical features and cross-table relationships from the structured data;

[0015] Extracting frequency domain and time domain features from semi-structured data;

[0016] Extract entity recognition features and semantic association features from unstructured data;

[0017] The extracted features are stored in the form of a knowledge graph, where nodes represent data entities, attributes, and features, and edges represent the weights between nodes.

[0018] As a preferred embodiment of the intelligent generation and optimization method for data quality rules described in this invention, the method involves: generating business rules based on the data feature map through multi-level analysis of multi-source data to obtain a rule base, including:

[0019] A first type of rule, including field integrity rules and numerical range rules, is generated based on the aforementioned field-level statistical features;

[0020] Based on the cross-table relationships, a second type of rule is generated, including primary and foreign key constraint rules and business logic consistency rules.

[0021] A third type of rule, including time-series anomaly detection rules and periodic fluctuation rules, is generated based on the frequency domain features and time domain features.

[0022] A fourth type of rule is generated based on the entity recognition features and semantic association features, including text semantic constraint rules and cross-modal association rules.

[0023] The beneficial effects of this preferred technical solution are as follows: by mapping the characteristics of different types of data to business rules, the subjectivity and oversights of traditional manual rule writing are avoided. It reduces human intervention, solves the problems of low rule generation efficiency and poor adaptability, and ensures that the rules are comprehensive and closely aligned with actual business needs.

[0024] As a preferred embodiment of the intelligent generation and optimization method for data quality rules described in this invention, it further includes:

[0025] When different rules produce conflicting judgments for the same data entity, the rule associated with the high-weight feature is selected as the valid rule.

[0026] The execution priority of rules is dynamically adjusted based on the edge weight values ​​between feature nodes, with rules corresponding to features with higher weight values ​​being executed first.

[0027] For rules that have an inclusion relationship, rules are merged through hierarchical associations between nodes to form composite rules.

[0028] The beneficial effects of this preferred technical solution are as follows: By introducing a weight system based on feature graphs to manage rule conflicts and execution priorities, it changes the inefficient mode of traditional rule management, which relies on manual intervention in conflicts and static priority settings. When multiple rules conflict with the same data, the system automatically selects the optimal rule based on feature weights to avoid contradictory judgments. Simultaneously, it dynamically adjusts the rule execution order according to the business scenario, ensuring that critical rules run first. For rules with inclusion relationships, they are automatically merged to form composite rules, reducing redundancy.

[0029] As a preferred embodiment of the intelligent generation and optimization method for data quality rules described in this invention, the method includes: iteratively updating the business rules based on data changes and anomalies through a learning and feedback mechanism, including:

[0030] Real-time monitoring of data changes and anomalies during the execution of business rules; triggering a rule update process when the actual data anomaly rate exceeds the first threshold.

[0031] The updated business rules are simulated and verified. The accuracy, coverage and execution efficiency of the new and old business rules on the historical dataset are compared to determine the effect of rule optimization.

[0032] If the verification passes, the updated rules will be synchronized to the rule base, and the edge weights of the corresponding nodes in the feature graph will be updated; if the verification fails, the update process will be restarted until the performance of the business rules meets the requirements.

[0033] Regularly conduct a global evaluation of the rule base, and automatically optimize low-frequency but critical business rules based on data quality monitoring logs and business feedback to maintain the dynamic adaptability and efficiency of the rule base.

[0034] As a preferred embodiment of the intelligent generation and optimization method for data quality rules described in this invention, the rule update process includes:

[0035] Extract the feature vector of the current abnormal data, perform similarity matching with the feature map corresponding to the historical rules, and locate the nodes of the failed rules or rules to be optimized;

[0036] Deep reinforcement learning algorithms are used to optimize the parameters of business rules, including adjusting the threshold of business rule conditions, weight coefficients and execution priorities. The optimization objective is to minimize the loss function of rule execution error and business impact.

[0037] The beneficial effects of this preferred technical solution are as follows: Rule updates are triggered by real-time monitoring of abnormal data; rule parameters are intelligently optimized using deep reinforcement learning; and the update effect is verified through simulation with historical data, ensuring the reliability of the new rules. Regardless of changes in the data environment or adjustments to business logic, the system can autonomously locate and rapidly iterate on invalid rules, synchronously updating feature map weights. This ensures the rule base remains adaptable to real-world scenarios, guaranteeing the timeliness and effectiveness of data quality rules.

[0038] As a preferred embodiment of the intelligent generation and optimization method for data quality rules described in this invention, the method includes: performing data quality detection using an updated rule base, including:

[0039] The first and second types of rules are used to detect the field integrity, numerical range, and cross-table consistency of structured data;

[0040] The third type of rule is used to detect outliers and fluctuation patterns in semi-structured data;

[0041] The fourth type of rule is used to detect semantic conflicts and missing information in unstructured data.

[0042] Secondly, the present invention provides a data quality rule intelligent generation and optimization system, comprising: a data acquisition module for acquiring multi-source data of a power system;

[0043] The feature extraction module is used to extract different features from the multi-source data of the power system and construct a data feature map;

[0044] The rule generation module is used to generate business rules based on the data feature map by analyzing multi-source data at multiple levels and to obtain a rule library.

[0045] The update detection module is used to iteratively update the business rules based on data changes and anomalies through a learning and feedback mechanism, and to perform data quality detection using the updated rule base.

[0046] Thirdly, the present invention provides an electronic device, comprising:

[0047] Memory and processor;

[0048] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the intelligent generation and optimization method for data quality rules are implemented.

[0049] Fourthly, the present invention provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the intelligent generation and optimization method for data quality rules.

[0050] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention achieves automated generation of business rules by extracting features and constructing graphs from multi-source data of the power system, reducing manual intervention; it resolves rule conflicts and dynamically adjusts execution priorities based on feature graph weights, improving rule execution efficiency; it combines real-time monitoring and deep reinforcement learning to achieve automatic iterative updates of rules, solving the problem of rule lag; and finally, it uses the updated rule base to perform targeted quality checks on different types of data and form a closed-loop feedback loop. This improves the generation efficiency, execution accuracy, and dynamic adaptability of data quality rules, reduces data governance costs, and ensures data quality. Attached Figure Description

[0051] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a schematic diagram of the overall process of the intelligent generation and optimization method for data quality rules according to an embodiment of the present invention.

[0053] Figure 2 This is a schematic diagram comparing the improvement effect of the intelligent generation and optimization method for data quality rules in equipment ledger data according to an embodiment of the present invention.

[0054] Figure 3 This is a schematic diagram comparing the anomaly detection effect of the intelligent generation and optimization method for data quality rules in smart meters according to an embodiment of the present invention. Detailed Implementation

[0055] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0056] Example 1, referring to Figure 1 As an embodiment of the present invention, a method for intelligent generation and optimization of data quality rules is provided, comprising:

[0057] S100: Acquire multi-source data from the power system;

[0058] S101: Extract different features from multi-source data of the power system and construct a data feature map;

[0059] S102: Based on data feature maps, generate business rules by analyzing multi-source data at multiple levels and obtain a rule base;

[0060] S103: Based on data changes and anomalies, iteratively update business rules through a learning and feedback mechanism, and use the updated rule base to perform data quality checks.

[0061] It should be noted that this invention first acquires multi-source data from the power system because the modern power data environment is complex and diverse. Only by integrating structured, semi-structured, and unstructured data can the data governance scenarios be fully covered. Feature extraction and graph construction of multi-source data aim to deeply explore the inherent relationships within the data, providing accurate basis for rule generation. Business rules generated based on graph analysis enable automated and intelligent rule production, eliminating reliance on manual experience. Iterative rule updates and data quality checks using learning and feedback mechanisms ensure that rules keep pace with data and business changes, continuously optimizing the system. This invention effectively solves the problems of high manual dependence and lagging updates in traditional data quality rule generation, improving rule generation efficiency and accuracy, ensuring data quality, reducing data governance costs, and enhancing the dynamic adaptability of the data rule system to complex and ever-changing data environments.

[0062] Example 2, refer to Figure 1 This is one embodiment of the present invention. Based on the above embodiment, a data quality rule intelligent generation and optimization method is provided.

[0063] In this embodiment of the invention, the multi-source data of the power system collected in step S100 includes structured, semi-structured and unstructured data.

[0064] For example, structured data includes equipment ledgers in power systems, power grid operation data tables in production and operation databases, and user permission tables in enterprise resource management systems; semi-structured data includes JSON-formatted electricity consumption data uploaded by smart meters, XML-formatted status reports generated by power inspection equipment, and time-series operation records with delimiters in log systems; unstructured data includes images and videos taken during transmission line inspections, fault description text in customer service work orders, and power industry technical specification documents.

[0065] In one optional implementation, after acquiring multi-source data, the data is preprocessed, including data cleaning, format conversion, and missing value handling. For structured data, duplicate equipment records in the equipment ledger table are deleted using a deduplication algorithm, the numerical format in the power grid operation data table is validated using regular expressions, and missing fields in the user permission table are filled with the mean value or completed using business logic.

[0066] For semi-structured data, the JSON data of smart meters is uniformly parsed into a standard format, the XML status report is processed with standardized tags, and the abnormal fluctuation values ​​in the log records are smoothed by the sliding window algorithm.

[0067] For unstructured data, optical character recognition technology is used to convert text information in inspection images into text. Stop word filtering and stemming techniques from NLP are used to process fault description text. Technical specification documents are parsed through document segmentation and key sentence extraction techniques to reduce data noise and unify data format, providing a high-quality data foundation for subsequent feature extraction and rule generation.

[0068] In this embodiment of the invention, step S101, which involves extracting different features from the multi-source data of the power system and constructing a data feature map, further includes sub-steps A1-A2:

[0069] A1: Extract field-level statistical features and cross-table relationships from structured data; extract frequency domain and time domain features from semi-structured data; extract entity recognition features and semantic association features from unstructured data;

[0070] A2: The extracted features are stored in the form of a knowledge graph, where nodes represent data entities, attributes, and features, and edges represent the weights between nodes.

[0071] Specifically, for structured data, field-level statistical features are extracted by calculating statistics such as the mean, standard deviation, maximum, and minimum values ​​of the fields. For example, the fluctuation range of voltage and current fields in the power grid operation data table can be calculated. Foreign key constraints and data dictionaries are used to analyze the relationships between tables, such as establishing a primary and foreign key relationship between the equipment ledger table and the maintenance record table through the equipment number.

[0072] For semi-structured data, Fourier transform is used to extract frequency domain features from smart meter electricity consumption data to identify periodic patterns of peak and off-peak electricity consumption; a sliding window algorithm is used to extract time domain features to detect abrupt changes and abnormal trends in the data. For unstructured data, named entity recognition technology is used to extract entities such as tower type and fault location from transmission line inspection texts; semantic role labeling and graph neural networks are used to mine semantic relationships between entities in the text, such as the causal relationship between insulator aging and discharge phenomena.

[0073] In this embodiment of the invention, the extracted features are transformed into nodes of a knowledge graph; for example, the device model and voltage value are used as data entity nodes, the value range and associated table name are used as attribute nodes, and periodic fluctuations and semantic dependencies are used as feature nodes.

[0074] Edges are weighted according to the importance and correlation strength of features. For example, the edge linking equipment ledgers and maintenance records is given a higher weight due to its close business relationship, while edges with secondary semantic connections are given a lower weight. In this way, a complete data feature map is constructed, which intuitively shows the complex internal relationships between data and provides a structured knowledge foundation for the subsequent generation of business rules.

[0075] In one alternative implementation, features can be acquired using machine learning algorithms. For structured data, a random forest algorithm is used to assess the feature importance of power grid operation data, identifying key fields that significantly impact voltage stability. Association rule mining algorithms are used to analyze potential associations between equipment ledgers and maintenance records, uncovering implicit relationships without foreign key constraints. For semi-structured data, an LSTM neural network is used to learn from the time-series data of smart meters, automatically extracting long-term dependent time-domain features. A convolutional neural network combined with Fourier-transformed frequency-domain data is used to identify frequency-domain features of complex electricity consumption patterns. For unstructured data, a BERT pre-trained model is fine-tuned to achieve high-precision entity recognition in transmission line inspection texts. A graph convolutional network is used to model semantic relationships in the text, uncovering deep semantic associations between entities.

[0076] In another alternative implementation, features can be obtained by combining domain expert knowledge with heuristic rules. Power industry experts, based on industry standards and operational experience, formulate field validation rules for structured data, such as equipment model coding standards and safety thresholds for power grid parameters, directly converting them into field-level statistical features. For semi-structured data, the sliding window size and anomaly detection threshold are set according to the business logic of power dispatching to extract time-domain features under specific scenarios; key periodic parameters for frequency domain analysis are preset based on seasonal electricity consumption patterns. For unstructured data, a power industry-specific dictionary and semantic templates are constructed to assist in named entity recognition; experts annotate semantic relationships in typical texts to generate heuristic rules to uncover related features such as insulator aging and discharge phenomena, ensuring that feature extraction meets actual business needs.

[0077] It should be noted that this invention effectively integrates the statistical and correlation features of structured data, the frequency and time domain features of semi-structured data, and the entity and semantic features of unstructured data by performing targeted feature extraction and knowledge graph construction on multi-source data of the power system. This transforms the data into a structured knowledge graph, clearly presenting the inherent connections and weight relationships of the data through nodes and edges. This approach breaks through the limitations of traditional data feature processing, which is characterized by fragmented processing and weak correlation. It not only accurately mines key information from different types of data but also provides systematic and visualized knowledge support for subsequent business rule generation, improving the efficiency and accuracy of data feature processing, enhancing the comprehensiveness and adaptability of rule generation, and laying a solid foundation for efficient data quality governance.

[0078] In this embodiment of the invention, step S102, which generates business rules based on data feature maps through multi-level analysis of multi-source data and obtains a rule base, also includes sub-steps B1-B3:

[0079] B1: The first type of rules, which includes field integrity rules and numerical range rules, are generated based on field-level statistical features; the second type of rules, which includes primary and foreign key constraint rules and business logic consistency rules, are generated based on cross-table relationships; the third type of rules, which includes time-series anomaly detection rules and periodic fluctuation rules, are generated based on frequency domain features and time domain features; and the fourth type of rules, which includes text semantic constraint rules and cross-modal association rules, are generated based on entity recognition features and semantic association features.

[0080] B2: When different rules produce conflicting judgments for the same data entity, the rule associated with the high-weight feature shall be selected as the valid rule;

[0081] B3: Dynamically adjust the rule execution priority based on the edge weight values ​​between feature nodes, with rules corresponding to features with higher weight values ​​being executed first;

[0082] B4: For rules that have an inclusion relationship, rules are merged through hierarchical association between nodes to form composite rules.

[0083] For example, for structured data in a power system, based on the field-level statistical characteristics of the equipment ledger table, if a null value is found in the equipment number field, a field integrity rule is automatically generated, mandating that the field be filled in. By analyzing the mean and standard deviation of the voltage field in the power grid operation data table, if the value exceeds the standard deviation of historical fluctuation range, a value range rule is generated to limit the voltage threshold. Based on the primary and foreign key relationships between the equipment ledger table and the maintenance record table, primary and foreign key constraint rules are generated to ensure that the equipment number is consistent in the two tables; according to the power business process, business logic consistency rules are formulated for the association between maintenance plans and equipment status.

[0084] For example, for semi-structured smart meter electricity consumption data, based on the frequency domain features obtained by Fourier transform, if abnormal electricity consumption cycle patterns are found, a periodic fluctuation rule is generated to warn of changes in electricity consumption patterns; if a sudden change in time domain data is detected using a sliding window algorithm, a time-series anomaly detection rule is generated to mark abnormal electricity consumption points.

[0085] For example, in unstructured transmission line inspection text, entities such as insulators and discharges are identified by named entities. Combined with semantic association analysis, text semantic constraint rules are generated to judge the rationality of fault descriptions. Furthermore, structured data from equipment ledgers are associated with the inspection text to generate cross-modal association rules to verify the consistency of equipment status descriptions.

[0086] For example, when the equipment model field in the equipment ledger table triggers both the numerical range rule and the business logic consistency rule and a conflict is determined, the business logic consistency rule is prioritized because it is related to the cross-table association feature with high weight. Based on the weight of the edges in the feature graph, the execution priority of the power grid voltage stability detection rule is set to the highest to ensure that key business rules are run first. For rules with inclusion relationships, such as equipment maintenance cycle rules and special equipment expedited maintenance rules, they are automatically merged into composite rules through the hierarchical association of the feature graph. This ensures the rigor of the rules while reducing rule redundancy, improving the execution efficiency and accuracy of the rule base, and making data quality detection more in line with the actual needs of the power business.

[0087] In one alternative implementation, rules can be generated by combining reinforcement learning with genetic algorithms; the nodes and edges in the data feature graph are transformed into the state space of reinforcement learning, and the data quality improvement effect after rule execution is used as the reward function. The agent continuously learns through trial and error to explore the optimal combination of rules.

[0088] For example, in generating rules for the numerical range of power grid operation data, the intelligent agent simulates rule execution under different threshold settings, and receives rewards based on the anomaly detection accuracy and false alarm rate, gradually optimizing the threshold range of parameters such as voltage and current. Simultaneously, a genetic algorithm is introduced to perform crossover and mutation operations on the generated rule group. For instance, the primary and foreign key constraint rules of equipment ledgers from different substations are cross-referenced to generate new cross-table association rules. A fitness function is then used to select better rules, which are then retained in the rule base.

[0089] It should be noted that this invention is based on data feature maps and generates multi-category business rules according to different types of data features, comprehensively covering data quality detection scenarios and effectively replacing the inefficient mode of traditional manual rule writing. By taking feature weights as the core, it intelligently resolves rule conflicts, dynamically configures execution priorities, and merges redundant rules, avoiding misjudgments and inefficient execution caused by rule contradictions.

[0090] In this embodiment of the invention, step S103, which iteratively updates business rules based on data changes and anomalies through a learning and feedback mechanism and uses the updated rule base to perform data quality checks, also includes the following sub-steps C1-C4:

[0091] C1: Real-time monitoring of data changes and anomalies during the execution of business rules; when the actual data anomaly rate exceeds the first threshold, triggering the rule update process.

[0092] C2: Simulate and verify the updated business rules. By comparing the accuracy, coverage, and execution efficiency of the new and old business rules on the historical dataset, determine the effect of rule optimization. If the verification is successful, synchronize the updated rules to the rule base and update the edge weights of the corresponding nodes in the feature graph. If the verification fails, re-enter the update process until the performance of the business rules meets the requirements.

[0093] C3: Regularly conduct a global evaluation of the rule base, and automatically optimize low-frequency but critical business rules based on data quality monitoring logs and business feedback to maintain the dynamic adaptability and efficiency of the rule base;

[0094] C4: Use the first and second types of rules to detect the field integrity, numerical range, and cross-table consistency of structured data; use the third type of rules to detect outliers and fluctuation patterns in semi-structured data; and use the fourth type of rules to detect semantic conflicts and missing information in unstructured data.

[0095] In this embodiment of the invention, the rule update process is as follows: extract the feature vector of the current abnormal data, perform similarity matching with the feature map corresponding to the historical rules, and locate the nodes of the failed rules or rules to be optimized.

[0096] Deep reinforcement learning algorithms are used to optimize the parameters of business rules, including adjusting the threshold of business rule conditions, weight coefficients and execution priorities. The optimization objective is to minimize the loss function of rule execution error and business impact.

[0097] In one alternative implementation, transfer learning combined with ensemble learning methods can be used to update business rules. When an anomaly is detected in the power grid data of a certain area and a rule update is triggered, the feature vector of the current anomaly data is first transferred to other similar power grid areas to perform transfer learning on the rule models that have been verified to be effective.

[0098] For example, if the voltage rules of the power grid in region A fail due to seasonal load changes, the parameters of the mature voltage rule model under similar load characteristics in region B can be migrated to region A and fine-tuned to adapt to local data characteristics.

[0099] Example 3, referring to Figure 2 This is one embodiment of the present invention. Based on the above embodiment, a data quality rule intelligent generation and optimization method is provided. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through experiments.

[0100] This embodiment aims to solve the problems of completeness, consistency, accuracy, and rule lag that are common in equipment ledger data in the power industry by utilizing the "intelligent generation and optimization method for data quality rules" proposed in this invention.

[0101] Structured equipment ledger data is obtained from multiple business systems in the company's data center, such as equipment ledger management and production management. The data is then cleaned, formatted, and missing values ​​are handled, such as standardizing equipment numbering, converting date formats, and filling in missing values ​​for non-critical fields. Field-level statistical features, including completeness, uniqueness, value range distribution, and cross-table relationships, are extracted from the processed data and stored in the form of a knowledge graph. Nodes represent data entities, attributes, and features, while edges represent relationships and are assigned weights.

[0102] Data quality business rules are automatically generated based on feature maps and through multi-level analysis, including:

[0103] The non-empty rate of the device number generates a non-empty rule;

[0104] The range is generated based on the voltage level numerical distribution rules;

[0105] Resolve rule conflicts, adjust priorities, and merge compound rules based on feature weights.

[0106] The system monitors the execution effect of rules in real time. When the anomaly rate exceeds the preset threshold, an update is triggered. The rule parameters are optimized using a deep reinforcement learning algorithm. After simulation verification, the algorithm is synchronized to the rule base and the graph weights are updated. The rule base is evaluated globally on a regular basis. The updated rules are used to perform quality checks on the equipment ledger data, such as field integrity, numerical range, and cross-table consistency, to ensure that the rule base dynamically adapts and operates efficiently.

[0107] Test results are as follows Figure 2 As shown, Figure 2 Comparison chart showing the improvement in equipment ledger data quality; Figure 2 In this invention, AI-driven rule adaptive generation and optimization can effectively identify and force the filling of key fields, such as automatically generating the rule "device number cannot be empty" and dynamically adjusting the triggering conditions, reducing data loss. The integrity (non-empty rate) of key fields is increased from 85% in traditional methods to 98% in this invention.

[0108] Meanwhile, the cross-system data consistency error rate decreased from 10% to 5%, and the detection rate of abnormal equipment parameters increased from 70% to 90%. This further demonstrates that the multi-level analysis method of this invention can accurately identify anomalies that exceed the normal range or business logic, and identify hidden anomalies based on historical data patterns and context, providing accurate data for equipment risk assessment. The rule false alarm rate has also decreased. By optimizing rule parameters through deep reinforcement learning, the anomaly detection rate is improved while the misjudgment of normal data is reduced, making alarms more accurate and reducing the burden of manual verification.

[0109] Example 4, refer to Figure 3 This is one embodiment of the present invention. Based on the above embodiment, a data quality rule intelligent generation and optimization method is provided. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through experiments.

[0110] This embodiment aims to utilize the "intelligent generation and optimization method for data quality rules" proposed in this invention to achieve accurate anomaly detection and dynamic optimization of rules for smart meter time-series data, thereby improving data quality and business responsiveness.

[0111] The system acquires massive amounts of smart meter electricity consumption data from the smart meter data acquisition system and the metering master station system, as well as structured auxiliary data such as user files and meter files; and preprocesses the raw time-series data.

[0112] For smart meter time-series data, time-domain features, frequency-domain features, and auxiliary features are extracted. Auxiliary features include user type, weather data, etc. The extracted features are stored in the form of a knowledge graph. Nodes represent meter ID, time point, electricity consumption, time-domain features, frequency-domain features, and auxiliary features; edges represent the relationships between these features and are assigned weights.

[0113] Using a pre-constructed data feature map, the system automatically generates time-series data anomaly detection rules through multi-level analysis. When multiple anomaly detection rules conflict in their judgments regarding data at the same time point, the system selects the valid judgment based on the weights of related features in the feature map. Simultaneously, the system dynamically adjusts the rule execution priority based on the edge weights between feature nodes.

[0114] Test results are as follows Figure 3 As shown, Figure 3 The data includes the harmonic average of the method of this invention and the conventional method, the false alarm rate of abnormal alarms, and the missed alarm rate of actual abnormalities.

[0115] from Figure 3 It is evident that the method described in this invention improves upon traditional methods in terms of harmonic mean, false alarm rate, and missed detection rate of true anomalies. This demonstrates that the method can accurately identify true anomalies in smart meter time-series data anomaly detection while effectively avoiding misclassifying normal data as abnormal. Furthermore, this invention refines the anomaly detection rules through deep reinforcement learning, enabling it to distinguish between normal fluctuations and true anomalies, reducing invalid alarms and improving alarm effectiveness. This alleviates the burden on maintenance personnel, allowing them to focus more on handling genuine anomalies. Through rapid learning of new anomaly patterns and rule updates, the system can promptly capture anomalies that are difficult to detect using traditional methods. This is of great significance for the safe and stable operation and economic benefits of the power system, enabling earlier detection of electricity theft and reducing power losses.

[0116] Example 5 illustrates a schematic scheme for an intelligent data quality rule generation and optimization method. It should be noted that the technical solution of this intelligent data quality rule generation and optimization system belongs to the same concept as the technical solution of the aforementioned intelligent data quality rule generation and optimization method. Details not described in detail in this embodiment can be found in the description of the technical solution of the aforementioned intelligent data quality rule generation and optimization method.

[0117] This embodiment also provides a data quality rule intelligent generation and optimization system, including:

[0118] The data acquisition module is used to acquire multi-source data from the power system;

[0119] The feature extraction module is used to extract different features from multi-source data of the power system and construct a data feature map;

[0120] The rule generation module is used to generate business rules based on data feature maps and multi-source data through multi-level analysis, and to obtain a rule library.

[0121] The update detection module is used to iteratively update business rules based on data changes and anomalies through a learning and feedback mechanism, and to perform data quality checks using the updated rule base.

[0122] This embodiment also provides an electronic device suitable for intelligent generation and optimization of data quality rules, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the intelligent generation and optimization method for data quality rules as proposed in the above embodiment.

[0123] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, it implements the intelligent generation and optimization method for data quality rules as proposed in the above embodiments.

[0124] The storage medium proposed in this embodiment and the intelligent generation and optimization method for data quality rules proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0125] Based on the above description of the implementation methods, those skilled in the art will clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0126] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for intelligent generation and optimization of data quality rules, characterized in that, include: Acquire multi-source data from the power system; Different features are extracted from the multi-source data of the power system to construct a data feature map; Based on the data feature map, business rules are generated through multi-level analysis of multi-source data to obtain a rule base; Based on data changes and anomalies, the business rules are iteratively updated through a learning and feedback mechanism, and the updated rule base is used to perform data quality checks.

2. The intelligent generation and optimization method for data quality rules as described in claim 1, characterized in that, Different features are extracted from the multi-source data of the power system to construct a data feature map, including: Collect and acquire structured data, semi-structured data, and unstructured data; Extract field-level statistical features and cross-table relationships from the structured data; Extracting frequency domain and time domain features from semi-structured data; Extract entity recognition features and semantic association features from unstructured data; The extracted features are stored in the form of a knowledge graph, where nodes represent data entities, attributes, and features, and edges represent the weights between nodes.

3. The intelligent generation and optimization method for data quality rules as described in claim 2, characterized in that, Based on the aforementioned data feature map, business rules are generated through multi-level analysis of multi-source data to obtain a rule base, including: A first type of rule, including field integrity rules and numerical range rules, is generated based on the aforementioned field-level statistical features; Based on the cross-table relationships, a second type of rule is generated, including primary and foreign key constraint rules and business logic consistency rules. A third type of rule, including time-series anomaly detection rules and periodic fluctuation rules, is generated based on the frequency domain features and time domain features. A fourth type of rule is generated based on the entity recognition features and semantic association features, including text semantic constraint rules and cross-modal association rules.

4. The intelligent generation and optimization method for data quality rules as described in claim 3, characterized in that, Also includes: When different rules produce conflicting judgments for the same data entity, the rule associated with the high-weight feature is selected as the valid rule. The execution priority of rules is dynamically adjusted based on the edge weight values ​​between feature nodes, with rules corresponding to features with higher weight values ​​being executed first. For rules that have an inclusion relationship, rules are merged through hierarchical associations between nodes to form composite rules.

5. The intelligent generation and optimization method for data quality rules as described in claim 4, characterized in that, Based on data changes and anomalies, the business rules are iteratively updated through a learning and feedback mechanism, including: Real-time monitoring of data changes and anomalies during the execution of business rules; triggering a rule update process when the actual data anomaly rate exceeds the first threshold. The updated business rules are simulated and verified. The accuracy, coverage and execution efficiency of the new and old business rules on the historical dataset are compared to determine the effect of rule optimization. If the verification passes, the updated rules will be synchronized to the rule base, and the edge weights of the corresponding nodes in the feature graph will be updated; if the verification fails, the update process will be restarted until the performance of the business rules meets the requirements. Regularly conduct a global evaluation of the rule base, and automatically optimize low-frequency but critical business rules based on data quality monitoring logs and business feedback to maintain the dynamic adaptability and efficiency of the rule base.

6. The intelligent generation and optimization method for data quality rules as described in claim 5, characterized in that, The rule update process includes: Extract the feature vector of the current abnormal data, perform similarity matching with the feature map corresponding to the historical rules, and locate the nodes of the failed rules or rules to be optimized; Deep reinforcement learning algorithms are used to optimize the parameters of business rules, including adjusting the threshold of business rule conditions, weight coefficients and execution priorities. The optimization objective is to minimize the loss function of rule execution error and business impact.

7. The intelligent generation and optimization method for data quality rules as described in claim 6, characterized in that, Perform data quality checks using an updated rule base, including: The first and second types of rules are used to detect the field integrity, numerical range, and cross-table consistency of structured data; The third type of rule is used to detect outliers and fluctuation patterns in semi-structured data; The fourth type of rule is used to detect semantic conflicts and missing information in unstructured data.

8. A data quality rule intelligent generation and optimization system, using the method described in any one of claims 1-7, characterized in that, include: The data acquisition module is used to acquire multi-source data from the power system; The feature extraction module is used to extract different features from the multi-source data of the power system and construct a data feature map; The rule generation module is used to generate business rules based on the data feature map by analyzing multi-source data at multiple levels and to obtain a rule library. The update detection module is used to iteratively update the business rules based on data changes and anomalies through a learning and feedback mechanism, and to perform data quality detection using the updated rule base.

9. An electronic device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the intelligent generation and optimization method for data quality rules according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores computer-executable instructions, which, when executed by a processor, implement the steps of the intelligent generation and optimization method for data quality rules as described in any one of claims 1 to 7.

Images