A power grid multi-format data fusion and management method based on data blood relationship

Abstract

The application discloses a power grid multi-format data fusion and management method based on data blood relationship, comprising the following steps: generating a field-level dynamic blood relationship and constructing a dynamic blood relationship graph by analyzing the execution logic of a power grid data fusion task; when an application system detects data anomalies, traversing reversely along the blood relationship edge in the dynamic blood relationship graph, locating the root cause field node causing the anomaly, reading the responsibility department code stored in the root cause field node, generating a structured work order and pushing it to the corresponding responsibility department; receiving the treatment scheme fed back by the responsibility department, storing the treatment scheme in the attribute of the corresponding node in the dynamic blood relationship graph, and loading and reusing the treatment scheme when subsequent pre-checking.

Description

Technical Field

[0001] This invention relates to the field of power system data governance technology, and more specifically, to a method for data fusion and governance of multi-business data in the power grid based on data lineage. Background Technology

[0002] In the context of new power system construction, the refined simulation, analysis, and control of the power grid rely on the deep integration of data from multiple sectors, including marketing, equipment, dispatching, and planning. For example, accurate dynamic load modeling of the distribution network requires mapping together user profiles from the marketing system, daily frozen load data from the consumption and acquisition system, distributed renewable energy measurement data, distribution network equipment models and topology from the distribution PMS system, and power grid model data and operational data from the dispatching system. However, current data fusion and subsequent governance face the following dual technical challenges: (1) Quality control failure in the fusion process: Existing data fusion technologies (such as ETL tools and data platform aggregation) focus more on the physical migration and format conversion of data, and lack a real-time calculation and verification mechanism for the business consistency and logical continuity between multi-source data during the execution of the fusion logic. For example, when fuzzily associating marketing user profiles with the "distribution transformer area" of the distribution PMS through "address text", existing technologies cannot monitor abnormal indicators such as "address field missing rate" or "text similarity matching threshold too low" in real time during the execution of the association task. This results in a large number of "continuous hidden defects" (such as users connecting the wrong distribution transformer) being discovered only after the fusion results are produced, and the quality of the fusion results is inherently uncontrollable.

[0003] (2) Passive and inefficient governance model: Data quality problems are usually only exposed in the final application stage (such as the discovery of abnormal load modeling results of a 220kV substation or abnormal line loss calculation of a transformer area). The subsequent governance process relies on manual cross-departmental (marketing department, operation and maintenance department, dispatch center) tracing, assigning responsibility and coordinating via email or meetings. Due to the lack of an automated tracing path from application data to source data, the accountability cycle is long and the phenomenon of shirking responsibility is serious, forming a vicious cycle of "late problem discovery, difficulty in root cause location, and high coordination costs". The overall effectiveness of data governance is difficult to meet the requirements of the new power system for data timeliness. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method for data fusion and governance of multiple power grid sectors based on data lineage.

[0005] According to one aspect of the present invention, a method for data fusion and governance of multi-business data in power grids based on data lineage is provided, comprising: By analyzing the execution logic of the power grid data fusion task, dynamic lineage relationships at the field level are generated, and a dynamic lineage graph is constructed. When the application system detects data anomalies, it traverses backward along the bloodline edges in the dynamic bloodline graph to locate the root cause field node that caused the anomaly, reads the responsible department code stored in the root cause field node, automatically generates a structured work order, and pushes it to the corresponding responsible department. Receive the processing solutions from the responsible department, and store the processing solutions in the attributes of the corresponding nodes in the dynamic lineage graph for later loading and reuse during pre-verification.

[0006] Optionally, the method further includes: before the fusion task is started, querying the matching success rate corresponding to the preset attributes of the historical execution records of the fusion task in the dynamic lineage graph; if the matching success rate is lower than a preset threshold, then blocking the task from starting; or Query the null value rate of key fields in the input source table. If the null value rate exceeds the threshold, block the task from starting.

[0007] Optionally, the method further includes: calculating the real-time matching rate during the execution of the fusion task, and comparing the real-time matching rate with a dynamic threshold calculated based on the historical success rate; if the real-time matching rate falls below the lower limit of the dynamic threshold, then interrupting the task execution.

[0008] Optionally, by parsing the execution logic of the power grid data fusion task, dynamic lineage relationships at the field level are generated, and a dynamic lineage graph is constructed, including: Metadata is extracted from multiple business systems, and an initial lineage graph is constructed using fields as nodes and physical dependencies or logical transformations as edges. Capture SQL statements in data fusion tasks, generate abstract syntax trees through SQL parsers, and generate logical plans through semantic analysis; Extract the transformation logic from source fields to target fields from the logical plan; Based on the initial lineage graph, a dynamic lineage graph is generated by creating lineage edges in the graph database that point from the source field node to the target field node and carry transformation logic attributes.

[0009] Optionally, the specific formula for calculating the real-time matching rate is as follows: Real-time matching rate = (Number of records successfully associated in the current batch / Total number of input records in the current batch) * 100%; The dynamic threshold is the lower limit of the confidence interval calculated based on the historical matching success rate.

[0010] Optionally, a structured application form shall include at least the following fields: TraceID, root cause field name, responsible department code, scope of impact, and recommended actions; The responsible department code is pre-stored in the attributes of field-level nodes in the dynamic lineage graph.

[0011] Optionally, the processing solution can be associated and stored in the attributes of the corresponding node in the dynamic kinship graph for reuse during subsequent pre-verification, including: The treatment plan text is associated and stored in the governance record attribute of the corresponding root cause field node in the dynamic lineage graph; When the same node triggers the pre-verification again, the processing solution text is automatically loaded as a prompt message.

[0012] According to another aspect of the present invention, a data fusion and governance device for multiple business sectors of a power grid based on data lineage is provided, comprising: The module is used to generate field-level dynamic lineage relationships and construct a dynamic lineage graph by parsing the execution logic of the power grid data fusion task. The positioning module is used to locate the root cause field node that caused the anomaly by traversing backwards along the bloodline edge in the dynamic bloodline graph when the application system detects data anomalies. It also reads the responsible department code stored in the root cause field node, automatically generates a structured work order, and pushes it to the corresponding responsible department. The receiving module is used to receive the processing solutions fed back by the responsible departments, and to associate and store the processing solutions in the attributes of the corresponding nodes in the dynamic lineage graph for subsequent pre-verification and reuse.

[0013] According to another aspect of the present invention, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the methods described in any of the above aspects of the present invention.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the method described in any of the preceding aspects of the present invention.

[0015] Therefore, this invention constructs a field-level dynamic data lineage graph to analyze and record the transformation logic of the data fusion process; it pre-blocks based on historical attributes of the graph before the fusion task is executed, and dynamically circuit breakers are implemented based on streaming indicator monitoring during task execution, ensuring fusion quality from the source; for abnormal issues, it uses graph database traversal technology to locate the root cause node along the lineage link and achieve automatic responsibility assignment, driving closed-loop governance of online work orders, and feeding back the handling knowledge to the graph nodes to achieve continuous optimization of governance rules. This invention solves the pain points of uncontrollable quality and low governance efficiency in the data fusion process of multiple power grid sectors, significantly reduces the error rate of fusion connectivity, and greatly shortens the data problem handling cycle. Attached Figure Description

[0016] Exemplary embodiments of the present invention can be more fully understood by referring to the following figures: Figure 1 This is a flowchart illustrating a method for data fusion and governance of power grid multi-business sectors based on data lineage, provided in an exemplary embodiment of the present invention. Figure 2 This is an overall architecture diagram of a data fusion and governance method for power grids based on data lineage, provided by an exemplary embodiment of the present invention. Figure 3 This is a flowchart of a closed-loop process for improving the "fusion-governance" dual-process based on data lineage, provided by an exemplary embodiment of the present invention. Figure 4 This is a schematic diagram of the bloodline-driven intelligent responsibility assignment and governance work order generation logic provided in an exemplary embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a power grid multi-business data fusion and governance device based on data lineage provided in an exemplary embodiment of the present invention; Figure 6 This is the structure of an electronic device provided in an exemplary embodiment of the present invention. Detailed Implementation

[0017] Hereinafter, exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein.

[0018] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0019] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of the present invention are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0020] It should also be understood that in the embodiments of the present invention, "multiple" can refer to two or more, and "at least one" can refer to one, two or more.

[0021] It should also be understood that any component, data or structure mentioned in the embodiments of the present invention can generally be understood as one or more unless explicitly defined or given contrary instructions in the context.

[0022] Furthermore, the term "and / or" in this invention is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this invention generally indicates that the preceding and following related objects have an "or" relationship.

[0023] It should also be understood that the description of the various embodiments in this invention emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0024] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0025] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0026] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the specification.

[0027] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0028] The embodiments of this invention can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Well-known examples of terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0029] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0030] Exemplary methods Figure 1This is a flowchart illustrating a method for data fusion and governance of power grid multi-business sectors based on data lineage, provided in an exemplary embodiment of the present invention. This embodiment can be applied to electronic devices, such as… Figure 1 As shown, the data fusion and governance method 100 for power grid multi-business data based on data lineage includes the following steps: Step 101: By parsing the execution logic of the power grid data fusion task, a field-level dynamic lineage relationship is generated, and a dynamic lineage graph is constructed. Step 102: When the application system detects data anomalies, it traverses backward along the bloodline edges in the dynamic bloodline graph to locate the root cause field node that caused the anomaly, reads the responsible department code stored in the root cause field node, automatically generates a structured work order, and pushes it to the corresponding responsible department. Step 103: Receive the processing plan from the responsible department, and store the processing plan in the attribute of the corresponding node in the dynamic lineage graph for subsequent pre-verification.

[0031] Specifically, in view of the deficiencies in the aforementioned background technology, the present invention aims to solve the following specific technical problems: (1) Quality built-in issues in the fusion process: How to verify the quality of business integration between multi-business data in real time and automatically before and during the execution of data fusion tasks, based on the inherent logical relationship between data, and provide an active blocking mechanism to prevent "fusion with defects".

[0032] (2) Self-driven improvement of governance effectiveness: How to establish a collaborative governance closed loop driven by field-level data lineage, online and automatically assigning responsibilities, compressing the average handling cycle of data problems from "days" to "hours", and realizing the continuous reuse of governance knowledge.

[0033] This invention proposes an integrated technical solution of "foundation building, quality assurance through fusion, and efficiency improvement through governance". Its core is to construct a cross-professional, field-level, dynamically mapped data lineage graph, and use it as the core computing engine and knowledge base to drive the optimization of data fusion quality control and governance processes.

[0034] The overall system architecture is as follows Figure 1 As shown, it mainly includes the following three core modules: Module 1: Construction and Dynamic Mapping of Multi-Source Data Lineage Maps This module forms the infrastructure layer of the solution. Unlike traditional static metadata collection, this module possesses the following dynamic construction and mapping capabilities: 1) Multi-source entity extraction and graph modeling: Utilizing data platform interfaces or database log parsing technology, metadata from marketing, power distribution, and dispatching systems is extracted. Specifically, a graph database (such as Neo4j) is used for storage, including: Nodes are designed for advanced business application requirements that rely on the integrated operation, distribution, and dispatching systems (such as automatic load modeling, line loss calculation, a unified power grid map, and the three capabilities of the power grid: observability, measurability, and controllability). These are refined to field-level granularity (e.g., Marketing_User Table.UserID, GIS_Distribution Transformer Table.Commissioning Date). Node attributes include: field name, business meaning, code of the responsible department (e.g., dept_id: market_01), and data source system identifier.

[0035] Edges contain two types of relationships. The first is the physical dependency relationship of the source system (such as foreign key associations between tables); the second is the logical transformation relationship generated by the fusion task (such as: the user table.address is associated with the distribution table.station ID after word segmentation and cosine similarity calculation).

[0036] 2) Dynamic Mapping Mechanism: The key innovation of this module lies in parsing and recording the execution logic of the fusion task. When an ETL script or Spark SQL task is executed, the module analyzes the field mapping logic in the INSERT ... SELECT ... statement using an SQL parser (such as Apache Calcite or Antlr), and automatically generates a lineage edge in the graph database with FUSION_TASK_ID and preset attributes (the preset attributes can be, but are not limited to, TRANSFORM_LOGIC attributes, such as Func: Address_Match(addr_a, addr_b)). This allows the lineage graph to reflect the dynamic processing of data from the source to the fusion wide table in real time, providing a calculation basis for subsequent "in-process monitoring".

[0037] This stage aims to automatically capture the execution logic of data fusion tasks (such as ETL and Spark SQL) and transform it into knowledge within a graph. The complete process is shown in the diagram below and can be broken down into the following four key steps: Step 1: Capture and Parse When a data fusion task (e.g., a Spark SQL job or a Kettle transformation) starts on the scheduling platform, the "lineage graph construction module" of this invention captures the core INSERT...SELECT... statement text through a listener or Hook mechanism. Subsequently, the SQL parser (Apache Calcite) performs lexical and syntactic analysis on the statement, generating an Abstract Syntax Tree (AST), which precisely expresses the structure of the SQL.

[0038] Step 2: Semantic Analysis and Logical Plan Generation The parser performs semantic analysis on the AST in conjunction with the metadata catalog (known data tables and field definitions). For example, it verifies whether the fields selected by the SELECT statement exist in the source table and whether the target table structure matches the INSERT statement. This step generates a logical plan that clearly defines the mapping path of "which fields in which source tables the data comes from, and which fields in the target table the data goes to".

[0039] Step 3: Extract the transformation logic This is crucial for generating the TRANSFORM_LOGIC attribute. System depth traversal logic plan: For a simple mapping like SELECT a, b FROM src, the logic might be DIRECT_COPY.

[0040] For SELECT UPPER(name), addr_a || addr_b FROM src, the logic will be recorded as FUNC:UPPER and EXPR: CONCAT.

[0041] Regarding Address_Match(addr_a, addr_b) you mentioned, this is typically a user-defined function or a built-in complex matching function. The parser recognizes the function name Address_Match and its parameters addr_a and addr_b, using this function signature as the core identifier of the conversion logic. In more advanced implementations, the system can even reference the function definition source code or configuration documentation of this UDF to understand its business meaning (such as "fuzzy address matching based on Jaro-Winkler distance").

[0042] Step 4: Generate dynamic bloodline edges The system generates a globally unique FUSION_TASK_ID for this fusion task. Subsequently, based on each pair of "source field -> target field" mappings extracted from the logical plan, it creates or updates an edge in a graph database (such as Neo4j).

[0043] The starting point of this edge is the "source table.source field" node.

[0044] The endpoint of this edge is the "target merge wide table.target field" node.

[0045] The properties of an edge must include at least: ①FUSION_TASK_ID: The ID of this task.

[0046] ②TRANSFORM_LOGIC: Transition logic identifier (e.g., Address_Match).

[0047] ③EXPRESSION: Specific expression or parameter details (optional, used for deep tracing).

[0048] ④EXECUTION_TIME: Task execution timestamp.

[0049] By dynamically capturing and solidifying the field-level transformation logic of fusion tasks through an SQL parser, this invention elevates data lineage from the static "where the data comes from" to the dynamic "how the data is processed." This builds a solid, transparent, and verifiable foundation for advanced power grid analysis applications, such as load modeling, and represents a key technological breakthrough in data governance, moving from "extensive management" to "precise and intelligent governance."

[0050] Module 2: Fusion Process Quality Collaborative Control Engine This engine is embedded in the scheduling process of the data fusion task, using the lineage map constructed in Module 1 for segmented quality control. Its working mechanism is as follows: Figure 3 As shown: 1) Pre-emptive verification (pre-emptive blocking): Before a fusion task (e.g., executing a user-allocation change relational SQL) is started, the engine queries the edge attributes of the historical execution records of the task in the lineage graph. If the historical average matching success rate corresponding to the relational logic is lower than a preset threshold (e.g., 85%), or if the query finds that the null value rate of the key field (address) in the input source table (user table) exceeds 20%, the engine will proactively throw an exception and block the task from starting, sending an alert to the operations and maintenance personnel.

[0051] 2) In-process monitoring and feedback (in-process circuit breaker): During the execution of the fusion task, the engine uses a streaming processing framework (such as Spark Streaming or Flink) to sample and compute the data blocks being processed.

[0052] The specific calculation formula is as follows: Real-time matching rate = (Number of successfully matched records in the current batch / Total number of input records in the current batch) * 100% The engine compares the real-time matching rate with a dynamic threshold (the lower limit of the confidence interval calculated based on historical success rates). If the real-time matching rate drops sharply and falls below the lower limit of the dynamic threshold, the engine determines that an anomaly has occurred in the fusion process (such as a sudden change in the data format of the source system), immediately interrupts the task execution, and prevents erroneous data from being written to the downstream result table, thus achieving "fusion as quality inspection".

[0053] Module 3: Bloodline-Driven Intelligent Governance Closed-Loop Scheduling Center This module is responsible for the automated scheduling and efficiency improvement of the governance process, and its logic is as follows: Figure 4 As shown: 1) Root cause identification and automatic accountability: When an application system (such as a load modeling algorithm) detects that the calculation results of a certain distribution area D_transformer_distribution area A are continuously abnormal, the dispatch center receives an abnormal signal.

[0054] The scheduling center uses graph traversal algorithms (such as Cypher queries) to trace back along the fusion link in the lineage graph: MATCH (target: target wide table {name:'load value of area A'})<-[r:FUSION*1..5]-(source: source table field) RETURN source Once the root cause node leading to the anomaly is located (e.g., the daily frozen display data field in the marketing data sheet), the system automatically reads the node's attribute "responsible department code" to automatically assign responsibility without requiring manual investigation.

[0055] 2) Structural engineering and knowledge reuse: The center automatically generates structured work orders (JSON format) containing the following fields and pushes them to the data governance work order system of the corresponding responsible department: Json: { "TraceID": "GID_20241022_001", "RootCause": "Marketing data collection. Daily frozen data", "ResponsibleDept": "Marketing Department, Measurement Section", "ImpactScope": "Load model acquisition deviation for transformer area A", "SuggestedAction": "Check the communication status of the data collection terminal and retrieve historical data." } Knowledge loop: Once the responsible department has completed its processing and provided a solution (e.g., "re-acquired historical data on frozen electricity on load days"), the system will associate and store the solution text with the corresponding node in the lineage graph. In the future, when this node triggers pre-verification again, the engine will automatically load this historical solution, prompting maintenance personnel to prioritize this check, thus enabling intelligent reuse and iterative optimization of governance experience.

[0056] Figure 2 The document clearly illustrates three core modules: the multi-source data lineage mapping and dynamic mapping module, the fusion process quality collaborative control engine, and the lineage-driven intelligent governance closed-loop scheduling center. It also demonstrates their data interaction relationships with external data sources, the fusion task scheduling platform, application systems, and the governance work order system. Arrows indicate the bidirectional transmission of data and control flows.

[0057] Figure 3The timeline shows the complete closed-loop process from "lineage building -> integration task triggering -> pre-event verification -> in-event monitoring -> problem occurrence -> root cause tracing -> automatic responsibility assignment -> work order dispatch -> processing feedback -> knowledge accumulation", highlighting the interweaving and feedback of the two main lines of "integration quality assurance" and "governance efficiency improvement".

[0058] Among them, the core quality control formula is: Real-time matching rate = (Number of successfully associated records in the current batch / Total number of input records in the current batch) * 100%; Circuit breaker trigger condition: Real-time matching rate < lower limit of historical success rate confidence interval.

[0059] Figure 4 Using graph database query views as the core, this paper demonstrates how, after receiving an application exception, the dispatch center uses Cypher queries to traverse the lineage graph in reverse order along the edges, locates the root cause node with the responsible department code attribute, and automatically assembles and generates a structured JSON work order.

[0060] In a specific embodiment of the present invention, taking the task of associating user profiles with distribution transformer relationships in the scenario of "integrated operation and distribution data fusion" as an example, the implementation of the present invention will be further described in detail with specific algorithms and data.

[0061] Scene setting: Task: Execute a Spark SQL task every morning at midnight to join the T_MK_USER table of the marketing system with the T_PMS_TRANSFORMER table of the power distribution PMS system to generate a merged wide table T_FUSION_USER_TRANS.

[0062] Association logic: The levenshtein() function is used to calculate the text similarity between the description of "user's electricity address" and "transformer power supply range". The record with the highest similarity and greater than 0.8 is selected to establish an association.

[0063] 1. Specific implementation of fusion process control 1) Pre-verification stage: The control engine queried the lineage graph and found that the historical average matching rate when the task was last executed was only 78% (below the threshold of 85%). Simultaneously, the engine performed metadata probing on the input table T_MK_USER and found that the missing address field for the current day had a missing rate as high as 22%. The engine determined the risk to be too high, blocked the submission of the Spark task via the message queue, and pushed an alert to the data operations dashboard: "Task [ID:101] pre-validation failed: historical matching rate is too low and the source address field missing rate is abnormal. Please check the completeness of the marketing file entry." 2) In-process monitoring stage: After the operations and maintenance personnel supplemented the missing addresses, the task was retried. During the Spark task execution, the engine monitored the metrics data of the execution plan and calculated the real-time matching rate within the sliding window every 30 seconds. When the task processed the 5th partition, the engine detected that the matching rate of that partition suddenly dropped to 55%. The engine immediately called Spark's cancelJobGroup interface to interrupt the task. After investigation, it was found that the area corresponding to the partition data had recently undergone GIS model splitting, but the marketing files had not been updated synchronously. Because the task was interrupted in time, the generation of approximately 20,000 erroneous data associations was avoided.

[0064] 2. Specific Implementation of Governance Closed-Loop Triggering 1) Problem Discovery and Tracing: When load modeling was used to calculate line loss for a specific transformer area, the value was consistently negative. The application layer called the REST API provided by this invention, passing in the abnormal transformer area ID. The dispatch center then performed a graph query in the lineage graph (Cypher statement example): MATCH p=(trans:TRANSFORMER {id:'T123'})-[*..5]-(src:SOURCE_FIELD) WHERE src.name CONTAINS 'Battery' RETURN src.system, src.dept_id, src.name The query results show that the root cause of the problem points to the "Marketing Collection Database.Collection Table.Reverse Active Total Energy" field, with the responsible department code dept_id: market_02 (Marketing Department Metering Class).

[0065] 2) Work order closure and knowledge reuse: The dispatch center automatically assembles JSON work orders and dispatches them to the marketing department's work order system. When the metering team processes the work order, they fill in the following in the system feedback column: "On-site inspection revealed that the meter wiring polarity was reversed. This has been corrected, and the meter parameters have been reset." The system backend uses text semantic analysis to extract the keywords "meter wiring" and "polarity," and connects this processing solution as a new attribute edge SOLUTION to the marketing data acquisition database, data acquisition meter, and reverse active total energy node. Three months later, when the replacement task of this type of data acquisition device triggers the pre-verification again, the control engine automatically pops up a prompt: "Note: This model of equipment has a history of wiring polarity errors. It is recommended to focus on checking the meter wiring." Compared with the prior art, the application of this invention can achieve the following significant technical effects: (1) Quality control at the source of fusion: By embedding quality verification based on lineage graphs into the pre- and in-process stages of fusion tasks, "post-event governance" is transformed into "in-process control". Actual tests show that the continuity error rate of fusion results (such as incorrect connection of household-transformation relationships) can be reduced by more than 60%.

[0066] (2) Leapfrog improvement in governance efficiency: By utilizing the graph traversal calculation capability of the field-level lineage graph, the root cause location and automatic responsibility assignment of data problems can be achieved in seconds. Combined with the online chemical single closed-loop scheduling, the average cycle of cross-departmental collaborative problem handling is reduced from the traditional 2-3 days to 2-4 hours, and the governance efficiency is improved by more than 70%.

[0067] (3) Continuous accumulation of governance knowledge: The disposal experience is structured and associated with the lineage nodes, forming a governance knowledge base that is automatically updated with business changes. This solves the problem that traditional governance work relies on "expert experience" and that experience is easily lost, and realizes the sustainable iterative improvement of the overall effectiveness of data governance.

[0068] Exemplary device Figure 5 This is a schematic diagram of the structure of a power grid multi-business data fusion and governance device based on data lineage, provided in an exemplary embodiment of the present invention. Figure 5 As shown, the device 500 includes: Module 510 is used to generate field-level dynamic lineage relationships and construct a dynamic lineage graph by parsing the execution logic of the power grid data fusion task. The positioning module 520 is used to traverse the bloodline edge in the dynamic bloodline graph in reverse when the application system detects data anomalies, locate the root cause field node that caused the anomaly, read the responsible department code stored in the root cause field node, automatically generate a structured work order and push it to the corresponding responsible department. The receiving module 530 is used to receive the processing plan fed back by the responsible department, and store the processing plan in the attribute of the corresponding node in the dynamic lineage graph for subsequent pre-verification and reuse.

[0069] Optionally, the device 500 further includes: a query module, used to query the matching success rate corresponding to preset attributes of the historical execution records of the fusion task in the dynamic lineage graph before the fusion task is started; if the matching success rate is lower than a preset threshold; or Query the null value rate of key fields in the input source table. If the null value rate exceeds the threshold, then block the task from starting.

[0070] Optionally, the device 500 further includes: a calculation module, used to calculate the real-time matching rate during the execution of the fusion task, and compare the real-time matching rate with a dynamic threshold calculated based on the historical success rate; if the real-time matching rate falls below the lower limit of the dynamic threshold, the task execution is interrupted. Optionally, by parsing the execution logic of the power grid data fusion task, dynamic lineage relationships at the field level are generated, and a dynamic lineage graph is constructed, including: Metadata is extracted from multiple business systems, and an initial lineage graph is constructed using fields as nodes and physical dependencies or logical transformations as edges. Capture SQL statements in data fusion tasks, generate abstract syntax trees through SQL parsers, and generate logical plans through semantic analysis; Extract the transformation logic from source fields to target fields from the logical plan; Based on the initial lineage graph, a dynamic lineage graph is generated by creating lineage edges in the graph database that point from the source field node to the target field node and carry transformation logic attributes.

[0071] Optionally, the specific formula for calculating the real-time matching rate is as follows: Real-time matching rate = (Number of records successfully associated in the current batch / Total number of input records in the current batch) * 100%; The dynamic threshold is the lower limit of the confidence interval calculated based on the historical matching success rate.

[0072] Optionally, a structured application form shall include at least the following fields: TraceID, root cause field name, responsible department code, scope of impact, and recommended actions; The responsible department code is pre-stored in the attributes of field-level nodes in the dynamic lineage graph.

[0073] Optionally, the processing solution can be associated and stored in the attributes of the corresponding node in the dynamic kinship graph for reuse during subsequent pre-verification, including: The treatment plan text is associated and stored in the governance record attribute of the corresponding root cause field node in the dynamic lineage graph; When the same node triggers the pre-verification again, the processing solution text is automatically loaded as a prompt message.

[0074] Exemplary electronic devices Figure 6 This is the structure of an electronic device provided in an exemplary embodiment of the present invention. For example... Figure 6 As shown, the electronic device 60 includes one or more processors 61 and a memory 62.

[0075] The processor 61 may be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0076] The memory 62 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 61 may execute the program instructions to implement the methods of the software programs of the various embodiments of the present invention described above, and / or other desired functions. In one example, the electronic device may also include an input device 63 and an output device 64, these components being interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0077] In addition, the input device 63 may also include, for example, a keyboard, a mouse, etc.

[0078] The output device 64 can output various information to the outside. The output device 64 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0079] Of course, for the sake of simplicity, Figure 6 Only some of the components of this electronic device relevant to the present invention are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0080] Exemplary computer program products and computer-readable storage media In addition to the methods and apparatus described above, embodiments of the present invention may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods according to various embodiments of the present invention described in the "Exemplary Methods" section above.

[0081] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of the present invention. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0082] Furthermore, embodiments of the present invention may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the methods according to various embodiments of the present invention described in the "Exemplary Methods" section above.

[0083] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0084] The basic principles of the present invention have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in the present invention are merely examples and not limitations, and should not be considered as essential features of each embodiment of the present invention. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the present invention to the necessity of employing the aforementioned specific details.

[0085] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0086] The block diagrams of devices, systems, devices, and systems involved in this invention are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, systems, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0087] The methods and systems of the present invention may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of the present invention are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, the present invention may also be implemented as a program recorded on a recording medium, the program comprising machine-readable instructions for implementing the methods according to the present invention. Thus, the present invention also covers recording media storing programs for performing the methods according to the present invention.

[0088] It should also be noted that in the systems, apparatus, and methods of the present invention, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered equivalents of the present invention. The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the invention. Therefore, the invention is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0089] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the invention to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for data fusion and governance of multi-business data in power grids based on data lineage, characterized in that, include: By analyzing the execution logic of the power grid data fusion task, dynamic lineage relationships at the field level are generated, and a dynamic lineage graph is constructed. When the application system detects data anomalies, it traverses backward along the bloodline edges in the dynamic bloodline graph to locate the root cause field node that caused the anomaly, reads the responsible department code stored in the root cause field node, automatically generates a structured work order, and pushes it to the corresponding responsible department. The system receives the processing solution from the responsible department and stores the processing solution in the attribute of the corresponding node in the dynamic kinship map for subsequent pre-verification and reuse.

2. The method according to claim 1, characterized in that, The method further includes: Before the fusion task is started, the matching success rate corresponding to the preset attributes of the historical execution records of the fusion task in the dynamic lineage graph is queried. If the matching success rate is lower than a preset threshold, the task start is blocked; or Query the null value rate of key fields in the input source table. If the null value rate exceeds the threshold, then block the task from starting.

3. The method according to claim 1, characterized in that, The method further includes: during the execution of the fusion task, calculating the real-time matching rate and comparing the real-time matching rate with a dynamic threshold calculated based on the historical success rate; if the real-time matching rate falls below the lower limit of the dynamic threshold, then interrupting the task execution.

4. The method according to claim 1, characterized in that, By analyzing the execution logic of the power grid data fusion task, dynamic lineage relationships at the field level are generated, and a dynamic lineage graph is constructed, including: Metadata is extracted from multiple business systems, and an initial lineage graph is constructed using fields as nodes and physical dependencies or logical transformations as edges. Capture SQL statements in data fusion tasks, generate abstract syntax trees through SQL parsers, and generate logical plans through semantic analysis; Extract the transformation logic from the source field to the target field from the logical plan; Based on the initial lineage graph, lineage edges are created in the graph database, pointing from the source field node to the target field node and carrying the transformation logic attribute, to generate the dynamic lineage graph.

5. The method according to claim 3, characterized in that, The specific formula for calculating the real-time matching rate is as follows: Real-time matching rate = (Number of successfully associated records in the current batch / Total number of input records in the current batch) * 100%; The dynamic threshold is the lower limit of the confidence interval calculated based on the historical matching success rate.

6. The method according to claim 1, characterized in that, The structured work order shall contain at least the following fields: TraceID, root cause field name, responsible department code, scope of impact, and recommended actions; The responsible department code is pre-stored in the attributes of the field-level nodes in the dynamic lineage graph.

7. The method according to claim 1, characterized in that, The processing scheme is associated and stored in the attributes of the corresponding node in the dynamic kinship graph for reuse during subsequent pre-verification, including: The treatment plan text is associated and stored in the governance record attribute of the corresponding root cause field node in the dynamic lineage graph; When the same node triggers the pre-verification again, the processing scheme text is automatically loaded as a prompt message.

8. A data fusion and governance device for multiple business sectors of a power grid based on data lineage, characterized in that, include: The module is used to generate field-level dynamic lineage relationships and construct a dynamic lineage graph by parsing the execution logic of the power grid data fusion task; The positioning module is used to, when the application system detects data anomalies, traverse backwards along the bloodline edges in the dynamic bloodline graph to locate the root cause field node that caused the anomaly, read the responsible department code stored in the root cause field node, automatically generate a structured work order and push it to the corresponding responsible department. The receiving module is used to receive the processing plan fed back by the responsible department, and to associate and store the processing plan in the attribute of the corresponding node in the dynamic lineage map for subsequent pre-verification and reuse.

9. The apparatus according to claim 8, characterized in that, Also includes: The query module is used to query the matching success rate corresponding to preset attributes of the historical execution records of the fusion task in the dynamic lineage graph before the fusion task is started. If the matching success rate is lower than a preset threshold, the task start is blocked; or Query the null value rate of key fields in the input source table. If the null value rate exceeds the threshold, then block the task from starting.

10. The apparatus according to claim 8, characterized in that, Also includes: The calculation module is used to calculate the real-time matching rate during the execution of the fusion task, and compare the real-time matching rate with a dynamic threshold calculated based on the historical success rate. If the real-time matching rate falls below the lower limit of the dynamic threshold, the task execution is interrupted.

11. The apparatus according to claim 8, characterized in that, By analyzing the execution logic of the power grid data fusion task, dynamic lineage relationships at the field level are generated, and a dynamic lineage graph is constructed, including: Metadata is extracted from multiple business systems, and an initial lineage graph is constructed using fields as nodes and physical dependencies or logical transformations as edges. Capture SQL statements in data fusion tasks, generate abstract syntax trees through SQL parsers, and generate logical plans through semantic analysis; Extract the transformation logic from the source field to the target field from the logical plan; Based on the initial lineage graph, lineage edges are created in the graph database, pointing from the source field node to the target field node and carrying the transformation logic attribute, to generate the dynamic lineage graph.

12. The apparatus according to claim 10, characterized in that, The specific formula for calculating the real-time matching rate is as follows: Real-time matching rate = (Number of successfully associated records in the current batch / Total number of input records in the current batch) * 100%; The dynamic threshold is the lower limit of the confidence interval calculated based on the historical matching success rate.

13. The apparatus according to claim 8, characterized in that, The structured work order shall contain at least the following fields: TraceID, root cause field name, responsible department code, scope of impact, and recommended actions; The responsible department code is pre-stored in the attributes of the field-level nodes in the dynamic lineage graph.

14. The apparatus according to claim 8, characterized in that, The processing scheme is associated and stored in the attributes of the corresponding node in the dynamic kinship graph for reuse during subsequent pre-verification, including: The treatment plan text is associated and stored in the governance record attribute of the corresponding root cause field node in the dynamic lineage graph; When the same node triggers the pre-verification again, the processing scheme text is automatically loaded as a prompt message.

15. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for performing the method described in any one of claims 1-7.

16. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1-7.

Images