Power enterprise data full-link fault positioning method based on data consanguinity

Abstract

The invention relates to a power enterprise data full-link fault positioning method based on data consanguinity. The method comprises the following steps: step 1, constructing a data consanguinity map; 2, based on the data consanguinity map constructed in the step 1, metadata consanguinity full links are collected and monitored in real time; 3, based on the data blood relationship map constructed in the step 1 and the real-time monitoring data collected in the step 2, a state transition matrix is constructed by analyzing state change information, and a Markov chain model is defined; by using the Markov chain model, the system can identify potential fault points and carry out accurate positioning. According to the invention, the problems of high data link complexity, difficulty in fault positioning, low troubleshooting efficiency and the like in the prior art can be solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data fault location, and relates to a method for locating a full-link fault of data in an electric power enterprise, and in particular to a method for locating a full-link fault of data in an electric power enterprise based on data lineage. Background Art

[0002] With the rapid development of big data technology, enterprises have accumulated a huge amount of data in their daily operations. These data cover various aspects such as user behavior, market trends, product performance, and financial data. However, as the amount of data continues to grow, the complexity of the data link has also increased significantly, bringing unprecedented challenges to the data management of enterprises. The data link refers to the entire life cycle of data from generation to consumption within the enterprise, including multiple links such as data generation, transmission, storage, processing, analysis, and display. In this complex environment, enterprises face many problems in the process of data management.

[0003] First, the current data link fault location capability is limited, and usually can only identify the problem in a single link, but cannot conduct a comprehensive impact analysis. This means that when a node fails, it is difficult to quickly assess its potential impact on upstream and downstream processes and even the entire business system. The lack of fault diagnosis from a full-link perspective not only prolongs the time for troubleshooting, but also makes it difficult for enterprises to take timely measures to reduce or avoid greater losses caused by chain reactions. This limitation significantly increases recovery time and has a serious impact on the business continuity and stability of enterprises.

[0004] In addition, traditional data management methods rely heavily on manual operations and empirical judgment, which is inadequate in the face of the increasing number of links. The new data links added each year further increase the workload of operation and maintenance personnel, resulting in rising maintenance costs year by year. At the same time, manual troubleshooting is inefficient and prone to errors, which not only prolongs the time required to solve the problem, but may also cause additional losses or system damage due to misjudgment, exacerbating the interference with normal business operations.

[0005] At the same time, the complexity of data links can easily lead to business islands and data redundancy within the enterprise. Due to the poor data flow between different business departments, each department has formed an independent data system, which is unable to achieve effective data sharing and collaboration, and eventually forms a "data island". This situation not only hinders the full utilization of data resources by the enterprise as a whole, but also causes the phenomenon of duplicate data storage, which wastes precious storage space and increases unnecessary management complexity, thereby reducing the overall operational efficiency of the enterprise.

[0006] At present, some technical solutions locate link failures by detecting anomalies in network traffic. These solutions usually use methods such as rule matching and pattern recognition to analyze the characteristics of network traffic, such as delay and packet loss rate. Once abnormal traffic is detected, the system will trigger an alarm and try to locate the fault point in the link. However, the disadvantage of such solutions is that they require manual setting of rules or thresholds, which leads to the inability to accurately reflect the actual situation when dealing with complex and changing network environments. In addition, since these solutions mainly rely on traffic characteristics for judgment, it is difficult to detect potential faults in time when the network traffic is normal but the link quality deteriorates, resulting in inaccurate fault location or even missed opportunities for fault location.

[0007] Another existing solution uses a machine learning model to locate link faults. This solution is trained with a large amount of historical network data to learn the patterns and rules in order to locate possible link faults in real time based on current data in future network monitoring. Although machine learning technology has achieved certain results in link fault location, the accuracy of its model depends on a large amount of high-quality training data. In addition, traditional machine learning models are usually black box models, and it is difficult to explain their location results, which is not conducive to further fault analysis and troubleshooting. At the same time, when faced with a complex network environment, relying solely on historical data for positioning has limited effect and is difficult to cope with dynamically changing network topology structures.

[0008] In summary, with the continuous growth of enterprise data volume and the increasingly complex data link structure, there is an urgent need for a full-link fault location method for power enterprise data.

[0009] After searching, no public documents of the prior art identical or similar to the present invention were found. Summary of the invention

[0010] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a method for locating faults in the entire data link of an electric power enterprise based on data lineage, which can solve the problems of high data link complexity, difficulty in fault locating, and low troubleshooting efficiency in the prior art.

[0011] The present invention solves the practical problem by adopting the following technical solutions:

[0012] A method for locating faults in the entire data link of a power enterprise based on data lineage includes the following steps:

[0013] Step 1: Construct a data lineage map;

[0014] Step 2: Based on the data lineage graph constructed in step 1, collect and monitor the entire metadata lineage chain in real time;

[0015] Step 3: Based on the data lineage map constructed in step 1 and the real-time monitoring data collected in step 2, the state change information is analyzed to construct a state transition matrix and define a Markov chain model. Using the Markov chain model, the system can identify potential fault points and accurately locate them.

[0016] Moreover, the specific method of step 1 is:

[0017] First, by parsing the transaction information in the system operation log, adopting the metadata management principle, and combining multi-dimensional analysis such as table name similarity, table structure similarity and field overlap, an accurate data lineage map is constructed.

[0018] Moreover, the specific method of step 2 is:

[0019] Based on the construction of the data lineage graph, log collection or event tracking tools are deployed at key nodes to capture system runtime information, including execution time, status, and error codes, to achieve full-link collection and real-time monitoring of metadata lineage.

[0020] Moreover, the specific steps of step 3 include:

[0021] (1) Extract state change data from the existing real-time monitoring system and store it in the database, including timestamp, link ID, component status, service health status, and error log;

[0022] (2) Based on the extracted state change information, a state transition matrix is ​​constructed to calculate the probability of each state transferring to another state;

[0023] (3) Construct a Markov chain model to reflect the transition probability between different states;

[0024] (4) Using the Markov chain model, the system can identify potential fault points and accurately locate them.

[0025] Moreover, the specific steps of step 3 (2) include:

[0026] ①Determine the state set

[0027] First, we need to determine all possible states and summarize them from the link state change information extracted from the log; suppose we have the following state set: S = {s1, s2, s3, …, s n}.

[0028] ②Count the number of state transitions

[0029] By traversing all transaction logs, we count the number of transitions from one state to another. For each pair of states (s i ,s j), record status s i s j This is achieved through a two-dimensional array C, where C[i][j] represents the state s i Transfer to state s j The number of times, and introduce a weight factor w ij ,This factor takes into account the changing trends of load, voltage and current;

[0030] w ij =α·load similarity(s i ,s j )+β·voltage similarity(s i ,s j )+γ

[0031] ·current similarity(s i ,s j )

[0032] Among them, α, β and γ are weight coefficients, which represent the importance of load, voltage and current similarity respectively. load_similarity, voltage_similarity and current_similarity are functions used to measure the similarity between two states in terms of load, voltage and current, which are also calculated by cosine similarity of formula (3).

[0033] ③Calculate state transition probability

[0034] Based on the counted number of transitions, the probability of state transition is calculated; each element P[i][j] in the state transition probability matrix P represents the transition from state s i Transfer to state s j The probability is calculated as follows:

[0035]

[0036] If the state s i If no transfer occurs (i.e., all C[i][k] = 0), then we can set Represents a uniform distribution.

[0037] Moreover, the specific method of step 3 (3) is:

[0038] ①Define the state space Define the state space S = {s1, s2, s3, ..., s n}, that is, the set of all possible states.

[0039] ②Define state transition probability

[0040] Define the state transition probability matrix P = [P[i][j]], which describes the probability of transitioning from one state to another.

[0041] ③Define the initial state distribution

[0042] Define an initial state distribution vector π=[π1,π2,…,π n ], which is a probability vector that represents the probability of the system being in each state at the beginning.

[0043] ④Simulation of Markov chain model

[0044] Based on the defined state space, state transition probability matrix and initial state distribution, the evolution process of the Markov chain can be simulated.

[0045] Moreover, the specific method of step (4) of step 3 is:

[0046] ① Determine the potential fault state set:

[0047] According to the state transition probability matrix P, combined with link logs and service characteristics, the possible set of potential fault states is identified;

[0048] ②Backtrack and analyze the state transfer path:

[0049] For each potential fault state, trace back its state transition path to locate the possible fault source:

[0050] 1) Extract state transition sequence:

[0051] Extract the current state from the link log f The predecessor state set s prev , and analyze its transfer to s f Path characteristics.

[0052] 2) Analyze key transfer points:

[0053] For each predecessor state s i ∈S prev , calculate its transfer to s f The weight factor w ij and the number of transfers C[i][j] if w ij If the value of C[i][j] is abnormally high, it is recorded as a possible cause of failure. If C[i][j] is abnormally frequent, the corresponding operation time and related upstream and downstream components are recorded.

[0054] 3) Locate the scope of influence based on data blood relationship:

[0055] Tracking fault status through data lineage graph f For the propagation paths of upstream and downstream nodes, clarify the chain reactions that may be triggered in the link.

[0056] ③ Refine fault location, combine the above analysis, and further clarify the fault location scope:

[0057] 1) Combine state transition features with blood relationship cross validation

[0058] If the fault state s f The upstream and downstream nodes of the ij Verify the transmission direction of the abnormal impact and locate whether the fault point is an internal problem of the node or an external propagation problem.

[0059] 2) Digging deeper into the similarity between states

[0060] Make a detailed comparison of the load, voltage and current changes at the potential fault point to clearly distinguish it from other states and ensure the accuracy of the positioning results.

[0061] 3) Generate a fault location report, record the final confirmed fault point, including the specific status f , path characteristics, triggering conditions and affected scope.

[0062] Through the above steps, the system successfully identified the potential fault points in the link and provided accurate positioning results and detailed upstream and downstream impact analysis.

[0063] Advantages and beneficial effects of the present invention:

[0064] 1. The present invention proposes a method for locating faults in the entire link of power enterprise data, which can support full-link impact analysis, that is, when an abnormality is detected somewhere, the location of the fault point can be quickly determined, helping repair personnel to quickly locate the fault, thereby significantly shortening the mean time to repair (MTTR) and reducing the negative impact caused by long-term downtime. The present invention can help enterprises manage and utilize their valuable information assets more efficiently in the challenging big data era.

[0065] 2. Accurate data lineage map construction: By comprehensively calculating the metadata table name similarity, structure similarity and field overlap, the data lineage relationship is effectively constructed. The cosine similarity is used to evaluate the table name similarity, and combined with the table structure comparison and field weight analysis, the related data tables are accurately identified. This method significantly improves the metadata quality, provides a solid foundation for data governance, and ensures the accuracy and reliability of the data link.

[0066] 3. Efficient troubleshooting: The state change information extracted from the log is used to construct the state transition matrix, and the Markov chain model is defined to identify potential fault points. This allows the fault source to be discovered and accurately located in time even when the network traffic is normal but the link quality deteriorates, thereby greatly improving the troubleshooting efficiency and reducing the problem repair time.

[0067] 4. Optimized data management and reduced redundancy: By classifying data nodes and calculating the degree of attribute influence to select key attributes, and aligning data accordingly, the number of duplicate data nodes is effectively reduced. This method not only ensures the accuracy and comprehensiveness of data lineage results, but also helps solve the problem of data islands formed within the enterprise due to poor data flow, promotes data sharing and collaboration, and thus reduces the complexity of data management and waste of storage resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 is a process flow chart of the present invention;

[0069] Figure 2 A structural diagram is constructed for the data lineage relationship of the present invention. DETAILED DESCRIPTION

[0070] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings:

[0071] A method for locating faults in the entire data chain of power enterprises based on data lineage, such as Figure 1 As shown, the following steps are included:

[0072] Step 1: Construct a data lineage map;

[0073] The specific method of step 1 is:

[0074] First, by parsing the transaction information in the system operation log, adopting the metadata management principles (comprehensive coverage, complete information, and real-time update), and combining multi-dimensional analysis such as table name similarity, table structure similarity, and field overlap, an accurate data lineage map is constructed.

[0075] In this embodiment, this basic step ensures the accuracy and reliability of the data link and provides necessary data support and theoretical basis for subsequent fault location.

[0076] In this embodiment, metadata management is very important in global data governance and application. In order to improve the metadata quality of data lineage analysis and provide a data foundation and quality assurance for the establishment of data links, three basic principles should be followed: First, metadata registration should cover all objects in the data supply architecture, including source data, data components, etc., and should be registered in a standardized manner from multiple source modules such as data acquisition to data service bus; second, the metadata information items of each type of object must be complete, covering all elements such as business attributes, data attributes, management attributes and historical versions to support data standards and lineage traceability capabilities; finally, following the principles of prior registration and real-time updating, metadata should be defined and registered in the data configuration design stage to ensure that the metadata database can be updated in a timely manner as the data design changes, thereby ensuring data quality and availability.

[0077] There are some metadata tables with association relationships in the database. The names of these tables are often very similar. The name of one of the tables may be obtained by transforming the name of another table. Therefore, the similarity between table names has become an important reference indicator for calculating data lineage. In addition, the structures of tables with association relationships are usually very similar. Derived tables are often created based on an original table by adding new fields, modifying existing fields, or deleting certain fields. When calculating data lineage, the similarity of table structures is also a key factor. The method proposed in this paper is based on the similarity analysis of data tables. The construction of data lineage takes into account the characteristics of data table names and table structures. This process is achieved by calculating the similarity of table names, table structures, and field overlap. A comprehensive evaluation is performed based on the calculation results of these three factors, and a reasonable judgment threshold is set through continuous testing and evaluation. Finally, these thresholds are used to determine whether there is a data lineage relationship between the two data tables. The structure diagram is shown in Figure 2 shown.

[0078] The table name similarity calculation uses cosine similarity. First, preprocess the table to ensure that all fields have the same name. If a field does not exist in one table, fill the missing value in the corresponding position in the other table, such as 0 or other appropriate default values. Then assign weights to the fields in each table, which can be adjusted according to actual needs. For example, you can give numeric fields a higher weight and text fields a lower weight. Then convert each table into a vector form, where the dimension of the vector is equal to the total number of fields, and the value of the vector represents the weight or importance of the field. Finally, use the cosine similarity formula to calculate the similarity between the two vectors.

[0079] Assume that we have two data tables A and B, each table contains several fields, each of which contains time series data (e.g., load, voltage, current, etc.). The features of these time series data are incorporated into the similarity calculation.

[0080] Let W A and W B are the field weight vectors of Table A and Table B respectively. The weight of each field reflects the importance of the field in the data table. A =[w A1 ,w A2 ,…,w An ],W B =[w B1 ,w B2 ,…,w Bn ].

[0081] Assume T A and T B are the time series vectors of Table A and Table B respectively, each of which contains data at multiple time points. Consider the time series data T of load, voltage and current A =[t A1 ,t A2 ,…,t An ], T B =[t B1 ,t B2 ,…,t Bn ]. Among them, each t Ai and t Bi is a vector containing time points, for example: t Ai =[L Ai (t1),V Ai (t1),I Ai (t1),L Ai (t2),V Ai (t2),I Ai (t2),…]. Where L Ai (t k ) represents field i at time point t k The load value, V Ai (t k ) represents the voltage value, I Ai (t k ) indicates the current value.

[0082] In order to make the length of the time series vector 1, which is convenient for combining with other weight vectors, the time series vector is normalized:

[0083]

[0084] Combining the field weights and time series weights, we define the comprehensive weight vector V A and V B :

[0085]

[0086] where w t is a coefficient that adjusts the importance of the time series weight.

[0087] Finally, the cosine similarity between the two vectors is calculated:

[0088]

[0089] The meaning of the formula:

[0090] Dot product: represents the sum of the products of two vectors in all dimensions.

[0091] Module: It represents the length of a vector, which is the square root of the sum of the squares of its components.

[0092] Cosine similarity: represents the cosine value of the angle between two vectors, ranging from [-1, 1].

[0093] When two vectors are completely parallel, the cosine similarity is 1; when two vectors are completely opposite, the cosine similarity is -1; when the two vectors are orthogonal, the cosine similarity is 0.

[0094] Application example: Assume that there are two tables, Table1 and Table2, which contain the following fields: Field1, Field2, Field3. Convert each table into a vector form and assign corresponding weights. Assume that the weight corresponding to Field1 is w1, the weight corresponding to Field2 is w2, and the weight corresponding to Field3 is w3. You can calculate their dot product and their respective moduli, and then substitute them into the cosine similarity formula to solve. The final cos_sim is the cosine similarity of Table1 and Table2. Determine whether the similarity standard is met based on the set threshold, so as to determine whether there is a data kinship relationship between them.

[0095] Step 2: Based on the data lineage graph constructed in step 1, collect and monitor the entire metadata lineage chain in real time;

[0096] The specific method of step 2 is:

[0097] Based on the construction of the data lineage graph, log collection or event tracking tools are deployed at key nodes to capture system runtime information, including execution time, status, and error codes, to achieve full-link collection and real-time monitoring of metadata lineage.

[0098] In this embodiment, the solution can not only capture information during system operation, but also ensure the real-time and integrity of data through efficient database storage technology. At the same time, by building a metadata management platform and a history record preservation mechanism that supports version control, as well as a rule engine based on business logic and performance indicators, automatic alarms for abnormal situations and in-depth problem mining are achieved, preparing a sufficient data foundation and analysis tools for the next step of fault location.

[0099] After building an accurate data lineage map, the next key step is to achieve the collection and real-time monitoring of the entire metadata lineage link. The construction of data lineage relationships provides a solid foundation for the collection and real-time monitoring of the entire link, ensuring the accuracy and reliability of the data. The data lineage map can clearly understand the flow and dependency of data between different links, which is crucial for real-time monitoring and fault location.

[0100] Based on the construction of data lineage, an integrated solution was designed and implemented to achieve comprehensive collection of metadata lineage links. By deploying log collection or event tracking tools (such as Apache Flume, Logstash, etc.) at key nodes, system runtime information, including execution time, status, and error codes, is captured. These tools can adapt to different data sources, such as data lakes, ETL jobs, data warehouses, and API calls, to ensure comprehensive coverage of the entire data processing process. After preprocessing, the collected data is stored in an efficient time series database (such as InfluxDB) or NoSQL database (such as Cassandra) to support fast query and analysis. At the same time, a dedicated metadata management platform is built to manage and maintain metadata information. The platform has version control functions and can save historical records for traceability. This platform not only ensures real-time updates of data, but also provides technical support for the continuous maintenance of data lineage.

[0101] In order to achieve effective real-time monitoring, a rule engine based on business logic and performance indicators is used to automatically trigger alarms when abnormal situations are detected. In addition, data analysis models are used to identify potential problem patterns, and deep mining is combined with blood relationship maps to discover hidden dangers between upstream and downstream dependencies. All these analysis results will be presented to users through intuitive dashboards, providing flexible filtering conditions to customize the viewing scope, and immediately notifying relevant personnel once a problem is found.

[0102] Finally, the entire system also sets up strict security and permission control measures to ensure that only authorized users can access sensitive data, and uses encryption transmission technology to protect data security. Through this series of steps, not only the operation and maintenance efficiency is improved, but also the understanding and control of the complex IT environment within the enterprise is enhanced.

[0103] Step 3: Based on the data lineage map constructed in step 1 and the real-time monitoring data collected in step 2, the state change information is analyzed to construct a state transition matrix and define a Markov chain model. Using the Markov chain model, the system can identify potential fault points and accurately locate them.

[0104] The specific steps of step 3 include:

[0105] (1) Extract state change data from the existing real-time monitoring system and store it in the database, including timestamp, link ID, component status, service health status, and error log;

[0106] (2) Based on the extracted state change information, a state transition matrix is ​​constructed to calculate the probability of each state transferring to another state;

[0107] The specific steps of step (2) of step 3 include:

[0108] ①Determine the state set

[0109] First, we need to determine all possible states and summarize them from the link state change information extracted from the log; suppose we have the following state set: S = {s1, s2, s3, …, s n}.

[0110] ②Count the number of state transitions

[0111] By traversing all transaction logs, we count the number of transitions from one state to another. For each pair of states (s i ,s j ), record status s i s j This is achieved through a two-dimensional array C, where C[i][j] represents the state s i Transfer to state s j The number of times, and introduce a weight factor w ij ,This factor takes into account the changing trends of load, voltage and current;

[0112] w ij =α·load similarity(s i ,s j )+β·voltage similarity(s i ,s j )+γ

[0113] ·current similarity(s i ,s j )

[0114] Among them, α, β and γ are weight coefficients, which represent the importance of load, voltage and current similarity respectively. load_similarity, voltage_similarity and current_similarity are functions used to measure the similarity between two states in terms of load, voltage and current, which are also calculated by cosine similarity of formula (3).

[0115] ③Calculate state transition probability

[0116] Based on the counted number of transitions, the probability of state transition is calculated; each element P[i][j] in the state transition probability matrix P represents the transition from state s i Transfer to state s j The probability is calculated as follows:

[0117]

[0118] If the state s i If no transfer occurs (i.e., all C[i][k] = 0), then we can set Represents a uniform distribution.

[0119] (3) Construct a Markov chain model to reflect the transition probability between different states;

[0120] The specific method of step (3) of step 3 is:

[0121] Using the state transition matrix to define a Markov chain With the state transition probability matrix P, we can define a Markov chain. A Markov chain is a random process in which the future state depends only on the current state and has nothing to do with the past state.

[0122] The Markov chain model is defined as follows:

[0123] 1. Define the state space

[0124] Define the state space S = {s1,s2,s3,…,s n}, that is, the set of all possible states.

[0125] 2. Define state transition probability

[0126] Define the state transition probability matrix P = [O[i][j]], which describes the probability of transitioning from one state to another.

[0127] 3. Define the initial state distribution

[0128] Define an initial state distribution vector π=[π1,π2,…,π n], which is a probability vector that represents the probability of the system being in each state at the beginning.

[0129] 4. Simulating Markov chain model

[0130] Based on the defined state space, state transition probability matrix and initial state distribution, the evolution process of the Markov chain can be simulated.

[0131] For example, if the current state is s i , then the next state s j It is from i The probability of transitioning out of a state is given by P[i][j].

[0132] (4) Using the Markov chain model, the system can identify potential fault points and accurately locate them;

[0133] This step not only improves the efficiency of troubleshooting and shortens the time to fix problems, but also enhances the stability and reliability of the system, directly reflecting the value and significance of the first two steps.

[0134] After achieving the collection and real-time monitoring of the entire metadata lineage link, the next key step is to build a Markov chain model for fault location. The collection and real-time monitoring of the entire metadata lineage link provides rich state change information and real-time data support for the construction of the Markov chain model, ensuring the accuracy and effectiveness of the model. Through real-time monitoring of the link state change information obtained, the state transition matrix can be constructed more accurately, thereby improving the accuracy of fault location.

[0135] The specific method of step (4) of step 3 is:

[0136] 1. Determine the set of potential failure states

[0137] According to the state transition probability matrix P, combined with link logs and service characteristics, the possible set of potential fault states is identified;

[0138] 2 Backtracking Analysis of State Transfer Path

[0139] For each potential fault state, trace back its state transition path to locate the possible fault source:

[0140] 1) Extract state transition sequence

[0141] Extract the current state from the link log f The predecessor state set s prev , and analyze its transfer to s f Path characteristics.

[0142] 2) Analyze key transfer points

[0143] For each predecessor state s i ∈S prev , calculate its transfer to s f The weight factor w ij and the number of transfers C[i][j] if w ij If the value of C[i][j] is abnormally high, it is recorded as a possible cause of failure. If C[i][j] is abnormally frequent, the corresponding operation time and related upstream and downstream components are recorded.

[0144] 3) Locate the impact scope based on data blood relationship

[0145] Tracking fault status through data lineage graph f For the propagation paths of upstream and downstream nodes, clarify the chain reactions that may be triggered in the link.

[0146] 3. Refined fault location

[0147] Based on the above analysis, the fault location scope is further clarified:

[0148] (1) Combining state transition features with blood relationship cross-validation

[0149] If the fault state s f The upstream and downstream nodes of the ij Verify the transmission direction of the abnormal impact and locate whether the fault point is an internal problem of the node or an external propagation problem.

[0150] (2) Digging Deeper into the Similarity Between States

[0151] Make a detailed comparison of the load, voltage and current changes at the potential fault point to clearly distinguish it from other states and ensure the accuracy of the positioning results.

[0152] (3) Generate a fault location report, recording the final confirmed fault point, including the specific status f , path characteristics, triggering conditions and affected scope.

[0153] Through the above steps, the system successfully identified the potential fault points in the link and provided accurate positioning results and detailed upstream and downstream impact analysis.

[0154] Embodiment 1:

[0155] The data lineage-based method for locating full-link faults in power enterprise data builds a data lineage map through the collected metadata table, and uses the Markov chain model to locate link faults, thereby improving data audit efficiency and fault location accuracy. The following will describe in detail the specific process of this method and its practical application steps, including the establishment and use of the data link fault analysis model and the data similarity analysis model.

[0156] 1. Data lineage relationship construction:

[0157] In order to improve the metadata quality of data lineage analysis and ensure the accuracy and reliability of data links, this method follows three basic principles: comprehensive coverage of all data objects, complete recording of various information items, and real-time metadata updates. For metadata tables with associations in the database, the data lineage relationship is comprehensively evaluated by calculating the table name similarity (using cosine similarity), table structure similarity, and field overlap. In the preprocessing stage, the tables are standardized and field weights are assigned; then each table is converted into a vector form, and the cosine similarity is calculated based on these vectors. Finally, a reasonable threshold is set based on the results of the three indicators to determine whether there is a lineage relationship between the two tables, thereby effectively supporting global data governance and application.

[0158] 2. Data collection and real-time monitoring of the entire lineage

[0159] Design an integrated solution to achieve the collection and real-time monitoring of the entire chain of metadata lineage: Deploy log collection or event tracking tools at key nodes to capture runtime information, ensuring coverage of all data sources from data lakes to API calls; store preprocessed data in an efficient time series database, and build a dedicated metadata management platform to support version control and historical record preservation; develop a rule engine based on business logic and performance indicators to automatically trigger alarms, and use data analysis models combined with lineage relationship maps to identify potential problems. All analysis results are displayed to users through intuitive dashboards, and support custom filtering and instant notifications.

[0160] 3. Markov chain construction

[0161] Storing link status: storing the extracted link status change information in the database, including timestamp, link ID, status, etc.

[0162] Construct a state transition matrix: count the number of transitions from one state to another and calculate the state transition probability.

[0163] Based on the extracted state change information, a state transfer matrix is ​​constructed to record the probability of each state transferring to another state. The state transfer matrix construction process is as follows:

[0164] [1]. Determine the state set

[0165] First, we need to determine all possible states. We can summarize the link state change information extracted from the log. Assume that we have the following state set: S = {s1, s2, s3, …, s n}.

[0166] [2] Counting the number of state transitions

[0167] By traversing all transaction logs, we count the number of transitions from one state to another. For each pair of states (s i ,s j ), record status s i s j This can be achieved by a two-dimensional array C, where C[i][j] represents the state s i Transfer to state s j The number of times, and introduce a weight factor w ij , which takes into account the changing trends of load, voltage and current.

[0168] w ij =α·load similarity(s i ,s j )+β·voltage similarity(s i ,s j )+γ

[0169] ·current similarity(s i ,s j )

[0170] Among them, α, β, and γ are weight coefficients, which represent the importance of load, voltage, and current similarity, respectively. load_similarity, voltage_similarity, and current_similarity are functions used to measure the similarity between two states in terms of load, voltage, and current, which are also calculated by the cosine similarity of formula 3.

[0171] [3]. Calculate the state transition probability

[0172] Based on the number of transitions counted, the probability of state transition is calculated. Each element P[i][j] in the state transition probability matrix P represents the transition from state s i Transfer to state s j The probability is calculated as follows:

[0173]

[0174] If the state s i If no transfer occurs (i.e., all C[i][k] = 0), then we can set , indicating uniform distribution.

[0175] Define Markov chain: Define the Markov chain model based on the state transition probability matrix.

[0176] Based on the defined state space, state transition probability matrix and initial state distribution, the evolution process of the Markov chain can be simulated. For example, if the current state is s i , then the next state s j It is from i The probability of transitioning out of a state is given by P[i][j].

[0177] 4. Fault location

[0178] Identify fault points: Identify possible fault points based on the results and locate them accurately.

[0179] The process of fault location using Markov chain can be divided into the following steps:

[0180] [1]. Determine the set of potential fault states

[0181] According to the state transition probability matrix P, combined with link logs and service characteristics, the possible potential fault state set is identified. The specific operations are as follows:

[0182] For each state s i The transition probability P[i][j] is analyzed to see whether it has abnormal characteristics, for example:

[0183] oThe transition probability is concentrated to abnormal states, such as link interruption, high load, etc.

[0184] oThe transfer probability deviates significantly from the historical normal range.

[0185] Mark the states that meet the abnormal characteristics as potential fault states, recorded as set S fault ={s f |s f

[0186] Satisfies abnormal characteristics}.

[0187] [2] Backtracking and analyzing state transition paths

[0188] For each potential fault state s f ∈S fault , trace back its state transition path to locate the possible source of the fault:

[0189] 1) Extract state transition sequence

[0190] Extract the current state from the link log f The predecessor state set s prev , and analyze its transfer to s f Path characteristics.

[0191] 2) Analyze key transfer points

[0192] For each predecessor state s i ∈S prev , calculate its transfer to s f The weight factor w ij And the number of transfers C[i][j]:

[0193] If ij Abnormally high values ​​(such as abnormally high similarity of load, voltage or current) are recorded as possible causes of faults.

[0194] oIf C[i][j] is abnormally frequent, the corresponding operation time and associated upstream and downstream components are recorded.

[0195] 3) Locate the impact scope based on data blood relationship

[0196] Tracking fault status through data lineage graph f For the propagation paths of upstream and downstream nodes, clarify the chain reactions that may be triggered in the link.

[0197] [3]. Fine-grained fault location

[0198] Based on the above analysis, the fault location scope is further clarified:

[0199] 1) Combine state transition features with blood relationship cross validation

[0200] If the fault state s f The upstream and downstream nodes of the ij Verify the transmission direction of the abnormal impact and locate whether the fault point is an internal problem of the node or an external propagation problem.

[0201] 2) Digging deeper into the similarity between states

[0202] Make a detailed comparison of the load, voltage and current changes at the potential fault point to clearly distinguish it from other states and ensure the accuracy of the positioning results.

[0203] 3) Generate a fault location report

[0204] Record the final confirmed fault point, including the specific status f , path characteristics, triggering conditions and affected scope.

[0205] [4]. Complete the identification and accurate positioning of potential fault points

[0206] Through the above steps, the system successfully identified the potential fault points in the link and provided accurate positioning results and detailed upstream and downstream impact analysis. The final results include:

[0207] · The status and transition characteristics of potential failure points.

[0208] · Fault propagation path and associated upstream and downstream nodes.

[0209] · Clear cause of failure.

[0210] Through the above steps, the system can automatically extract link status change information from the system operation log and build a data lineage map. It not only utilizes the rich data provided by the full-link collection and real-time monitoring of metadata lineage, but also improves the accuracy of fault location through the Markov chain model. This succession relationship ensures the efficiency and accuracy of fault location and further enhances the stability and reliability of the system.

[0211] It should be emphasized that the embodiments described in the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific implementation manner. Any other implementation manners derived by those skilled in the art based on the technical solution of the present invention also fall within the scope of protection of the present invention.

Claims

1. A method for locating faults in the entire data chain of a power enterprise based on data lineage, characterized by: The following steps are involved: Step 1: Construct a data lineage map; Step 2: Based on the data lineage graph constructed in step 1, collect and monitor the entire metadata lineage chain in real time; Step 3: Based on the data lineage map constructed in step 1 and the real-time monitoring data collected in step 2, the state change information is analyzed to construct a state transition matrix and define a Markov chain model. Using the Markov chain model, the system can identify potential fault points and accurately locate them.

2. According to claim 1, a method for locating faults in the entire data link of a power enterprise based on data lineage, characterized in that: The specific method of step 1 is: First, by parsing the transaction information in the system operation log, adopting the metadata management principle, and combining multi-dimensional analysis such as table name similarity, table structure similarity and field overlap, an accurate data lineage map is constructed.

3. According to claim 1, a method for locating faults in the entire data link of a power enterprise based on data lineage, characterized in that: The specific method of step 2 is: Based on the construction of the data lineage graph, log collection or event tracking tools are deployed at key nodes to capture system runtime information, including execution time, status, and error codes, to achieve full-link collection and real-time monitoring of metadata lineage.

4. According to claim 1, a method for locating faults in the entire data link of a power enterprise based on data lineage, characterized in that: The specific steps of step 3 include: (1) Extract state change data from the existing real-time monitoring system and store it in the database, including timestamp, link ID, component status, service health status, and error log; (2) Based on the extracted state change information, a state transition matrix is ​​constructed to calculate the probability of each state transferring to another state; (3) Construct a Markov chain model to reflect the transition probability between different states; (4) Using the Markov chain model, the system can identify potential fault points and accurately locate them.

5. A method for locating faults in the entire data link of a power enterprise based on data lineage according to claim 4, characterized in that: The specific steps of step (2) of step 3 include: ①Determine the state set First, we need to determine all possible states and summarize them from the link state change information extracted from the log; suppose we have the following state set: S = {s1, s2, s3, …, s n }; ②Count the number of state transitions By traversing all transaction logs, we count the number of transitions from one state to another; for each pair of states (s i ,s j ), record status s i s j The number of times; implemented through a two-dimensional array C, where C[i][j] represents the state s i Transfer to state s j The number of times, and introduce a weight factor w ij ,This factor takes into account the changing trends of load, voltage and current; w ij =α·load similarity(s i ,s j )+β·voltage similarity(s i ,s j )+γ·currentsimilarity(s i ,s j ) Among them, α, β and γ are weight coefficients, which represent the importance of load, voltage and current similarity respectively; load_similarity, voltage_similarity and current_similarity are functions used to measure the similarity between two states in terms of load, voltage and current, which are also calculated by the cosine similarity of formula (3); ③Calculate state transition probability Based on the counted number of transitions, the probability of state transition is calculated; each element P[i][j] in the state transition probability matrix P represents the transition from state s i Transfer to state s j The probability is calculated as follows: If the state s i If no transfer occurs (i.e., all C[i][k] = 0), then we can set Represents a uniform distribution.

6. A method for locating faults in the entire data link of a power enterprise based on data lineage according to claim 4, characterized in that: The specific method of step (3) of step 3 is: ①Define the state space Define the state space S = {s1,s2,s3,…,s n }, that is, the set of all possible states; ②Define state transition probability Define the state transition probability matrix P = [P[i][j]], which describes the probability of transitioning from one state to another; ③Define the initial state distribution Define an initial state distribution vector π=[π1,π2,…,π n ], which is a probability vector, indicating the probability of the system being in each state at the beginning; ④Simulation of Markov chain model Based on the defined state space, state transition probability matrix and initial state distribution, the evolution process of the Markov chain can be simulated.

7. A method for locating faults in the entire data link of a power enterprise based on data lineage according to claim 4, characterized in that: The specific method of step (4) of step 3 is: ① Determine the potential fault state set: According to the state transition probability matrix P, combined with link logs and service characteristics, the possible set of potential fault states is identified; ②Backtrack and analyze the state transfer path: For each potential fault state, trace back its state transition path to locate the possible fault source: 1) Extract state transition sequence: Extract the current state from the link log f The predecessor state set s prev , and analyze its transfer to s f Path characteristics; 2) Analyze key transfer points: For each predecessor state s i ∈S prev , calculate its transfer to s f The weight factor w ij and the number of transfers C[i][j], if w ij If the abnormality is too high, it is recorded as a possible cause of the fault; if C[i][j] is abnormally frequent, the corresponding operation time and the associated upstream and downstream components are recorded; 3) Locate the scope of influence based on data blood relationship: Tracking fault status through data lineage graph f For the propagation paths of upstream and downstream nodes, clarify the chain reactions that may be triggered in the link; ③ Refine fault location, combine the above analysis, and further clarify the fault location scope: 1) Combine state transition features with blood relationship cross validation If the fault state s f The upstream and downstream nodes of the ij Verify the transmission direction of the abnormal impact and locate whether the fault point is an internal node problem or an external propagation problem; 2) Digging deeper into the similarity between states Make a detailed comparison of the load, voltage and current changes at the potential fault point to clearly distinguish it from other states and ensure the accuracy of the positioning results; 3) Generate a fault location report, record the final confirmed fault point, including the specific status f , path characteristics, triggering conditions and affected scope; Through the above steps, the system successfully identified the potential fault points in the link and provided accurate positioning results and detailed upstream and downstream impact analysis.

Images