Cross-system anomaly repair method based on global heterogeneous blood graph and dependency matrix

Abstract

Embodiments of the present disclosure disclose a cross-system anomaly repair method based on a global heterogeneous blood relation graph and a dependency matrix. A specific embodiment of the method includes: obtaining an anomaly indicator identifier and an anomaly occurrence time; constructing a global heterogeneous blood relation graph; determining an input-output dependency matrix of a data computing node; determining a starting node in the global heterogeneous blood relation graph; determining a time series graph snapshot of a business indicator; performing pruning processing on the time series graph snapshot to obtain a target attribution factor graph; generating standardized data probes for each node in the target attribution factor graph; performing distributed execution processing on each obtained standardized data probe; determining a node corresponding to an execution result as a root cause node; and outputting a repair driving instruction to a target device corresponding to the root cause node. The embodiment can reduce the situation that all target devices have a CPU running at a high load, memory frequently reading and writing, and a disk continuously operating at a high IO, slow down the aging rate of device hardware, and prolong the service life.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of big data processing technology, specifically to a cross-system anomaly repair method based on global heterogeneous lineage maps and dependency matrices. Background Technology

[0002] Cross-system anomaly repair refers to the process of tracing back from the anomaly indicators to locate the root cause node in a complex data chain composed of heterogeneous data systems (such as BI reports, API services, data warehouses, stream computing engines, etc.) when a data anomaly occurs at a certain link, and then repairing it. It is widely used in data observability platforms, BI report maintenance, and end-to-end data governance. Currently, the common approach for cross-system anomaly repair is as follows: A data tracing chain is constructed based on table-level lineage. Starting from the node containing the anomaly indicator, the data flow is traversed backwards, checking the task running status of all upstream nodes. Nodes with failed or timed-out statuses are identified as root cause nodes. For nodes with normal status but abnormal data content, the output data is quality checked using preset calculation rules (such as row count verification and null value rate verification). Based on the detection results, the process is repeated upstream until the root cause node is located, and then the target device corresponding to the root cause node is repaired.

[0003] However, when using the above methods to repair cross-system anomalies, the following technical problems often arise: Existing technologies only maintain table-level lineage relationships and cannot perceive computational dependencies at the field level within computing nodes. This results in the inability to remove irrelevant input fields and their upstream links that the current abnormal field does not depend on during reverse tracing, leading to the contamination of the investigation scope by numerous bypass branches. Repair requires a full data scan and verification of all target devices, causing all target devices to run at high CPU loads, frequently read and write memory, and continuously perform high disk I / O operations, accelerating hardware aging (such as CPU silicon electromigration, SSD write lifespan depletion, fan bearing wear, etc.), thus shortening device lifespan. In addition, preset computing rules need to be pre-configured for each node, which is costly and has low coverage. Nodes without configured rules cannot be effectively detected when an anomaly occurs, causing the computer to skip the node and miss it, or perform a full data scan, exhausting device computing resources, further accelerating hardware aging and shortening device lifespan.

[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not form prior art known to those skilled in the art. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure propose a cross-system anomaly repair method based on global heterogeneous lineage maps and dependency matrices to solve one or more of the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of this disclosure provide a cross-system anomaly repair method based on a global heterogeneous lineage graph and a dependency matrix. The method includes: in response to receiving an anomaly repair request from various target devices corresponding to a target heterogeneous system, obtaining the anomaly indicator identifier and anomaly occurrence time of the business indicators of the target heterogeneous system where anomalies occur; constructing a global heterogeneous lineage graph of the target heterogeneous system, wherein the global heterogeneous lineage graph includes nodes under the target heterogeneous system and dependency edges between nodes; and for each data computation node in the global heterogeneous lineage graph, determining the input-output dependency matrix of the data computation node. Based on the aforementioned anomaly indicator identifiers, the starting node corresponding to the aforementioned anomaly indicator identifier in the aforementioned global heterogeneous lineage graph is determined; based on the aforementioned anomaly occurrence time and the aforementioned global heterogeneous lineage graph, a time-series graph snapshot of the aforementioned business indicator is determined; based on the obtained input-output dependency matrices and the aforementioned starting node, the aforementioned time-series graph snapshot is pruned to obtain the target attribution graph corresponding to the aforementioned business indicator; based on the aforementioned target attribution graph and the anomaly type identifier corresponding to the aforementioned business indicator, standardized data probes for each node in the aforementioned target attribution graph are generated; distributed execution processing is performed on the obtained standardized data probes to obtain the execution results for each node in the aforementioned target attribution graph; in response to the determination that there is an execution result that meets the preset positioning conditions among the obtained execution results of each node, the node corresponding to the aforementioned execution result is determined as the root cause node; based on the node type of the aforementioned root cause node, a repair drive command is output to the target device corresponding to the aforementioned root cause node to restore the aforementioned business indicator to normal.

[0008] Secondly, some embodiments of this disclosure provide a cross-system anomaly repair device based on a global heterogeneous lineage graph and dependency matrix. The device includes: an acquisition unit configured to, in response to receiving an anomaly repair request from each target device corresponding to a target heterogeneous system, acquire anomaly indicator identifiers and anomaly occurrence times of business indicators where anomalies occur in the target heterogeneous system; a construction unit configured to construct a global heterogeneous lineage graph of the target heterogeneous system, wherein the global heterogeneous lineage graph includes nodes under the target heterogeneous system and dependency edges between nodes; a first determination unit configured to, for each data computing node in the global heterogeneous lineage graph, determine the input-output dependency matrix of the data computing node; a second determination unit configured to, based on the anomaly indicator identifier, determine the starting node in the global heterogeneous lineage graph corresponding to the anomaly indicator identifier; and a third determination unit configured to, based on the anomaly occurrence time and the global heterogeneous lineage... The system comprises: a time-series graph snapshot of the aforementioned business indicators; a pruning unit configured to prune the time-series graph snapshot based on the obtained input-output dependency matrices and the aforementioned starting node, to obtain the target attribution graph corresponding to the aforementioned business indicators; a generation unit configured to generate standardized data probes for each node in the target attribution graph based on the target attribution graph and the anomaly type identifier corresponding to the aforementioned business indicators; an execution unit configured to perform distributed execution processing on the obtained standardized data probes to obtain the execution results of each node in the target attribution graph; a fourth determination unit configured to determine the node corresponding to the execution result as the root cause node in response to the determination that there is an execution result that meets the preset positioning conditions among the execution results of the obtained nodes; and an output unit configured to output a repair drive command to the target device corresponding to the root cause node based on the node type of the root cause node, so that the aforementioned business indicators return to normal.

[0009] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0010] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0011] The above embodiments of this disclosure have the following beneficial effects: the cross-system anomaly repair method based on global heterogeneous lineage map and dependency matrix of some embodiments of this disclosure can reduce the occurrence of continuous high CPU load operation, frequent memory read and write, and continuous high disk IO operation of all target devices, thus slowing down the aging rate of device hardware and extending its lifespan. Specifically, the reason why all target devices experience continuous high CPU load, frequent memory read / write, and continuous high disk I / O operations, accelerating hardware aging and shortening device lifespan, is as follows: Existing technology only maintains table-level lineage relationships and cannot perceive field-level computational dependencies within computing nodes. This means that during reverse tracing, irrelevant input fields and their upstream links that the current abnormal field does not depend on cannot be eliminated, resulting in the investigation scope being contaminated by numerous bypass branches. Repair requires a full data scan and verification of all target devices, leading to continuous high CPU load, frequent memory read / write, and continuous high disk I / O operations, accelerating hardware aging (such as CPU silicon electromigration, SSD write lifespan depletion, and fan bearing wear), thus shortening device lifespan. Furthermore, preset computation rules need to be pre-configured for each node, which is costly and has low coverage. Nodes without configured rules cannot be effectively detected when anomalies occur, causing the computer to skip those nodes, resulting in missed detections, or perform a full data scan, exhausting device computing resources, further accelerating hardware aging and shortening device lifespan. Based on this, some embodiments of the present disclosure of a cross-system anomaly repair method based on a global heterogeneous lineage graph and dependency matrix firstly, in response to receiving anomaly repair requests from various target devices corresponding to the target heterogeneous system, obtain the anomaly indicator identifier and anomaly occurrence time of the business indicators that have anomaly occurred in the target heterogeneous system. Thus, the anomaly indicator identifier and anomaly occurrence time of the business indicators that have anomaly occurred can be obtained. Next, a global heterogeneous lineage graph of the target heterogeneous system is constructed. Thus, a global heterogeneous lineage graph of the multi-target heterogeneous system can be obtained. Then, for each data computing node in the global heterogeneous lineage graph, the input-output dependency matrix of the data computing node is determined. Thus, the input-output dependency matrix of each data computing node can be obtained. Then, based on the anomaly indicator identifier, the starting node corresponding to the anomaly indicator identifier in the global heterogeneous lineage graph is determined. Thus, the starting node of the corresponding business indicator can be obtained. Next, based on the anomaly occurrence time and the global heterogeneous lineage graph, a time-series snapshot of the business indicator is determined. Thus, the time-series snapshot corresponding to the time of the anomaly occurrence can be obtained. Subsequently, based on the obtained input-output dependency matrices and the aforementioned starting nodes, the time-series graph snapshots are pruned to obtain the target factorization graphs corresponding to the aforementioned business metrics. Thus, the pruned target factorization graph can be obtained.Next, based on the aforementioned target attribution graph and the anomaly type identifiers corresponding to the aforementioned business indicators, standardized data probes for each node in the target attribution graph are generated. This yields a standardized data probe for each node in the target attribution graph. Then, distributed execution processing is performed on each of the obtained standardized data probes to obtain the execution results for each node in the target attribution graph. This yields the execution results after executing each standardized data probe. Next, in response to the determination that there are execution results among the obtained node execution results that satisfy preset location conditions, the node corresponding to the above execution results is identified as the root cause node. This yields the root cause node. Finally, based on the node type of the root cause node, a repair drive command is output to the target device corresponding to the root cause node to restore the aforementioned business indicators to normal. Because this solution does not only maintain table-level lineage relationships but also constructs a field-level input-output dependency matrix for each data computing node, it can automatically eliminate irrelevant input fields and their upstream links corresponding to columns with element values ​​of 0 based on the row vectors corresponding to the abnormal fields during reverse tracing, significantly narrowing the investigation scope and reducing computational resource waste and location time. Because it doesn't rely solely on preset calculation rules for general quality checks, but instead dynamically matches verification operators based on anomaly type identifiers and generates standardized data probes on demand using the pruned target attribution graph, it avoids pre-configuring rules for all nodes, reducing configuration costs and improving detection coverage. It also reduces repair time because it doesn't rely solely on physical blocking based on task execution status; instead, it first restores a snapshot of the timeline graph at the time of the anomaly, then performs logical pruning using the dependency matrix, and locates the root cause node through distributed execution probes and logic for judging normal input and abnormal output. This avoids invalid verification and full scans, improving the efficiency of anomaly root cause localization. Therefore, it reduces situations where the computer skips the node, causing missed detections, or performs a full data scan, exhausting computing resources, resulting in timely repairs and reducing the computational resources consumed during repair. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0013] Figure 1 This is a flowchart of some embodiments of the cross-system anomaly repair method based on global heterogeneous lineage maps and dependency matrices according to this disclosure; Figure 2 This is a schematic diagram of the structure of some embodiments of the cross-system anomaly repair device based on global heterogeneous lineage map and dependency matrix according to the present disclosure; Figure 3This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0016] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0017] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0018] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0019] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] Figure 1 The flowchart 100 illustrates some embodiments of a cross-system anomaly repair method, apparatus, device, and computer-readable medium based on a global heterogeneous lineage map and dependency matrix according to this disclosure. The cross-system anomaly repair method, apparatus, device, and computer-readable medium based on a global heterogeneous lineage map and dependency matrix includes the following steps: Step 101: In response to receiving the abnormal repair request from each target device corresponding to the target heterogeneous system, obtain the abnormal indicator identifier and the time of abnormality of the business indicators of the target heterogeneous system.

[0021] In some embodiments, in response to receiving anomaly repair requests from various target devices corresponding to the target heterogeneous system, the executing entity (e.g., a computer device) of the cross-system anomaly repair method based on the global heterogeneous lineage graph and dependency matrix can obtain the anomaly indicator identifier and anomaly occurrence time of the business indicators of the aforementioned target heterogeneous system where anomalies occur. The aforementioned target heterogeneous system can be a data processing system with different technical architectures, data formats, access protocols, or metadata specifications. For example, the target heterogeneous system can be a database, a computing engine, an API service, and a BI report. The target devices among the aforementioned target devices can be physical servers running the systems within the aforementioned target heterogeneous system. For example, the target device corresponding to the MySQL database is the physical server running the MySQL service, the target device corresponding to the Flink computing engine is the physical server running Flink tasks, and the target device corresponding to the BI report system is the physical server running the BI report service. The aforementioned anomaly repair requests can be data packets collected by monitoring agents (such as Node Exporter and FlinkMetric Reporter) deployed on the aforementioned target devices and reported through hardware interfaces. The abnormal business metrics of the aforementioned heterogeneous system could be as follows: Under normal circumstances, GMV (Gross Merchandise Volume) is 1 million RMB / hour, but it suddenly drops to 500,000 RMB / hour, a decrease of 50%. These business metrics can represent the measurement of business data that needs to be monitored. For example, business metrics could be GMV, order volume, user conversion rate, or congestion index. The abnormal metric identifier can be a unique identifier for the aforementioned business metric. For example, the abnormal metric identifier could be a metric name in a BI report or a field name in a data table. The abnormal occurrence time can be the time when the aforementioned business metric occurred. In practice, the aforementioned executing entity can obtain the abnormal metric identifier and abnormal occurrence time of the aforementioned business metrics in the following ways: in response to receiving an alarm sent by the monitoring system or data quality platform, the abnormal metric identifier and abnormal occurrence time are parsed from the alarm; in response to detecting a diagnostic request entered by a user through the diagnostic interface, the abnormal metric identifier and abnormal occurrence time of the business metrics are obtained from the diagnostic request. For example, if the alarm is in JSON format: {"alertname":"GMV abnormal","metric":"gmv","timestamp":"2026-06-23 10:00:00"}, then the abnormal indicator can be read from the metric field of the alarm, and the time of occurrence of the abnormality can be read from the timestamp field.

[0022] Step 102: Construct a global heterogeneous lineage map of the target heterogeneous system.

[0023] The aforementioned execution entity can construct a global heterogeneous lineage graph of the target heterogeneous system. This graph represents the cross-system metadata relationship network. It can include nodes within the target heterogeneous system and the dependency edges between them. Each node can be a unified abstraction of physical objects within the target heterogeneous system. This unified abstraction can be the process of uniformly converting physical objects of different formats and structures within the target heterogeneous system into a data structure with the same attribute fields. These attribute fields can represent node identifiers, node types, and node attributes. Each node can be a data storage node, a data computation node, a data service node, or a data presentation node. Data storage nodes can be nodes used for persistent or temporary data storage. For example, a data storage node can be a MySQL table, a Hive partition, a Kafka Topic, or an HDFS file. Data computation nodes can be nodes that include computational logic (such as SQL statements and data processing code). For example, a data computation node can be a Spark task, a Flink task, or an ODPS task. Data service nodes can be nodes that provide data access interfaces. For example, a data service node can be an API interface, a Web Service, or a gRPC service. The data visualization nodes mentioned above can be used for data visualization and report display. For example, data visualization nodes can be BI reports or dashboards. The dependency edges mentioned above can represent the data flow or dependency relationship between two nodes. The global heterogeneous lineage graph mentioned above can be used to record field-level dependencies between data entities in a target heterogeneous system.

[0024] In some optional implementations of certain embodiments, the aforementioned execution entity may construct the global heterogeneous lineage map of the target heterogeneous system through the following steps: The first step is to collect metadata from the target heterogeneous system, resulting in a heterogeneous metadata set. This set of heterogeneous metadata represents the metadata collected from the target heterogeneous system. In practice, firstly, for tasks containing SQL computation logic (such as Hive tasks or Spark SQL tasks), the execution entity can identify the SQL code text of the task as its heterogeneous metadata. For stream computing tasks (such as Flink tasks or Kafka tasks), the task's configuration files (such as YAML files or Properties files) are identified as heterogeneous metadata. For BI reports, the metadata information of the BI report is identified as heterogeneous metadata. This metadata information can include the name of the BI report, the names of the metrics in the BI report, and the data source configuration. Finally, the obtained heterogeneous metadata is defined as a heterogeneous metadata set.

[0025] The second step involves normalizing the heterogeneous metadata set to abstract physical objects in the target heterogeneous system into general graph nodes, resulting in individual nodes. These physical objects can represent real data entities existing within the target heterogeneous system, including but not limited to database tables, message queues, message topics, computational tasks, API interfaces, and BI reports. The general graph nodes represent a unified data structure for representing physical objects in the global heterogeneous lineage graph. In practice, for each piece of heterogeneous metadata in the set, in response to determining that the physical object of the heterogeneous metadata is a data storage object (such as a MySQL table, Hive partition, or KafkaTopic), the executing entity can abstract the heterogeneous metadata into a data storage node, serving as a general graph node. The node attributes of these data storage nodes can include the storage engine type, connection information, table name or topic name, and a list of fields. For example, the node attributes of a data storage node could be "Storage engine type: MySQL, Connection information: jdbc:mysql: / / 192.168.1.100:3306 / ecommerce, Table name: orders, Field list: ["order_id", "user_id", "amount", "create_time"]". In response to determining that the physical object of the above heterogeneous metadata is a data computation object (such as a Spark task or Flink task), the above heterogeneous metadata is abstracted into data computation nodes as general graph nodes. The node attributes of the above data computation nodes can include the type of computation engine, task name, computation logic code, input field list, and output field list. For example, the node attributes of a data computation node could be "Computation engine type: Flink, Task name: RealtimeGMVJob, Computation logic code: SQL statement, Input field list: ["amount", "status", "create_time"] (extracted from SQL's SELECT, WHERE, GROUP BY, etc. clauses), Output field list: ["window_end", "gmv"] (extracted from SQL's SELECT clause)". In response to the determination that the physical object of the aforementioned heterogeneous metadata is a data service object (such as an API interface), the heterogeneous metadata is abstracted into data service nodes, serving as general graph nodes. The node attributes of the aforementioned data service nodes include protocol type, URL path, request method, input parameters, and return fields.For example, the node attributes of a data service node could be "Protocol Type: HTTP, URL Path: http: / / api.example.com / order / query, Request Method: POST, Input Parameter: ["orderId"], Return Fields: ["orderId", "amount", "status"]". In response to determining that the above physical object is a data visualization object (such as a BI report), the above heterogeneous metadata is abstracted into data visualization nodes as general graph nodes. The node attributes of the above data visualization nodes include report tool type, report name, metric name, and data source. For example, the node attributes of a data visualization node could be "Report Tool Type: Tableau, Report Name: Sales Dashboard, Metric Name: ["GMV", "Order Quantity"], Data Source: http: / / api.example.com / sales / summary". Abstracting physical objects into generic graph nodes can be achieved through the following steps: First, create an empty node object and fill the node identifier field with the unique identifier from the heterogeneous metadata corresponding to the physical object. Then, set the node type field (data storage node, data computation node, data service node, or data display node). Next, package other attributes (such as field list, computation logic, etc.) into JSON format and fill them into the node attribute field to obtain the generic graph node.

[0026] The third step involves establishing dependency edges between the nodes based on the table-level and field-level dependencies extracted from the heterogeneous metadata set. Table-level dependencies characterize the data dependencies between tables, describing the flow of data from the source table to the target table. Field-level dependencies characterize the data dependencies between fields, describing which source fields and how the target field is obtained through which computational logic. In practice, for tasks involving SQL computation logic, the execution entity can parse the abstract syntax tree of the SQL statement to identify the source table in the FROM or JOIN clause and the target table in the INSERT INTO clause (table-level dependencies), as well as the correspondence between the target field and the source table field in the SELECT clause (field-level dependencies), thus obtaining the dependency edges from the source table to the target table and from the source table field to the target field. For the stream computing task (based on configuration-bound metadata), the source configuration and sink configuration (table-level dependencies) are extracted by parsing the stream computing task's configuration file, obtaining the dependency edges from the source configuration to the sink configuration. For metadata collected through declarative registration, obtain the dependency edges that the user manually binds.

[0027] The fourth step is to construct a global heterogeneous lineage graph based on the nodes and dependent edges described above. In practice, the executing entity can store the nodes as a set of nodes and the dependent edges as a set of edges in a graph database (such as Neo4j or JanusGraph) to obtain the global heterogeneous lineage graph. It should be noted that each node in the node set and each edge in the edge set is appended with a timestamp attribute. This timestamp attribute represents the effective time range of a node or edge (including the start and end times of effectiveness). For example, node ID: N001, node type: data computation node, valid_from (start time of effectiveness): 2025-06-01 00:00:00, valid_to (end time of effectiveness): 2025-06-14 23:59:59.

[0028] Step 103: For each data computation node in the global heterogeneous lineage graph, determine the input-output dependency matrix of the data computation node.

[0029] In some embodiments, for each data computation node in the aforementioned global heterogeneous lineage map, the executing entity may determine the input-output dependency matrix of the data computation node. This input-output dependency matrix is ​​a binary matrix that describes the input-output dependencies between output fields and input fields in the data computation node. Rows in the input-output dependency matrix correspond to output fields, columns correspond to input fields, and an element value of 1 indicates the existence of a dependency, while 0 indicates the absence of a dependency.

[0030] In addressing the technical challenges mentioned above, and considering the specific application scenario—during e-commerce promotions, real-time GMV dashboards often experience fluctuations due to upstream data anomalies—the following technical problem arises: When attributing anomalies to GMV data output by Flink real-time computing tasks, existing technologies only maintain table-level lineage, failing to distinguish the computational dependencies between GMV metrics and fields in the upstream order table. This leads to the introduction of irrelevant upstream links during reverse tracing, requiring a full data scan and verification of all target devices during repair. This results in continuous high CPU loads, frequent memory read / write operations, and continuous high disk I / O operations on all target devices, accelerating hardware aging (e.g., CPU silicon electromigration, SSD write lifespan depletion, fan bearing wear), and ultimately shortening device lifespan. Given the specific requirements of this application scenario—high real-time performance for anomaly repair and high accuracy for root cause localization—we have decided to adopt the following solution: In some optional implementations of certain embodiments, the execution entity may determine the input-output dependency matrix of each data computation node in the aforementioned global heterogeneous lineage graph through the following steps: The first step, in response to the determination that the computational logic contained in the aforementioned data computation node is in the form of Structured Query Language (SCL), parses the computational logic to obtain the corresponding abstract syntax tree. The computational logic can represent the processing rules in the data computation node used to convert input data into output data. For example, in a Flink real-time computing task, the computational logic in SCL form might specifically be: reading the amount (order amount) and status (order status) fields from the order_stream as input fields, filtering records with a status of "paid" (paid), summing the amount field by minute time window to obtain the gmv (total transaction value) per minute as the output field. This computational logic can be in the form of SCL or unstructured query language statements. For example, structured query language statements could be Hive SQL, Spark SQL, or Flink SQL, while unstructured query language statements could be Python or Scala scripts. In practice, in response to the determination that the above computational logic is in the form of a structured query language (such as Hive SQL, Spark SQL, Flink SQL), the above execution entity can use ANTLR or Calcite to parse and process the above structured query language to obtain an abstract syntax tree.

[0031] The second step involves traversing the abstract syntax tree to obtain the output fields and input fields of the corresponding data computation nodes. The input fields represent the source fields read by the data computation node during the execution of computational logic. The output fields represent the target fields produced by the data computation node after executing computational logic. In practice, the executing entity can traverse the abstract syntax tree to extract output fields from the SELECT clause, extract fields from the source table as input fields from the FROM and JOIN clauses, and extract referenced fields from the WHERE, GROUP BY, and HAVING clauses as input fields.

[0032] The third step involves, in response to the determination that the aforementioned computational logic is in the form of an unstructured query language, parsing the logic using a static code analysis tool to obtain the output fields and input fields of the corresponding data computation nodes. This static code analysis tool can be a JavaCC or CPython parser. In practice, the executing entity can use a static code analysis tool to parse the unstructured query language code, using fields referenced in data reading interfaces (such as `spark.read` and `env.fromSource`) as input fields and fields output in data writing interfaces (such as `df.write` and `sink`) as output fields.

[0033] The fourth step involves constructing an initial binary dependency matrix, using the output fields as rows and the input fields as columns. This initial binary dependency matrix can be a two-dimensional array recording the dependencies between output and input fields. All elements in the initial binary dependency matrix are initialized to 0. The number of rows in the initial binary dependency matrix is ​​equal to the number of output fields, and the number of columns is equal to the number of input fields. In practice, the execution entity can first create a two-dimensional array with the number of output fields as the number of rows and the number of input fields as the number of columns. Finally, all elements in the two-dimensional array are initialized to 0 to obtain the initial binary dependency matrix.

[0034] Fifth, for each of the above output fields, perform the following overwrite steps: In the first sub-step, for each of the aforementioned input fields, in response to determining that the computational logic of the aforementioned output field references the aforementioned input field, a preset value is overwritten into the row corresponding to the aforementioned output field and the column corresponding to the aforementioned input field in the aforementioned initial binary dependency matrix. The preset value can be 1. In practice, in response to determining that the aforementioned computational logic is in the form of a structured query language and that the expression tree corresponding to the aforementioned output field contains the aforementioned input field, it is determined that the computational logic of the aforementioned output field references the aforementioned input field. For example, for the output field gmv, whose expression is SUM(amount), the input field amount is determined to be referenced, while the input field user_id is determined not to be referenced. In response to determining that the aforementioned computational logic is in the form of an unstructured query language, the abstract syntax tree of the aforementioned computational logic is parsed using the aforementioned static code analysis tool. If the assignment statement or return statement of the aforementioned output field references the aforementioned input field, it is determined that the computational logic of the aforementioned output field references the aforementioned input field. For example, in a Python script, if the assignment statement for the output field `total` is `total = df['amount'].sum()`, then the input field `amount` is considered to be referenced, while the input field `user_id` is considered not to be referenced.

[0035] The sixth step is to determine the initial binary dependency matrix after overwriting as the input-output dependency matrix of the aforementioned data computation nodes.

[0036] The above-described technical solution, as an inventive point of this disclosure, solves technical problem two: "causing all target devices to run under continuous high CPU load, frequent memory read / write, and continuous high disk I / O operations, accelerating device hardware aging and thus shortening device lifespan." The reasons for this are as follows: When attributing anomalies to GMV data output by Flink real-time computing tasks, existing technologies only maintain table-level lineage relationships and cannot distinguish the computational dependencies between GMV metrics and fields in the upstream order table. This leads to the introduction of irrelevant upstream links during reverse tracing, requiring a full data scan and verification of all target devices during repair. This results in all target devices running under continuous high CPU load, frequent memory read / write, and continuous high disk I / O operations, accelerating device hardware aging (such as CPU silicon electromigration, SSD write lifespan depletion, fan bearing wear, etc.), and thus shortening device lifespan. To achieve this effect, the cross-system anomaly repair method based on global heterogeneous lineage graphs and dependency matrices disclosed herein constructs field-level input-output dependency matrices for data computing nodes, recording the dependency relationships between output fields and each input field. This enables automatic pruning, cutting off upstream links of irrelevant fields, thereby reducing the amount of irrelevant data scanning, lowering computational resource consumption, and shortening anomaly repair time. Consequently, it can reduce situations where all target devices experience continuous high CPU loads, frequent memory read / write operations, and continuous high disk I / O operations, delaying hardware aging and thus extending device lifespan.

[0037] Step 104: Determine the starting node of the corresponding abnormal indicator in the global heterogeneous lineage map based on the abnormal indicator identifier.

[0038] In some embodiments, the executing entity can determine the starting node corresponding to the abnormal indicator in the heterogeneous lineage graph based on the abnormal indicator identifier. The starting node can be the node corresponding to the abnormal indicator identifier in the heterogeneous lineage graph. In practice, firstly, the executing entity can search for a node whose node identifier or node name matches the abnormal indicator identifier in the node set of the heterogeneous lineage graph. Then, the matched node is determined as the starting node. For example, if the abnormal indicator identifier is "gmv", and there is a data display node (corresponding to the GMV indicator in the BI report) in the heterogeneous lineage graph with node identifier N_GMV_001 and node name "gmv", then the node name of this data display node matches the abnormal indicator identifier, and this data display node is determined as the starting node.

[0039] Step 105: Based on the anomaly occurrence time and the global heterogeneous lineage map, determine the time-series snapshot of the business indicators.

[0040] In some embodiments, the execution entity may determine a time-series snapshot of the business metrics based on the anomaly occurrence time and the global heterogeneous lineage graph. The time-series snapshot can characterize the topological structure at the moment corresponding to the anomaly occurrence time, reconstructed from the global heterogeneous lineage graph.

[0041] In some optional implementations of certain embodiments, the aforementioned execution entity may determine a time-series snapshot of the aforementioned business metrics based on the aforementioned anomaly occurrence time and the aforementioned global heterogeneous lineage graph: The first step is to define each node in the aforementioned heterogeneous lineage graph as a temporal node in response to determining that the occurrence time of the aforementioned anomaly falls within the node's valid time range. The valid time range characterizes the lifecycle of a node or dependent edge on the time axis, including the start time of its effectiveness and the end time of its failure. The temporal node characterizes a node whose anomaly occurrence time falls within its valid time range.

[0042] The second step involves defining each dependency edge in the aforementioned global heterogeneous lineage graph as a temporal dependency edge, in response to determining that the anomaly occurrence time falls within the valid time range of the dependency edge. Here, the temporal dependency edge characterizes a dependency edge whose anomaly occurrence time falls within the valid time range.

[0043] The third step involves constructing a snapshot of the time-series graph by combining the obtained time-series nodes as a node set and the obtained time-series dependent edges as an edge set. In practice, the execution entity can associate and combine the node set and the edge set (i.e., each edge connects the corresponding source node and target node) to obtain the snapshot of the time-series graph.

[0044] Optionally, after step 104, the aforementioned execution entity may also perform the following steps: First, obtain the running instance status of each node in the aforementioned time-series graph snapshot. This running instance status characterizes the actual running status of the node at the time of the anomaly, used to determine whether the node completed execution normally. The running instance status can be Success, Failed, or Pending. In practice, the executing entity can call the application programming interface (API) of a scheduling center (such as Airflow, DolphinScheduler, or YARN) to obtain the running instance status of the node at the time of the anomaly.

[0045] Then, in response to the determination that there are failed or timed-out running instance states among the obtained running instance states, the node corresponding to the above running instance states is determined as the root cause node.

[0046] Step 106: Based on the obtained input-output dependency matrices and starting nodes, prune the time series graph snapshot to obtain the target factorization graph corresponding to the business indicators.

[0047] In some embodiments, the execution entity can prune the time series graph snapshot based on the obtained input-output dependency matrices and the starting node to obtain the target factorization graph corresponding to the business metrics. The target factorization graph can be a subgraph obtained after pruning the time series graph snapshot, containing only nodes and dependency edges that have direct or indirect dependencies with the metric or field that has anomaly.

[0048] In addressing the technical challenges mentioned above, and considering the specific application scenario—an e-commerce user profiling system experiencing a sharp drop in the number of high-value user tags due to upstream data anomalies (high-value user tags are calculated based on a combination of three indicators: spending amount, login days, and add-to-cart frequency)—the following technical problem often arises: When attributing anomalies to high-value user tags output by Spark offline computation tasks, existing pruning methods can only physically block them based on task execution status (success / failure). For scenarios where all task statuses are successful, existing technology cannot perform logical pruning based on field-level computational dependencies. This necessitates retaining all upstream data sources during reverse tracing, requiring a full data scan and verification of all target devices during repair. This results in continuous high CPU load, frequent memory read / write operations, and continuous high disk I / O operations on all target devices, accelerating hardware aging and shortening device lifespan. Given the following requirements for this application scenario—high real-time performance in anomaly repair and high accuracy in root cause localization—we have decided to adopt the following solution: In some optional implementations of certain embodiments, the execution entity can perform pruning on the time series snapshot based on the obtained input-output dependency matrices and the starting node to obtain the target attribution graph corresponding to the business indicators: The first step is to add the starting node to the queue to be traversed. This queue represents the data structure used to store the nodes to be traversed. Initially, the queue can be empty.

[0049] The second step, in response to determining that the queue to be traversed is not empty, is to perform the following pruning steps based on the input-output dependency matrices and the initial full quantum graph described above: The first sub-step involves taking the first node from the queue to be traversed and using it as the target node.

[0050] The second sub-step, in response to determining that the target node is not a data computation node, adds the upstream node of the target node to the traversal queue. It should be noted that the upstream node has not yet been added to the traversal queue.

[0051] The third sub-step, in response to determining that the target node is a data computation node, determines the row vector of the abnormal output field corresponding to the aforementioned business metric based on the input-output dependency matrix of the data computation node. The abnormal output field can be any abnormal output field corresponding to the aforementioned business metric. For example, the abnormal output field could be `gmv`. The row vector can represent a vector formed by directly concatenating the values ​​of each element in the row corresponding to the business metric in the input-output dependency matrix. In practice, the executing entity can concatenate the values ​​of each element in the row of the abnormal output field in the input-output dependency matrix to obtain the row vector of the abnormal output field corresponding to the aforementioned business metric.

[0052] The fourth sub-step involves adding the nodes from which the input fields corresponding to the columns in the row vector with element values ​​of the preset values ​​originate to the queue to be traversed. It should be noted that the nodes from which the input fields corresponding to the columns in the row vector with element values ​​of the preset values ​​originate are nodes that have not yet been added to the queue to be traversed.

[0053] The fifth sub-step involves determining the input field corresponding to the column in the aforementioned row vector whose element value is the target value as the pruning field. The target value can be 0.

[0054] The sixth sub-step involves deleting the node corresponding to the aforementioned pruning field and all its predecessor paths from the initial full quantum graph, resulting in the modified initial full quantum graph. Each predecessor path can be represented as all nodes and all dependent edges traversed upstream from the node corresponding to the aforementioned pruning field, in the opposite direction of the dependent edges, back to the source node.

[0055] The seventh sub-step, in response to determining that the above queue to be traversed is empty, determines the above modified initial full quantum graph as the target factorization graph.

[0056] The eighth sub-step, in response to determining that the queue to be traversed is not empty, re-executes the pruning step and repairs the heterogeneous target system according to the target factorization graph to restore the business indicators to normal. It should be noted that the specific implementation steps for repairing the heterogeneous target system according to the target factorization graph to restore the business indicators to normal can be found in steps 107-110, and will not be repeated here.

[0057] The above-described technical solution, as an inventive point of this disclosure, solves technical problem three: "causing all target devices to run under continuous high CPU load, frequent memory read / write, and continuous high disk I / O operations, accelerating device hardware aging and thus shortening device lifespan." The reasons for this are as follows: When performing anomaly attribution on high-value user tags output by Spark offline computing tasks, existing pruning methods can only physically block based on task running status (success / failure). For scenarios where all task statuses are successful, existing technologies cannot perform logical pruning based on field-level computational dependencies. This results in all upstream data sources being retained during reverse tracing, and a full data scan and verification of all target devices is required during repair. This leads to continuous high CPU load, frequent memory read / write, and continuous high disk I / O operations on all target devices, accelerating device hardware aging and thus shortening device lifespan. To achieve this effect, the cross-system anomaly repair method based on global heterogeneous lineage graphs and dependency matrices disclosed herein utilizes input-output dependency matrices for logical pruning. When traversing to a Spark compute node, its input-output dependency matrix is ​​loaded. Irrelevant fields and their upstream nodes are automatically pruned based on the columns with values ​​of 0 in the row vectors of the anomaly output fields, retaining only the upstream paths corresponding to fields with values ​​of 1. This narrows the investigation scope and reduces the amount of data scanned by the computer. Consequently, it can reduce CPU utilization, memory read / write operations, and disk I / O operations on the target devices corresponding to non-root cause nodes, slowing down hardware aging and extending device lifespan.

[0058] Step 107: Generate standardized data probes for each node in the target attribution graph based on the anomaly type identifiers corresponding to the target attribution graph and business indicators.

[0059] In some embodiments, the execution entity can generate standardized data probes for each node in the target attribution graph based on the target attribution graph and the anomaly type identifiers corresponding to the business indicators. The anomaly type identifiers can represent classification labels used to describe the nature of the business indicators. These classification labels may include, but are not limited to: integrity anomalies (missing data), consistency anomalies (calculation errors), and timeliness anomalies (delays). The standardized data probes can be executable verification instructions generated for each node in the target attribution graph. These executable verification instructions can represent instructions used to verify whether the data status or calculation results of a node are normal within the time range corresponding to the anomaly occurrence time. The time range corresponding to the anomaly occurrence time can be a time interval of a preset length centered or ending at the anomaly occurrence time, used to determine the start and end times of data queries when the standardized data probes are executed. Here, the specific length of the preset length is not limited and can be adjusted according to actual needs. For example, for real-time computing tasks scheduled every minute, the preset length can be 5 minutes; for offline batch processing tasks scheduled daily, the preset length can be 24 hours.

[0060] In some optional implementations of certain embodiments, the execution entity can generate standardized data probes for each node in the target attribution graph based on the target attribution graph and the anomaly type identifiers corresponding to the business metrics. The first step is to determine the verification operator corresponding to the aforementioned anomaly type identifier based on the preset operator information set. The preset operator information in the preset operator information set represents the correspondence between the anomaly type identifier and the verification operator. The verification operator represents the operator used to verify whether a node is functioning correctly. The verification operator can be a row count comparison operator, a logical recalculation operator, or a timestamp check operator. The row count comparison operator can be an operator used to count the number of records in the node's output data and compare it with a benchmark value. The logical recalculation operator can be an operator used to re-execute the calculation according to the node's calculation logic and compare the recalculation result with the actual output result. The timestamp check operator can be an operator used to check the difference between the timestamp of the node's output data and the time the anomaly occurred. For example, the preset operator information could be: when the anomaly type identifier is an integrity anomaly, the corresponding verification operator is a row count comparison operator; when the anomaly type identifier is a timeliness anomaly, the corresponding verification operator is a timestamp check operator.

[0061] The second step, for each node in the above target attribution factor graph, is to perform the following steps: The first sub-step involves determining the standardized contract template for the aforementioned nodes based on their node types and a predefined set of contract information relationships. The predefined contract information relationships in this set represent the mapping between node types and standardized contract templates. The standardized contract template represents a predefined framework for generating probes containing placeholders. These placeholders are used to replace the attribute values ​​of specific nodes during instantiation. For example, if the node type is a data storage node and the database type is MySQL, the corresponding standardized contract template could be `SELECT {operator}({field}) FROM {table} WHERE {time_range_condition}`. If the node type is a data service node and the protocol type is HTTP, the corresponding standardized contract template could be `curl -H "{auth_header}" "{url}?{params}"`. In practice, the executing entity can obtain the standardized contract template corresponding to the node type of the target node by querying the predefined set of contract information relationships.

[0062] The second sub-step involves injecting the aforementioned verification operator, the node attributes of the aforementioned node, and the anomaly occurrence time of the aforementioned business metrics into the aforementioned standardized contract template to obtain the executable probe corresponding to the aforementioned node. The aforementioned node attributes can characterize the parameters carried by the node itself. These node attributes may include at least one of the following: access address (e.g., IP and port), authentication credentials (e.g., account password or token), and data location (e.g., table name). In practice, the aforementioned execution entity can fill the aforementioned verification operator into the {operator} placeholder of the aforementioned standardized contract template, fill the connection information, authentication credentials, and data location from the aforementioned node attributes into their respective placeholders, and fill the aforementioned anomaly occurrence time into the time condition placeholder. For example, for a MySQL storage node, after filling, an executable probe can be generated: SELECT COUNT( ) asrow_count FROM order_detail WHERE create_time BETWEEN '2025-06-09 09:00:00'AND '2025-06-09 11:00:00'.

[0063] The third sub-step involves associating and storing the executable probes with the nodes to obtain standardized data probes for the nodes. In practice, the executing entity can use the node ID as the key and the executable probe as the value to obtain standardized data probes for the nodes, thereby establishing a correspondence between the executable probes and the nodes.

[0064] Step 108: Perform distributed execution processing on the obtained standardized data probes to obtain the execution results of each node in the target attribution factor graph.

[0065] In some embodiments, the execution entity can perform distributed execution processing on the obtained standardized data probes to obtain the execution results of each node in the target attribution factor graph. The execution results can characterize normal or abnormal behavior.

[0066] Step 109: In response to the determination that there is an execution result that satisfies the preset positioning conditions among the execution results of each node, the node corresponding to the execution result is determined as the root cause node.

[0067] In some embodiments, in response to determining that among the execution results obtained from each node there is an execution result that satisfies a preset location condition, the execution entity can identify the node corresponding to the execution result as the root cause node. The preset location condition can be that the execution result is abnormal, and the standardized data probe execution result of the upstream node corresponding to the execution result is normal. The upstream node can represent a node that is located before the current node in the data flow direction and directly or indirectly provides data to the current node. If there exists a dependency edge from node A to node B (i.e., data flows from node A to node B), then node A is an upstream node of node B. The root cause node can represent the source node that caused the aforementioned business metric to occur.

[0068] Step 110: Based on the node type of the root cause node, output a repair drive command to the target device corresponding to the root cause node so that the business indicators return to normal.

[0069] In some embodiments, the execution entity can output repair drive instructions to the target device corresponding to the root cause node based on the node type of the root cause node, so as to restore the business indicators to normal. The repair drive instructions can be instructions that drive the target device to perform repairs. The target device corresponding to the root cause node can represent the physical server running the root cause node. In practice, firstly, the execution entity can match the repair drive instructions corresponding to the node type from a preset repair correspondence set. The preset repair correspondence in the preset repair correspondence set can represent the correspondence between node types and repair drive instructions. For example, when the node type of the root cause node is a data computing node, the matched repair drive instruction is to send a task restart instruction to the target device running the data computing node to instruct the target device to restart the corresponding computing task; when the node type of the root cause node is a data storage node, the matched repair drive instruction is to send a service restart instruction or a data recovery instruction to the target device running the data storage node; when the node type of the root cause node is a data service node, the matched repair drive instruction is to send a service process restart instruction to the target device running the data service node. Then, the target device executes the repair driver command, thereby restoring the aforementioned business indicators to normal.

[0070] The above embodiments of this disclosure have the following beneficial effects: the cross-system anomaly repair method based on global heterogeneous lineage map and dependency matrix of some embodiments of this disclosure can reduce the occurrence of continuous high CPU load operation, frequent memory read and write, and continuous high disk IO operation of all target devices, thus slowing down the aging rate of device hardware and extending its lifespan. Specifically, the reason why all target devices experience continuous high CPU load, frequent memory read / write, and continuous high disk I / O operations, accelerating hardware aging and shortening device lifespan, is as follows: Existing technology only maintains table-level lineage relationships and cannot perceive field-level computational dependencies within computing nodes. This means that during reverse tracing, irrelevant input fields and their upstream links that the current abnormal field does not depend on cannot be eliminated, resulting in the investigation scope being contaminated by numerous bypass branches. Repair requires a full data scan and verification of all target devices, leading to continuous high CPU load, frequent memory read / write, and continuous high disk I / O operations, accelerating hardware aging (such as CPU silicon electromigration, SSD write lifespan depletion, and fan bearing wear), thus shortening device lifespan. Furthermore, preset computation rules need to be pre-configured for each node, which is costly and has low coverage. Nodes without configured rules cannot be effectively detected when anomalies occur, causing the computer to skip those nodes, resulting in missed detections, or perform a full data scan, exhausting device computing resources, further accelerating hardware aging and shortening device lifespan. Based on this, some embodiments of the present disclosure of a cross-system anomaly repair method based on a global heterogeneous lineage graph and dependency matrix firstly, in response to receiving anomaly repair requests from various target devices corresponding to the target heterogeneous system, obtain the anomaly indicator identifier and anomaly occurrence time of the business indicators that have anomaly occurred in the target heterogeneous system. Thus, the anomaly indicator identifier and anomaly occurrence time of the business indicators that have anomaly occurred can be obtained. Next, a global heterogeneous lineage graph of the target heterogeneous system is constructed. Thus, a global heterogeneous lineage graph of the multi-target heterogeneous system can be obtained. Then, for each data computing node in the global heterogeneous lineage graph, the input-output dependency matrix of the data computing node is determined. Thus, the input-output dependency matrix of each data computing node can be obtained. Then, based on the anomaly indicator identifier, the starting node corresponding to the anomaly indicator identifier in the global heterogeneous lineage graph is determined. Thus, the starting node of the corresponding business indicator can be obtained. Next, based on the anomaly occurrence time and the global heterogeneous lineage graph, a time-series snapshot of the business indicator is determined. Thus, the time-series snapshot corresponding to the time of the anomaly occurrence can be obtained. Subsequently, based on the obtained input-output dependency matrices and the aforementioned starting nodes, the time-series graph snapshots are pruned to obtain the target factorization graphs corresponding to the aforementioned business metrics. Thus, the pruned target factorization graph can be obtained.Next, based on the aforementioned target attribution graph and the anomaly type identifiers corresponding to the aforementioned business indicators, standardized data probes for each node in the target attribution graph are generated. This yields a standardized data probe for each node in the target attribution graph. Then, distributed execution processing is performed on each of the obtained standardized data probes to obtain the execution results for each node in the target attribution graph. This yields the execution results after executing each standardized data probe. Next, in response to the determination that there are execution results among the obtained node execution results that satisfy preset location conditions, the node corresponding to the above execution results is identified as the root cause node. This yields the root cause node. Finally, based on the node type of the root cause node, a repair drive command is output to the target device corresponding to the root cause node to restore the aforementioned business indicators to normal. Because this solution does not only maintain table-level lineage relationships but also constructs a field-level input-output dependency matrix for each data computing node, it can automatically eliminate irrelevant input fields and their upstream links corresponding to columns with element values ​​of 0 based on the row vectors corresponding to the abnormal fields during reverse tracing, significantly narrowing the investigation scope and reducing computational resource waste and location time. Because it doesn't rely solely on preset calculation rules for general quality checks, but instead dynamically matches verification operators based on anomaly type identifiers and generates standardized data probes on demand using the pruned target attribution graph, it avoids pre-configuring rules for all nodes, reducing configuration costs and improving detection coverage. It also reduces repair time because it doesn't rely solely on physical blocking based on task execution status; instead, it first restores a snapshot of the timeline graph at the time of the anomaly, then performs logical pruning using the dependency matrix, and locates the root cause node through distributed execution probes and logic for judging normal input and abnormal output. This avoids invalid verification and full scans, improving the efficiency of anomaly root cause localization. Therefore, it reduces situations where the computer skips the node, causing missed detections, or performs a full data scan, exhausting computing resources, resulting in timely repairs and reducing the computational resources consumed during repair.

[0071] Further reference Figure 2 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a cross-system anomaly repair method based on global heterogeneous lineage maps and dependency matrices. These device embodiments are similar to... Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0072] like Figure 2As shown, a cross-system anomaly repair device 200 based on global heterogeneous lineage maps and dependency matrices in some embodiments includes: an acquisition unit 201, a construction unit 202, a first determination unit 203, a second determination unit 204, a third determination unit 205, a pruning unit 206, a generation unit 207, an execution unit 208, a fourth determination unit 209, and an output unit 210. The acquisition unit 201 is configured to, in response to receiving an anomaly repair request from each target device corresponding to the target heterogeneous system, acquire the anomaly indicator identifier and anomaly occurrence time of the business indicator of the target heterogeneous system that is experiencing an anomaly; the construction unit 202 is configured to construct a global heterogeneous lineage graph of the target heterogeneous system, wherein the global heterogeneous lineage graph includes nodes under the target heterogeneous system and dependency edges between nodes; the first determination unit 203 is configured to determine the input-output dependency matrix of each data computing node in the global heterogeneous lineage graph; the second determination unit 204 is configured to determine the starting node corresponding to the anomaly indicator identifier in the global heterogeneous lineage graph based on the anomaly indicator identifier; the third determination unit 205 is configured to determine a time-series graph snapshot of the business indicator based on the anomaly occurrence time and the global heterogeneous lineage graph; and the pruning unit 2... 06 is configured to prune the time series map snapshot based on the obtained input-output dependency matrices and the aforementioned starting node to obtain the target attribution map corresponding to the aforementioned business indicators; generation unit 207 is configured to generate standardized data probes for each node in the aforementioned target attribution map based on the aforementioned target attribution map and the anomaly type identifiers corresponding to the aforementioned business indicators; execution unit 208 is configured to perform distributed execution processing on the obtained standardized data probes to obtain the execution results of each node in the aforementioned target attribution map; fourth determination unit 209 is configured to determine the node corresponding to the aforementioned execution result as the root cause node in response to determining that there is an execution result that satisfies the preset positioning conditions among the execution results of the obtained nodes; output unit 210 is configured to output a repair drive command to the target device corresponding to the aforementioned root cause node according to the node type of the aforementioned root cause node, so that the aforementioned business indicators return to normal.

[0073] It is understandable that the units and references described in the cross-system anomaly repair device 200 based on global heterogeneous lineage maps and dependency matrices are... Figure 1 The steps described in the method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method are also applicable to the cross-system anomaly repair device 200 based on global heterogeneous lineage maps and dependency matrices, and the units contained therein, and will not be repeated here.

[0074] The following is for reference. Figure 3It shows a schematic diagram of the structure of an electronic device (such as a computing device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0075] like Figure 3 As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0076] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0077] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0078] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0079] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0080] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: in response to receiving an anomaly repair request from each target device corresponding to the target heterogeneous system, obtain the anomaly indicator identifier and anomaly occurrence time of the business indicators of the target heterogeneous system where the anomaly occurred; construct a global heterogeneous lineage graph of the target heterogeneous system, wherein the global heterogeneous lineage graph includes nodes under the target heterogeneous system and dependency edges between nodes; and for each data computing node in the global heterogeneous lineage graph, determine the input-output dependency matrix of the data computing node; Based on the aforementioned anomaly indicator identifiers, the starting node corresponding to the aforementioned anomaly indicator identifier in the aforementioned global heterogeneous lineage graph is determined; based on the aforementioned anomaly occurrence time and the aforementioned global heterogeneous lineage graph, a time-series graph snapshot of the aforementioned business indicator is determined; based on the obtained input-output dependency matrices and the aforementioned starting node, the aforementioned time-series graph snapshot is pruned to obtain the target attribution graph corresponding to the aforementioned business indicator; based on the aforementioned target attribution graph and the anomaly type identifier corresponding to the aforementioned business indicator, standardized data probes for each node in the aforementioned target attribution graph are generated; distributed execution processing is performed on the obtained standardized data probes to obtain the execution results for each node in the aforementioned target attribution graph; in response to the determination that there is an execution result that meets the preset positioning conditions among the obtained execution results of each node, the node corresponding to the aforementioned execution result is determined as the root cause node; based on the node type of the aforementioned root cause node, a repair drive command is output to the target device corresponding to the aforementioned root cause node to restore the aforementioned business indicator to normal.

[0081] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0082] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0083] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including an acquisition unit, a construction unit, a first determination unit, a second determination unit, a third determination unit, a pruning unit, a generation unit, an execution unit, a fourth determination unit, and an output unit. The names of these units do not necessarily limit the specific unit itself. For example, the acquisition unit may also be described as a unit that "in response to receiving an anomaly repair request from each target device corresponding to the target heterogeneous system, acquires the anomaly indicator identifier and anomaly occurrence time of the business indicators of the aforementioned target heterogeneous system where an anomaly occurred."

[0084] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0085] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A cross-system anomaly repair method based on global heterogeneous lineage maps and dependency matrices, comprising: In response to receiving an anomaly repair request from each target device corresponding to the target heterogeneous system, the abnormal indicator identifier and anomaly occurrence time of the business indicators of the target heterogeneous system that are abnormal are obtained; Construct a global heterogeneous lineage graph of the target heterogeneous system, wherein the global heterogeneous lineage graph includes nodes in the target heterogeneous system and dependency edges between nodes; For each data computation node in the global heterogeneous lineage map, determine the input-output dependency matrix of the data computation node; Based on the abnormal indicator identifier, determine the starting node corresponding to the abnormal indicator identifier in the global heterogeneous lineage map; Based on the anomaly occurrence time and the global heterogeneous lineage map, determine the time-series snapshot of the business indicators; Based on the obtained input-output dependency matrices and the starting node, the time series graph snapshot is pruned to obtain the target factorization graph corresponding to the business indicator; Based on the target attribution factor graph and the anomaly type identifiers corresponding to the business indicators, generate standardized data probes for each node in the target attribution factor graph; Distributed execution processing is performed on each of the obtained standardized data probes to obtain the execution results of each node in the target attribution factor graph; In response to the determination that there is an execution result that satisfies the preset positioning conditions among the execution results of each node, the node corresponding to the execution result is determined as the root cause node; Based on the node type of the root cause node, a repair drive command is output to the target device corresponding to the root cause node so that the business indicators return to normal.

2. The method according to claim 1, wherein, The preset positioning condition is that the execution result is abnormal, and the standardized data probe execution result of the upstream node of the node corresponding to the execution result is normal.

3. The method according to claim 1, wherein, The construction of the global heterogeneous lineage map of the target heterogeneous system includes: Collect metadata from the target heterogeneous system to obtain a heterogeneous metadata set; The heterogeneous metadata set is normalized and modeled to abstract the physical objects in the target heterogeneous system into general graph nodes, and each general graph node is used as a node. Based on the table-level and field-level dependencies extracted from the heterogeneous metadata set, dependency edges are established between the nodes. Based on the nodes and the resulting dependency edges, a global heterogeneous lineage graph is constructed.

4. The method according to claim 1, wherein, The step of determining the time-series snapshot of the business metrics based on the anomaly occurrence time and the global heterogeneous lineage graph includes: For each node in the global heterogeneous lineage map, in response to determining that the time of the anomaly occurs is within the valid time range of the node, the node is determined as a time-series node; For each dependency edge in the global heterogeneous lineage graph, in response to determining that the time of the anomaly occurrence is within the valid time range of the dependency edge, the dependency edge is determined as a temporal dependency edge. The obtained time-series nodes are used as a node set, and the obtained time-series dependent edges are used as an edge set to construct a time-series graph snapshot.

5. The method according to claim 1, wherein, The method further includes: Obtain the running instance status of each node in the time series graph snapshot; In response to the determination that there is a failed or timed-out running instance state among the obtained running instance states, the node corresponding to the running instance state is determined as the root cause node.

6. The method according to claim 1, wherein, The step of generating standardized data probes for each node in the target attribution graph based on the target attribution graph and the anomaly type identifiers corresponding to the business metrics includes: Based on the preset operator information set, determine the verification operator corresponding to the anomaly type identifier; For each node in the target attribution graph, perform the following steps: Based on the node type and the preset contract information relationship set of the node, determine the standardized contract template of the node; The verification operator, the node attributes of the node, and the abnormal occurrence time of the business metric are injected into the standardized contract template to obtain the executable probe corresponding to the node. The executable probe is associated with the node and stored to obtain the standardized data probe of the node.

7. A cross-system anomaly repair device based on global heterogeneous lineage maps and dependency matrices, comprising: The acquisition unit is configured to, in response to receiving an anomaly repair request from each target device corresponding to the target heterogeneous system, acquire the anomaly indicator identifier and anomaly occurrence time of the business indicators of the target heterogeneous system where the anomaly occurred; The construction unit is configured to construct a global heterogeneous lineage graph of the target heterogeneous system, wherein the global heterogeneous lineage graph includes nodes in the target heterogeneous system and dependency edges between nodes; The first determining unit is configured to determine the input-output dependency matrix of each data computing node in the global heterogeneous lineage map. The second determining unit is configured to determine the starting node corresponding to the abnormal indicator in the global heterogeneous lineage map based on the abnormal indicator identifier. The third determining unit is configured to determine a time-series snapshot of the business indicator based on the time of the anomaly occurrence and the global heterogeneous lineage graph. The pruning unit is configured to prune the time series map snapshot based on the obtained input-output dependency matrices and the starting node to obtain the target factorization map corresponding to the business indicator. The generation unit is configured to generate standardized data probes for each node in the target attribution graph based on the target attribution graph and the anomaly type identifiers corresponding to the business indicators. The execution unit is configured to perform distributed execution processing on the obtained standardized data probes to obtain the execution results of each node in the target attribution factor graph; The fourth determining unit is configured to determine the node corresponding to the execution result as the root cause node in response to determining that there is an execution result that satisfies the preset positioning conditions among the execution results of each node. The output unit is configured to output a repair drive command to the target device corresponding to the root cause node according to the node type of the root cause node, so as to restore the business indicators to normal.

8. An electronic device, comprising: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 6.

9. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

Images