Data processing method and related device

Abstract

The invention provides a data processing method and a related device. The method comprises the steps of generating a data blood relationship map of first data based on a first database statement of the first data; in the data consanguinity map, determining a first object downstream of first data, the first data being a calculation dependency of the first object; determining a second object causing the upstream of the first data in the data consanguinity map, the first data being obtained in dependence on the second object; and performing task processing on the first data, the first object and the second object to obtain a processing result. According to the invention, the task processing efficiency of the data can be improved.

Description

Technical Field

[0001] The present application relates to database technology, and in particular to a data processing method and related devices. Background Art

[0002] With the ever-increasing volume of data in databases, it's crucial to analyze some of this data's upstream sources and downstream impacts to facilitate task processing. For example, identifying the upstream root cause and downstream dependencies of abnormal data before performing remediation tasks. However, related technologies suffer from inefficient data task processing. Summary of the Invention

[0003] The embodiments of the present application provide a data processing method and related devices, which can improve the efficiency of data task processing.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] This embodiment of the present application provides a data processing method, the method comprising:

[0006] generating a data lineage graph of the first data based on a first database statement of the first data;

[0007] In the data lineage graph, determining a first object downstream of the first data, where the first data is a dependency of the first object;

[0008] In the data lineage graph, determining a second object upstream of the first data, the first data being obtained by relying on the second object;

[0009] Task processing is performed on the first data, the first object, and the second object to obtain a processing result.

[0010] An embodiment of the present application provides a data processing device, including:

[0011] A graph generation module, configured to generate a data lineage graph of the first data based on a first database statement of the first data;

[0012] A first object module is configured to determine, in the data lineage graph, a first object downstream of the first data, where the first data is a dependency of the first object;

[0013] A second object module is configured to determine, in the data lineage graph, a second object upstream of the first data, the first data being obtained by relying on the second object;

[0014] The task processing module is used to perform task processing on the first data, the first object and the second object to obtain a processing result.

[0015] An embodiment of the present application provides an electronic device, comprising:

[0016] a memory for storing computer-executable instructions or computer programs;

[0017] The processor is used to implement the data processing method provided in the embodiment of the present application when executing the computer-executable instructions or computer programs stored in the memory.

[0018] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the data processing method provided in the embodiment of the present application when executed by a processor.

[0019] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the data processing method provided in the embodiment of the present application is implemented.

[0020] The embodiments of the present application have the following beneficial effects: when writing the first data into the database, a data lineage graph is generated for the first data using the first database statement, and then the first object is determined for the first data based on the data lineage graph, that is, the downstream influence range of the first data is determined, thereby completing the forward tracing of the first data, and then the second object upstream of the first data is determined based on the data lineage graph, that is, the object that directly or indirectly affects the first data, thereby completing the reverse tracing of the first data, and finally task processing is performed on the obtained first data, first object and second object. In this way, not only can two-way traceability of the first data be achieved, thereby quickly determining the upstream source and downstream influence of the first data, but also task processing is performed on the upstream source of the first data, the first data itself and the downstream influence, so as to more efficiently complete the task processing of the first data, thereby improving the efficiency of task processing of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a schematic diagram of an application environment of the data processing method provided in an embodiment of the present application;

[0022] Figure 2 This is a flow diagram of the data processing method provided in the embodiment of the present application. Figure 1 ;

[0023] Figure 3 is a schematic diagram of the upstream and downstream of the first data provided in an embodiment of the present application;

[0024] Figure 4 This is a flow diagram of the data processing method provided in the embodiment of the present application. Figure 2 ;

[0025] Figure 5 This is a flow diagram of the data processing method provided in the embodiment of the present application. Figure 3 ;

[0026] Figure 6 This is a flow diagram of the data processing method provided in the embodiment of the present application. Figure 4 ;

[0027] Figure 7 This is a flowchart of processing outbound call data in a database provided by an embodiment of the present application;

[0028] Figure 8 This is a flowchart of reverse tracing provided by an embodiment of the present application;

[0029] Figure 9 This is a schematic diagram of the architecture of the multi-layer quality assessment provided by the embodiment of the present application;

[0030] Figure 10 This is an architectural diagram of the bidirectional traceability and repair provided by an embodiment of the present application;

[0031] Figure 11 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0033] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0034] In the following description, the terms "first\second\third\fourth" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third\fourth" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0035] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0036] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0037] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0038] Before further explaining the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0039] 1) A database is a systematic electronic data storage method used to store, manage, query, and update data. In a database, data is stored according to specific structures, such as tables, documents, and key-value pairs, to facilitate efficient query and management.

[0040] 2) Database statements are a programming language used to interact with the database. Through database statements, operations such as querying, inserting, updating, and deleting data in the database can be performed.

[0041] 3) Data lineage graph is a graph data that describes the flow path of data. Through the data lineage graph, the generation, transmission, processing and use of data can be reflected to facilitate the management of database data.

[0042] The amount of data in the database is increasing day by day. For some data, it is necessary to analyze its upstream sources and downstream impacts to facilitate task processing. For example, first determine the upstream root cause and downstream dependencies of abnormal data, and then repair it to realize the processing of the repair task.

[0043] In the related art, for the data that needs to be analyzed, it is necessary to manually trace the data to determine the upstream source and downstream impact range of the data, and then perform task processing to complete the entire task processing process for the data. However, manual tracing is time-consuming, which means it takes a long time to determine the upstream source and downstream dependencies of the data, and ultimately it takes a long time to complete the task processing of the data, which is inefficient. Therefore, it can be seen that in the related art, there is a problem of low efficiency in data task processing.

[0044] In response to at least one of the above-mentioned problems existing in the related art, an embodiment of the present application provides a data processing method and related devices, the method comprising: generating a data lineage graph of the first data based on a first database statement of the first data; determining a first object downstream of the first data in the data lineage graph, the first data being a dependency of the first object; determining a second object upstream of the first data in the data lineage graph, the first data being obtained by being dependent on the second object; performing task processing on the first data, the first object, and the second object to obtain a processing result. In this way, not only can bidirectional traceability be achieved for the first data, thereby quickly determining the upstream source and downstream impact of the first data, but task processing can also be performed on the upstream source of the first data, the first data itself, and the downstream impact, so as to more efficiently complete the task processing of the first data, thereby improving the efficiency of task processing of the data. At the same time, since the electronic device first determines the first object downstream of the first data, when there is an anomaly in the first data, its downstream impact can be determined more quickly, which is more conducive to preventing the downstream spread of the anomaly.

[0045] In order to better understand the data processing method and related devices provided in the embodiments of the present application, the application environment applicable to the embodiments of the present application is described below.

[0046] See also Figure 1 , Figure 1 Schematic diagram of an application environment of the data processing method provided in the embodiment of the present application. As an implementation method, the data processing method provided in the embodiment of the present application can be applied to an electronic device, wherein the electronic device can be such as Figure 1 The server 110 shown can be connected to the terminal 120 through a network 130. The network 130 is used to provide a medium for a communication link between the server 110 and the terminal 120. The network 130 can include various connection types, such as wired communication links, wireless communication links, etc., which are not limited in this embodiment of the present application.

[0047] It should be understood that Figure 1The server 110, terminal 120, and network 130 are merely illustrative. Any number of servers, networks, and terminals may be used as needed. For example, the server 110 may be a physical server or a server cluster consisting of multiple servers, and the terminal 120 may be a smartphone, tablet computer, desktop computer, laptop computer, smartwatch, or other device. It will be appreciated that in embodiments of the present application, multiple terminals 120 may be allowed to access the server 110 simultaneously.

[0048] The embodiment of the present application can be implemented by a server. For example, when the server 110 detects that the first data needs to be written to the database, it will perform task processing on the first data based on the data processing method provided in the embodiment of the present application to facilitate efficient management of the first data.

[0049] The data processing method provided in the embodiments of the present application can be applied to various scenarios requiring data processing, such as repairing abnormal transaction data in the financial field, monitoring the flow of sensitive data in the data engineering field, etc. The following describes the scenarios in which the data processing method provided in the embodiments of the present application can be applied.

[0050] 1) Abnormal transaction data repair scenario. For example, when the abnormal transaction amount needs to be written into the database, the server generates a data lineage graph for the abnormal transaction amount according to the data processing method provided in the embodiment of the present application, determines the risk control model input downstream of the abnormal transaction amount in the data lineage graph, determines the amount calculation logic of the transaction amount that caused the abnormality in the data lineage graph, and then corrects the amount calculation logic. Thereafter, the transaction amount is recalculated according to the corrected amount calculation logic, and the risk control model input is re-determined to achieve full-link processing of the transaction amount.

[0051] 2) Sensitive data flow monitoring scenario. For example, when sensitive data needs to be written to a database, the server first generates a data lineage graph for the sensitive data according to the data processing method provided in the embodiment of the present application, and determines the data table located downstream of the sensitive data in the data lineage graph, as well as the input interface of the sensitive data in the data lineage graph, and then records the input interface, data table, etc. of the sensitive data to monitor the flow of sensitive data.

[0052] The following describes the data processing method provided in the embodiment of the present application. As mentioned above, the electronic device that implements the data processing method in the embodiment of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution entity of each step will not be repeated below.

[0053] See also Figure 2 , Figure 2 This is a flow diagram of the data processing method provided in the embodiment of the present application. Figure 1 , will combine Figure 2 The steps shown are explained, Figure 2 The main body of the step is the electronic device.

[0054] Step 101: Generate a data lineage graph of the first data based on a first database statement of the first data.

[0055] The electronic device first detects whether data is currently being written to the database using a database statement. If data is currently being written to the database using a database statement, the electronic device sets the data currently being written to the database as the first data and then identifies the database statement used to write the first data to the database as the first database statement. The electronic device then determines data lineage for the first data based on the first database statement and stores the data lineage as graph data, thereby generating a data lineage graph for the first data.

[0056] It should be noted that the first data can be any data that needs to be written to the database. It can be data generated and uploaded during the operation of a business application or business system, or it can be data obtained from a third party through an application programming interface. The first data can be a transaction amount in the financial field, an estimated delivery time in the logistics field, or a blood glucose measurement value in the medical field, etc., and the embodiments of this application are not limited here.

[0057] The database in the embodiments of the present application can be a relational database or a non-relational database. When the database is a relational database, the first database statement can be implemented as a Structured Query Language (SQL); when the database is a non-relational database, the first database statement can be implemented as a simple imperative statement or a JavaScript Object Notation (JSON) statement.

[0058] In some embodiments of the present application, Figure 2 Step 101, i.e., generating a data lineage graph of the first data based on the first database statement of the first data, can be achieved by the following processing: extracting statement elements from the first database statement and constructing a syntax tree based on the statement elements; performing lineage recognition based on the syntax tree, and generating a graph structure for the recognized data lineage to obtain a data lineage graph.

[0059] The electronic device can extract sentence elements from the first database statement through natural language processing technology, such as lexical analysis and syntactic analysis, wherein the sentence elements refer to basic elements such as keywords, table names, and column names in the first database statement. Afterwards, the electronic device generates a syntax tree using the hierarchical relationship between the sentence elements. Next, the electronic device traverses the syntax tree to identify the data lineage at the table level and the data lineage at the field level, and then uses the table and field as graph nodes, and establishes table-level edges based on the data flow in the table-level data lineage, and establishes field-level edges for data conversion and transmission in the field-level data lineage, and then integrates the graph nodes and edges into a directed graph, which is the data lineage graph.

[0060] Among them, the electronic device can first determine the target table of the first database statement operation, extract the source table from some specific clauses, such as FROM, JOIN, subquery and other clauses, and then establish the flow relationship between the source table and the target table, so as to obtain the data lineage at the table level; the electronic device can identify the write fields of the target table, such as the INSERT field list, SELECT result column, and then parse the calculation logic of the write fields to identify the corresponding source table fields, and finally record the conversion logic from the source table fields to the target fields to obtain the data lineage at the field level.

[0061] In other embodiments of the present application, Figure 2 Step 101, i.e., generating a data lineage graph of the first data based on the first database statement of the first data, can also be achieved by the following processing: parsing the first database statement to obtain statement elements, and using the statement elements to generate a semantic model of the business; establishing a semantic association relationship between tables and columns according to the semantic model and the grammatical structure of the first database statement, and generating a data lineage graph based on the semantic association relationship.

[0062] The electronic device can parse the first database statement to obtain statement elements such as table names, field names, and operations, and then map the statement elements to the semantic space of the business, such as mapping the user ID to the user's unique identifier, while simultaneously building a semantic model containing business rules, such as calculation logic and foreign key constraints. Subsequently, the electronic device first extracts direct dependencies based on the grammatical structure of the first database statement, and then determines indirect dependencies that are not directly reflected in the grammatical structure of the first database statement based on the semantic model containing business rules and contextual information, such as data directory annotations, and then generates semantic associations using direct and indirect dependencies. Finally, the electronic device performs lineage tracing in the semantic associations, that is, starting from the data source, such as the original table, external application interface, etc., it traces the data flow path layer by layer along the semantic associations to obtain data lineage, and then stores the data lineage as a directed graph, that is, obtaining a data lineage graph.

[0063] Step 102: In the data lineage graph, determine a first object downstream of the first data, where the first data is a dependency of the first object.

[0064] Among them, the dependency refers to the data that the object depends on during the generation or processing process. In the embodiment of the present application, after obtaining the data lineage graph, the electronic device will first combine the data lineage graph to determine the first object downstream of the first data, that is, the downstream object affected by the first data, so that the first data is the upstream dependency of the first object when it is generated or processed, and the first object is the downstream influence of the first data. It should be noted that the object here can refer to data, or it can refer to a system or application, or it can refer to data processing logic, business logic, etc., and the embodiment of the present application does not limit this.

[0065] Here, upstream refers to the earlier link in the data flow process, which is used to explain the source direction of the first data; downstream refers to the subsequent link in the data flow process, which is used to explain the subsequent impact of the first data. Figure 3 is a schematic diagram of the upstream and downstream of the first data provided in the embodiment of the present application. Figure 3 In the figure, the direction of data flow is represented by arrows. If the first data 3-1 is the sales volume in a report, it comes from the upstream 3-2, that is, the data interface of the report, and flows to its downstream 3-3, such as the risk control model.

[0066] It should also be noted that dependencies can include direct dependencies or indirect dependencies. Thus, the first object can include sub-objects that directly depend on the first data, as well as sub-objects that indirectly depend on the first data. A sub-object that directly depends on the first data refers to a sub-object that directly references the first data and has an explicit reference relationship with the first data, meaning that the dependency path between the first data and the first sub-object does not include other processing. A sub-object that indirectly depends on the first data refers to a sub-object that is indirectly associated with the first data through an intermediate node and has an implicit reference relationship with the first data.

[0067] See also Figure 4 , Figure 4 This is a flow diagram of the data processing method provided in the embodiment of the present application. Figure 2 In some embodiments of the present application, the first object includes a first sub-object and a second sub-object, the first data is a direct dependency of the first sub-object, and the first data is an indirect dependency of the second sub-object. In this case, Figure 2 Step 102, i.e., determining the first object downstream of the first data in the data lineage graph, can be implemented by the following process:

[0068] Step 1021: Locate the first graph node of the first data from the data lineage graph.

[0069] The electronic device first searches for a graph node corresponding to the first data in the data lineage graph and identifies the found graph node as the first graph node. It should be noted that the node information of the graph node may store information about the object to which it corresponds, and the electronic device can locate the graph node using the node information of the graph node.

[0070] Step 1022: Determine from the data lineage graph a second graph node that is located downstream of the first graph node and directly connected to the first graph node, and determine the object corresponding to the second graph node as the first child object.

[0071] Because all edges in a data lineage graph are directed, the electronic device can use the first graph node as a starting point and identify the graph node directly connected to the first graph node as the second graph node. This makes the second graph node the direct downstream node of the first graph node. The electronic device then identifies the object corresponding to the second graph node, such as data, application, system, or even processing logic, as the first sub-object, thereby obtaining the first sub-object that directly depends on the first data.

[0072] Step 1023: Using the first model and based on the data lineage graph, predict a third graph node that is downstream of the first graph node and not directly connected to the first graph node, and determine the object corresponding to the third graph node as the second child object.

[0073] The electronic device calls the first model and provides the data lineage graph to the first model, so as to predict the graph nodes that depend on the first graph node but are not directly connected to the first graph node from the data lineage graph through the first model, and determine these graph nodes as second graph nodes. In other words, the electronic device can use the first model to predict the nodes that are located downstream of the first graph node but are not directly connected to the first graph node, that is, the indirect downstream nodes of the first graph node, and predict the graph nodes that are affected by the first graph node but are not connected to the first graph node, that is, the graph nodes that have no link connection with the first graph node and are only associated with the business logic (for example, the graph node corresponding to the user's portrait field, and the graph node corresponding to the advertising delivery effect have no direct link connection and are only associated with the business logic), and use these graph nodes as third graph nodes, and determine the objects corresponding to the third graph nodes, such as data, systems, applications, etc., as second sub-objects.

[0074] It should be noted that the first model can be implemented as a graph neural network model (GNN), and the first model can obtain the prediction ability of the third graph node through training, wherein the electronic device can use the historical lineage graph of a certain data as input and use the real third graph nodes determined from the historical lineage graph based on the data as labels to perform model training.

[0075] It can be understood that in an embodiment of the present application, the electronic device can first determine the second graph node of the first graph node from the data lineage graph, that is, the direct downstream node, and then predict the third graph node through the first model, that is, the indirect downstream node of the first graph node and the graph node affected by the first graph node but not directly connected to it by a link, and based on the second graph node and the third graph node, it can simultaneously determine the explicit sub-object of the first sub-object and the more hidden second sub-object, so that a more comprehensive and accurate downstream influencing object can be obtained for the first data.

[0076] Step 103: In the data lineage graph, determine the second object upstream of the first data, and the first data depends on the second object.

[0077] After the electronic device completes the determination of the first object, it will continue to determine the second object in the data lineage diagram, where the second object is the object upstream of the first data. In other words, the second object is the upstream source of the first data, so that the first data is directly or indirectly dependent on the second object. The second object can be a data, or a data processing step, system, etc.

[0078] See also Figure 5 , Figure 5 This is a flow diagram of the data processing method provided in the embodiment of the present application. Figure 3 In some embodiments of the present application, Figure 2 Step 103 in the data lineage graph, i.e., determining the second object upstream of the first data, can be achieved by the following process:

[0079] Step 1031: For the first data, detect and obtain at least one first path from the data lineage graph.

[0080] It should be noted that the electronic device will detect the existence of potential data flow paths in the data lineage graph for the first data, and determine the detected data flow path as the first path, so that the electronic device can obtain one or more first paths, that is, the number of first paths can be one or more.

[0081] Among them, when the first data is abnormal data, that is, data whose quality does not meet the quality requirements, the first path can refer to the path in the data lineage map where the data flow deviates from the preset requirements. The preset requirements here can be preset technical processing rules, preset business logic or preset compliance standards. Therefore, if the first data is abnormal data, then the root cause of the abnormality of the first data is more likely to appear in the first path. Therefore, the first path is a potential abnormal path and needs to be analyzed in depth. Therefore, the electronic device needs to first determine the first path from the data lineage map, so that the root cause can be located faster in the first path later, rather than locating the root cause of the entire data lineage map, thereby improving the efficiency of determining the root cause of the abnormal first data.

[0082] Exemplarily, when the preset requirement is a preset technical processing rule, the first path may be a technical exception path, that is, a link that violates the data processing technical rules, such as a path where data loss, transmission interruption, conversion logic error, etc. occur; when the preset requirement is a preset business logic, the first path may be a path that violates the preset business logic, such as a path where the calculation logic is inconsistent with the business definition; when the preset requirement is a preset compliance standard, the first path may be a path that violates the data compliance requirement, such as a path where sensitive data is transmitted in plain text, etc.

[0083] In the case where the first data is normal data, that is, data whose quality meets the quality requirements, the first path may also be any path containing the first data.

[0084] In some embodiments of the present application, Figure 4 Step 1031, i.e., detecting and obtaining at least one first path from the data lineage graph for the first data, can be achieved by the following processing: sampling multiple candidate second paths from the data lineage graph; predicting the first probability of each second path through the second model; and determining the second path whose first probability is greater than the probability threshold as the first path.

[0085] When the electronic device determines the first path from the data lineage graph for the first data, it first samples the paths in the data lineage graph to obtain a second path. The second path is a candidate path for the first path, meaning the electronic device needs to identify the second path to determine whether it is a potential path requiring further analysis. The electronic device then invokes a second model and uses the second model to perform a probability prediction for each second path to obtain a first probability representing the second path. This first probability can be the probability that the second path is an abnormal path or the probability that the second path is a path requiring further analysis. The electronic device then determines as the first path a number of second paths whose first probabilities are greater than a probability threshold, or whose first probabilities are the highest.

[0086] It should be noted that the electronic device can use the graph node corresponding to the first data in the data lineage graph, i.e., the first graph node, as the end point and extract its upstream path, i.e., extract the path that can ultimately reach the first graph node, as the second path. Here, when extracting the upstream path, the electronic device can set a maximum number of hops, such as 5 hops, and determine all upstream paths that do not exceed the maximum number of hops as the second path. The electronic device can also use a random walk method to sample the second path from the data lineage graph.

[0087] The second model can be implemented as a graph neural network model or as a generative adversarial network (GAN). The second model can acquire the ability to predict the first probability of the second path through training. Taking the first probability as the abnormal probability, the training process is explained. In this case, during training, the historical database statements, that is, the database statements used to operate the database at historical time, can be used as the input of the generative network. The generative network generates abnormal database statement samples based on the historical database statements, such as dynamically spliced ​​malicious query statements. Then, based on the abnormal database statement samples, it can generate a trained abnormal path. The discriminant network predicts the abnormal probability of the abnormal path and calculates the loss value based on the predicted abnormal probability and the actual abnormal probability of the abnormal path to adjust the parameters of the discriminant network so that the discriminant network has the ability to identify abnormal paths, that is, predict the abnormal probability of the path.

[0088] It can be understood that in the embodiment of the present application, the electronic device can first sample multiple candidate second paths and then call the second model to perform a probabilistic prediction on the second path. It can not only quickly complete the determination of the first path based on probability, but also use the ability obtained by training the second model to identify unconventional paths that are difficult for human beings to directly identify, thereby helping to more accurately determine the path that needs to be analyzed in the next step.

[0089] Step 1032: Determine a second object upstream of the first data based on at least one first path.

[0090] After obtaining at least one first path, the electronic device locates the upstream impact of the first data based on the obtained one or more first paths to obtain the second object. If there is only one first path, i.e., if at least one first path only includes one first path, the electronic device will directly backtrack on that first path to locate the root cause. If there are multiple first paths, i.e., if at least one first path includes multiple first paths, the electronic device can further select from the multiple first paths to determine the path that actually needs to be backtracked, and then backtrack on the determined path to locate the root cause.

[0091] That is to say, in some embodiments of the present application, Figure 5 Step 1032, i.e., determining the second object upstream of the first data based on at least one first path, can be achieved by the following processing: in response to the number of the first paths being multiple, selecting a third path from the multiple first paths; starting from the end graph node of the third path, determining the states of the graph nodes upstream of the end node in sequence, and determining a fourth graph node based on the states of the graph nodes, and determining that the object corresponding to the fourth graph node is the second object.

[0092] There are multiple first paths, which means that there are multiple data flow paths that may contain the first data in the data lineage graph. However, considering factors such as repair efficiency, it is usually necessary to select one from the multiple first paths to locate the upstream source. Therefore, the electronic device needs to continue to screen the multiple first paths to obtain a priority backtracking path, and determine the priority backtracking path as the third path. Here, the priority backtracking path can refer to the shortest backtracking path, such as a path that directly points to the upstream source, or it can refer to the path with the greatest possibility of causing anomalies, such as a path with a node failure rate higher than the corresponding threshold, or it can be a path with the highest business priority, such as a path that affects core business, etc. It can be seen from this that the third path can also be understood as the path with the highest comprehensive benefit of backtracking among the multiple first paths.

[0093] It should be noted that the electronic device can perform dynamic path evaluation on multiple first paths through reinforcement learning to determine the third path therefrom. In some embodiments, the electronic device can calculate the risk weight of each graph node in each first path through a reinforcement learning model, calculate the selection reward of each first path based on the risk weight of each graph node, and then select the first path with the highest reward as the third path. The risk weight of each graph node can be determined based on the attributes of the object corresponding to the graph node. For example, for objects involving sensitive data, the corresponding graph node will be set with a higher risk weight, and for interfaces with unauthorized access, the corresponding graph node will also be set with a higher risk weight; the selection reward of each first path can be determined by a reward mechanism. For example, when approaching a graph node with a risk weight higher than a corresponding threshold, a certain reward value is increased, and when approaching a graph node with a risk weight lower than the corresponding threshold, a certain reward value is reduced, and so on.

[0094] For example, if the first data item is a user's importance level, the electronic device can determine multiple first paths, i.e., multiple potential data flow paths, from the data lineage graph for the first data. These paths include: Path 1: original user table data error → uncorrected cleaning task → level calculation error; Path 2: external data source interface return data error → aggregation task reference error. The electronic device calculates a risk score for each node in the first path and finds that the external data source interface timeout rate is higher than a threshold. Therefore, the external data source interface is assigned a higher risk weight. Based on this, the electronic device selects Path 2 as the third path for priority backtracking.

[0095] If the first data is abnormal, after obtaining the third path, the electronic device can use the graph node at the end of the third path (i.e., the terminal node) as the backtracking starting point, backtrack to the upstream graph nodes in sequence, and check the status of the upstream graph nodes to determine whether the upstream graph nodes are in an abnormal state. The electronic device can then use the object corresponding to the abnormal graph node, such as data, application, or system, as the second object. If the first data is normal, the electronic device can use all graph nodes in the third path upstream of the terminal node as the second object.

[0096] It should be noted that the state of the upstream graph node can be determined based on the properties of the object corresponding to the upstream graph node. For example, if the object corresponding to the upstream graph node is data, whether the upstream graph node is in an abnormal state can be determined based on whether the data is abnormal (for example, whether it is empty or negative); if the object corresponding to the upstream graph node is data cleaning processing, whether the upstream graph node is in an abnormal state can be determined based on whether the logic of the cleaning processing is correct (for example, whether the filtering rules are correct, whether the null value processing is correct); if the object corresponding to the upstream graph node is a third-party interface, whether the upstream graph node is in an abnormal state can be determined based on whether the third-party interface runs normally (for example, whether the call has timed out, whether the returned data format is correct).

[0097] It is understandable that in the embodiment of the present application, the electronic device will first determine a potential first path from the data lineage map for the first data, so as to narrow the range of paths that need to be traced back when locating the upstream source through the first path, and determine the second object based on the first path, so as to be able to quickly and accurately locate the second object of the first data in the data lineage map, thereby improving the efficiency of data task processing. In addition, the electronic device will further distinguish from multiple potential abnormal paths, that is, multiple first paths, to select a third path that is preferentially traced back from the first path, and then trace back on the third path to determine the upstream source of the first data, that is, the second object. In this way, the number of paths that need to be traced back can be further reduced, thereby further improving the efficiency of upstream source determination, and thus further improving the efficiency of data task processing.

[0098] Step 104: Perform task processing on the first data, the first object, and the second object to obtain a processing result.

[0099] After obtaining the first data, the first object and the second object, the electronic device performs task processing on the determined content and obtains a processing result after the task processing is completed.

[0100] It should be noted that in the embodiment of the present application, task processing can be exception repair processing, for example, when the first data is abnormal data, the second object is repaired to obtain the repaired second object, and based on the repaired second object, the first data and the first object are repaired to obtain the repaired first data and the repaired first object; task processing can be data directory construction, that is, automatically generating a data asset list based on the first object and the second object of the first data; task processing can be metadata management, that is, storing corresponding metadata for the first data, the first object and the second object of the first data, so as to form a more complete blood relationship for the first data; task processing can be sensitive data flow tracking to determine the flow path of the sensitive first data and ensure the processing of sensitive data Management compliance; task processing can be audit tracking, that is, recording the origin and scope of changes to the first data, and providing verifiable path evidence for the audit of the first data; task processing can be data desensitization verification to confirm the source of sensitive first data and whether the first data is correctly desensitized downstream; task processing can be permission management, that is, determining whether the second object downstream is the same as the preset object allowed to access the first data; task processing can be feature engineering traceability, used to track the generation path of model input features, such as the path of original data, feature table and training set, to enhance the interpretability of the model; task processing can also be version control of experimental environment and production environment, that is, associating the data flow of experimental environment and production environment to avoid inconsistency between training data and production data.

[0101] Taking abnormality repair processing as an example, after the electronic device determines the second object of the first data, it will first repair the second object to repair the root cause of the abnormality of the first data, and then repair the first data and the first object downstream of the first data in sequence, so as to ensure the consistency and integrity of the data. It should be noted that the electronic device can repair the second object through a preset repair operation, and then update the first data based on the repaired second object to achieve the repair of the first data. Then, based on the repaired first data, the downstream first object is repaired to obtain the repaired first object.

[0102] In some embodiments of the present application, the processing result includes: the repaired first data, the repaired first object, and the repaired second object. Figure 2 Step 104 in the above, i.e., performing task processing on the first data, the first object, and the second object to obtain a processing result, can be achieved by the following processing: predicting the execution probability of multiple preset second repair operations based on the second object using the fourth model; selecting a third repair operation from the multiple second repair operations based on the execution probability, and generating a second database statement for the third repair operation; repairing the first object using the second database statement to obtain a repaired first object; repairing the first data based on the repaired first object to obtain the repaired first data; and repairing the first object based on the repaired first data to obtain the repaired first object.

[0103] That is, when the first data is abnormal data, the first object is the downstream impact of the abnormal data, the second object is the upstream source of the abnormal data, and the task processing is abnormal repair processing, the electronic device calls the fourth model and provides the fourth model with information about the second object and information about each preset second repair operation. The fourth model then predicts the execution probability of each preset second repair operation. The execution probability represents the probability that the second repair operation can correctly repair the second object. The electronic device can then select the second repair operation with the highest execution probability as the third repair operation, or select any second repair operation with an execution probability above a corresponding threshold as the third repair operation. A database statement is then generated for the third repair operation, and the generated database statement is used as the second database statement. The electronic device then executes the second database statement to repair the second object using the second database statement, thereby obtaining the repaired second object.

[0104] It should be noted that the preset multiple second repair operations may include data backfill operations, permission repair operations, field mapping operations, syntax completion operations, and the like. The third model may be implemented as a Deep Reinforcement Learning (DRL) Decision Maker, which can be trained to predict the probability of executing the second repair operation. The training input may be information about different repair operations and root causes, and the label may be the final selected repair action. The electronic device may pre-set some database statement templates, and then backfill the database statement templates according to the third repair action to obtain a complete second database statement.

[0105] The electronic device can then reprocess the original data used to generate the first data based on the repaired second object to obtain normal data, which is the repaired first data. Finally, the electronic device can provide the repaired first data to the downstream first object, allowing the downstream first object to be rebuilt or updated based on the repaired first data, thereby repairing the first object.

[0106] In other embodiments of the present application, after selecting a third repair operation from multiple second repair operations based on the execution probability and generating a second database statement for the third repair operation, repairing the first data based on the repaired second object to obtain the repaired first data, and repairing the first object based on the repaired first data to obtain the repaired first object, the method may also include the following processing: performing semantic correction on the second database statement to obtain the third database statement; and repairing the second object through the third database statement to obtain the repaired second object.

[0107] It should be noted that since the second database statement generated directly based on the third repair operation may have some field type conflicts or missing dependencies, the electronic device needs to perform semantic correction on the second database statement to eliminate field type conflicts or complete dependencies, thereby obtaining a more reasonable third database statement. The electronic device then uses the third database statement to repair the second object, thereby better repairing the second object. The electronic device can perform semantic correction on the second database statement using a large language model.

[0108] It can be understood that in the embodiment of the present application, when the electronic device repairs the second object, it can first use the fourth model to predict the execution probability of different second repair operations, and then select the third repair operation from the different second repair operations based on the execution probability, so that a more reasonable third repair operation can be used to repair the second object, thereby improving the accuracy of the repair; in addition, the electronic device can first generate an initial second database statement for the third repair operation, and then optimize the second database statement through semantic correction to obtain a third database statement, so that a more reasonable database statement can be used to repair the second object, further improving the repair accuracy of the second object.

[0109] At this point, the electronic device has completed processing the first data.

[0110] It can be understood that compared with the relevant technology, since it is necessary to manually trace the abnormal data first to determine the root cause of the data anomaly and the downstream impact range, and then repair it to complete the processing of the abnormal data, in the embodiment of the present application, the electronic device can use the first database statement to generate a data lineage graph for the first data when writing the first data to the database, and then first determine the first object for the first data based on the data lineage graph, that is, determine the downstream impact range of the first data, thereby completing the forward tracing of the first data, and then determine the second object upstream of the first data based on the data lineage graph, that is, the object that directly or indirectly affects the first data, thereby completing the reverse tracing of the first data, and finally perform task processing on the first data, first object and second object obtained. In this way, not only can two-way tracing of the first data be achieved, thereby quickly determining the upstream source and downstream impact of the first data, but also the upstream source of the first data, the first data itself and the downstream impact are sequentially processed to more efficiently complete the task processing of the first data, thereby improving the efficiency of task processing of the data. At the same time, because the electronic device first determines the first object downstream of the first data, when an abnormality exists in the first data, its downstream impact can be determined more quickly, which is more conducive to preventing the abnormality from spreading downstream.

[0111] The quality assessment process of the first data is described below. The quality assessment is used to determine whether the first data has any anomalies.

[0112] based on Figure 2 , see Figure 6 , Figure 6 This is a flow diagram of the data processing method provided in the embodiment of the present application. Figure 4 In some embodiments of the present application, Figure 2 After step 102 and before step 103, that is, after determining the first object downstream of the first data in the data lineage graph and before determining the second object upstream of the first data in the data lineage graph, the method may further include the following processing:

[0113] Step 105: Determine, for the first data, first quality information corresponding to the first data in multiple dimensions.

[0114] The electronic device performs a quality assessment on the first data in each of multiple dimensions to obtain first quality information of each dimension, wherein the first quality information of each dimension represents relevant indicators of whether the first data meets preset standards of each dimension. The preset standards here can be accuracy standards, completeness standards, reliability standards, etc. Therefore, the first quality information reflects information such as the credibility and applicability of the first data.

[0115] It should be noted that the multiple dimensions in the embodiments of this application may include technical dimensions, business dimensions, and management dimensions. The technical dimension is used to ensure the basic quality of data, and may include sub-dimensions such as completeness, accuracy, consistency, and timeliness. The business dimension is used to ensure the business availability of data, and may include sub-dimensions such as logical correctness, business compliance, and timeliness. The management dimension is used to drive data value optimization, and may include sub-dimensions such as compliance auditing, cost-effectiveness, and risk scoring.

[0116] In some embodiments of the present application, the plurality of dimensions includes: a first dimension, in which case, Figure 6 Step 105, i.e., determining the first quality information of the first data in each of the multiple dimensions, can be achieved by the following processing: obtaining the quality requirement information of the first dimension, and generating a regular expression for quality verification based on the quality requirement information through a third model; performing quality verification on the first data using the regular expression of the first data, and determining the verification result as the first quality information of the first dimension.

[0117] The electronic device can obtain quality requirement information for the first dimension, then parse the quality requirement information using a third model to determine how quality verification should be performed for the first dimension, and then generate a corresponding regular expression. The regular expression then describes the quality verification rules for the first dimension. The electronic device then substitutes the first data into the regular expression and performs an operation to complete the quality verification of the first data. The output of the regular expression is the verification result, and the electronic device determines the verification result as the first quality information for the first dimension.

[0118] It should be noted that the first dimension here can refer to any one of the technical dimension, business dimension, and management dimension. Therefore, the quality requirement information in the first dimension can be any one of the technical, business, and management dimension quality requirements. The third model can be a large language model.

[0119] For example, if the first dimension is a business dimension and its quality requirement information is "the email field must comply with the standard email format", the third model can generate a regular expression for quality verification based on the quality requirement information: ^[a-zA-Z0-9._%+-] +@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$ .

[0120] It can be understood that in an embodiment of the present application, the electronic device can parse the quality requirement information of the first dimension through the third model field to determine a regular expression for quality verification, and implement quality verification through the regular expression, so that the quality verification of the first data can be completed efficiently.

[0121] In some other embodiments of the present application, the multiple dimensions include: a second dimension, and the second dimension includes multiple sub-dimensions. In this case, Figure 6 Step 105, i.e., determining the first quality information of the first data in each of the multiple dimensions, can be achieved by the following processing: determining the quality scores of the first data in the multiple sub-dimensions; summing the quality scores of the multiple sub-dimensions, and determining the summed result as the first quality information of the first data in the second dimension.

[0122] It should be noted that in embodiments of the present application, quality score calculation rules can be preset for each of the multiple sub-dimensions in the second dimension. The electronic device then calculates a quality score for each sub-dimension of the first data using the preset quality score calculation rules for each sub-dimension. The electronic device can then directly sum or weightedly sum the quality scores of all sub-dimensions, and use the summed result as the first instruction information. The second dimension can be any one of a business dimension, a computing dimension, and a management dimension.

[0123] For example, if the second dimension is a management dimension, it has sub-dimensions such as compliance audit, cost-effectiveness, and risk scoring. Therefore, the electronic device can first use the preset quality score calculation rules of each sub-dimension. For example, the calculation rule of compliance audit is the weighted sum of all compliance indicator scores and the weights of the compliance indicators. The calculation rule of cost-effectiveness is to first compare the business value of the data with the storage cost, and then calculate the product of the ratio and the access frequency of the data, etc., and then use the preset weights of these sub-dimensions to perform weighted summation of the quality scores of these sub-dimensions to obtain the first quality information.

[0124] It is understandable that in the embodiment of the present application, the electronic device may further comprehensively consider the quality scores of different sub-dimensions in the second dimension to obtain the first quality information, thereby making the calculation of the first quality information more accurate.

[0125] Step 106: When the first quality information of any dimension meets the quality abnormality condition, determine that the first data is abnormal.

[0126] If the first quality information of any dimension meets the corresponding quality abnormality condition, the electronic device will determine that the first data is abnormal, thereby obtaining a quality verification result indicating that the first data has a quality abnormality. It should be noted that the quality abnormality conditions of different dimensions can be obtained by analyzing historical data using a clustering algorithm, and can be manually set as needed, and this embodiment of the present application is not limited thereto.

[0127] It can be understood that in an embodiment of the present application, the electronic device can perform quality assessment on the first data from multiple different dimensions, and complete the quality assessment of the first data based on the relationship between the first quality information of any dimension and the corresponding quality abnormality condition. In this way, the quality assessment of the first data can be performed more comprehensively, so that the accuracy of the quality assessment of the first quantity can be improved.

[0128] In some embodiments of the present application, Figure 2 After step 102, that is, after determining the first object downstream of the first data in the data lineage graph, the method may further include the following processing: when the first object and the preset downstream object of the first data are inconsistent, determining a first repair operation for the first object; repairing the first object through the first repair operation to obtain a repaired first object.

[0129] The electronic device can obtain a preset downstream object for the first data, and then compare the first object with the preset downstream object. If the first object and the preset downstream object are inconsistent, that is, an object that should not reference the first data references the first data, or should reference the first data but does not, or the business logic of the first object is inconsistent with the preset, then it means that the downstream impact range of the first data is inconsistent with expectations, and the downstream impact range of the first data is abnormal and needs to be repaired to avoid the first data from being used incorrectly. Afterwards, the electronic device can use the first repair operation to repair the first object, that is, generate a corresponding database statement for the first repair operation, and then execute the database statement to complete the repair, such as increasing or decreasing the first object's reference permission to the first data, or modifying the business logic of the first object, etc.

[0130] The first repair operation here can be selected from multiple preset second repair operations. The selection process is similar to the selection process of the third repair operation in the above content, and will not be repeated here.

[0131] It is understood that in the embodiments of the present application, the electronic device can proactively repair the first object when the first object downstream of the first data does not conform to expectations, so that the repaired first object is consistent with the preset downstream object. In this way, the electronic device can also proactively repair the first object downstream of the first data when an anomaly occurs, thereby further achieving full-link processing of the first data.

[0132] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0133] The embodiment of the present application is implemented in a scenario where the server processes the outbound call data of the call system stored in the database, so that the root cause of the abnormal outbound call data in the database can be located and the downstream impact range can be analyzed, and the data can be automatically repaired.

[0134] Figure 7 This is a flowchart of processing outbound call data in the database provided by the embodiment of the present application. Figure 7 , the process may include the following processing:

[0135] Step 201: Collect SQL statements (called first database statements) in real time.

[0136] Step 202: Transmit the SQL statement to the SQL parsing layer via asynchronous transmission.

[0137] The SQL collection component transmits SQL statements to the SQL parsing layer through asynchronous transmission.

[0138] Step 203: Parse the SQL statement.

[0139] The SQL parsing layer is not limited to static syntax parsing, but tracks changes in the SQL execution environment in real time, deeply analyzes complex nested queries, distributed database interactions, and function calls, and splits SQL statements into basic elements such as keywords, table names, column names, operators, etc. (called statement elements).

[0140] Step 204: Construct an abstract syntax tree according to the parsing result.

[0141] Afterwards, the SQL parsing layer constructs an abstract syntax tree and presents the statement structure of the SQL statement.

[0142] For example, the common SQL statements for the call system are:

[0143]

[0144] The electronic device can parse the data table names "call_records" and "customer_info," the association condition "call_records.customer_id = customer_info.customer_id," and the filter condition "call_duration>60."

[0145] Step 205: Bloodline tracing.

[0146] The server can perform multi-level lineage tracking based on the syntax tree. For example, at the table association level, the lineage between tables is determined by analyzing the JOIN clause. At the subquery level, a sampling recursive algorithm queries each layer of subqueries to determine the source and destination of the data. For functions and expressions, their impact on data lineage is analyzed.

[0147] In addition, electronic devices can also achieve lineage tracing through the following processing. For example, electronic devices build a semantic model, map the basic elements parsed from the SQL statement to semantic concepts, and then establish semantic association relationships between tables and columns based on the semantic model. At the same time, considering the business meaning and contextual information of the data, the semantic association relationships between the data are judged, and then lineage tracing is performed based on the semantic association relationships. For example, the semantic association relationship is identified through a recursive formula expression, and starting from the specified target table and column, all indirect upstream columns that meet specific semantic conditions are traced back layer by layer, and traced upstream through the recursive connection table, and finally the complete data lineage is output.

[0148] Step 206: Construct a lineage graph (called a data lineage graph).

[0149] Step 207: Perform a bidirectional traceability query on the data.

[0150] Among them, two-way traceability includes forward traceability and reverse traceability.

[0151] The server can perform forward tracing for all data and reverse tracing only for data with anomalies. Whether the data has anomalies can be determined through quality assessment.

[0152] For example, Figure 8 This is a flowchart of reverse tracing provided by the embodiment of this application. Figure 8 , the process includes the following processing:

[0153] Step 2071: The data pipeline submits data batches to the quality assessment engine.

[0154] Step 2072: The quality assessment engine performs quality assessment.

[0155] Step 2073: If an exception exists, the quality assessment engine sends information triggering the exception event to the traceability and repair engine.

[0156] Step 2074: The source tracing and repair engine reversely traces the source and locates the root cause.

[0157] Step 2075: The source tracing and repair engine performs repair operations on the data in the data pipeline.

[0158] Step 2076: The data pipeline resubmits the data batch to the quality assessment engine.

[0159] Step 2077: If there is no abnormality, the quality assessment engine sends a release message to the data pipeline to the downstream.

[0160] In the embodiment of the present application, the server's quality assessment engine can perform quality assessments on data at multiple levels (referred to as dimensions). The quality assessment engine's clustering algorithm sets dynamic quality baselines for each level based on historical data characteristics. Once the baseline threshold is exceeded, the data is determined to be abnormal, thereby triggering a reverse tracing and repair mechanism. The multiple levels here include the technical layer, the business layer, and the management layer.

[0161] For example, Figure 9 This is a schematic diagram of the architecture of the multi-layer quality assessment provided by the embodiment of the present application. The server performs quality assessment on the data source 9-1 at the technical layer 9-2, the business layer 9-3, and the management layer 9-4. If an abnormality occurs at any layer, it will trigger the quality repair 9-5, and feedback will be given to the data source 9-1 after the repair is completed. Among them, the technical layer 9-2 can include dimensions such as completeness, accuracy, consistency, and timeliness. The business layer 9-3 can include dimensions such as logical consistency, business rule compliance, and timeliness. The management layer 9-4 can include dimensions such as compliance audit, cost-effectiveness, and risk scoring.

[0162] It's important to note that the technical layer ensures the fundamental quality of data. Integrity can refer to field non-null rates and data partition coverage, accuracy can refer to data type matching and range validation pass rates, consistency can refer to foreign key constraint satisfaction rates and redundant data consistency, and timeliness can refer to data collection latency and processing task SLA compliance rates. Quality assessment at the technical layer can be achieved through automated rule engine pre-verification rules (e.g., regular expressions) and real-time monitoring (e.g., real-time calculation of quality indicators).

[0163] The business layer is used to ensure data availability; logical consistency can refer to the consistency of the derived indicator calculation logic with the requirements document, business rule compliance can refer to the desensitization coverage of sensitive fields, etc., and timeliness refers to the on-time report generation rate, application interface response delay, etc. Quality assessment at the business layer can be achieved through indicator lineage binding and dynamic rule generation. Among them, indicator lineage binding refers to associating quality rules with nodes in the lineage graph. For example, the original field that the "sales" field of the report depends on must pass the integrity check; dynamic rule generation refers to using a large language model to parse business documents (called quality requirement information) to generate verification logic.

[0164] The management level is used to drive data value optimization; compliance audits can refer to sensitive data tracking coverage and data retention policy compliance; cost-effectiveness can refer to storage redundancy and the proportion of low-value data; and risk scoring can be calculated based on the frequency of anomalies at the technical and business levels. The management level's quality assessment can be determined through a compliance knowledge graph, a cost optimization model, and a risk scoring formula. Among them, the compliance knowledge graph refers to building a relationship between rule requirements and data to automatically identify risk points; the cost optimization model predicts cold data based on historical access patterns and triggers automatic archiving; the risk scoring formula is obtained by weighted summation of the frequency of technical layer anomalies, business impact weights, and the severity of compliance violations (called the quality score of the sub-dimension).

[0165] Step 208: Display the traceability query results.

[0166] The server can display the traceability query results on a graphical interface for users to understand.

[0167] Step 209: Determine the repair operation based on the traceability query result and provide feedback for optimization.

[0168] Figure 10 This is the architecture diagram of the two-way traceability and repair provided by the embodiment of this application. Figure 10 , for any data 10-1, it is first processed by the forward tracing module 10-2, wherein the forward tracing module 10-2 may include a stream processing engine 10-21 and a dependency chain predictor 10-22. If the output of the forward tracing module 10-2 produces an abnormal prediction result, the repair center 10-3 will repair it. The repair center 10-3 includes a DRL decision maker 10-31 and a semantic corrector 10-32. The repair center 10-3 will generate a repair instruction for repair. After that, the repaired data will flow to the storage and feedback layer 10-4. The storage and feedback layer 10-4 includes a real-time lineage data storage 10-41 and a repair log storage 10-42. The repair center 10-3 will also provide log records to the repair log storage 10-42. Abnormal data 10-8 needs to be processed by the reverse tracing module 10-5, which includes an adversarial detector 10-51 and a root cause location DRL 10-52. The root cause determined by the root cause location DRL 10-52, i.e., the source of the abnormality, can be provided to the DRL decision maker 10-31 for decision making. The data in the repair log storage 10-42 can be provided to the root cause location DRL 10-52 and the dependency chain predictor 10-22 for model optimization. In addition, the data in the real-time lineage data storage 10-41 can be provided to the external lineage data visualization module 10-6 and the external real-time application program interface 10-7.

[0169] Among them, the forward tracing module is used to track the flow of data in real time and predict the impact of changes. The flow processing engine subcomponent can detect SQL execution logs or DDL change events, parse them into field-level operations, and build a real-time dependency chain. The dependency chain subcomponent can be implemented through a graph neural network, which trains a graph neural network based on a historical lineage graph and predicts the downstream impact path of the data (called the second sub-object) through the trained graph neural network, such as ETL tasks spanning more than 10 layers. In more detail, the graph neural network can learn the embedding representation by aggregating the information of the neighbor nodes of the node in the lineage graph, and then use a graph convolution layer to extract and activate local neighborhood features, and then use another graph convolution network to integrate high-order graph structure information, and finally output node features to characterize its downstream impact path.

[0170] The reverse tracing module is used to locate the root cause of data anomalies and detect hidden paths. Its adversarial detector subcomponent can be implemented through an adversarial network. The adversarial network generates obfuscated SQL samples, such as dynamically spliced ​​malicious queries, and then uses the paths of the obfuscated SQL samples to train the model to identify unconventional lineage paths (called the first path). The adversarial network generator simulates the grammatical patterns of legitimate SQL statements and injects hidden abnormal payloads to generate obfuscated SQL statements that can bypass regular expressions or engine rules. Its core is to capture SQL context dependencies through long-short-term memory networks and use the softmax layer to output grammatically correct but implicitly abnormal SQL statements word by word. The root cause locator subcomponent can be implemented through a deep reinforcement learning model, which converts the lineage graph into a Markov decision process and then selects the optimal backtracking path (called the third path) based on the Q value to locate the root cause.

[0171] The repair center is used to automatically generate and execute repair operations. Its DRL decision maker subcomponent receives abnormal signals from the forward tracing module or the reverse tracing module, and then generates repair operations, such as field mapping and syntax completion, and optimizes the strategy through the reward function. Among them, the reward function can guide the DRL decision maker to prioritize efficient repair strategies with minimal system disturbances by weighted balancing the effectiveness, efficiency and side effects of data repair operations; the DRL decision maker models the generation of repair operations as a reinforcement learning problem, defines the action space, such as automatic completion, rollback, manual intervention, as well as the state space and reward mechanism, so that the DRL decision maker can learn the optimal repair strategy through interaction with the environment, and make automated decisions to repair data and minimize the impact on downstream. The semantic corrector subcomponent can be implemented by a large language model, which can parse the semantic context of SQL statements (called second database statements) and correct field type conflicts or missing dependencies.

[0172] The real-time lineage storage subcomponent in the storage and feedback layer can store lineage data through persistent storage and caching of real-time changes; the repair log feedback subcomponent indexes repair records for continuous model training.

[0173] At this point, two-way traceability and repair of data can be completed.

[0174] It is understandable that in the embodiments of the present application, when user information is involved, such as outbound call data and other related data, when the embodiments of the present application are applied to specific products or technologies, user permission or consent must be obtained, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards.

[0175] See also Figure 11 , Figure 11 is a structural diagram of an electronic device provided in an embodiment of the present application, Figure 11 The electronic device 400 shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the electronic device 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 11 Various buses are labeled as bus system 440 .

[0176] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0177] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0178] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0179] The memory 450 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0180] In some embodiments, the memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0181] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0182] A network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include Bluetooth, Wi-Fi, and Universal Serial Bus (USB);

[0183] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0184] The input processing module 454 is configured to detect one or more user inputs or interactions from one of the one or more input devices 432 and to translate the detected inputs or interactions.

[0185] In some embodiments, the data processing device provided in the embodiments of the present application can be implemented in software. Figure 11 The data processing device 455 stored in the memory 450 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: a graph generation module 4551, a first object module 4552, a second object module 4553, a task processing module 4554, and an anomaly determination module 4555. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0186] In other embodiments, the data processing device provided in the embodiments of the present application can be implemented in hardware. As an example, the data processing device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components.

[0187] In some embodiments, the electronic device can implement the data processing method provided by the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be commands, machine instructions or software instructions at the microprogram level. The computer program can be a native program or software module in the operating system; it can be a local (Native) application (APPlication, APP), that is, a program that needs to be installed in the operating system to run, such as a data processing APP; it can also be a small program that can be embedded in any APP, that is, a program that only needs to be downloaded to a browser environment to run. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0188] The following continues to describe the exemplary structure of the data processing device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 11 As shown, the software modules stored in the data processing device 455 of the memory 450 may include:

[0189] A graph generation module 4551 is configured to generate a data lineage graph of the first data based on a first database statement of the first data;

[0190] A first object module 4552 is configured to determine, in the data lineage graph, a first object downstream of the first data, where the first data is a dependency of the first object;

[0191] A second object module 4553 is configured to determine, in the data lineage graph, a second object upstream of the first data, the first data being dependent on the second object;

[0192] The task processing module 4554 is used to perform task processing on the first data, the first object and the second object to obtain a processing result.

[0193] In the above scheme, the first object includes a first sub-object and a second sub-object, the first data is a direct dependency of the first sub-object, and the first data is an indirect dependency of the second sub-object; the first object module 4552 is also used to locate the first graph node of the first data from the data lineage graph; determine the second graph node that is downstream of the first graph node and directly connected to the first graph node from the data lineage graph, and determine the object corresponding to the second graph node as the first sub-object; through the first model, based on the data lineage graph, predict the third graph node that is downstream of the first graph node and not directly connected to the first graph node, and determine the object corresponding to the third graph node as the second sub-object.

[0194] In the above solution, the second object module 4553 is further used to detect at least one first path from the data lineage graph for the first data; and determine the second object upstream of the first data based on the at least one first path.

[0195] In the above scheme, the second object module 4553 is also used to sample multiple candidate second paths from the data lineage graph; predict the first probability of each second path through the second model; and determine the second path whose first probability is greater than the probability threshold as the first path.

[0196] In the above scheme, the second object module 4553 is also used to select a third path from multiple first paths in response to the number of the first paths being multiple; starting from the end node of the third path, determine the status of the graph nodes located upstream of the end node in sequence, and determine the fourth graph node based on the status of the graph node, and determine that the object corresponding to the fourth graph node is the second object.

[0197] In the above scheme, the data processing device 455 also includes: an abnormality determination module 4555, which is used to determine the first quality information corresponding to the first data in multiple dimensions; when the first quality information of any one of the dimensions meets the quality abnormality condition, it is determined that the first data has an abnormality.

[0198] In the above scheme, the abnormality determination module 4555 is also used to obtain the quality requirement information of the first dimension, and generate a regular expression for quality verification based on the quality requirement information through a third model; use the regular expression of the first data to perform quality verification on the first data, and determine the verification result as the first quality information of the first dimension.

[0199] In the above scheme, the multiple dimensions include: a second dimension, which includes multiple sub-dimensions; the anomaly determination module 4555 is also used to determine the quality scores of the first data in the multiple sub-dimensions; summing the quality scores of the multiple sub-dimensions, and determining the summation result as the first quality information of the first data in the second dimension.

[0200] In the above scheme, the task processing module 4554 is also used to determine a first repair operation for the first object when the first object and the preset downstream object of the first data are inconsistent; repair the first object through the first repair operation to obtain the repaired first object.

[0201] In the above scheme, the processing results include: the repaired first data, the repaired first object and the repaired second object; the task processing module 4554 is also used to predict the execution probability of multiple preset second repair operations based on the second object through a fourth model; based on the execution probability, select a third repair operation from multiple second repair operations, and generate a second database statement for the third repair operation; repair the second object through the second database statement to obtain the repaired second object; repair the first data based on the repaired second object to obtain the repaired first data, and repair the first object based on the repaired first data to obtain the repaired first object.

[0202] An embodiment of the present application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the data processing method described in the embodiment of the present application.

[0203] The embodiment of the present application provides a computer-readable storage medium in which computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the data processing method provided in the embodiment of the present application, for example, Figure 2 The data processing method is shown.

[0204] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0205] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0206] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0207] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0208] In summary, through the embodiments of the present application, it is possible to achieve two-way traceability for the first data, thereby quickly determining the upstream source and downstream impact of the first data, and at the same time perform task processing on the upstream source of the first data, the first data itself, and the downstream impact, so as to more efficiently complete the task processing of the first data, thereby improving the efficiency of task processing of the data. At the same time, because the electronic device first determines the first object downstream of the first data, it can more quickly determine its downstream impact when there is an anomaly in the first data, which is more conducive to preventing the downstream spread of the anomaly; the quality assessment of the first data can be performed from multiple different dimensions, and the quality assessment of the first data can be completed based on the relationship between the first quality information of any dimension and the corresponding quality anomaly condition. In this way, the quality assessment of the first data can be more comprehensive, so that the accuracy of the quality assessment of the first quantity can be improved.

[0209] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: generating a data lineage graph of the first data based on a first database statement of the first data; In the data lineage graph, determining a first object downstream of the first data, where the first data is a dependency of the first object; In the data lineage graph, determining a second object upstream of the first data, the first data being obtained by relying on the second object; Task processing is performed on the first data, the first object, and the second object to obtain a processing result.

2. The method according to claim 1, characterized in that The first object includes a first sub-object and a second sub-object, the first data is a direct dependency of the first sub-object, and the first data is an indirect dependency of the second sub-object; Determining the first object downstream of the first data in the data lineage graph includes: Locating a first graph node of the first data from the data lineage graph; Determine, from the data lineage graph, a second graph node that is downstream of the first graph node and directly connected to the first graph node, and determine an object corresponding to the second graph node as the first child object; Through the first model, based on the data lineage graph, a third graph node located downstream of the first graph node and not directly connected to the first graph node is predicted, and the object corresponding to the third graph node is determined as the second sub-object.

3. The method according to claim 1, characterized in that Determining the second object upstream of the first data in the data lineage graph includes: For the first data, detecting and obtaining at least one first path from the data lineage graph; Based on at least one of the first paths, the second object upstream of the first data is determined.

4. The method according to claim 3, characterized in that The detecting and obtaining at least one first path from the data lineage graph for the first data includes: Sampling a plurality of second paths from the data lineage graph; Predicting a first probability of each of the second paths using a second model; The second path for which the first probability is greater than a probability threshold is determined as the first path.

5. The method according to claim 3, characterized in that The determining, based on at least one of the first paths, the second object upstream of the first data includes: In response to the number of the first paths being plural, selecting a third path from the plurality of the first paths; Starting from the end node of the third path, the states of the graph nodes upstream of the end node are determined in sequence, and a fourth graph node is determined based on the states of the graph nodes, and the object corresponding to the fourth graph node is determined to be the second object.

6. The method according to any one of claims 1 to 5, characterized in that After determining the first object downstream of the first data in the data lineage graph and before determining the second object upstream of the first data in the data lineage graph, the method further includes: Determining first quality information corresponding to each of the first data in multiple dimensions; When the first quality information of any one of the dimensions meets a quality abnormality condition, it is determined that the first data is abnormal.

7. The method according to claim 6, characterized in that The multiple dimensions include: a first dimension; and determining the first quality information corresponding to the first data in the multiple dimensions includes: Acquire quality requirement information of the first dimension, and generate a regular expression for quality verification based on the quality requirement information using a third model; The first data is quality checked by using the regular expression, and a check result is determined as the first quality information of the first data in the first dimension.

8. The method according to claim 6, characterized in that The multiple dimensions include: a second dimension, the second dimension includes multiple sub-dimensions; and determining the first quality information corresponding to the first data in the multiple dimensions includes: determining quality scores of the first data in the plurality of sub-dimensions; The quality scores of the multiple sub-dimensions are summed, and the sum result is determined as the first quality information of the first data in the second dimension.

9. The method according to any one of claims 1 to 5, characterized in that After determining the first object downstream of the first data in the data lineage graph, the method further includes: In a case where the first object and a preset downstream object of the first data are inconsistent, determining a first repair operation for the first object; The first object is repaired by performing the first repair operation to obtain the repaired first object.

10. The method according to claim 6, characterized in that The processing result includes: the repaired first data, the repaired first object and the repaired second object; The performing task processing on the first data, the first object, and the second object to obtain a processing result includes: predicting, by a fourth model, execution probabilities for a plurality of preset second repair operations based on the second object; selecting a third repair operation from a plurality of second repair operations based on the execution probability, and generating a second database statement for the third repair operation; Repairing the second object using the second database statement to obtain a repaired second object; The first data is repaired based on the repaired second object to obtain the repaired first data, and the first object is repaired based on the repaired first data to obtain the repaired first object.

11. A data processing device, characterized in that: The device comprises: A graph generation module, configured to generate a data lineage graph of the first data based on a first database statement of the first data; A first object module is configured to determine, in the data lineage graph, a first object downstream of the first data, where the first data is a dependency of the first object; A second object module is configured to determine, in the data lineage graph, a second object upstream of the first data, the first data being obtained by relying on the second object; The task processing module is used to perform task processing on the first data, the first object and the second object to obtain a processing result.

12. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer-executable instructions or computer programs; A processor, configured to implement the method according to any one of claims 1 to 10 when executing computer-executable instructions or computer programs stored in the memory.

13. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images