Data processing method, device, electronic device and storage medium

Abstract

The present disclosure relates to a data processing method, device, electronic device, and storage medium, the method comprising: obtaining a current map and a historical map corresponding to the data to be processed; determining change information between the current map and the historical map; performing attribution analysis and / or early warning analysis on the change information to obtain a change analysis result corresponding to the change information; the early warning analysis is used to analyze the change of the change information in the current map or the historical map. The technical solution of the present disclosure can realize automatic monitoring of data changes in the multimedia data processing process, perform real-time attribution analysis and / or early warning analysis, reduce the cost of multimedia data change monitoring, avoid data failures caused by unintended changes, and ensure the accuracy and timeliness of data change analysis in the multimedia data processing process.

Description

Technical Field

[0001] The present disclosure relates to the field of Internet technology, and in particular to a data processing method, device, electronic device, and storage medium. Background Art

[0002] Data graphs (i.e., data lineage relationship graphs) are a key method for achieving data traceability and data fusion processing. Data graph analysis describes the chain of origin of big data, specifically how the data is generated and the processes and stages it passes through. During data processing, from the source of the data to the final data generation, data quality issues can arise at every stage, potentially leading to serious data processing incidents.

[0003] Related technologies typically rely on manual intervention to analyze data changes using data lineage relationship maps to avoid data quality issues. However, these methods require high labor costs and are often inaccurate and inefficient in analyzing data quality issues. Summary of the Invention

[0004] The present disclosure provides a data processing method, device, electronic device, and storage medium to at least address the problems of high cost, low accuracy, and poor timeliness in data change analysis in related technologies. The technical solutions of the present disclosure are as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided a data processing method, including:

[0006] Obtain a current graph and a historical graph corresponding to the data to be processed; the nodes in the current graph represent the data to be processed and the first data generated between the starting time point and the current time point depending on the data to be processed, the first edge in the current graph is used to describe the dependency relationship between the data represented by the first node, and the first node is a node connected to the first edge; the nodes in the historical graph represent the data to be processed and the second data generated between the starting time point and the historical time point depending on the data to be processed, the second edge in the historical graph is used to describe the dependency relationship between the data represented by the second node, and the second node is a node connected to the second edge;

[0007] Determining change information between the current map and the historical map;

[0008] Perform attribution analysis and / or early warning analysis on the change information to obtain a change analysis result corresponding to the change information; the early warning analysis is used to analyze the change of the change information in the current map or the historical map.

[0009] In an exemplary embodiment, the change analysis result includes a change attribution result, the change information includes at least one change node, and performing attribution analysis on the change information to obtain the change analysis result corresponding to the change information includes:

[0010] Obtaining a first downstream node corresponding to each change node; data represented by the first downstream node corresponding to each change node is generated based on the data represented by each change node;

[0011] In a case where the first downstream node corresponding to each change node is included in the at least one change node, determining the change attribution result corresponding to each change node is as follows: based on a cascade result caused by the first downstream node corresponding to each change node being included in the at least one change node:

[0012] In the case where a third node exists in the first downstream node corresponding to each change node, the association information between the data represented by each change node and the data represented by the first downstream node corresponding to each change node is determined, and based on the association information, the change attribution result corresponding to each change node is determined; the third node does not belong to the at least one change node.

[0013] In an exemplary embodiment, the graph to which the at least one change node belongs is the historical graph, and determining the change attribution result corresponding to each change node based on the association information includes:

[0014] According to the association information, the change attribution result corresponding to each change node is determined as: a result caused by disconnection of a target edge, where the target edge is an edge between each change node and a first downstream node corresponding to each change node.

[0015] In an exemplary embodiment, the graph to which the at least one change node belongs is the current graph, and determining, based on the association information, a change attribution result corresponding to each change node includes:

[0016] According to the association information, the change attribution result corresponding to each change node is determined as: a result caused by the connection of the newly added target edge, where the newly added target edge is an edge between each change node and the first downstream node corresponding to each change node.

[0017] In an exemplary embodiment, the change analysis result includes a warning result, the change information includes a change edge, and performing warning analysis on the change information to obtain a change analysis result corresponding to the change information includes:

[0018] Obtaining a second downstream node corresponding to the change edge; data represented by the second downstream node is generated based on data represented by a fourth node connected to the change edge, the nodes connected to the change edge include the fourth node and a fifth node, and the data represented by the fourth node is generated based on data represented by the fifth node;

[0019] Performing a warning analysis on the changed edge, the second downstream node corresponding to the changed edge, and the graph to which the changed edge belongs to obtain the warning result; the graph to which the changed edge belongs is the current graph or the historical graph.

[0020] In an exemplary embodiment, performing early warning analysis on the changed edge, the second downstream node corresponding to the changed edge, and the graph to which the changed edge belongs to obtain the early warning result includes:

[0021] Determine a sixth node from the second downstream node; the out-degree of the sixth node is 0;

[0022] In the case where the sixth node has labeled object identification information, determining that the graph to which the changed edge belongs is an object labeled graph; and the object is the object represented by the object identification information;

[0023] In a case where the object annotation graph is the historical graph, a result of removing the changed edge from the object annotation graph is used as the warning result;

[0024] In a case where the object annotation graph is the current graph, a result of adding the changed edge to the object annotation graph is used as the warning result.

[0025] In an exemplary embodiment, the method further comprises:

[0026] If the object identification information does not exist at the sixth node and the object annotation graph is the historical graph, the result of removing the change edge from the historical graph is used as the warning result;

[0027] When the object identification information does not exist at the sixth node and the object annotation graph is the current graph, the result of adding the changed edge to the current graph is used as the warning result.

[0028] In an exemplary embodiment, obtaining the current graph and the historical graph corresponding to the data to be processed includes:

[0029] Acquire the data to be processed, the first data, and the second data;

[0030] The current graph is generated based on current data and the dependency relationship between the current data, and the historical graph is generated based on historical data and the dependency relationship between the historical data; the current data includes the data to be processed and the first data, and the historical data includes the data to be processed and the second data.

[0031] In an exemplary embodiment, the method further comprises:

[0032] In response to a node labeling instruction, a current labeled graph is labeled from the current graph, and a historical labeled graph is labeled from the historical graph; the current labeled graph is a partial graph in the current graph, and the historical labeled graph is a partial graph in the historical graph;

[0033] Determining target change information between the historical annotated graph and the current annotated graph;

[0034] The attribution analysis and / or the early warning analysis are performed on the target change information to obtain a target change analysis result corresponding to the target change information.

[0035] In an exemplary embodiment, in response to the node labeling instruction, labeling a current labeled graph from the current graph and labeling a historical labeled graph from the historical graph includes:

[0036] In response to the node marking instruction, marking a current marked node from the current graph;

[0037] Determining a historical annotation node corresponding to the current annotation node from the historical graph;

[0038] Obtaining a first upstream node corresponding to the historical annotation node and a second upstream node corresponding to the current annotation node, wherein data represented by the historical annotation node is generated based on data represented by the first upstream node, and data represented by the current annotation node is generated based on data represented by the second upstream node;

[0039] The historical annotation graph is obtained according to the historical annotation node and the first upstream node, and the current annotation graph is obtained according to the current annotation node and the second upstream node.

[0040] According to a second aspect of an embodiment of the present disclosure, there is provided a data processing apparatus, including:

[0041] A graph acquisition module is configured to execute acquisition of a current graph and a historical graph corresponding to the data to be processed; the nodes in the current graph represent the data to be processed and the first data generated between the starting time point and the current time point depending on the data to be processed; the first edge in the current graph is used to describe the dependency relationship between the data represented by the first node, and the first node is a node connected to the first edge; the nodes in the historical graph represent the data to be processed and the second data generated between the starting time point and the historical time point depending on the data to be processed; the second edge in the historical graph is used to describe the dependency relationship between the data represented by the second node, and the second node is a node connected to the second edge;

[0042] a change information determination module, configured to determine change information between the current graph and the historical graph;

[0043] The analysis module is configured to perform attribution analysis and / or early warning analysis on the change information to obtain a change analysis result corresponding to the change information; the early warning analysis is used to analyze the change of the change information in the current map or the historical map.

[0044] In an exemplary embodiment, the change analysis result includes a change attribution result, the change information includes at least one change node, and the analysis module includes:

[0045] A first downstream node acquisition unit is configured to acquire a first downstream node corresponding to each changed node; data represented by the first downstream node corresponding to each changed node is generated based on the data represented by each changed node;

[0046] The first change attribution result determining unit is configured to, when the first downstream node corresponding to each change node is included in the at least one change node, determine the change attribution result corresponding to each change node as follows: based on a cascade result caused by the first downstream node corresponding to each change node being included in the at least one change node:

[0047] The second change attribution result determination unit is configured to determine the association information between the data represented by each change node and the data represented by the first downstream node corresponding to each change node when there is a third node in the first downstream node corresponding to each change node, and determine the change attribution result corresponding to each change node based on the association information; the third node does not belong to the at least one change node.

[0048] In an exemplary embodiment, the graph to which the at least one change node belongs is the historical graph, and the second change attribution result determination unit is configured to determine, based on the association information, the change attribution result corresponding to each change node: the result caused by the disconnection of the target edge, where the target edge is the edge between each change node and the first downstream node corresponding to each change node.

[0049] In an exemplary embodiment, the graph to which the at least one change node belongs is the current graph, and the second change attribution result determination unit is configured to determine the change attribution result corresponding to each change node based on the association information: the result caused by the connection of the newly added target edge, and the newly added target edge is the edge between each change node and the first downstream node corresponding to each change node.

[0050] In an exemplary embodiment, the change analysis result includes an early warning result, the change information includes a change edge, and the analysis module includes:

[0051] a second downstream node acquisition unit configured to acquire a second downstream node corresponding to the change edge; data represented by the second downstream node is generated based on data represented by a fourth node connected to the change edge, the nodes connected to the change edge include the fourth node and a fifth node, and the data represented by the fourth node is generated based on data represented by the fifth node;

[0052] The warning result determination unit is configured to perform warning analysis on the change edge, the second downstream node corresponding to the change edge, and the graph to which the change edge belongs to obtain the warning result; the graph to which the change edge belongs is the current graph or the historical graph.

[0053] In an exemplary embodiment, the warning result determination unit includes:

[0054] a sixth node determination subunit, configured to determine a sixth node from the second downstream node; the out-degree of the sixth node is 0;

[0055] The object annotation graph determination subunit is configured to determine, when the sixth node has annotated object identification information, that the graph to which the changed edge belongs is an object annotation graph; and the object is the object represented by the object identification information;

[0056] A first warning result determination subunit is configured to execute, when the object annotation graph is the historical graph, a result of removing the changed edge from the object annotation graph as the warning result;

[0057] The second warning result determination subunit is configured to execute, when the object annotation graph is the current graph, a result of adding the changed edge to the object annotation graph as the warning result.

[0058] In an exemplary embodiment, the apparatus further comprises:

[0059] A third warning result determination subunit is configured to, when the object identification information does not exist at the sixth node and the object annotation graph is the historical graph, remove the change edge from the historical graph as the warning result;

[0060] The fourth warning result determination subunit is configured to execute, when the object identification information does not exist at the sixth node and the object annotation graph is the current graph, the result of adding the change edge to the current graph as the warning result.

[0061] In an exemplary embodiment, the atlas acquisition module includes:

[0062] a data acquisition unit, configured to acquire the data to be processed, the first data, and the second data to be processed;

[0063] The graph generation unit is configured to generate the current graph based on current data and the dependency relationship between the current data, and to generate the historical graph based on historical data and the dependency relationship between the historical data; the current data includes the data to be processed and the first data, and the historical data includes the data to be processed and the second data.

[0064] In an exemplary embodiment, the apparatus further comprises:

[0065] a labeling module configured to execute, in response to a node labeling instruction, labeling a current labeling graph from the current graph and labeling a historical labeling graph from the historical graph; the current labeling graph is a portion of the graph in the current graph, and the historical labeling graph is a portion of the graph in the historical graph;

[0066] a target change information determination module, configured to determine target change information between the historical annotation graph and the current annotation graph;

[0067] The target change analysis result determination module is configured to perform attribution analysis and / or early warning analysis on the target change information to obtain a target change analysis result corresponding to the target change information.

[0068] In an exemplary embodiment, the annotation module includes:

[0069] a current marking node unit, configured to execute, in response to the node marking instruction, marking a current marking node from the current graph;

[0070] a historical annotation node determining unit, configured to determine a historical annotation node corresponding to the current annotation node from the historical graph;

[0071] an upstream node acquisition unit configured to acquire a first upstream node corresponding to the historically annotated node and a second upstream node corresponding to the currently annotated node, wherein data represented by the historically annotated node is generated based on data represented by the first upstream node, and data represented by the currently annotated node is generated based on data represented by the second upstream node;

[0072] The annotation graph generation unit is configured to obtain the historical annotation graph according to the historical annotation node and the first upstream node, and obtain the current annotation graph according to the current annotation node and the second upstream node.

[0073] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0074] processor;

[0075] A memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the data processing method described above.

[0076] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device executes the data processing method as described above.

[0077] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the above-mentioned data processing method when executed by a processor.

[0078] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

[0079] The disclosed embodiment determines the change information between the current graph and the historical graph corresponding to the data to be processed, and performs attribution analysis and / or early warning analysis on the change information to obtain the change analysis result corresponding to the change information, thereby realizing automatic monitoring of data changes in the process of processing the data to be processed, performing real-time attribution analysis and / or early warning analysis, reducing the cost of monitoring changes in the data to be processed, avoiding data failures caused by unexpected changes, and ensuring the accuracy and timeliness of the data change analysis in the process of processing the data to be processed.

[0080] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0082] Figure 1 The figure is an application environment diagram of a data processing method according to an exemplary embodiment.

[0083] Figure 2 The figure is a flow chart showing a data processing method according to an exemplary embodiment.

[0084] Figure 3 The figure is a schematic diagram showing change information between a current map and a historical map according to an exemplary embodiment.

[0085] Figure 4 The present invention is a flowchart showing a method of obtaining a current map and a historical map corresponding to data to be processed according to an exemplary embodiment.

[0086] Figure 5 is a flow chart of another data processing method according to an exemplary embodiment.

[0087] Figure 6 The present invention is a flowchart showing a method of performing attribution analysis on change information and obtaining a change analysis result corresponding to the change information according to an exemplary embodiment.

[0088] Figure 7 The figure is a schematic diagram showing a change attribution analysis and early warning analysis according to an exemplary embodiment.

[0089] Figure 8 The present invention is a flowchart showing a method of performing early warning analysis on change information and obtaining a change analysis result corresponding to the change information according to an exemplary embodiment.

[0090] Figure 9 The figure is a block diagram of a data processing device according to an exemplary embodiment.

[0091] Figure 10 It is a block diagram of an electronic device for data processing according to an exemplary embodiment. DETAILED DESCRIPTION

[0092] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0093] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0094] See also Figure 1 , Figure 1 Figure 1 is an application environment diagram illustrating a data processing method according to an exemplary embodiment. The application environment may include a client 01 and a server 02. Client 01 can be used to collect data to be processed and view change analysis results of the data to be processed returned by the server. Alternatively, client 01 may include a smartphone, desktop computer, tablet computer, laptop computer, digital assistant, augmented reality (AR) / virtual reality (VR) device, smart wearable device, or other terminal device.

[0095] Among them, server 02 can be used to obtain the current map and historical map corresponding to the data to be processed; and to determine the change information between the above current map and the above historical map; and to perform attribution analysis and / or early warning analysis on the above change information to obtain the change analysis results corresponding to the above change information. Optionally, the server 02 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0096] It should be noted that the above application environment is only an example, and the embodiments of the present disclosure may also include other application environments.

[0097] Figure 2 is a flow chart showing a data processing method according to an exemplary embodiment. Figure 2 As shown, this method is used to Figure 1 In a system including a client and a server, the following steps are included.

[0098] In step S11, the current graph and historical graph corresponding to the data to be processed are obtained; the nodes in the above-mentioned current graph represent the above-mentioned data to be processed and the first data generated between the starting time point and the current time point depending on the above-mentioned data to be processed, the first edge in the above-mentioned current graph is used to describe the dependency relationship between the data represented by the first node, and the above-mentioned first node is the node connected to the above-mentioned first edge, the nodes in the above-mentioned historical graph represent the above-mentioned data to be processed and the second data generated between the above-mentioned data to be processed between the above-mentioned starting time point and the historical time point, the second edge in the above-mentioned historical graph is used to describe the dependency relationship between the data represented by the second node, and the above-mentioned second node is the node connected to the above-mentioned second edge.

[0099] Optionally, the data to be processed, the first data, and the second data may be data in the entire big data processing chain, from data acquisition to ETL processing. The data processing chain is a chain formed by individual data, as well as the dependencies and output order between the data. The data to be processed may be the original data collected during the data acquisition process, i.e., the first data in the data processing chain. The starting time point may be the time when processing of the data to be processed begins.

[0100] Specifically, ETL is the abbreviation of Extract-Transform-Load, which is used to describe the process of extracting (extract), transforming (transform), and loading (load) data from the source end to the destination end.

[0101] Optionally, the above-mentioned data to be processed, first data and second data may include but are not limited to: kafka topic, hive library table, druID data source, clickhouse table, indicators, dimensions, business intelligence dashboard (BI dashboard), etc. Among them, Kafka is a high-throughput distributed publish-subscribe messaging system that can process consumer action stream data on the website. Kafka topic refers to the subject in the distributed publish-subscribe messaging system. Hive refers to the data warehouse tool. DruID is an efficient data query system that mainly solves the problem of aggregated query for large amounts of time-series-based data. Clickhouse refers to a columnar database management system.

[0102] For example, the current graph may correspond to the current data processing link of the data to be processed, and the first data may be data in the current data processing link. The historical graph may correspond to the historical data processing link of the data to be processed, and the second data may be data in the historical data processing link. The current time point may be the time point at which the tail data of the current data processing link was generated, and the historical time point may be the time point at which the tail data of the historical data processing link was generated.

[0103] For example, the nodes in the current graph can represent the data in the current data processing link, and the first edge in the current graph can be used to describe the dependency relationship between the data represented by the first node connected to the first edge. The nodes in the historical graph can represent the data in the historical data processing link, and the second edge in the historical graph can be used to describe the dependency relationship between the data represented by the second node connected to the second edge. Specifically, the relationship can be a dependency or output relationship between data. For example, the data to be processed is Table A, the second data generated based on Table A at the historical time point is Table B (i.e., Table B is processed from Table A), and the second data generated based on Table A at the intermediate time point (the time between the starting time point and the historical time point) is Table C (i.e., Table C is processed from Table B). Then Table A, Table B, and Table C are nodes in the historical graph, and the dependency and output relationship between Table A and Table B (e.g., the processing relationship, i.e., how Table A is processed to obtain Table B), and the dependency and output relationship between Table B and Table C are edges in the historical graph.

[0104] In step S13, the change information between the current map and the historical map is determined.

[0105] Optionally, the above-mentioned change information may include at least one changed node and / or at least one changed edge.

[0106] Exemplarily, a current node set consisting of nodes in the current graph and a historical node set consisting of nodes in the historical graph may be obtained, and at least one changed node may be obtained by performing a difference between the sets.

[0107] The nodes obtained by subtracting the current node set from the historical node set belong to the current node set and do not belong to the historical node set. The nodes obtained by subtracting the historical node set from the current node set belong to the historical node set and do not belong to the current node set.

[0108] Exemplarily, a current edge set consisting of edges in the current graph and a historical edge set consisting of edges in the historical graph may be obtained, and at least one changed edge may be obtained by performing set difference.

[0109] Among them, the edge obtained by taking the difference between the current edge set and the historical edge set belongs to the current edge set and does not belong to the historical edge set. The edge obtained by taking the difference between the historical edge set and the current edge set belongs to the historical edge set and does not belong to the current edge set.

[0110] Figure 3 FIG. 1 is a schematic diagram showing change information between a current map and a historical map according to an exemplary embodiment. Figure 3 As shown, nodes D7, D10, and D11 are changed nodes, the edges between D3->D7, the edge between D8->D11, and the edge between D5->D9 are mutated edges, the edges between D7, D10, D3->D7, and the edge between D5->D9 belong to the historical graph (D7 and D10 are nodes removed from the historical graph, the edge between D3->D7 is an edge removed from the historical graph, and the edge between D5->D9 is an edge removed from the historical graph), and the edge between D11 and D8->D11 belongs to the current graph (D11 is a newly added node in the current graph, and D8->D11 is a newly added edge in the current breakthrough).

[0111] In step S15, attribution analysis and / or warning analysis are performed on the above-mentioned change information to obtain change analysis results corresponding to the above-mentioned change information; the above-mentioned warning analysis is used to analyze the changes of the above-mentioned change information in the above-mentioned current map or the above-mentioned historical map.

[0112] Specifically, attribution analysis refers to analyzing the reasons for the generation of change information. The above-mentioned early warning analysis is used to analyze the changes of the above-mentioned change information in the current map or the above-mentioned historical map, so as to issue early warnings based on the changes.

[0113] In the disclosed embodiment, after obtaining the change analysis result, the current graph may be processed according to the change analysis result.

[0114] For example, the change analysis results can be used to determine whether the change in the change information is in line with expectations. If so, no processing can be performed. If not, the first data corresponding to the current graph and / or the dependency relationship between the first data can be processed based on the change analysis results. For example, if the current graph is missing a node / edge compared to the historical graph, and the change is in line with expectations, the data / dependency relationship represented by the changed node / edge can be left unchanged. If the change is not in line with expectations, the data / dependency relationship represented by the node / edge can be repaired.

[0115] For example, if a failure occurs during subsequent data processing, the change analysis results can be used to trace the source, quickly locate the problem, and roll back the first data corresponding to the current graph and / or the dependencies between the first data. Rollback refers to the act of restoring data to its last correct state when a data processing error occurs.

[0116] The disclosed embodiment determines the change information between the current graph and the historical graph, and performs attribution analysis and / or early warning analysis on the change information to obtain the change analysis result corresponding to the change information, thereby realizing automatic monitoring of data changes in the process of processing the data to be processed, performing real-time attribution analysis and / or early warning analysis, reducing the cost of monitoring changes in the data to be processed, avoiding data failures caused by unexpected changes, and ensuring the accuracy and timeliness of the data change analysis in the process of processing the data to be processed.

[0117] Figure 4 This is a flow chart showing a method of obtaining the current graph and historical graph corresponding to the data to be processed according to an exemplary embodiment. Figure 4 As shown, in a feasible embodiment, in the above step S11, the above-mentioned acquisition of the current graph and the historical graph corresponding to the data to be processed may include:

[0118] In step S1101, the data to be processed, the first data and the second data are obtained.

[0119] In step S1103, the current graph is generated based on the current data and the dependency relationship between the current data, and the historical graph is generated based on the historical data and the dependency relationship between the historical data; the current data includes the data to be processed and the first data, and the historical data includes the data to be processed and the second data.

[0120] In the disclosed embodiment, the entire data processing link can be circled as the monitoring focus, so as to perform full-scale graph monitoring. That is, the current graph can be generated based on the data to be processed, the first data generated between the starting time point and the current time point that depends on the data to be processed, and the dependency relationship between the data. The historical graph can be generated based on the data to be processed, the second data generated between the starting time point and the current time point that depends on the data to be processed, and the dependency relationship between the data, thereby realizing the monitoring of the full amount of data in the data processing link of the data to be processed, improving the monitoring scope of the data to be processed, and ensuring the comprehensiveness and timeliness of the data change analysis during the processing of the data to be processed.

[0121] In the embodiment of the present disclosure, the scope of the data processing links that affect the analysis and monitoring scope can also be defined, that is, the data processing links that need to be protected can be circled, so as to achieve attribution analysis and / or the above-mentioned early warning analysis on the data of the data processing links that need to be protected.

[0122] Figure 5 FIG. 1 is a flow chart of another data processing method according to an exemplary embodiment. Figure 5 As shown, in another feasible embodiment, the above method may further include:

[0123] In step S21, in response to the node labeling instruction, the current labeled graph is labeled from the above-mentioned current graph, and the historical labeled graph is labeled from the above-mentioned historical graph; the above-mentioned current labeled graph is a partial graph in the above-mentioned current graph, and the above-mentioned historical labeled graph is a partial graph in the above-mentioned historical graph.

[0124] In step S23, target change information between the historical annotation map and the current annotation map is determined.

[0125] In step S25, the attribution analysis and / or the early warning analysis are performed on the target change information to obtain a target change analysis result corresponding to the target change information.

[0126] Optionally, in the embodiment of the present disclosure, in the above step S21, a variety of methods can be used to mark the current marked graph from the above current graph, and to mark the historical marked graph from the above historical graph, which are not specifically limited in the embodiment of the present disclosure. For example, in response to a node marking instruction triggered by a user, the current marked graph that needs to be monitored can be marked from the current graph, and the historical marked graph that needs to be marked can be marked from the historical graph (i.e., the data processing link that needs to be protected can be circled).

[0127] Optionally, in the embodiment of the present disclosure, the process of determining the target change information in the above step S23 is similar to the above step S13 and is not repeated here.

[0128] Optionally, in the embodiment of the present disclosure, the process of performing the attribution analysis and / or the warning analysis on the target change information in the step S25 is similar to the step S15, and is not described in detail here.

[0129] The disclosed embodiment can define the scope of data processing links that affect the analysis and monitoring scope, mark the current marking map and the historical marking map, that is, circle the data processing links that need to be protected, and perform attribution analysis and / or the above-mentioned early warning analysis on the target change information between the historical marking map and the above-mentioned current marking map, and obtain the target change analysis results corresponding to the above-mentioned target change information, thereby realizing the monitoring of data processing links that need to be protected, reducing the data monitoring scope, further reducing the cost of monitoring changes in data to be processed, and improving the flexibility of monitoring changes in data to be processed, ensuring the accuracy and timeliness of data change analysis of data processing links that need to be protected during the processing process.

[0130] In an exemplary embodiment, in the above-mentioned step S21, the above-mentioned response to the node labeling instruction, labeling the current labeled graph from the above-mentioned current graph, and labeling the historical labeled graph from the above-mentioned historical graph, may include: responding to the node labeling instruction, labeling the current labeled node from the above-mentioned current graph.

[0131] Determine the historical annotation node corresponding to the current annotation node from the historical graph.

[0132] Obtain the first upstream node corresponding to the above historical annotation node and the second upstream node corresponding to the above current annotation node, the data represented by the above historical annotation node is generated based on the data represented by the above first upstream node, and the data represented by the above current annotation node is generated based on the data represented by the above second upstream node.

[0133] A historical annotation graph is obtained based on the historical annotation node and the first upstream node, and a current annotation graph is obtained based on the current annotation node and the second upstream node.

[0134] In the embodiment of the present disclosure, the scope of the data processing link that affects the analysis and monitoring scope can be delineated, that is, the data processing link that needs to be protected can be circled, and the end data of the data processing link that needs to be protected can be provided.

[0135] After determining the end data of the key data processing link, the end data of the key data processing link can be annotated from the current graph, representing the current annotated node, and the historical annotated node corresponding to the current annotated node can be determined from the historical graph. It should be noted that the current annotated node and the historical annotated node are theoretically the same node, that is, both nodes representing the end data of the key data processing link.

[0136] Exemplarily, a historical graph and a current graph may be retrieved through a graph traversal language (eg, Gremlin) to obtain the first upstream node and the second upstream node.

[0137] Specifically, a historical annotation graph can be obtained based on the historical annotation node, the first upstream node, and the connection relationship between the nodes, and a current annotation graph can be obtained based on the current annotation node, the second upstream node, and the connection relationship between the nodes. The historical annotation graph is a subgraph in the historical graph, and the current annotation graph is a subgraph in the current graph.

[0138] In the embodiment of the present disclosure, based on the current annotation node represented by the end data of the data processing link that needs to be protected, the current annotation map and the historical annotation map can be accurately determined, thereby realizing the monitoring of the data processing link that needs to be protected, reducing the data monitoring scope, further reducing the cost of monitoring changes in pending data, improving the flexibility of monitoring changes in pending data, and ensuring the accuracy and timeliness of data change analysis during the processing of the data processing link that needs to be protected.

[0139] In an exemplary embodiment, the current graph may be transferred to a graph database to obtain a current snapshot, and the historical graph may be transferred to a graph database to obtain a historical snapshot. The current snapshot and the historical snapshot are compared to obtain change information.

[0140] Optionally, data dump or data backup can be performed through relevant interfaces.

[0141] Optionally, the graph structure of the current graph or historical graph can be inserted into the corresponding graph structure of the graph database by adding points (addVertex()) / adding edges (addEdge()) and other methods according to the syntax of the graph database while keeping the graph structure unchanged.

[0142] In another exemplary embodiment, the current graph can be serialized and stored in a traditional relational database to obtain a current snapshot, and corresponding read or deserialization methods can be provided to facilitate reading the current snapshot in the loaded database. Furthermore, the historical graph can be serialized and stored in a traditional relational database to obtain a historical snapshot, and corresponding read or deserialization methods can be provided to facilitate reading the historical snapshot in the loaded database.

[0143] Optionally, the relational database uses two tables for storage. One is the vertex table, which is used to store the nodes in the graph, involving the identity identification number (IDentityDocument, ID) and attribute key-value pairs corresponding to the node; the other is the edge table, which is used to store the edges in the graph, involving the starting node ID, the ending node ID and the attribute key-value pairs on the edge. The serialization process refers to storing each node in the graph in the Vertex table and each edge in the Edge table. The deserialization process refers to taking out the full amount of information from the Vertex table, and then traversing the Edge table according to the starting and ending nodes to connect the corresponding two nodes.

[0144] Since a snapshot is a record of the state of data storage at a certain moment, transferring the graph to obtain a snapshot can prevent damage to the current data graph and historical data graph. When an application failure occurs in the storage device or the graph is damaged, timely data recovery can be performed to restore the data to the state at the time when the snapshot was generated, thereby ensuring the accuracy and reliability of data change analysis.

[0145] In a feasible embodiment, performing attribution analysis and / or early warning analysis on the change information to obtain a change analysis result corresponding to the change information may include:

[0146] Attribution analysis and / or early warning analysis are performed on the above-mentioned change information and the correlation relationship between the downstream nodes corresponding to the above-mentioned change information to obtain a change analysis result corresponding to the above-mentioned change information.

[0147] Optionally, when the change information is a change node, the downstream node corresponding to the change information may be a node located downstream of the change information and dependent on the generation of the change information. When the change information is a change edge, the downstream node corresponding to the change information may be a downstream node corresponding to the downstream node connected to the change edge.

[0148] Optionally, the association relationship between the change information and the downstream nodes corresponding to the above change information can be: the dependency and output relationship between the data represented by the change information and the data represented by the downstream nodes corresponding to the change information, the attribute information of the data represented by the change information, and the relationship between the attribute information of the data represented by the downstream nodes corresponding to the change information, etc.

[0149] In an optional embodiment, Figure 6 This is a flow chart showing a method of performing attribution analysis on change information and obtaining change analysis results corresponding to the change information according to an exemplary embodiment. Figure 6As shown, the change analysis result includes a change attribution result, and the change information includes at least one change node. In step S15, the change information is subjected to attribution analysis to obtain a change analysis result corresponding to the change information, which may include:

[0150] In step S15101, the first downstream node corresponding to each change node is obtained; the data represented by the first downstream node corresponding to each change node is generated based on the data represented by each change node.

[0151] For example, a graph traversal language can be used to retrieve all first downstream nodes of each change node in the graph to which it belongs. The first downstream node corresponding to each change node is the node generated by the dependency of each change node, that is, the first downstream node corresponding to each change node is located downstream of each change node.

[0152] The graph to which each change node belongs is a historical graph or a current graph.

[0153] In step S15103, when the first downstream node corresponding to each of the above-mentioned change nodes is included in the above-mentioned at least one change node, the change attribution result corresponding to each change node is determined as: the cascade result caused by the first downstream node corresponding to each change node being included in the above-mentioned at least one change node.

[0154] Specifically, the first downstream node corresponding to each of the change nodes is included in the at least one change node, which means that the first downstream node corresponding to each of the change nodes is included in the at least one change node.

[0155] Specifically, when the first downstream nodes corresponding to a certain change node A are all included in at least one change node, the change attribution result of the change of the change node A is: the cascade result caused by the first downstream node of the change node A being included in the at least one change node. That is, because the first downstream node of the change node A is included in the at least one change node, the change node A is also a change node. The cascade can represent the mapping relationship between a node and its corresponding downstream node. Whatever operation the downstream node performs or what state it is in, the associated node will also perform the same operation or be in the same state.

[0156] Optionally, if at least one change node belongs to the historical graph, indicating that the at least one change node is a set of nodes that is larger in the historical graph than in the current graph (i.e., a set of nodes that is smaller in the current graph than in the historical graph), then the above-mentioned change attribution result can be a downgrade attribution result, and the above-mentioned cascade result can be a cascade downgrade result.

[0157] Optionally, if at least one change node belongs to the current graph, indicating that the at least one change node is a node set that is larger in the current graph than in the historical graph (i.e., a node set that is smaller in the historical graph than in the current graph), then the above-mentioned change attribution result can be an upgrade attribution result, and the above-mentioned cascade result can be a cascade upgrade result.

[0158] In step S15105, when there is a third node in the first downstream node corresponding to each change node, the association information between the data represented by each change node and the data represented by the first downstream node corresponding to each change node is determined, and based on the above association information, the change attribution result corresponding to each change node is determined; the above third node does not belong to the above at least one change node.

[0159] Specifically, if there is a third node in the first downstream node that does not belong to at least one change node (that is, the third node belongs to the first downstream node, but does not belong to at least one change node), then it is not a cascade result. Then, the data represented by each change node and the association information between the data represented by the first downstream node corresponding to each change node can be obtained, and the change attribution result corresponding to each change node can be determined through the association information.

[0160] Exemplarily, the association information may include: dependency and output relationships between data represented by the nodes, attribute information of the data represented by the nodes, etc.

[0161] In the embodiment of the present disclosure, when the first downstream nodes corresponding to each change node are all included in the first change node, the change attribution result is determined to be a cascade result; otherwise, the change attribution result is determined based on the association information between the data represented by each change node and the data represented by the first downstream node corresponding to each change node, thereby improving the accuracy of the change attribution result determination and ensuring the accuracy of the data change analysis in the data processing link during the processing process.

[0162] In a feasible embodiment, the graph to which the at least one change node belongs is the historical graph. Then, in step S15105, determining the change attribution result corresponding to each change node based on the association information may include:

[0163] According to the above association information, the change attribution result corresponding to each change node is determined as: the result caused by the disconnection of the target edge, and the above target edge is the edge between each change node and the first downstream node corresponding to each change node.

[0164] Specifically, if the graph to which at least one change node belongs is a historical graph, indicating that the at least one change node is a node set with more nodes in the historical graph than in the current graph (that is, a node set with fewer nodes in the current graph than in the historical graph), then when there is a third node in the above-mentioned first downstream node, it can be determined that the change attribution result corresponding to each change node is a degraded attribution result caused by the disconnection of the edge between each change node and its corresponding first downstream node.

[0165] In another feasible embodiment, the graph to which the at least one change node belongs is the current graph. Then, in step S15105, determining the change attribution result corresponding to each change node based on the association information may include:

[0166] According to the above association information, the change attribution result corresponding to each change node is determined as: the result caused by the connection of the newly added target edge, and the above newly added target edge is the edge between each change node and the first downstream node corresponding to each change node.

[0167] Specifically, if the graph to which at least one change node belongs is the current graph, indicating that the at least one change node is a node set with more nodes in the current graph than in the historical graph (that is, a node set with fewer nodes in the historical graph than in the current graph), then when there is a third node in the above-mentioned first downstream node, it can be determined that the change attribution result corresponding to each change node is the upgrade attribution result caused by the connection of the newly added target edge, and the newly added target edge is the edge between each change node and its corresponding first downstream node in the current graph.

[0168] In an embodiment of the present disclosure, when at least one of the above-mentioned change nodes belongs to a historical graph, the change attribution result corresponding to each change node is determined to be the result caused by the disconnection of a certain edge; when at least one of the above-mentioned change nodes belongs to a current graph, the change attribution result corresponding to each change node is determined to be the result caused by the addition of a certain edge, so that different change attribution results can be analyzed for the graph to which the change node belongs, thereby improving the accuracy and flexibility of determining the change attribution results.

[0169] Optionally, the above steps S15101 to S15105 may be executed serially or in parallel.

[0170] The following describes steps S15101 to S15105 by taking serial execution as an example:

[0171] 1. When at least one change node belongs to the historical graph, the change attribution analysis is a downgraded change attribution analysis, which is mainly based on the historical graph. It can traverse all nodes in at least one change node and use the currently traversed node as the current change node.

[0172] Use the graph traversal language to retrieve all first downstream nodes of the current changed node in the historical graph.

[0173] When all first downstream nodes are included in at least one change node (i.e., each first downstream node is a change node), the downgrade change attribution result of the current change node is determined as: the cascading downgrade result caused by the inclusion of all first downstream nodes in at least one change node.

[0174] In the case that there is a third node in the first downstream node that does not belong to at least one change node, the downgrade change attribution result corresponding to the current change node is determined based on the association information between the data represented by the current change node and the data represented by the first downstream node, which is the result caused by the disconnection of the edge between the current change node and the first downstream node.

[0175] Repeat the above steps until each change node in at least one change node is attributed, indicating that the downgrade attribution is completed.

[0176] 2. When at least one change node belongs to the current graph, the change attribution analysis is an upgraded change attribution analysis, which is mainly based on the current graph, can traverse all nodes in at least one change node, and use the currently traversed node as the current change node.

[0177] Use the graph traversal language to retrieve all first downstream nodes of the current changed node in the current graph.

[0178] When all first downstream nodes are included in at least one change node (i.e., each first downstream node is a change node), the upgrade change attribution result of the current change node is determined as: the cascade upgrade result caused by the fact that all first downstream nodes belong to at least one change node.

[0179] When there is a third node in the first downstream node that does not belong to at least one change node, the upgrade change attribution result corresponding to the current change node is determined based on the association information between the data represented by the current change node and the data represented by the first downstream node, which is the result caused by the newly added edge between the current change node and the first downstream node.

[0180] Repeat the above steps until each change node in at least one change node is attributed, indicating that the upgrade attribution is complete.

[0181] Figure 7 FIG. 1 is a schematic diagram showing a change attribution analysis and early warning analysis according to an exemplary embodiment. Figure 7 As shown, D7 and D10 are at least one changed node (the graph they are in is a historical graph), the first downstream node of D7 is D3, and the first downstream node of D10 is D7. Therefore, the downgrade change attribution result of D10 is cascade downgrade (that is, the reason for the change of D10 is the cascade change caused by D7 being a changed node). D7 is not cascade downgraded, but actively downgraded. That is, the downgrade change attribution result of D7 is: the result caused by the disconnection of the edge D3->D7.

[0182] Continue as Figure 7 As shown, D11 is at least one changed node (the graph it belongs to is the current graph), the first downstream node of D11 is D8, and D8 does not belong to at least one changed node. Then the upgrade attribution result of D11 is: the result caused by the newly added edge D8->D11.

[0183] Figure 8 This is a flow chart showing a method of performing early warning analysis on change information and obtaining change analysis results corresponding to the change information according to an exemplary embodiment. Figure 8 As shown, the change analysis result includes an early warning result, and the change information includes a change edge. In the step S15, the change information is subjected to early warning analysis to obtain a change analysis result corresponding to the change information, including:

[0184] In step S15301, the second downstream node corresponding to the above-mentioned change edge is obtained; the data represented by the above-mentioned second downstream node depends on the data represented by the fourth node connected to the above-mentioned change edge, and the nodes connected to the above-mentioned change edge include the above-mentioned fourth node and the fifth node, and the data represented by the above-mentioned fourth node depends on the data represented by the above-mentioned fifth node.

[0185] Specifically, the nodes connected by the above-mentioned change edge include the fourth node and the fifth node. The fifth node is located upstream of the change edge, and the fourth node is located downstream of the change edge (that is, the fourth node depends on the fifth node to be generated), and the second downstream node is a node that depends on the fourth node to be generated.

[0186] In step S15303, a warning analysis is performed on the above-mentioned change edge, the second downstream node corresponding to the above-mentioned change edge, and the graph to which the above-mentioned change edge belongs to obtain the above-mentioned warning result; the graph to which the above-mentioned change edge belongs is the above-mentioned current graph or the above-mentioned historical graph.

[0187] In the embodiment of the present disclosure, after obtaining the second downstream node corresponding to the change edge, an early warning analysis can be performed on the above-mentioned change edge, the second downstream node corresponding to the above-mentioned change edge, and the graph to which the above-mentioned change edge belongs to obtain an early warning result, thereby realizing real-time early warning analysis of data changes in the process of processing the data to be processed, avoiding data failures caused by unexpected changes, and ensuring the accuracy and timeliness of the data change analysis in the process of processing the data to be processed.

[0188] In an optional embodiment, in step S15303, the warning analysis of the changed edge, the second downstream node corresponding to the changed edge, and the graph to which the changed edge belongs to obtain the warning result may include:

[0189] 1) Determine a sixth node from the second downstream node; the out-degree of the sixth node is 0.

[0190] Specifically, the sixth node is a node with no out-degree, that is, a node with an out-degree of 0.

[0191] The degree of a node refers to the number of edges connected to the node. In a directed graph, the number of edges starting with a node is called the in-degree of the node, and the number of edges ending with a node is called the out-degree of the node.

[0192] 2) When the sixth node has labeled object identification information, it is determined that the graph to which the change edge belongs is an object labeled graph; and the object is the object represented by the object identification information.

[0193] Specifically, when there is labeled object identification information at the sixth node, it indicates that the sixth node is a node with pre-labeled objects, which represents the end data of the data processing link that needs to be protected. Then, the graph to which the change edge belongs can be used as the object labeling graph, and the warning result can be determined based on the change edge and the object labeling graph.

[0194] 3) In the case where the object annotation graph is the historical graph, the result of removing the change edge from the object annotation graph is used as the warning result.

[0195] Optionally, when the object annotation graph is a historical graph, the warning result may be a degraded warning result, specifically, it may be that a changed edge is removed from the object annotation graph.

[0196] Optionally, the object annotation map may be a map corresponding to a data processing link that requires special protection. Therefore, from the perspective of the data processing link, the warning result may also be: the associated relationship is removed from the data processing link of the object annotation.

[0197] 4) When the object annotation graph is the current graph, the result of adding the changed edge to the object annotation graph is used as the warning result.

[0198] Specifically, when the object annotation graph is the current graph, the warning result is an upgraded warning result. This upgraded warning result can be understood as a high-priority processing prompt, meaning that a changed edge has been added to the middle link of the high-priority guaranteed data processing chain, and therefore this part of the data will be executed and processed with high priority.

[0199] Optionally, the object annotation graph can be a graph corresponding to the data processing link that needs to be protected. Therefore, from the dimension of the data processing link, the warning result can also be: a new association relationship is added to the object annotation data processing link.

[0200] In an embodiment of the present disclosure, a sixth node with a degree of 0 is determined from the second downstream node corresponding to the change edge, and when there is labeled object identification information at the sixth node, the warning result is determined based on the determination that the graph to which the change edge belongs is the object labeling graph, and based on the change edge and the object labeling graph, thereby improving the accuracy of the warning result determination, and thus ensuring the accuracy of the data change analysis during the processing of the data processing link that needs to be focused on; in addition, when the object labeling graph is a historical graph, the warning result of the change edge is determined to be "a change edge is removed from the object labeling graph of the object labeling", and when the object labeling graph is the current graph, the warning result of the change edge is determined to be "a change edge is added to the object labeling graph of the object labeling", so that different warning results can be analyzed for the graph to which the change edge belongs, thereby improving the accuracy and flexibility of the warning result determination.

[0201] In a feasible embodiment, after finding the sixth node, the object marked with the fourth node can be determined based on the object identification information, and the warning result can be notified to the object so that the object can promptly understand the data changes and check whether the data changes are in line with expectations, whether any erroneous operations have occurred, etc.; in the event of a failure in the subsequent data processing process, the object can trace the source based on the change analysis results, quickly locate the problem point, and roll back the first data corresponding to the current graph and / or the association relationship between the first data, thereby analyzing the accuracy and timeliness of the data quality problem.

[0202] In an exemplary embodiment, the above method may further include:

[0203] In the case that the object identification information does not exist at the sixth node and the object annotation graph is the historical graph, the result of removing the change edge from the historical graph is used as the warning result.

[0204] When the object identification information does not exist in the sixth node and the object annotation graph is the current graph, the result of adding the change edge to the current graph is used as the warning result.

[0205] In an embodiment of the present disclosure, when there is no labeled object identification information at the sixth node, it indicates that the sixth node is not the end data of the data processing link that is pre-labeled by the object and represents the need for key protection. At this time, a full warning is performed based on the current graph and the historical graph. When the object labeling graph is the above-mentioned historical graph, the warning result can be "changed edges are removed from the historical graph", and when the object labeling graph is the above-mentioned current graph, the warning result can be "changed edges are added to the above-mentioned current graph", thereby realizing automatic monitoring of data changes in the process of processing the data to be processed, and performing real-time full warning analysis, avoiding data failures caused by unexpected changes, and ensuring the accuracy, timeliness and comprehensiveness of the data change analysis in the process of processing the data to be processed.

[0206] In a feasible embodiment, if the sixth node is not marked by an object, the early warning result may not be specifically notified to the relevant object. In this case, the early warning result can be notified to the system administrator or the technical person in charge of all downstream nodes.

[0207] It should be noted that the number of the above-mentioned changed edges may be at least one, that is, one or more.

[0208] Optionally, the above steps S15301 to S15303 may be executed serially or in parallel.

[0209] The following describes steps S15301 to S15303 by taking serial execution as an example:

[0210] 1. When at least one changed edge belongs to the historical graph, the warning analysis can be a downgraded warning result. It is mainly based on the historical graph, can traverse all edges in at least one changed edge, and use the currently traversed edge as the current changed edge.

[0211] Through the graph traversal language, starting from the fourth node (ie, downstream node) connected to the current change edge and taking the query downstream as the direction, all sixth nodes with no out-degree are queried from the second downstream node.

[0212] When the sixth node has labeled object identification information, the graph to which the current change edge belongs is determined to be the object labeling graph, and the degradation warning result is determined to be: the current change edge is removed from the object labeling graph of the object labeling.

[0213] Repeat the above steps until all edges in at least one change edge are traversed, indicating that the degradation warning is completed.

[0214] 2. When at least one changed edge belongs to the current graph, the warning analysis can be an upgraded warning result, that is, a high-priority processing prompt. It is mainly based on the current graph, and can traverse all edges in at least one changed edge and use the currently traversed edge as the current changed edge.

[0215] Through the graph traversal language, starting from the fourth node connected to the current change edge and taking the query downstream as the direction, all sixth nodes with no out-degree are queried from the second downstream node.

[0216] When the sixth node has labeled object identification information, the graph to which the current change edge belongs is determined to be the object labeling graph, and the upgrade warning result is determined to be: a new change edge is added to the object labeling graph of the object labeling.

[0217] Repeat the above steps until all edges in at least one change edge are traversed, indicating that the upgrade warning is completed.

[0218] Continue as Figure 7 As shown, the edge between D3->D7 is a changed edge, and the downgrade warning result is: the edge between D3->D7 is removed from the historical graph.

[0219] Continue as Figure 7 As shown, the edge between D8->D11 is a changed edge, and the upgrade warning result is: the edge between D8->D11 is added to the current graph.

[0220] Figure 9 FIG. 1 is a block diagram of a data processing device according to an exemplary embodiment. Figure 8 The device includes a map acquisition module 31, a change information determination module 33 and an analysis module 35.

[0221] The graph acquisition module 31 is configured to execute the acquisition of the current graph and historical graph corresponding to the data to be processed; the nodes in the above-mentioned current graph represent the above-mentioned data to be processed and the first data generated between the starting time point and the current time point depending on the above-mentioned data to be processed, the first edge in the above-mentioned current graph is used to describe the dependency relationship between the data represented by the first node, and the above-mentioned first node is the node connected to the above-mentioned first edge, the nodes in the above-mentioned historical graph represent the above-mentioned data to be processed and the second data generated between the above-mentioned data to be processed between the above-mentioned starting time point and the historical time point, the second edge in the above-mentioned historical graph is used to describe the dependency relationship between the data represented by the second node, and the above-mentioned second node is the node connected to the above-mentioned second edge.

[0222] The change information determination module 33 is configured to determine the change information between the current graph and the historical graph.

[0223] The analysis module 35 is configured to perform attribution analysis and / or early warning analysis on the above-mentioned change information to obtain the change analysis results corresponding to the above-mentioned change information; the above-mentioned early warning analysis is used to analyze the changes of the above-mentioned change information in the above-mentioned current map or the above-mentioned historical map.

[0224] In an exemplary embodiment, the change analysis result includes a change attribution result, the change information includes at least one change node, and the analysis module includes:

[0225] A first downstream node acquisition unit is configured to acquire a first downstream node corresponding to each changed node; data represented by the first downstream node corresponding to each changed node is generated based on the data represented by each changed node;

[0226] The first change attribution result determining unit is configured to, when the first downstream node corresponding to each change node is included in the at least one change node, determine the change attribution result corresponding to each change node as follows: based on a cascade result caused by the first downstream node corresponding to each change node being included in the at least one change node:

[0227] The second change attribution result determination unit is configured to determine the association information between the data represented by each change node and the data represented by the first downstream node corresponding to each change node when there is a third node in the first downstream node corresponding to each change node, and determine the change attribution result corresponding to each change node based on the above association information; the above third node does not belong to the above at least one change node.

[0228] In an exemplary embodiment, the graph to which the above-mentioned at least one change node belongs is the above-mentioned historical graph, and the above-mentioned second change attribution result determination unit is configured to execute, based on the above-mentioned association information, determination of the change attribution result corresponding to each change node as follows: the result caused by the disconnection of the target edge, and the above-mentioned target edge is the edge between each change node and the first downstream node corresponding to each change node.

[0229] In an exemplary embodiment, the graph to which the at least one change node belongs is the current graph, and the second change attribution result determination unit is configured to determine the change attribution result corresponding to each change node based on the association information: the result caused by the connection of the newly added target edge, and the newly added target edge is the edge between each change node and the first downstream node corresponding to each change node.

[0230] In an exemplary embodiment, the change analysis result includes an early warning result, the change information includes a change edge, and the analysis module includes:

[0231] a second downstream node acquisition unit configured to acquire a second downstream node corresponding to the change edge; data represented by the second downstream node is generated based on data represented by a fourth node connected to the change edge, the nodes connected to the change edge include the fourth node and a fifth node, and the data represented by the fourth node is generated based on data represented by the fifth node;

[0232] The warning result determination unit is configured to perform warning analysis on the above-mentioned change edge, the second downstream node corresponding to the above-mentioned change edge, and the graph to which the above-mentioned change edge belongs, to obtain the above-mentioned warning result; the graph to which the above-mentioned change edge belongs is the above-mentioned current graph or the above-mentioned historical graph.

[0233] In an exemplary embodiment, the warning result determination unit includes:

[0234] a sixth node determining subunit, configured to determine a sixth node from the second downstream node; the out-degree of the sixth node is 0;

[0235] The object annotation graph determination subunit is configured to determine, when the sixth node has annotated object identification information, that the graph to which the changed edge belongs is an object annotation graph; and the object is the object represented by the object identification information;

[0236] The first warning result determination subunit is configured to, when the object annotation graph is the historical graph, remove the change edge from the object annotation graph as the warning result;

[0237] The second warning result determination subunit is configured to execute, when the object annotation graph is the current graph, adding the change edge to the object annotation graph as the warning result.

[0238] In an exemplary embodiment, the apparatus further comprises:

[0239] The third warning result determination subunit is configured to, when the object identification information does not exist at the sixth node and the object annotation graph is the historical graph, remove the change edge from the historical graph as the warning result;

[0240] The fourth warning result determination subunit is configured to execute the result of adding the above-mentioned change edge to the above-mentioned current graph as the above-mentioned warning result when the above-mentioned object identification information does not exist in the above-mentioned sixth node and the above-mentioned object annotation graph is the above-mentioned current graph.

[0241] In an exemplary embodiment, the above-mentioned atlas acquisition module includes:

[0242] A data acquisition unit is configured to acquire the data to be processed, the first data, and the second data to be processed;

[0243] The graph generation unit is configured to generate the above-mentioned current graph based on the current data and the dependency relationship between the above-mentioned current data, and to generate the above-mentioned historical graph based on the historical data and the dependency relationship between the above-mentioned historical data; the above-mentioned current data includes the above-mentioned data to be processed and the above-mentioned first data, and the above-mentioned historical data includes the above-mentioned data to be processed and the above-mentioned second data.

[0244] In an exemplary embodiment, the apparatus further comprises:

[0245] a labeling module configured to execute, in response to a node labeling instruction, labeling a current labeling graph from the current graph and labeling a historical labeling graph from the historical graph; the current labeling graph is a portion of the graph in the current graph, and the historical labeling graph is a portion of the graph in the historical graph;

[0246] a target change information determination module configured to determine target change information between the historical annotation graph and the current annotation graph;

[0247] The target change analysis result determination module is configured to perform attribution analysis and / or early warning analysis on the target change information to obtain a target change analysis result corresponding to the target change information.

[0248] In an exemplary embodiment, the annotation module includes:

[0249] A current marking node unit is configured to execute a response to the node marking instruction and mark a current marking node from the current graph;

[0250] A historical annotation node determining unit is configured to determine a historical annotation node corresponding to the current annotation node from the historical graph;

[0251] an upstream node acquisition unit configured to acquire a first upstream node corresponding to the historically annotated node and a second upstream node corresponding to the currently annotated node, wherein data represented by the historically annotated node is generated based on data represented by the first upstream node, and data represented by the currently annotated node is generated based on data represented by the second upstream node;

[0252] The annotation graph generation unit is configured to obtain the historical annotation graph based on the historical annotation node and the first upstream node, and obtain the current annotation graph based on the current annotation node and the second upstream node.

[0253] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0254] In an exemplary embodiment, an electronic device is also provided, including a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the steps of any data processing method in the above embodiments when executing the instructions stored in the memory.

[0255] The electronic device may be a terminal, a server or a similar computing device. For example, the electronic device is a server. Figure 10 This is a block diagram of an electronic device for data processing according to an exemplary embodiment. The electronic device 40 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 41 (the central processing unit 41 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 43 for storing data, and one or more storage media 42 for storing application programs 423 or data 422 (for example, one or more mass storage devices). Among them, the memory 43 and the storage medium 42 can be temporary storage or permanent storage. The program stored in the storage medium 42 may include one or more modules, and each module may include a series of instruction operations on the electronic device. Furthermore, the central processing unit 41 can be configured to communicate with the storage medium 42 to execute a series of instruction operations in the storage medium 42 on the electronic device 40. The electronic device 40 may also include one or more power supplies 46, one or more wired or wireless network interfaces 45, one or more input and output interfaces 44, and / or one or more operating systems 421, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0256] The input / output interface 44 can be used to receive or send data via a network. Specific examples of the aforementioned network may include a wireless network provided by a communications provider of the electronic device 40. In one embodiment, the input / output interface 44 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In an exemplary embodiment, the input / output interface 44 can be a radio frequency (RF) module for wireless communication with the Internet.

[0257] It can be understood by those skilled in the art that Figure 10 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 10 More or fewer components than shown, or with Figure 10 Different configurations shown.

[0258] In an exemplary embodiment, a computer-readable storage medium is further provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the steps of any data processing method in the above embodiments.

[0259] In an exemplary embodiment, a computer program product is further provided, including a computer program, which implements the data processing method provided in any one of the above embodiments when executed by a processor.

[0260] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, which can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present disclosure can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0261] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0262] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data processing method, characterized in that: include: Obtain a current graph and a historical graph corresponding to the data to be processed; the nodes in the current graph represent the data to be processed and the first data generated between the starting time point and the current time point depending on the data to be processed, the first edge in the current graph is used to describe the dependency relationship between the data represented by the first node, and the first node is a node connected to the first edge; the nodes in the historical graph represent the data to be processed and the second data generated between the starting time point and the historical time point depending on the data to be processed, the second edge in the historical graph is used to describe the dependency relationship between the data represented by the second node, and the second node is a node connected to the second edge; Determining change information between the current map and the historical map; Performing attribution analysis and / or early warning analysis on the change information to obtain a change analysis result corresponding to the change information; The early warning analysis is used to analyze changes in the change information in the current map or the historical map.

2. The data processing method according to claim 1, wherein: The change analysis result includes a change attribution result, the change information includes at least one change node, and performing attribution analysis on the change information to obtain a change analysis result corresponding to the change information includes: Obtaining a first downstream node corresponding to each change node; data represented by the first downstream node corresponding to each change node is generated based on the data represented by each change node; In a case where the first downstream node corresponding to each change node is included in the at least one change node, determining the change attribution result corresponding to each change node is as follows: based on a cascade result caused by the first downstream node corresponding to each change node being included in the at least one change node: In the case where a third node exists in the first downstream node corresponding to each change node, the association information between the data represented by each change node and the data represented by the first downstream node corresponding to each change node is determined, and based on the association information, the change attribution result corresponding to each change node is determined; the third node does not belong to the at least one change node.

3. The data processing method according to claim 2, characterized in that: The graph to which the at least one change node belongs is the historical graph, and determining, based on the association information, a change attribution result corresponding to each change node includes: According to the association information, the change attribution result corresponding to each change node is determined as: a result caused by disconnection of a target edge, where the target edge is an edge between each change node and a first downstream node corresponding to each change node.

4. The data processing method according to claim 2, wherein: The graph to which the at least one change node belongs is the current graph, and determining, based on the association information, a change attribution result corresponding to each change node includes: According to the association information, the change attribution result corresponding to each change node is determined as: a result caused by the connection of the newly added target edge, where the newly added target edge is an edge between each change node and the first downstream node corresponding to each change node.

5. The data processing method according to claim 1, wherein: The change analysis result includes a warning result, the change information includes a change edge, and performing warning analysis on the change information to obtain a change analysis result corresponding to the change information includes: Obtaining a second downstream node corresponding to the change edge; data represented by the second downstream node is generated based on data represented by a fourth node connected to the change edge, the nodes connected to the change edge include the fourth node and a fifth node, and the data represented by the fourth node is generated based on data represented by the fifth node; Performing a warning analysis on the changed edge, the second downstream node corresponding to the changed edge, and the graph to which the changed edge belongs to obtain the warning result; the graph to which the changed edge belongs is the current graph or the historical graph.

6. The data processing method according to claim 5, characterized in that: The performing early warning analysis on the changed edge, the second downstream node corresponding to the changed edge, and the graph to which the changed edge belongs to obtain the early warning result, including: Determine a sixth node from the second downstream node; the out-degree of the sixth node is 0; In the case where the sixth node has labeled object identification information, determining that the graph to which the changed edge belongs is an object labeled graph; and the object is the object represented by the object identification information; In a case where the object annotation graph is the historical graph, a result of removing the changed edge from the object annotation graph is used as the warning result; In a case where the object annotation graph is the current graph, a result of adding the changed edge to the object annotation graph is used as the warning result.

7. The data processing method according to claim 6, characterized in that: The method further comprises: If the object identification information does not exist at the sixth node and the object annotation graph is the historical graph, the result of removing the change edge from the historical graph is used as the warning result; When the object identification information does not exist at the sixth node and the object annotation graph is the current graph, the result of adding the changed edge to the current graph is used as the warning result.

8. The data processing method according to any one of claims 1 to 7, characterized in that: The obtaining of the current graph and the historical graph corresponding to the data to be processed includes: Acquire the data to be processed, the first data, and the second data; The current graph is generated based on current data and the dependency relationship between the current data, and the historical graph is generated based on historical data and the dependency relationship between the historical data; the current data includes the data to be processed and the first data, and the historical data includes the data to be processed and the second data.

9. The data processing method according to any one of claims 1 to 7, characterized in that: The method further comprises: In response to a node labeling instruction, a current labeled graph is labeled from the current graph, and a historical labeled graph is labeled from the historical graph; the current labeled graph is a partial graph in the current graph, and the historical labeled graph is a partial graph in the historical graph; Determining target change information between the historical annotated graph and the current annotated graph; The attribution analysis and / or the early warning analysis are performed on the target change information to obtain a target change analysis result corresponding to the target change information.

10. The data processing method according to claim 9, characterized in that: The step of labeling a current labeled graph from the current graph and labeling a historical labeled graph from the historical graph in response to the node labeling instruction includes: In response to the node marking instruction, marking a current marked node from the current graph; Determining a historical annotation node corresponding to the current annotation node from the historical graph; Obtaining a first upstream node corresponding to the historical annotation node and a second upstream node corresponding to the current annotation node, wherein data represented by the historical annotation node is generated based on data represented by the first upstream node, and data represented by the current annotation node is generated based on data represented by the second upstream node; The historical annotation graph is obtained according to the historical annotation node and the first upstream node, and the current annotation graph is obtained according to the current annotation node and the second upstream node.

11. A data processing device, characterized in that: include: A graph acquisition module is configured to execute acquisition of a current graph and a historical graph corresponding to the data to be processed; the nodes in the current graph represent the data to be processed and the first data generated between the starting time point and the current time point depending on the data to be processed; the first edge in the current graph is used to describe the dependency relationship between the data represented by the first node, and the first node is a node connected to the first edge; the nodes in the historical graph represent the data to be processed and the second data generated between the starting time point and the historical time point depending on the data to be processed; the second edge in the historical graph is used to describe the dependency relationship between the data represented by the second node, and the second node is a node connected to the second edge; a change information determination module, configured to determine change information between the current graph and the historical graph; an analysis module configured to perform attribution analysis and / or early warning analysis on the change information to obtain a change analysis result corresponding to the change information; The early warning analysis is used to analyze changes in the change information in the current map or the historical map.

12. The data processing device according to claim 11, characterized in that The change analysis result includes a change attribution result, the change information includes at least one change node, and the analysis module includes: A first downstream node acquisition unit is configured to acquire a first downstream node corresponding to each changed node; data represented by the first downstream node corresponding to each changed node is generated based on the data represented by each changed node; The first change attribution result determining unit is configured to, when the first downstream node corresponding to each change node is included in the at least one change node, determine the change attribution result corresponding to each change node as follows: based on a cascade result caused by the first downstream node corresponding to each change node being included in the at least one change node: The second change attribution result determination unit is configured to determine the association information between the data represented by each change node and the data represented by the first downstream node corresponding to each change node when there is a third node in the first downstream node corresponding to each change node, and determine the change attribution result corresponding to each change node based on the association information; the third node does not belong to the at least one change node.

13. The data processing device according to claim 12, characterized in that The graph to which the at least one change node belongs is the historical graph, and the second change attribution result determination unit is configured to determine, based on the association information, a change attribution result corresponding to each change node: a result caused by the disconnection of a target edge, wherein the target edge is an edge between each change node and a first downstream node corresponding to each change node.

14. The data processing device according to claim 12, characterized in that The graph to which the at least one change node belongs is the current graph, and the second change attribution result determination unit is configured to determine, based on the association information, the change attribution result corresponding to each change node as follows: a result caused by the connection of a newly added target edge, wherein the newly added target edge is an edge between each change node and the first downstream node corresponding to each change node.

15. The data processing device according to claim 11, characterized in that The change analysis result includes an early warning result, the change information includes a change edge, and the analysis module includes: a second downstream node acquisition unit configured to acquire a second downstream node corresponding to the change edge; data represented by the second downstream node is generated based on data represented by a fourth node connected to the change edge, the nodes connected to the change edge include the fourth node and a fifth node, and the data represented by the fourth node is generated based on data represented by the fifth node; The warning result determination unit is configured to perform warning analysis on the change edge, the second downstream node corresponding to the change edge, and the graph to which the change edge belongs to obtain the warning result; the graph to which the change edge belongs is the current graph or the historical graph.

16. The data processing device according to claim 15, characterized in that The early warning result determination unit includes: a sixth node determination subunit, configured to determine a sixth node from the second downstream node; the out-degree of the sixth node is 0; The object annotation graph determination subunit is configured to determine, when the sixth node has annotated object identification information, that the graph to which the changed edge belongs is an object annotation graph; and the object is the object represented by the object identification information; A first warning result determination subunit is configured to execute, when the object annotation graph is the historical graph, a result of removing the changed edge from the object annotation graph as the warning result; The second warning result determination subunit is configured to execute, when the object annotation graph is the current graph, a result of adding the changed edge to the object annotation graph as the warning result.

17. The data processing device according to claim 16, characterized in that The device further comprises: A third warning result determination subunit is configured to, when the object identification information does not exist at the sixth node and the object annotation graph is the historical graph, remove the change edge from the historical graph as the warning result; The fourth warning result determination subunit is configured to execute, when the object identification information does not exist at the sixth node and the object annotation graph is the current graph, the result of adding the change edge to the current graph as the warning result.

18. The data processing device according to any one of claims 11 to 17, characterized in that: The atlas acquisition module includes: a data acquisition unit, configured to acquire the data to be processed, the first data, and the second data; The graph generation unit is configured to generate the current graph based on current data and the dependency relationship between the current data, and to generate the historical graph based on historical data and the dependency relationship between the historical data; the current data includes the data to be processed and the first data, and the historical data includes the data to be processed and the second data.

19. The data processing device according to any one of claims 11 to 17, characterized in that: The device further comprises: a labeling module configured to execute, in response to a node labeling instruction, labeling a current labeling graph from the current graph and labeling a historical labeling graph from the historical graph; the current labeling graph is a portion of the graph in the current graph, and the historical labeling graph is a portion of the graph in the historical graph; a target change information determination module, configured to determine target change information between the historical annotation graph and the current annotation graph; The target change analysis result determination module is configured to perform attribution analysis and / or early warning analysis on the target change information to obtain a target change analysis result corresponding to the target change information.

20. The data processing device according to claim 19, characterized in that The marking module includes: a current marking node unit, configured to execute, in response to the node marking instruction, marking a current marking node from the current graph; a historical annotation node determining unit, configured to determine a historical annotation node corresponding to the current annotation node from the historical graph; an upstream node acquisition unit configured to acquire a first upstream node corresponding to the historically annotated node and a second upstream node corresponding to the currently annotated node, wherein data represented by the historically annotated node is generated based on data represented by the first upstream node, and data represented by the currently annotated node is generated based on data represented by the second upstream node; The annotation graph generation unit is configured to obtain the historical annotation graph according to the historical annotation node and the first upstream node, and obtain the current annotation graph according to the current annotation node and the second upstream node.

21. An electronic device, characterized in that: include: processor; A memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the data processing method according to any one of claims 1 to 10.

22. A computer-readable storage medium, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, causes the electronic device to execute the data processing method according to any one of claims 1 to 10.

23. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 10 is implemented.

Images