A data processing system and method based on lineage analysis

Abstract

This invention discloses a data processing system and method based on data lineage analysis, belonging to the field of big data analysis technology. The invention extracts operational information throughout the entire data lifecycle and labels it with a unique identifier containing data source type code, initial acquisition node ID, and a globally unique sequence number; preprocesses the operational information to construct dynamic associations; uses these dynamic associations to construct a data lineage network containing nodes and directed edges; determines whether the lineage is broken based on the average number of nodes; if broken, it is repaired and an anomaly report is generated. The system includes modules for information extraction and identification, preprocessing and association, lineage network construction, lineage breakage judgment, and breakage repair reporting. This invention solves the problem of missing and broken data lineages caused by static construction in existing technologies, improving the continuity, accuracy, and traceability of data processing, and is suitable for data quality control and problem tracing scenarios.

Description

Technical Field

[0001] This invention relates to the field of big data analytics, specifically a data processing system and method based on data lineage analysis. Background Technology

[0002] In today's big data era, the volume of data is exploding, and data processing is becoming increasingly complex. Information regarding the source, destination, and processing methods of data during its flow and processing is crucial for quality control, problem tracing, and compliance auditing.

[0003] Existing data processing technologies have shortcomings in data lineage analysis. Most existing technologies rely on static data to construct lineage relationships, which may lead to data gaps and broken lineage relationships, thus requiring improvements in the efficiency and accuracy of data processing. Summary of the Invention

[0004] The purpose of this invention is to provide a data processing system and method based on data lineage analysis to solve the problems raised in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a data processing method based on data lineage analysis, the data processing method comprising the following steps:

[0006] Step S1: Extract the full lifecycle operation information of a specific piece of data from the database and label it with a unique identifier;

[0007] Step S2: Preprocess the full lifecycle operation information and construct dynamic associations;

[0008] Step S3: Construct a data lineage network containing nodes and edges based on dynamic associations;

[0009] Step S4: Calculate the weighted score based on the average node value and edge density of the data lineage network to determine whether a data lineage break has occurred;

[0010] Step S5: If a data lineage break occurs, repair it and generate an anomaly handling report.

[0011] Furthermore, in step S1, the unique identifier includes the data source type code, the initial acquisition node ID, and the globally unique serial number.

[0012] Furthermore, the specific steps of step S2 are as follows:

[0013] Step S2-1: Classify the full lifecycle operation information according to the operation type, which includes addition, modification, deletion, query and transformation, to form a standard operation dataset;

[0014] Step S2-2: Use a syntax parser to parse the standard operation dataset, obtain the operation topic, operation object, operation timestamp, and operation logic, and store them as structured fields;

[0015] Step S2-3: Using the operation object of the structured field as the anchor point, establish a dynamic association by uniquely identifying the upstream source and downstream destination of the data throughout its entire lifecycle and arranging them in ascending order using timestamps. The upstream source is the preceding data and operations of a certain data, and the downstream destination is the derived data and operations generated after the data has been processed.

[0016] Furthermore, the specific steps of step S3 are as follows:

[0017] Step S3-1: Extract structured fields, treat the operation object as a node, assign a unique identifier to each node, and associate node attribute information, including data type and timestamp;

[0018] Step S3-2: Extract the upstream source and downstream destination of the data, establish directional edges from the upstream source to the current node, establish directional edges from the current node to the downstream destination, mark the operation type on each directional edge, and connect all nodes and directional edges to construct a data lineage network.

[0019] Furthermore, the specific steps of step S4 are as follows:

[0020] Step S4-1: Count the number of nodes, the number of directed edges, and the total number of data lineage networks. Use the number of nodes in all data lineage networks as the numerator and the total number of data lineage networks as the denominator. Calculate the average number of nodes. Subtract the absolute value of the average number of nodes from the average number of data lineage networks and denote the ratio to the average number of nodes as 'a'. Calculate the ratio of the number of directed edges to the number of nodes in a data lineage network and mark it as the edge density as 'b'.

[0021] Step S4-2: Calculate a weighted score based on the number of nodes and edge density. Data lineage networks with scores below a preset threshold are marked as candidate abnormal networks. The formula for calculating the weighted score is:

[0022] ;

[0023] s represents the score, a represents the ratio of the absolute value of the number of nodes in the data lineage network minus the average value to the average value, and b represents the edge density of the data lineage network.

[0024] Step S4-3: Verify whether the association records of the upstream source and downstream destination of the candidate anomaly network are complete. The verification process is as follows: traverse the candidate anomaly network whose upstream source is not an initial node and mark it as an upstream associated broken network, and mark the initial node as complete. The initial node is the first node of the complete data lineage network. Traverse the candidate anomaly network whose downstream destination is not a terminal node and mark it as a downstream associated broken network, and mark the terminal node as complete. The terminal node is the last node of the complete data lineage network.

[0025] Furthermore, the specific steps of step S5 are as follows:

[0026] Step S5-1: If it is an upstream associated fracture network, extract the unique identifier and timestamp of the first node and mark it as an upstream fracture node. Search for operation objects with timestamps earlier than that node in the structured field. According to the operation logic stored in the structured field, if the operation logic of the operation object and the operation logic of the upstream fracture node have a logical derivation relationship, mark the operation object as a candidate upstream node.

[0027] Step S5-2: Calculate the time interval between each pair of candidate upstream nodes and the time interval threshold between upstream associated broken network nodes. Extract nodes whose time intervals meet the time interval threshold. Logically decompose the logical operations of the upstream broken nodes. The logical decomposition is based on the functional positioning of the upstream broken nodes in the data lineage network. The functional positioning includes data transformation and calculation. The core operation elements are extracted by a syntax parser through structured fields. The same decomposition is performed on the nodes that meet the time interval threshold to generate an operation logic element set. According to the logical rules, nodes that meet the logical rules are selected from the operation logic element set to repair the upstream associated broken network.

[0028] Step S5-3: If it is a downstream associated fracture network, extract the unique identifier of the last node and mark it as a downstream fracture node. Search for operation objects with timestamps later than that node in the structured field. According to the operation logic stored in the structured field, if there is a logical derivation relationship between the operation logic of the operation object and the operation logic of the downstream fracture node, mark the operation object as a candidate downstream node.

[0029] Step S5-4: Calculate the time interval between each pair of candidate downstream nodes and the time interval threshold between downstream associated broken network nodes, extract data whose time intervals meet the time interval threshold, and perform logical decomposition on the logical operations of downstream broken nodes. The logical decomposition is based on the functional positioning of the downstream broken nodes in the data lineage network. The functional positioning includes data transformation and calculation. The core operation elements are extracted by a syntax parser through structured fields. The same decomposition is performed on nodes that meet the time interval threshold to generate a set of operation logic elements. According to the logical rules, nodes that meet the logical rules are selected from the set of operation logic elements to repair the downstream associated broken network.

[0030] Step S5-5: Store the repaired data lineage network in the database, verify the data lineage network, count the data types marked on all its directional edges, and form a ratio based on the number of types to form a type comparison set. Use the same method to extract the repaired data lineage network to form a set to be verified. If the similarity between the type comparison set and the set to be verified exceeds the threshold, the repair is complete. If complete, an anomaly handling report is generated for the repair process. If incomplete, the data lineage network is repaired again.

[0031] Furthermore, the data processing system based on data lineage analysis includes an information extraction and identification module, a preprocessing and association module, a lineage network construction module, a lineage break judgment module, and a break repair report module;

[0032] The information extraction and identification module is used to extract the full lifecycle operation information of a certain data in the database and mark it with a unique identifier; the preprocessing and association module is used to preprocess the full lifecycle operation information and construct dynamic associations; the lineage network construction module is used to construct a data lineage relationship network containing nodes and edges based on the dynamic associations; the lineage break judgment module is used to calculate a weighted score based on the average node value and edge density of the data lineage relationship network to determine whether a data lineage break has occurred; the break repair report module is used to repair the data lineage break and generate an anomaly handling report when a data lineage break occurs.

[0033] The output of the information extraction and identification module is electrically connected to the input of the preprocessing association module; the output of the preprocessing association module is electrically connected to the input of the bloodline network construction module; the output of the bloodline network construction module is electrically connected to the input of the bloodline breakage judgment module; and the output of the bloodline breakage judgment module is electrically connected to the input of the breakage repair report module.

[0034] Furthermore, the information extraction and identification module includes a full lifecycle extraction unit, a data source encoding unit, and a global serial number generation unit;

[0035] The full lifecycle extraction unit is used to extract the full lifecycle operation information of a certain data in the database; the data source encoding unit is used to generate a data source type encoding as a component of the unique identifier; the global serial number generation unit is used to generate a globally unique serial number as a component of the unique identifier.

[0036] The preprocessing association module includes an operation type classification unit, a syntax parsing structure unit, and a timestamp association unit;

[0037] The operation type classification unit is used to classify the operation information of the entire life cycle according to the operation type and form a standard operation dataset; the syntax parsing and structuring unit is used to parse the standard operation dataset with a syntax parser and obtain the operation information and store it as a structured field; the timestamp association unit is used to establish a dynamic association by using the operation object of the structured field as the anchor point, and by uniquely identifying the upstream and downstream of the associated data throughout the entire life cycle and arranging them in ascending order of timestamp.

[0038] Furthermore, the bloodline network construction module includes a node attribute configuration unit, a directional edge marking unit, and a network integration unit;

[0039] The node attribute configuration unit is used to extract structured fields, treat the operation object as a node and assign a unique identifier, and associate node attribute information; the directional edge marking unit is used to extract the upstream and downstream destinations of the data, establish directional edges and mark the operation type on each directional edge; the network integration unit is used to connect all nodes and directional edges to construct a data lineage network.

[0040] The bloodline breakage judgment module includes a node mean calculation unit, an abnormal network screening unit, and an association integrity verification unit.

[0041] The node mean calculation unit is used to count the number of nodes and the total number of all data lineage networks, calculate the average value of the nodes, and filter out data lineage networks with values ​​less than the average value as candidate abnormal networks; the abnormal network filtering unit is used to assist in filtering out candidate abnormal networks; the association integrity verification unit is used to verify whether the association records of upstream and downstream sources and destinations of candidate abnormal networks are complete, and mark the upstream or downstream association broken networks.

[0042] Furthermore, the fracture repair report module includes a fracture node localization unit, a similar node matching unit, and an anomaly report generation unit;

[0043] The fracture node localization unit is used to locate upstream or downstream fracture nodes in the upstream and downstream associated fracture network; the similar node matching unit is used to retrieve operation objects that meet the conditions, calculate feature cosine similarity, and repair the fracture network using data exceeding the threshold; the anomaly report generation unit is used to store the fully repaired data lineage network into the database and generate an anomaly handling report for the repair process.

[0044] Compared with the prior art, the beneficial effects of the present invention are:

[0045] 1. This invention improves the continuity and accuracy of data lineage analysis by extracting the full lifecycle operation information of data and marking it with a unique identifier, constructing dynamic associations after preprocessing, and then constructing a data lineage relationship network containing nodes and edges.

[0046] 2. This invention filters candidate abnormal networks by calculating the average number of nodes in the data lineage network, verifies the integrity of associated records to identify upstream and downstream breaks, and then repairs them through operational logic and time intervals, thus solving the problems of data loss and lineage breakage and ensuring the traceability of data flow.

[0047] 3. This invention comprehensively improves data processing efficiency by constructing a complete process from data extraction and association building to fracture repair and report generation, which helps to control data quality and trace the source of problems, and better meets the complex data processing needs of the big data era. Attached Figure Description

[0048] Figure 1 This is a flowchart illustrating a data processing method based on data lineage analysis according to the present invention.

[0049] Figure 2 This is a schematic diagram of the structure of a data processing system based on data lineage analysis according to the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] Example 1: As Figure 1 As shown, the present invention provides a technical solution, a data processing method based on data lineage analysis, the data processing method comprising the following steps:

[0052] Step S1: Extract the full lifecycle operation information of a specific piece of data from the database and label it with a unique identifier;

[0053] In step S1, the unique identifier includes the data source type code, the initial acquisition node ID, and the globally unique serial number.

[0054] For example:

[0055] The system's information extraction and identification module extracts the full lifecycle operation information of a customer's transfer transaction data and generates a unique identifier for the transaction data, consisting of three parts: data source type code identifier, initial collection node ID, and globally unique serial number.

[0056] Step S2: Preprocess the full lifecycle operation information and construct dynamic associations;

[0057] The specific steps of step S2 are as follows:

[0058] Step S2-1: Classify the full lifecycle operation information according to the operation type, which includes addition, modification, deletion, query and transformation, to form a standard operation dataset;

[0059] Step S2-2: Use a syntax parser to parse the standard operation dataset, obtain the operation topic, operation object, operation timestamp, and operation logic, and store them as structured fields;

[0060] Step S2-3: Using the operation object of the structured field as the anchor point, establish a dynamic association by uniquely identifying the upstream source and downstream destination of the data throughout its entire lifecycle and arranging them in ascending order using timestamps. The upstream source is the preceding data and operations of a certain data, and the downstream destination is the derived data and operations generated after the data has been processed.

[0061] For example:

[0062] By classifying the operations into four categories—add, query, modify, and transform—using the preprocessing module's operation type classification unit, a standard operation dataset is formed. This dataset is then parsed using the syntax parsing structure unit to obtain structured fields. Using the operation object as the anchor point, upstream and downstream operations are linked through unique identifiers.

[0063] Arranged in ascending order by timestamp, forming a dynamic chain of relationships.

[0064] Step S3: Construct a data lineage network containing nodes and edges based on dynamic associations;

[0065] The specific steps of step S3 are as follows:

[0066] Step S3-1: Extract structured fields, treat the operation object as a node, assign a unique identifier to each node, and associate node attribute information, including data type and timestamp;

[0067] Step S3-2: Extract the upstream source and downstream destination of the data, establish directional edges from the upstream source to the current node, establish directional edges from the current node to the downstream destination, mark the operation type on each directional edge, and connect all nodes and directional edges to construct a data lineage network.

[0068] For example:

[0069] Through the node attribute configuration unit of the lineage network construction module, the operation object is treated as a node, assigned a unique identifier and associated with attributes. Through the directional edge marking unit, directional edges are established from upstream to downstream and the operation type is marked. Node 1 to node 2 is marked as query, node 2 to node 3 as modification, node 3 to node 4 as conversion, and node 4 to node 5 as conversion. All nodes are connected to the directional edges to form a complete data lineage network.

[0070] Step S4: Calculate the weighted score based on the average node value and edge density of the data lineage network to determine whether a data lineage break has occurred;

[0071] The specific steps of step S4 are as follows:

[0072] Step S4-1: Count the number of nodes, the number of directed edges, and the total number of data lineage networks. Use the number of nodes in all data lineage networks as the numerator and the total number of data lineage networks as the denominator. Calculate the average number of nodes. Subtract the absolute value of the average number of nodes from the average number of data lineage networks and denote the ratio to the average number of nodes as 'a'. Calculate the ratio of the number of directed edges to the number of nodes in a data lineage network and mark it as the edge density as 'b'.

[0073] Step S4-2: Calculate a weighted score based on the number of nodes and edge density. Data lineage networks with scores below a preset threshold are marked as candidate abnormal networks. The formula for calculating the weighted score is:

[0074] ;

[0075] s represents the score, a represents the ratio of the absolute value of the number of nodes in the data lineage network minus the average value to the average value, and b represents the edge density of the data lineage network.

[0076] Step S4-3: Verify whether the association records of the upstream source and downstream destination of the candidate anomaly network are complete. The verification process is as follows: traverse the candidate anomaly network whose upstream source is not an initial node and mark it as an upstream associated broken network, and mark the initial node as complete. The initial node is the first node of the complete data lineage network. Traverse the candidate anomaly network whose downstream destination is not a terminal node and mark it as a downstream associated broken network, and mark the terminal node as complete. The terminal node is the last node of the complete data lineage network.

[0077] For example:

[0078] The total number of nodes in 100 data lineage networks is 800, and the total number of directed edges is 1200. The average number of nodes is calculated to be 8. Network X has 5 nodes and 6 directed edges. The calculated index 'a' is 0.375, the calculated edge density 'b' is 1.2, and the calculated weighted score 's' is 0.6225. Based on historical data, the normal score threshold for financial transaction networks is set to 0.8. Because network X's score 's' is less than the threshold, it is marked as a candidate abnormal network.

[0079] Step S5: If a data lineage break occurs, repair it and generate an anomaly handling report.

[0080] The specific steps of step S5 are as follows:

[0081] Step S5-1: If it is an upstream associated fracture network, extract the unique identifier and timestamp of the first node and mark it as an upstream fracture node. Search for operation objects with timestamps earlier than that node in the structured field. According to the operation logic stored in the structured field, if the operation logic of the operation object and the operation logic of the upstream fracture node have a logical derivation relationship, mark the operation object as a candidate upstream node.

[0082] Step S5-2: Calculate the time interval between each pair of candidate upstream nodes and the time interval threshold between upstream associated broken network nodes. Extract nodes whose time intervals meet the time interval threshold. Logically decompose the logical operations of the upstream broken nodes. The logical decomposition is based on the functional positioning of the upstream broken nodes in the data lineage network. The functional positioning includes data transformation and calculation. The core operation elements are extracted by a syntax parser through structured fields. The same decomposition is performed on the nodes that meet the time interval threshold to generate an operation logic element set. According to the logical rules, nodes that meet the logical rules are selected from the operation logic element set to repair the upstream associated broken network.

[0083] Step S5-3: If it is a downstream associated fracture network, extract the unique identifier of the last node and mark it as a downstream fracture node. Search for operation objects with timestamps later than that node in the structured field. According to the operation logic stored in the structured field, if there is a logical derivation relationship between the operation logic of the operation object and the operation logic of the downstream fracture node, mark the operation object as a candidate downstream node.

[0084] Step S5-4: Calculate the time interval between each pair of candidate downstream nodes and the time interval threshold between downstream associated broken network nodes, extract data whose time intervals meet the time interval threshold, and perform logical decomposition on the logical operations of downstream broken nodes. The logical decomposition is based on the functional positioning of the downstream broken nodes in the data lineage network. The functional positioning includes data transformation and calculation. The core operation elements are extracted by a syntax parser through structured fields. The same decomposition is performed on nodes that meet the time interval threshold to generate a set of operation logic elements. According to the logical rules, nodes that meet the logical rules are selected from the set of operation logic elements to repair the downstream associated broken network.

[0085] Step S5-5: Store the repaired data lineage network in the database, verify the data lineage network, count the data types marked on all its directional edges, and form a ratio based on the number of types to form a type comparison set. Use the same method to extract the repaired data lineage network to form a set to be verified. If the similarity between the type comparison set and the set to be verified exceeds the threshold, the repair is complete. If complete, an anomaly handling report is generated for the repair process. If incomplete, the data lineage network is repaired again.

[0086] For example:

[0087] For the aforementioned downstream associated fracture network, the last node is extracted as node 3 by the fracture node localization unit and marked as a downstream fracture node.

[0088] Searching for operation objects with timestamps later than node 3 in the structured fields reveals that the operation logic of nodes 4 and 5 has a logical derivative relationship with node 3; calculating the time interval between nodes 3 and 4, which meets the time interval threshold for the bank's transaction processing; extracting the core operation elements of node 3, and similarly extracting the core operation elements of candidate nodes to form a set of operation logic elements, and selecting matching nodes according to logical rules to repair the data lineage network;

[0089] The similar node matching unit completes the directional edges from node 3 to node 4 to node 5, repairs the lineage network, and extracts data lineage networks of the same type for verification. It checks whether the operation type of the data node is complete. If it is incomplete, it is repaired again. If it is complete, the anomaly report generation unit records the cause of the break, generates an anomaly handling report, and stores it in the database.

[0090] Example 2, as Figure 2 As shown, the present invention provides a data processing system based on data lineage analysis. The data processing system includes an information extraction and identification module, a preprocessing and association module, a lineage network construction module, a lineage break judgment module, and a break repair report module.

[0091] The information extraction and identification module is used to extract the full lifecycle operation information of a certain data in the database and mark it with a unique identifier; the preprocessing and association module is used to preprocess the full lifecycle operation information and construct dynamic associations; the lineage network construction module is used to construct a data lineage relationship network containing nodes and edges based on the dynamic associations; the lineage break judgment module is used to calculate a weighted score based on the average node value and edge density of the data lineage relationship network to determine whether a data lineage break has occurred; the break repair report module is used to repair the data lineage break and generate an anomaly handling report when a data lineage break occurs.

[0092] The output of the information extraction and identification module is electrically connected to the input of the preprocessing association module; the output of the preprocessing association module is electrically connected to the input of the bloodline network construction module; the output of the bloodline network construction module is electrically connected to the input of the bloodline breakage judgment module; and the output of the bloodline breakage judgment module is electrically connected to the input of the breakage repair report module.

[0093] The information extraction and identification module includes a full lifecycle extraction unit, a data source encoding unit, and a global serial number generation unit;

[0094] The full lifecycle extraction unit is used to extract the full lifecycle operation information of a certain data in the database; the data source encoding unit is used to generate a data source type encoding as a component of the unique identifier; the global serial number generation unit is used to generate a globally unique serial number as a component of the unique identifier.

[0095] The preprocessing association module includes an operation type classification unit, a syntax parsing structure unit, and a timestamp association unit;

[0096] The operation type classification unit is used to classify the operation information of the entire life cycle according to the operation type and form a standard operation dataset; the syntax parsing and structuring unit is used to parse the standard operation dataset with a syntax parser and obtain the operation information and store it as a structured field; the timestamp association unit is used to establish a dynamic association by using the operation object of the structured field as the anchor point, and by uniquely identifying the upstream and downstream of the associated data throughout the entire life cycle and arranging them in ascending order of timestamp.

[0097] The bloodline network construction module includes a node attribute configuration unit, a directional edge marking unit, and a network integration unit;

[0098] The node attribute configuration unit is used to extract structured fields, treat the operation object as a node and assign a unique identifier, and associate node attribute information; the directional edge marking unit is used to extract the upstream and downstream destinations of the data, establish directional edges and mark the operation type on each directional edge; the network integration unit is used to connect all nodes and directional edges to construct a data lineage network.

[0099] The bloodline breakage judgment module includes a node mean calculation unit, an abnormal network screening unit, and an association integrity verification unit.

[0100] The node mean calculation unit is used to count the number of nodes and the total number of all data lineage networks, calculate the average value of the nodes, and filter out data lineage networks with values ​​less than the average value as candidate abnormal networks; the abnormal network filtering unit is used to assist in filtering out candidate abnormal networks; the association integrity verification unit is used to verify whether the association records of upstream and downstream sources and destinations of candidate abnormal networks are complete, and mark the upstream or downstream association broken networks.

[0101] The fracture repair report module includes a fracture node localization unit, a similar node matching unit, and an anomaly report generation unit;

[0102] The fracture node localization unit is used to locate upstream or downstream fracture nodes in the upstream and downstream associated fracture network; the similar node matching unit is used to retrieve operation objects that meet the conditions, calculate feature cosine similarity, and repair the fracture network using data exceeding the threshold; the anomaly report generation unit is used to store the fully repaired data lineage network into the database and generate an anomaly handling report for the repair process.

[0103] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A data processing method based on lineage analysis, characterized in that: The data processing method includes the following steps: Step S1: Extract the full lifecycle operation information of a specific piece of data from the database and label it with a unique identifier; Step S2: Preprocess the full lifecycle operation information and construct dynamic associations; Step S3: Construct a data lineage network containing nodes and edges based on dynamic associations; Step S4: Based on the average node value and edge density of the data lineage network, determine whether the data lineage network is a candidate abnormal network, and determine whether the candidate abnormal network has experienced a data lineage break. Determine the type of data lineage break, where the lineage break type includes upstream associated break networks and downstream associated break networks; Step S5: Repair different types of data lineage breaks and generate an anomaly handling report; For the upstream associated fracture network, the unique identifier and timestamp of the first node are extracted and marked as the upstream fracture node. The operation object with a timestamp earlier than that node is retrieved in the structured field. According to the operation logic stored in the structured field, if the operation logic of the operation object and the operation logic of the upstream fracture node have a logical derivation relationship, the operation object is marked as a candidate upstream node. Based on logical rules, upstream nodes that conform to the logical rules are selected from the set of operational logical elements, and the upstream associated broken network is repaired. For the downstream associated fracture network, the unique identifier of the last node is extracted and marked as the downstream fracture node. The operation object with a timestamp later than that node is retrieved in the structured field. According to the operation logic stored in the structured field, if the operation logic of the operation object and the operation logic of the downstream fracture node have a logical derivation relationship, the operation object is marked as a candidate downstream node. Based on logical rules, downstream nodes that conform to the logical rules are selected from the set of operational logical elements, and the downstream associated broken network is repaired.

2. The data processing method based on data lineage analysis according to claim 1, characterized in that: In step S1, the unique identifier includes the data source type code, the initial acquisition node ID, and the globally unique serial number.

3. The data processing method based on data lineage analysis according to claim 1, characterized in that: The specific steps of step S2 are as follows: Step S2-1: Classify the full lifecycle operation information according to the operation type, which includes addition, modification, deletion, query and transformation, to form a standard operation dataset; Step S2-2: Use a syntax parser to parse the standard operation dataset, obtain the operation topic, operation object, operation timestamp, and operation logic, and store them as structured fields; Step S2-3: Using the operation object of the structured field as the anchor point, establish a dynamic association by uniquely identifying the upstream source and downstream destination of the data throughout its entire lifecycle and arranging them in ascending order using timestamps. The upstream source is the preceding data and operations of a certain data, and the downstream destination is the derived data and operations generated after the data has been processed.

4. The data processing method based on data lineage analysis according to claim 1, characterized in that: The specific steps of step S3 are as follows: Step S3-1: Extract structured fields, treat the operation object as a node, assign a unique identifier to each node, and associate node attribute information, including data type and timestamp; Step S3-2: Extract the upstream source and downstream destination of the data, establish directional edges from the upstream source to the current node, establish directional edges from the current node to the downstream destination, mark the operation type on each directional edge, and connect all nodes and directional edges to construct a data lineage network.

5. The data processing method based on data lineage analysis according to claim 1, characterized in that: The specific steps of step S4 are as follows: Step S4-1: Count the number of nodes, the number of directed edges, and the total number of data lineage networks. Use the number of nodes in all data lineage networks as the numerator and the total number of data lineage networks as the denominator. Calculate the average number of nodes. Subtract the absolute value of the average number of nodes from the average number of data lineage networks and denote the ratio to the average number of nodes as 'a'. Calculate the ratio of the number of directed edges to the number of nodes in a data lineage network and mark it as the edge density as 'b'. Step S4-2: Calculate a weighted score based on the number of nodes and edge density. Data lineage networks with scores below a preset threshold are marked as candidate abnormal networks. The formula for calculating the weighted score is: ； s represents the score, a represents the ratio of the absolute value of the number of nodes in the data lineage network minus the average value to the average value, and b represents the edge density of the data lineage network. Step S4-3: Verify whether the association records of the upstream source and downstream destination of the candidate anomaly network are complete. The verification process is as follows: traverse the candidate anomaly network whose upstream source is not an initial node and mark it as an upstream associated broken network, and mark the initial node as complete. The initial node is the first node of the complete data lineage network. Traverse the candidate anomaly network whose downstream destination is not a terminal node and mark it as a downstream associated broken network, and mark the terminal node as complete. The terminal node is the last node of the complete data lineage network.

6. The data processing method based on data lineage analysis according to claim 1, characterized in that: The specific steps of step S5 are as follows: Step 5-1: If it is an upstream associated broken network, extract candidate upstream nodes, calculate the time interval between each pair of candidate upstream nodes and the time interval threshold between nodes in the upstream associated broken network, extract data whose time intervals meet the time interval threshold, and perform logical decomposition on the logical operations of the upstream broken nodes. The logical decomposition is based on the functional positioning of the upstream broken nodes in the data lineage network. The functional positioning includes data transformation and calculation. The core operation elements are extracted through a syntax parser using structured fields. The same decomposition is performed on nodes that meet the time interval threshold to generate a set of operation logic elements. Nodes that meet the logic rules are selected from the set of operation logic elements according to the logic rules to repair the upstream associated broken network. Step 5-2: If it is a downstream associated fracture network, extract candidate downstream nodes, calculate the time interval between each pair of candidate downstream nodes and the time interval threshold between nodes in the downstream associated fracture network, extract data whose time intervals meet the time interval threshold, and perform logical decomposition on the logical operations of the downstream fracture nodes. The logical decomposition is based on the functional positioning of the downstream fracture nodes in the data lineage network. The functional positioning includes data transformation and calculation. The core operation elements are extracted through a syntax parser using structured fields. The same decomposition is performed on nodes that meet the time interval threshold to generate a set of operation logic elements. Nodes that meet the logic rules are selected from the set of operation logic elements according to the logic rules to repair the downstream associated fracture network. Step S5-3: Store the repaired data lineage network in the database, verify the data lineage network, count the data types marked on all its directional edges, and form a ratio based on the number of types to form a type comparison set. Use the same method to extract the repaired data lineage network to form a set to be verified. If the similarity between the type comparison set and the set to be verified exceeds the threshold, the repair is complete. If complete, an anomaly handling report is generated for the repair process. If incomplete, the data lineage network is repaired again.

7. A data processing system based on data lineage analysis, applied to the data processing method based on data lineage analysis as described in any one of claims 1-6, characterized in that: The data processing system includes an information extraction and identification module, a preprocessing and association module, a bloodline network construction module, a bloodline breakage judgment module, and a breakage repair report module. The information extraction and identification module is used to extract the full lifecycle operation information of a certain data in the database and mark it with a unique identifier; The preprocessing association module is used to preprocess the full lifecycle operation information and construct dynamic associations; the lineage network construction module is used to construct a data lineage relationship network containing nodes and edges based on the dynamic associations; the lineage break judgment module is used to calculate a weighted score based on the average node value and edge density of the data lineage relationship network to determine whether a data lineage break has occurred. The fracture repair report module is used to repair and generate an anomaly handling report when a data lineage break occurs. The output of the information extraction and identification module is electrically connected to the input of the preprocessing association module; the output of the preprocessing association module is electrically connected to the input of the bloodline network construction module; the output of the bloodline network construction module is electrically connected to the input of the bloodline breakage judgment module; and the output of the bloodline breakage judgment module is electrically connected to the input of the breakage repair report module.

8. A data processing system based on data lineage analysis according to claim 7, characterized in that: The information extraction and identification module includes a full lifecycle extraction unit, a data source encoding unit, and a global serial number generation unit; The full lifecycle extraction unit is used to extract the full lifecycle operation information of a certain data in the database; The data source encoding unit is used to generate a data source type encoding as a component of the unique identifier; the global serial number generation unit is used to generate a globally unique serial number as a component of the unique identifier. The preprocessing association module includes an operation type classification unit, a syntax parsing structure unit, and a timestamp association unit; The operation type classification unit is used to classify the operation information of the entire life cycle according to the operation type and form a standard operation dataset; the syntax parsing and structuring unit is used to parse the standard operation dataset with a syntax parser and obtain the operation information and store it as structured fields. The timestamp association unit is used to establish dynamic associations by using the operation object of the structured field as the anchor point, uniquely identifying the upstream and downstream of the associated data throughout its entire lifecycle, and arranging them in ascending order of timestamps.

9. A data processing system based on data lineage analysis according to claim 7, characterized in that: The bloodline network construction module includes a node attribute configuration unit, a directional edge marking unit, and a network integration unit; The node attribute configuration unit is used to extract structured fields, treat the operation object as a node and assign a unique identifier, and associate node attribute information. The directional edge marking unit is used to extract the upstream and downstream destinations of data, establish directional edges, and mark the operation type on each directional edge; the network integration unit is used to connect all nodes and directional edges to construct a data lineage network; The bloodline breakage judgment module includes a node mean calculation unit, an abnormal network screening unit, and an association integrity verification unit. The node mean calculation unit is used to count the number of nodes and the total number of all data lineage networks, calculate the average value of the nodes, and filter out data lineage networks with values ​​less than the average value as candidate abnormal networks; the abnormal network filtering unit is used to assist in filtering out candidate abnormal networks; the association integrity verification unit is used to verify whether the association records of upstream and downstream sources and destinations of candidate abnormal networks are complete, and mark the upstream or downstream association broken networks.

10. A data processing system based on data lineage analysis according to claim 7, characterized in that: The fracture repair report module includes a fracture node localization unit, a similar node matching unit, and an anomaly report generation unit; The fracture node localization unit is used to locate upstream or downstream fracture nodes in the upstream and downstream associated fracture network; the similar node matching unit is used to retrieve operation objects that meet the conditions, calculate feature cosine similarity, and repair the fracture network using data exceeding the threshold; the anomaly report generation unit is used to store the fully repaired data lineage network into the database and generate an anomaly handling report for the repair process.

Images