Data processing system and method based on data consanguinity analysis

Abstract

The invention discloses a data processing system and method based on data consanguinity analysis, and relates to the technical field of big data analys.The data processing method comprises the steps that data full life cycle operation information is extracted, and a unique identifier containing a data source type code, an initial collection node ID and a global unique serial number is marked; preprocessing the operation information, and constructing dynamic association; constructing a data blood relationship network containing nodes and directional edges through dynamic association; judging whether the blood relationship is fractured according to the node number average value; and if so, repairing and generating an abnormal report. The system comprises an information extraction identification module, a preprocessing association module, a blood relationship network construction module, a blood relationship fracture judgment module and a fracture repair report module. According to the method, the problems of missing and breakage caused by data blood relationship static construction in the prior art are solved, the continuity, accuracy and traceability of data processing are improved, and the method is suitable for data quality control and problem traceability scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of big data analysis, and in particular to a data processing system and method based on data lineage analysis. Background Art

[0002] In today's big data era, data volumes are exploding, and data processing is becoming increasingly complex. During data flow and processing, information about its source, destination, and processing methods is crucial for ensuring data quality, tracing issues, and conducting compliance audits.

[0003] Existing data processing technologies have shortcomings in data lineage analysis. Existing technologies mostly rely on static data to build lineage relationships, which may lead to problems such as missing data and broken lineage relationships, making the efficiency and accuracy of data processing need to be improved. Summary of the Invention

[0004] The purpose of the present invention is to provide a data processing system and method based on data lineage analysis to solve the problems raised in the prior art.

[0005] To achieve the above object, the present invention provides the following technical solution: a data processing method based on data lineage analysis, the data processing method comprising the following steps: Step S1: extract the full life cycle operation information of a certain data in the database and mark it with a unique identifier; Step S2: Preprocess the full life cycle operation information and build dynamic associations; Step S3: constructing a data kinship network including nodes and edges based on dynamic association; Step S4: Calculate a weighted score based on the node average and edge density of the data lineage relationship network to determine whether a data lineage break occurs; Step S5: If data lineage is broken, repair it and generate an exception handling report.

[0006] Furthermore, in step S1, the unique identifier includes a data source type code, an initial collection node ID, and a globally unique serial number.

[0007] Furthermore, the specific steps of step S2 are as follows: Step S2-1: Classify the full lifecycle operation information according to the operation type, including adding, modifying, deleting, querying and converting, to form a standard operation data set; Step S2-2: Use a syntax parser to parse the standard operation data set, obtain the operation subject, operation object, operation timestamp and operation logic, and store them as structured fields; Step S2-3: Using the operation object of the structured field as the anchor point, establish a dynamic association by uniquely identifying the upstream source and downstream destination of the associated data throughout its life cycle, and use timestamps in ascending order. The upstream source is the predecessor data and operations of a certain data, and the downstream destination is the derivative data and operations generated after a certain data is processed.

[0008] Furthermore, the specific steps of step S3 are as follows: Step S3-1: extract structured fields, treat the operation objects as nodes, assign a unique identifier to each node, and associate node attribute information, wherein the node attribute information includes a data type and a timestamp; Step S3-2: Extract the upstream source and downstream destination of the data, establish a directed edge from the upstream source to the current node, establish a directed edge from the current node to the downstream destination, mark the operation type on each directed edge, and connect all nodes and directed edges to build a data lineage network.

[0009] Furthermore, the specific steps of step S4 are as follows: Step S4-1: Count the number of nodes, the number of directed edges, and the total number of data lineage networks in all data lineage networks. Use the number of nodes in all data lineage networks as the numerator and the total number of data lineage networks as the denominator. Calculate the average value of the nodes. Deduct the absolute value of the number of nodes in the data lineage network minus the average value from the average value, and denote it as a. Calculate the ratio of the number of directed edges to the number of nodes in a particular data lineage network, mark it as edge density, and denote it as b. Step S4-2: Perform weighted scoring calculation on the number of nodes and edge density, and mark the data lineage network with a score lower than a preset threshold as a candidate abnormal network. The formula for weighted scoring calculation is: ; The s is the score, a is the ratio of the absolute value of the number of nodes in the data lineage network minus the average value to the average value, and b is the edge density of the data lineage network; Step S4-3, verify whether the association records between the upstream source and downstream destination of the candidate abnormal network are complete. The verification process is: traverse the candidate abnormal network whose upstream source is not the initial node and mark it as an upstream association broken network, and mark it as an initial node as complete. The initial node is the first node of the complete data lineage network. Traverse the candidate abnormal network whose downstream destination is not the terminal node and mark it as a downstream association broken network, and mark it as a terminal node as complete. The terminal node is the last node of the complete data lineage network.

[0010] Furthermore, the specific steps of step S5 are as follows: Step S5-1: If the upstream association is broken, extract the unique identifier and timestamp of the first node and mark it as the upstream broken node. Search the structured field for operation objects with timestamps earlier than that of the node. Based on the operation logic stored in the structured field, if the operation logic of the operation object has a logical derivative relationship with the operation logic of the upstream broken node, mark the operation object as a candidate upstream node. Step S5-2: Calculate the time intervals between candidate upstream nodes and the time interval threshold between nodes in the upstream associated broken network, extract nodes whose time intervals meet the time interval threshold, and perform logical decomposition on the logical operations of the upstream broken nodes. The logical decomposition is based on the functional positioning of the upstream broken nodes in the data lineage network. The functional positioning includes data conversion and calculation. A syntax parser is used to extract core operation elements through structured fields. The same decomposition is performed on the nodes that meet the time interval threshold to generate an operation logic element set. Nodes that meet the logical rules are screened from the operation logic element set according to the logical rules, and the upstream associated broken network is repaired. Step S5-3: If the downstream associated network is broken, extract the unique identifier of the last node and mark it as the downstream broken node. Search the structured field for operation objects with a timestamp later than that of the node. Based on the operation logic stored in the structured field, if the operation logic of the operation object and the operation logic of the downstream broken node have a logical derivative relationship, mark the operation object as a candidate downstream node. Step S5-4: Calculate the time intervals between candidate downstream nodes and the time interval threshold between nodes in the downstream associated broken network, extract data whose time intervals meet the time interval threshold, and perform logical decomposition on the logical operations of the downstream broken nodes. The logical decomposition is based on the functional positioning of the downstream broken nodes in the data lineage network. The functional positioning includes data conversion and calculation. A syntax parser is used to extract core operation elements through structured fields. The same decomposition is performed on nodes that meet the time interval threshold to generate an operation logic element set. Nodes that meet the logical rules are screened from the operation logic element set according to the logical rules, and the downstream associated broken network is repaired. Step S5-5: store the completely repaired data lineage network into the database, verify the data lineage network, count the data types marked on all its directional edges, and form a ratio based on the number of types to form a type comparison set. Use the same method to extract the repaired data lineage network to form a set to be verified. If the similarity between the type comparison set and the set to be verified exceeds the threshold, the repair is complete. If it is complete, an exception handling report will be generated for the repair process. If it is incomplete, the data lineage network will be repaired again.

[0011] Furthermore, the data processing system based on data lineage analysis includes an information extraction and identification module, a pre-processing association module, a lineage network construction module, a lineage break judgment module, and a break repair reporting module; The information extraction and identification module is used to extract the full life cycle operation information of a certain data in the database and mark it with a unique identifier; the preprocessing association module is used to preprocess the full life cycle operation information and build dynamic associations; the lineage network construction module is used to build a data lineage relationship network containing nodes and edges based on dynamic associations; the lineage break judgment module is used to calculate a weighted score based on the node average value and edge density of the data lineage relationship network to determine whether a data lineage break has occurred; the break repair report module is used to repair a data lineage break when it occurs and generate an exception handling report; The output end of the information extraction and identification module is electrically connected to the input end of the preprocessing association module; the output end of the preprocessing association module is electrically connected to the input end of the bloodline network construction module; the output end of the bloodline network construction module is electrically connected to the input end of the bloodline fracture judgment module; the output end of the bloodline fracture judgment module is electrically connected to the input end of the fracture repair reporting module.

[0012] Furthermore, the information extraction and identification module includes a full life cycle extraction unit, a data source encoding unit, and a global sequence number generation unit; The full life cycle extraction unit is used to extract the full life cycle operation information of a certain data in the database; the data source encoding unit is used to generate a data source type code as a component of the unique identifier; the global sequence number generation unit is used to generate a globally unique sequence number as a component of the unique identifier; The pre-processing association module includes an operation type classification unit, a syntax parsing and structuring unit, and a timestamp association unit; The operation type classification unit is used to classify the full life cycle operation information according to the operation type and form a standard operation data set; the syntax parsing and structuring unit is used to use a syntax parser to parse the standard operation data set and obtain the operation information and store it as a structured field; the timestamp association unit is used to use the operation object of the structured field as an anchor point, and establish a dynamic association by uniquely identifying the upstream and downstream of the entire life cycle of the associated data and arranging them in ascending order by timestamps.

[0013] Furthermore, the kinship network construction module includes a node attribute configuration unit, a directional edge labeling unit, and a network integration unit; The node attribute configuration unit is used to extract structured fields, treat the operation object as a node and assign a unique identifier, and associate node attribute information; the directed edge marking unit is used to extract the upstream and downstream destinations of the data, establish directed edges and mark the operation type on each directed edge; the network integration unit is used to connect all nodes and directed edges to construct a data lineage network; The bloodline break judgment module includes a node mean value calculation unit, an abnormal network screening unit and an association integrity verification unit; The node mean calculation unit is used to count the number of nodes and the total number of all data lineage networks, calculate the average value of the nodes and filter out the data lineage networks that are smaller than the average value and mark them as candidate abnormal networks; the abnormal network screening unit is used to assist in screening out candidate abnormal networks; the association integrity verification unit is used to verify whether the association records of the upstream and downstream sources and destinations on the candidate abnormal networks are complete, and mark the upstream or downstream association broken networks.

[0014] Furthermore, the fracture repair report module includes a fracture node positioning unit, a similar node matching unit and an abnormality report generating unit; The broken node positioning unit is used to locate the upstream or downstream broken nodes in the upstream and downstream associated broken network; the similar node matching unit is used to retrieve qualified operation objects, calculate the feature cosine similarity, and use data exceeding the threshold to repair the broken network; the exception report generation unit is used to store the completely repaired data lineage network into the database, and generate an exception handling report for the repair process.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention extracts data operation information throughout its life cycle and marks it with a unique identifier, constructs dynamic associations through preprocessing, and then constructs a data lineage relationship network containing nodes and edges, thereby improving the continuity and accuracy of data lineage analysis.

[0016] 2. The present invention screens candidate abnormal networks by calculating the average number of nodes in the data lineage network, verifies the integrity of related records to identify upstream and downstream breaks, and then repairs them through operational logic and time intervals, thereby solving the problems of data missing and lineage breaks and ensuring the traceability of data flow.

[0017] 3. This invention improves data processing efficiency comprehensively by constructing a complete process from data extraction, association construction to fracture repair and report generation, assists in data quality control and problem tracing, and better meets the complex data processing needs in the big data era. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flow chart of a data processing method based on data lineage analysis according to the present invention; Figure 2This is a structural diagram of a data processing system based on data lineage analysis according to the present invention. DETAILED DESCRIPTION

[0019] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0020] Example 1: Figure 1 As shown, the present invention provides a technical solution, a data processing method based on data lineage analysis, the data processing method comprises the following steps: Step S1: extract the full life cycle operation information of a certain data in the database and mark it with a unique identifier; In step S1, the unique identifier includes a data source type code, an initial collection node ID, and a globally unique serial number.

[0021] For example: Through the system's information extraction and identification module, the full life cycle operation information of a customer's transfer transaction data is extracted, and a unique identifier consisting of three parts: data source type code identifier, initial collection node ID and globally unique serial number is generated for the transaction data.

[0022] Step S2: Preprocess the full life cycle operation information and build dynamic associations; The specific steps of step S2 are as follows: Step S2-1: Classify the full lifecycle operation information according to the operation type, including adding, modifying, deleting, querying and converting, to form a standard operation data set; Step S2-2: Use a syntax parser to parse the standard operation data set, obtain the operation subject, operation object, operation timestamp and operation logic, and store them as structured fields; Step S2-3: Using the operation object of the structured field as the anchor point, establish a dynamic association by uniquely identifying the upstream source and downstream destination of the associated data throughout its life cycle, and use timestamps in ascending order. The upstream source is the predecessor data and operations of a certain data, and the downstream destination is the derivative data and operations generated after a certain data is processed.

[0023] For example: Through the operation type classification unit of the pre-processing association module, the above operations are divided into four categories: adding, querying, modifying, and converting to form a standard operation data set. Then, through the syntax parsing structure unit, the structured field is obtained. With the operation object as the anchor point, the upstream and downstream are associated through the unique identifier. Arrange in ascending order by timestamp to form a dynamic association chain.

[0024] Step S3: constructing a data kinship network including nodes and edges based on dynamic association; The specific steps of step S3 are as follows: Step S3-1: extract structured fields, treat the operation objects as nodes, assign a unique identifier to each node, and associate node attribute information, wherein the node attribute information includes a data type and a timestamp; Step S3-2: Extract the upstream source and downstream destination of the data, establish a directed edge from the upstream source to the current node, establish a directed edge from the current node to the downstream destination, mark the operation type on each directed edge, and connect all nodes and directed edges to build a data lineage network.

[0025] For example: Through the node attribute configuration unit of the bloodline network construction module, the operation object is used as a node, a unique identifier is assigned and attributes are associated. Through the directed edge marking unit, a directed edge is established from upstream to downstream and the operation type is marked. Node 1 to node 2 is marked as query, node 2 to node 3 is marked as modification, node 3 to node 4 is marked as conversion, and node 4 to node 5 is marked as conversion. All nodes are connected with directed edges to form a complete data bloodline network.

[0026] Step S4: Calculate a weighted score based on the node average and edge density of the data lineage relationship network to determine whether a data lineage break occurs; The specific steps of step S4 are as follows: Step S4-1: Count the number of nodes, the number of directed edges, and the total number of data lineage networks in all data lineage networks. Use the number of nodes in all data lineage networks as the numerator and the total number of data lineage networks as the denominator. Calculate the average value of the nodes. Deduct the absolute value of the number of nodes in the data lineage network minus the average value from the average value, and denote it as a. Calculate the ratio of the number of directed edges to the number of nodes in a particular data lineage network, mark it as edge density, and denote it as b. Step S4-2: Perform weighted scoring calculation on the number of nodes and edge density, and mark the data lineage network with a score lower than a preset threshold as a candidate abnormal network. The formula for weighted scoring calculation is: ; The s is the score, a is the ratio of the absolute value of the number of nodes in the data lineage network minus the average value to the average value, and b is the edge density of the data lineage network; Step S4-3, verify whether the association records between the upstream source and downstream destination of the candidate abnormal network are complete. The verification process is: traverse the candidate abnormal network whose upstream source is not the initial node and mark it as an upstream association broken network, and mark it as an initial node as complete. The initial node is the first node of the complete data lineage network. Traverse the candidate abnormal network whose downstream destination is not the terminal node and mark it as a downstream association broken network, and mark it as a terminal node as complete. The terminal node is the last node of the complete data lineage network.

[0027] For example: Statistics for the 100 data lineage networks show a total of 800 nodes and 1200 directed edges. The node average is calculated as 8 nodes. Network X has 5 nodes and 6 directed edges. The calculated index a is 0.375, the calculated edge density b is 1.2, and the calculated weighted score s is 0.6225. Based on historical data, the normal score threshold for financial transaction networks is set at 0.8. Since network X's score s is less than the threshold, it is marked as a candidate anomaly network.

[0028] Step S5: If data lineage is broken, repair it and generate an exception handling report.

[0029] The specific steps of step S5 are as follows: Step S5-1: If the upstream association is broken, extract the unique identifier and timestamp of the first node and mark it as the upstream broken node. Search the structured field for operation objects with timestamps earlier than that of the node. Based on the operation logic stored in the structured field, if the operation logic of the operation object has a logical derivative relationship with the operation logic of the upstream broken node, mark the operation object as a candidate upstream node. Step S5-2: Calculate the time intervals between candidate upstream nodes and the time interval threshold between nodes in the upstream associated broken network, extract nodes whose time intervals meet the time interval threshold, and perform logical decomposition on the logical operations of the upstream broken nodes. The logical decomposition is based on the functional positioning of the upstream broken nodes in the data lineage network. The functional positioning includes data conversion and calculation. A syntax parser is used to extract core operation elements through structured fields. The same decomposition is performed on the nodes that meet the time interval threshold to generate an operation logic element set. Nodes that meet the logical rules are screened from the operation logic element set according to the logical rules, and the upstream associated broken network is repaired. Step S5-3: If the downstream associated network is broken, extract the unique identifier of the last node and mark it as the downstream broken node. Search the structured field for operation objects with a timestamp later than that of the node. Based on the operation logic stored in the structured field, if the operation logic of the operation object and the operation logic of the downstream broken node have a logical derivative relationship, mark the operation object as a candidate downstream node. Step S5-4: Calculate the time intervals between candidate downstream nodes and the time interval threshold between nodes in the downstream associated broken network, extract data whose time intervals meet the time interval threshold, and perform logical decomposition on the logical operations of the downstream broken nodes. The logical decomposition is based on the functional positioning of the downstream broken nodes in the data lineage network. The functional positioning includes data conversion and calculation. A syntax parser is used to extract core operation elements through structured fields. The same decomposition is performed on nodes that meet the time interval threshold to generate an operation logic element set. Nodes that meet the logical rules are screened from the operation logic element set according to the logical rules, and the downstream associated broken network is repaired. Step S5-5: store the completely repaired data lineage network into the database, verify the data lineage network, count the data types marked on all its directional edges, and form a ratio based on the number of types to form a type comparison set. Use the same method to extract the repaired data lineage network to form a set to be verified. If the similarity between the type comparison set and the set to be verified exceeds the threshold, the repair is complete. If it is complete, an exception handling report will be generated for the repair process. If it is incomplete, the data lineage network will be repaired again.

[0030] For example: For the above downstream associated fracture network, the last node is extracted as node 3 through the fracture node location unit and marked as the downstream fracture node; The structured field was searched for operation objects with timestamps later than node 3. It was discovered that the operation logic of nodes 4 and 5 had a logical derivative relationship with node 3. The time interval between nodes 3 and 4 was calculated, which met the time interval threshold for bank transaction processing. The core operation elements of node 3 were extracted. Similarly, the core operation elements of candidate nodes were extracted to form an operation logic element set. According to the logical rules, the nodes that met the requirements were selected to repair the data lineage network. The directed edge from node 3 to node 4 to node 5 is completed through the similar node matching unit, the bloodline network is repaired and the same type of data bloodline network is extracted for verification, and the operation type of the data node is detected to see if it is complete. If it is incomplete, it is repaired again. If it is complete, the exception report generation unit is used to record the cause of the break, generate an exception handling report and store it in the database.

[0031] Example 2, as Figure 2As shown, the present invention provides a data processing system based on data lineage analysis, which includes an information extraction and identification module, a preprocessing and association module, a lineage network construction module, a lineage break judgment module, and a break repair reporting module; The information extraction and identification module is used to extract the full life cycle operation information of a certain data in the database and mark it with a unique identifier; the preprocessing association module is used to preprocess the full life cycle operation information and build dynamic associations; the lineage network construction module is used to build a data lineage relationship network containing nodes and edges based on dynamic associations; the lineage break judgment module is used to calculate a weighted score based on the node average value and edge density of the data lineage relationship network to determine whether a data lineage break has occurred; the break repair report module is used to repair a data lineage break when it occurs and generate an exception handling report; The output end of the information extraction and identification module is electrically connected to the input end of the preprocessing association module; the output end of the preprocessing association module is electrically connected to the input end of the bloodline network construction module; the output end of the bloodline network construction module is electrically connected to the input end of the bloodline fracture judgment module; the output end of the bloodline fracture judgment module is electrically connected to the input end of the fracture repair reporting module.

[0032] The information extraction and identification module includes a full life cycle extraction unit, a data source encoding unit and a global sequence number generation unit; The full life cycle extraction unit is used to extract the full life cycle operation information of a certain data in the database; the data source encoding unit is used to generate a data source type code as a component of the unique identifier; the global sequence number generation unit is used to generate a globally unique sequence number as a component of the unique identifier; The pre-processing association module includes an operation type classification unit, a syntax parsing and structuring unit, and a timestamp association unit; The operation type classification unit is used to classify the full life cycle operation information according to the operation type and form a standard operation data set; the syntax parsing and structuring unit is used to use a syntax parser to parse the standard operation data set and obtain the operation information and store it as a structured field; the timestamp association unit is used to use the operation object of the structured field as an anchor point, and establish a dynamic association by uniquely identifying the upstream and downstream of the entire life cycle of the associated data and arranging them in ascending order by timestamps.

[0033] The kinship network construction module includes a node attribute configuration unit, a directional edge marking unit and a network integration unit; The node attribute configuration unit is used to extract structured fields, treat the operation object as a node and assign a unique identifier, and associate node attribute information; the directed edge marking unit is used to extract the upstream and downstream destinations of the data, establish directed edges and mark the operation type on each directed edge; the network integration unit is used to connect all nodes and directed edges to construct a data lineage network; The bloodline break judgment module includes a node mean value calculation unit, an abnormal network screening unit and an association integrity verification unit; The node mean calculation unit is used to count the number of nodes and the total number of all data lineage networks, calculate the average value of the nodes and filter out the data lineage networks that are smaller than the average value and mark them as candidate abnormal networks; the abnormal network screening unit is used to assist in screening out candidate abnormal networks; the association integrity verification unit is used to verify whether the association records of the upstream and downstream sources and destinations on the candidate abnormal networks are complete, and mark the upstream or downstream association broken networks.

[0034] The fracture repair report module includes a fracture node positioning unit, a similar node matching unit and an abnormality report generation unit; The broken node positioning unit is used to locate the upstream or downstream broken nodes in the upstream and downstream associated broken network; the similar node matching unit is used to retrieve qualified operation objects, calculate the feature cosine similarity, and use data exceeding the threshold to repair the broken network; the exception report generation unit is used to store the completely repaired data lineage network into the database, and generate an exception handling report for the repair process.

[0035] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A data processing method based on data lineage analysis, characterized by: The data processing method comprises the following steps: Step S1: extract the full life cycle operation information of a certain data in the database and mark it with a unique identifier; Step S2: Preprocess the full life cycle operation information and build dynamic associations; Step S3: constructing a data kinship network including nodes and edges based on dynamic association; Step S4: judging whether the data lineage network is a candidate abnormal network based on the node average value and edge density of the data lineage network, and judging whether the candidate abnormal network has a data lineage break; Determine the type of data lineage break, where the lineage break type includes upstream association break network and downstream association break network; Step S5: Repair different types of data lineage breaks and generate an exception handling report; For the upstream associated fracture network, the unique identifier and timestamp of the first node are extracted and marked as the upstream fracture node. The operation objects with timestamps earlier than that of the node are searched in the structured field. According to the operation logic stored in the structured field, if the operation logic of the operation object has a logical derivative relationship with the operation logic of the upstream fracture node, the operation object is marked as a candidate upstream node. According to the logic rules, upstream nodes that meet the logic rules are selected from the set of operation logic elements, and the upstream associated broken network is repaired; For the downstream associated fracture network, the unique identifier of the last node is extracted and marked as the downstream fracture node. The operation objects with a timestamp later than that of the node are searched in the structured field. According to the operation logic stored in the structured field, if the operation logic of the operation object has a logical derivative relationship with the operation logic of the downstream fracture node, the operation object is marked as a candidate downstream node. According to the logic rules, the downstream nodes that meet the logic rules are selected in the operation logic element set, and the downstream associated broken network is repaired.

2. The data processing method based on data lineage analysis according to claim 1, characterized in that: In step S1, the unique identifier includes a data source type code, an initial collection node ID, and a globally unique serial number.

3. The data processing method based on data lineage analysis according to claim 1, characterized in that: The specific steps of step S2 are as follows: Step S2-1: Classify the full lifecycle operation information according to the operation type, including adding, modifying, deleting, querying and converting, to form a standard operation data set; Step S2-2: Use a syntax parser to parse the standard operation data set, obtain the operation subject, operation object, operation timestamp and operation logic, and store them as structured fields; Step S2-3: Using the operation object of the structured field as the anchor point, establish a dynamic association by uniquely identifying the upstream source and downstream destination of the associated data throughout its life cycle, and use timestamps in ascending order. The upstream source is the predecessor data and operations of a certain data, and the downstream destination is the derivative data and operations generated after a certain data is processed.

4. The data processing method based on data lineage analysis according to claim 1, characterized in that: The specific steps of step S3 are as follows: Step S3-1: extract structured fields, treat the operation objects as nodes, assign a unique identifier to each node, and associate node attribute information, wherein the node attribute information includes a data type and a timestamp; Step S3-2: Extract the upstream source and downstream destination of the data, establish a directed edge from the upstream source to the current node, establish a directed edge from the current node to the downstream destination, mark the operation type on each directed edge, and connect all nodes and directed edges to build a data lineage network.

5. The data processing method based on data lineage analysis according to claim 1, characterized in that: The specific steps of step S4 are as follows: Step S4-1: Count the number of nodes, the number of directed edges, and the total number of data lineage networks in all data lineage networks. Use the number of nodes in all data lineage networks as the numerator and the total number of data lineage networks as the denominator. Calculate the average value of the nodes. Deduct the absolute value of the number of nodes in the data lineage network minus the average value from the average value, and denote it as a. Calculate the ratio of the number of directed edges to the number of nodes in a particular data lineage network, mark it as edge density, and denote it as b. Step S4-2: Perform weighted scoring calculation on the number of nodes and edge density, and mark the data lineage network with a score lower than a preset threshold as a candidate abnormal network. The formula for weighted scoring calculation is: ； The s is the score, a is the ratio of the absolute value of the number of nodes in the data lineage network minus the average value to the average value, and b is the edge density of the data lineage network; Step S4-3, verify whether the association records between the upstream source and downstream destination of the candidate abnormal network are complete. The verification process is: traverse the candidate abnormal network whose upstream source is not the initial node and mark it as an upstream association broken network, and mark it as an initial node as complete. The initial node is the first node of the complete data lineage network. Traverse the candidate abnormal network whose downstream destination is not the terminal node and mark it as a downstream association broken network, and mark it as a terminal node as complete. The terminal node is the last node of the complete data lineage network.

6. The data processing method based on data lineage analysis according to claim 1, characterized in that: The specific steps of step S5 are as follows: Step 5-1: If the upstream association network is broken, extract candidate upstream nodes, calculate the time intervals between candidate upstream nodes and the time interval threshold between nodes in the upstream association network, extract data whose time intervals meet the time interval threshold, and perform logical decomposition on the logical operations of the upstream broken nodes. The logical decomposition is based on the functional positioning of the upstream broken nodes in the data lineage network. The functional positioning includes data conversion and calculation. The core operation elements are extracted using a syntax parser through structured fields. The same decomposition is performed on the nodes that meet the time interval threshold to generate an operation logic element set. The nodes that meet the logical rules are screened from the operation logic element set according to the logical rules to repair the upstream association broken network. Step 5-2: If the downstream associated network is broken, extract candidate downstream nodes, calculate the time intervals between candidate downstream nodes and the time interval threshold between nodes in the downstream associated broken network, extract data whose time intervals meet the time interval threshold, and perform logical decomposition on the logical operations of the downstream broken nodes. The logical decomposition is based on the functional positioning of the downstream broken nodes in the data lineage network. The functional positioning includes data conversion and calculation. The core operation elements are extracted using a syntax parser through structured fields. The same decomposition is performed on the nodes that meet the time interval threshold to generate an operation logic element set. The nodes that meet the logical rules are screened from the operation logic element set according to the logical rules to repair the downstream associated broken network. Step S5-3: store the completely repaired data lineage network into the database, verify the data lineage network, count the data types marked on all its directional edges, and form a ratio based on the number of types to form a type comparison set. Use the same method to extract the repaired data lineage network to form a set to be verified. If the similarity between the type comparison set and the set to be verified exceeds the threshold, the repair is complete. If it is complete, an exception handling report will be generated for the repair process. If it is incomplete, the data lineage network will be repaired again.

7. A data processing system based on data lineage analysis, applied to a data processing method based on data lineage analysis according to any one of claims 1 to 6, characterized in that: The data processing system includes an information extraction and identification module, a pre-processing association module, a bloodline network construction module, a bloodline break judgment module, and a break repair reporting module; The information extraction and identification module is used to extract the full life cycle operation information of a certain data in the database and mark it with a unique identification; The pre-processing association module is used to pre-process the full life cycle operation information and build dynamic associations; the lineage network construction module is used to build a data lineage relationship network containing nodes and edges based on dynamic associations; the lineage break judgment module is used to calculate a weighted score based on the node average value and edge density of the data lineage relationship network to determine whether a data lineage break has occurred; The break repair report module is used to repair data lineage breaks and generate an exception handling report when a break occurs; The output end of the information extraction and identification module is electrically connected to the input end of the preprocessing association module; the output end of the preprocessing association module is electrically connected to the input end of the bloodline network construction module; the output end of the bloodline network construction module is electrically connected to the input end of the bloodline fracture judgment module; the output end of the bloodline fracture judgment module is electrically connected to the input end of the fracture repair reporting module.

8. The data processing system based on data lineage analysis according to claim 7, characterized in that: The information extraction and identification module includes a full life cycle extraction unit, a data source encoding unit and a global sequence number generation unit; The full life cycle extraction unit is used to extract the full life cycle operation information of a certain data in the database; The data source encoding unit is used to generate a data source type code as a component of the unique identifier; the global sequence number generating unit is used to generate a globally unique sequence number as a component of the unique identifier; The pre-processing association module includes an operation type classification unit, a syntax parsing and structuring unit, and a timestamp association unit; The operation type classification unit is used to classify the full life cycle operation information according to the operation type and form a standard operation data set; the syntax parsing and structuring unit is used to parse the standard operation data set using a syntax parser and obtain the operation information and store it as a structured field; The timestamp association unit is used to establish a dynamic association by taking the operation object of the structured field as an anchor point, uniquely identifying the upstream and downstream of the entire life cycle of the associated data and arranging them in ascending order of timestamps.

9. The data processing system based on data lineage analysis according to claim 7, characterized in that: The kinship network construction module includes a node attribute configuration unit, a directional edge marking unit and a network integration unit; The node attribute configuration unit is used to extract structured fields, treat the operation object as a node and assign a unique identifier, and associate node attribute information; The directed edge marking unit is used to extract the upstream and downstream destinations of the data, establish directed edges and mark the operation type on each directed edge; the network integration unit is used to connect all nodes and directed edges to build a data lineage network; The bloodline break judgment module includes a node mean value calculation unit, an abnormal network screening unit and an association integrity verification unit; The node mean calculation unit is used to count the number of nodes and the total number of all data lineage networks, calculate the average value of the nodes and filter out the data lineage networks that are smaller than the average value and mark them as candidate abnormal networks; the abnormal network screening unit is used to assist in screening out candidate abnormal networks; the association integrity verification unit is used to verify whether the association records of the upstream and downstream sources and destinations on the candidate abnormal networks are complete, and mark the upstream or downstream association broken networks.

10. The data processing system based on data lineage analysis according to claim 7, characterized in that: The fracture repair report module includes a fracture node positioning unit, a similar node matching unit and an abnormality report generation unit; The broken node positioning unit is used to locate the upstream or downstream broken nodes in the upstream and downstream associated broken network; the similar node matching unit is used to retrieve qualified operation objects, calculate the feature cosine similarity, and use data exceeding the threshold to repair the broken network; the exception report generation unit is used to store the completely repaired data lineage network into the database, and generate an exception handling report for the repair process.

Images