Data blood relationship tracing method based on block chain

Abstract

The invention discloses a data consanguinity tracing method based on a block chain. The method mainly comprises three stages of data acquisition and structured evidence storage, consanguinity information extraction and chaining, and consanguinity verification and visual display. According to the method, the source, the destination and the operation information of the data in the acquisition, processing, transmission and output processes are extracted by automatically analyzing the data processing log or the SQL statement, and the data blood relationship model is constructed according to the preset blood relationship modeling rule, so that the full-link traceability of the data between different systems and links is realized, and the data processing efficiency is improved. The method realizes credible collection, verifiable evidence storage and atlas display of blood relationship information, and has the advantages of high automation degree, strong auditing performance and good expandability.

Description

Technical Field

[0001] This invention relates to a blockchain-based data lineage tracing method, belonging to the field of data management and information security technology. Background Technology

[0002] Current data provenance / data lineage technology is primarily used to record information about the entire process of data from generation, transmission, processing to result formation, illustrating the data's source, processing procedures, and responsible parties. Early research proposed theoretical models such as "why / where / how provenance," laying the foundation for lineage computing. To achieve cross-system and cross-organizational data traceability, the W3C released the PROV series of specifications to uniformly describe the relationships between data entities, activities, and agents, improving the interoperability of lineage data.

[0003] With the widespread application of big data, cloud computing, and the Internet of Things, data flow processes have become more complex, and traditional lineage recording methods have limitations in automated extraction, cross-platform integration, and reliable verification. On the one hand, extracting lineage relationships from logs, SQL, or ETL tools requires complex parsing, making it difficult to cover dynamic or unstructured data processing. On the other hand, the aggregation and indexing of large-scale lineage data leads to high storage and computational overhead, affecting query performance. Furthermore, the reliability of lineage data remains a prominent issue, as traditional centralized storage is susceptible to tampering or loss.

[0004] In recent years, blockchain technology has been introduced into the field of data lineage due to its decentralized, immutable, and traceable characteristics. Some studies have achieved verifiable evidence storage of data operation records by putting lineage information or its hash value on the blockchain. However, existing solutions generally suffer from problems such as high on-chain costs, privacy risks, and complex on-chain and off-chain coordination, making it difficult to balance trustworthiness and scalability. Summary of the Invention

[0005] The purpose of this invention is to provide a blockchain-based data lineage tracing method that enables cross-system, verifiable, and low-overhead reliable data tracing, thereby solving a technical problem.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows: A blockchain-based method for tracing kinship data includes the following steps: Step 1: Data Collection and Structured Evidence Storage The system automatically parses data processing relationships from data processing logs or SQL scripts, generates structured lineage description files and lineage model files conforming to the W3C PROV standard, calculates hash values ​​for the original files, generates digital signatures, and stores them on the blockchain, while simultaneously saving complete metadata in an off-chain database. Step 2: Extraction and Uploading of Bloodline Information to the Blockchain The lineage file is uploaded to the knowledge graph database to establish the lineage relationship between entities, activities and subjects, generate a structured lineage record file, and complete on-chain notarization through smart contracts to achieve verifiable traceability of data throughout its entire lifecycle. Step 3: Bloodline Verification and Visual Display A unified verification interface provides lineage lookup based on file hash, task identifier, or time interval. By recalculating off-chain file hashes and comparing them with on-chain evidence, combined with signature verification and timestamp mechanisms, verifiable traceability of file content and responsible parties can be achieved. A visual interface displays the entire lineage relationship of data from source, processing to result.

[0007] As a preferred embodiment, specifically, the first step includes the following steps: S1. Deploy off-chain modules on local servers or data warehouses to handle log parsing, SQL statement analysis, and lineage extraction; on-chain modules implement on-chain storage of data lineage hashes and proofs based on consortium blockchains; deploy query and verification modules on the user end for visualization and integrity verification. S2. Database and on-chain node initialization: assign a unique identity ID to each participant and generate a corresponding key pair; blockchain nodes complete initialization and consensus configuration to ensure the immutability of subsequent lineage information writing; S3. Lineage Model Initialization: Presets a lineage representation model conforming to the W3C PROV standard, defines three types of nodes—entities, activities, and proxies—and their relationships, and establishes mapping rules to automatically convert field dependencies in SQL into the PROV model structure. S4. Data structure conventions define the data structures of three types of files that need to be uploaded to the blockchain during execution: SQL / log files, JSON files, and .prov files.

[0008] As a preferred embodiment, specifically, the second step includes the following steps: S5. Automatically collect execution logs and SQL files during data processing. Identify data operation types through the log monitoring module and SQL parsing module, extract the data source table, target table, and field mapping relationship, and store the parsing results in a structured JSON file. The generated JSON file is the intermediate result file for lineage extraction. S6. The parsed JSON file is converted into a data traceability representation file conforming to the W3C PROV specification by the lineage modeling module. The generated data traceability representation file records the complete path of data from generation, transmission, processing to result generation, and calculates the file hash value ProvHash for subsequent on-chain verification. S7. On-chain evidence storage and hash registration: The on-chain smart contract interface is called to write the generated SON file and the hash digest and metadata of the data traceability representation file into the blockchain; the on-chain data structure includes: file hash, corresponding off-chain identifier, uploader identity, timestamp, file description or version information; the consistency between hash and signature is automatically verified through the on-chain smart contract, and the result is written into the ledger to achieve tamper-proof and traceable evidence storage of lineage records; after the on-chain evidence storage is completed, the transaction hash and block height are returned as the unique and trusted index of the lineage record; S8. After on-chain verification is completed, an on-chain and off-chain correlation index record is generated to ensure that lineage data can be mutually queried between the two storage layers; the original off-chain lineage file can be quickly located through on-chain transaction hash or file hash, and the corresponding on-chain record can be queried through the UID of the off-chain database to realize a two-way verification mechanism; when a user requests verification, the real-time hash of the off-chain file is calculated and compared with the hash of the on-chain evidence; if they match, it proves that the lineage file has not been tampered with; if they do not match, an integrity alarm is issued. As a preferred option, specifically, the third step includes the following steps: S9. Users initiate verification requests through a unified lineage query interface; S10. After the lineage verification is completed, the knowledge graph engine is invoked to map the entities, activities and subjects in the lineage file into a graph structure. The formation path of the data lineage is displayed through the front-end visualization interface, including the source entity of the data and its generating activities, the input and output relationship of each data processing step, the responsible subject of each node and the operation time.

[0009] As a preferred option, specifically, in step S9, the user can initiate a verification request in the following three ways: On-chain hash query: The user inputs the hash of the on-chain file, the system locates the corresponding off-chain file uid according to the blockchain record, and reads the original file metadata from the off-chain database; Query based on filename or task identifier: Retrieve the corresponding record in the off-chain database based on the task ID or file path, and automatically compare it with the hash and signature registered on the chain; Search by time or uploader identity: Retrieve historical version records of related files by uploader and timestamp, supporting time range filtering and version backtracking.

[0010] Beneficial effects: Compared with existing technologies, this invention is an end-to-end data lineage tracing method that combines high-precision lineage analysis, knowledge graph modeling and blockchain evidence storage mechanism, realizing cross-system, verifiable and low-overhead reliable data traceability.

[0011] This invention automatically parses data processing logs or SQL statements to extract the source, destination, and operation information of data during the collection, processing, transmission, and output processes. It then constructs a data lineage relationship model based on preset lineage modeling rules to achieve full-link traceability of data across different systems and stages. This enables reliable collection, verifiable evidence storage, and graphical display of lineage information, and has the advantages of high automation, strong auditability, and good scalability.

[0012] This invention has the following characteristics: 1. In the process of lineage modeling, this invention adopts a lineage representation method conforming to the W3C PROV specification, abstracting data objects as entities, processing procedures as activities, and executing entities as agents, forming a directed lineage graph composed of "entity-activity-entity". Through knowledge graph modeling technology, lineage information is stored in a graph database (such as Neo4j), supporting cross-system and cross-level data tracking and visualization queries.

[0013] 2. Regarding data trust assurance, this invention utilizes blockchain technology to store kinship information. Specifically, a hash digest is calculated for key kinship relationships (including data node identifiers, operation types, timestamps, and executing entities), and written to the blockchain via a smart contract to achieve tamper-proof and verifiable storage of kinship records. Simultaneously, a complete kinship graph and business data are stored in an off-chain database, realizing a hybrid storage structure of "on-chain hash + off-chain data," thus balancing performance and trustworthiness.

[0014] 3. In the kinship query and verification stage, this invention implements kinship path retrieval based on a graph database through a query interface, and verifies the integrity and authenticity of kinship data by comparing on-chain hashes. When a user initiates a kinship tracing request, the system automatically matches the data's source path, operation history, and responsible entity, achieving end-to-end reliable data traceability. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the overall structure of the present invention; Figure 2 This is a schematic diagram of the data lineage processing flow of the present invention. Detailed Implementation

[0016] To further illustrate the technical means and effects of the present invention in achieving the intended purpose, the following detailed description of the specific implementation methods, structures, features and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0017] A blockchain-based data lineage tracing method comprises three stages: data collection and structured notarization, lineage information extraction and on-chain storage, and lineage verification and visualization. First, the system automatically parses data processing relationships from data processing logs or SQL scripts, generating a structured lineage description file (JSON) and a lineage model file (.prov) conforming to the W3C PROV standard. The system calculates the hash value of the original file, generates a digital signature, and stores it on-chain, while simultaneously saving complete metadata in an off-chain database. Second, the system uploads the lineage file to a knowledge graph database (Neo4j), establishing lineage associations between entities, activities, and agents, achieving end-to-end traceability. Third, the system provides a unified verification interface, supporting lineage queries based on file hashes, task identifiers, or time intervals. By recalculating the off-chain file hash and comparing it with the on-chain notarization, combined with signature verification and timestamp mechanisms, verifiable traceability of file content and responsible parties is achieved. Ultimately, the system displays the entire data lineage from source, processing to result through a front-end visual interface, supporting path tracing and exporting traceability reports.

[0018] Specifically, it includes the following steps: S1 to S4 constitute the first initialization stage of this invention: S1. The off-chain module is deployed on the local server or data warehouse side and is responsible for log parsing, SQL statement analysis and lineage extraction; the on-chain module is based on the consortium blockchain (FISCO BCOS) to realize the on-chain storage of data lineage hash and proof; the query and verification module is deployed on the user side for visualization and integrity verification. S2. Database and on-chain node initialization: The system administrator assigns a unique identifier (ID) to each participant (data provider, processor, user) and generates a corresponding key pair (PK, SK). Blockchain nodes complete initialization and consensus configuration to ensure the immutability of subsequent lineage information writing. S3. Lineage Model Initialization: The system pre-defines a lineage representation model conforming to the W3C PROV standard, defining three types of nodes: Entity, Activity, and Agent, and their relationships (wasDerivedFrom, wasGeneratedBy, used, etc.). Simultaneously, mapping rules are established to automatically convert field dependencies in SQL into the PROV model structure. S4. Data Structure Conventions: During the execution of this system, three types of files need to be uploaded to the blockchain: SQL / log files, JSON files, and .prov files. The data structures are defined as follows: sql / log file: on-chain struct{fileHash;offchainId;Uploader; Timestamp;Description;dataSignatureHash;Signer; signatureAlgorithm;signatureTimestamp;}, off-chain Struct{uid; json file: on-chain struct{ fileHash; offchainId; Uploader; Timestamp;Description;}, off-chain Struct{ uid; file_path; file_sha256; generated_at;uploader; source_file; created_at; updated_at;} .prov file: on-chain Struct{ ProvHash; prov_id; Uploader; Timestamp;Description; Version;}, off-chain Struct{ prov_uid; prov_file_path; prov_file_sha256; generated_at; uploader; version; description; source_file; created_at; updated_at;} In the above data structure, the following fields are the core control fields of the system: uid / offchainId: Used to achieve a one-to-one correspondence between on-chain and off-chain files, and is a unique identifier for cross-layer tracing.

[0019] fileHash / ProvHash: Hash values ​​calculated from the file content, used to verify the integrity of the file during transmission and storage.

[0020] dataSignatureHash / dataSignature: These represent the on-chain digest and off-chain original text of the digital signature, respectively, used to verify the identity of the uploader and the authenticity of the file.

[0021] Timestamp / signatureTimestamp: Used to identify the time sequence of a file being uploaded to the blockchain and signed, thus achieving time verifiability.

[0022] Version: Used for version control when lineage files are updated or revised to ensure that the evolution of lineage relationships is traceable.

[0023] S5 to S8 constitute the second stage of this invention: lineage extraction and on-chain processing. This stage primarily involves parsing data processing logs and SQL statements, automatically extracting data lineage relationships, generating structured lineage record files (JSON and .prov), and completing on-chain notarization through smart contracts to achieve verifiable traceability throughout the entire data lifecycle. S5. The system first automatically collects execution logs and SQL files during data processing. The middleware, through its log monitoring and SQL parsing modules, identifies the data operation type (SELECT, INSERT, UPDATE, JOIN, etc.) and extracts key information such as the source table, target table, and field mapping relationships. The parsing results are stored in a structured JSON file, with fields including source table, target table, transformation rules, operation time, and executing user; and a unique uid and fileHash are generated as on-chain and off-chain identifiers. The JSON file generated in this process is the intermediate result file for lineage extraction. S6. The parsed JSON file is converted into a data tracing representation file (.prov) conforming to the W3C PROV specification by the lineage modeling module. At this stage, the system automatically constructs three types of entities: Entity: representing a data table, dataset, or file; Activity: representing a data processing or transformation process; Agent: representing the user or system component performing the operation. A complete data lineage graph is formed by extracting relationships such as "used," "wasGeneratedBy," and "wasAssociatedWith." The generated .prov file records the complete path of data from generation, transmission, processing to result generation, and calculates the file hash value (ProvHash) for subsequent on-chain verification. S7. On-chain Notarization and Hash Registration: After the JSON file and .prov file are generated, the system calls the smart contract interface to write their hash digests and metadata to the blockchain. The on-chain data structure includes: file hash (fileHash / ProvHash), corresponding off-chain identifier (uid / prov_uid), uploader identity (Uploader), timestamp (Timestamp), file description (Description), and version information (Version, optional). The on-chain smart contract automatically verifies the consistency between the hash and the signature and writes the result to the ledger, achieving tamper-proof and traceable notarization of lineage records. After on-chain notarization is completed, the transaction hash (TxHash) and block height (BlockNumber) are returned as a unique and trusted index to the lineage record. S8. After on-chain verification is completed, the system generates an on-chain and off-chain correlation index record to ensure that lineage data can be cross-referenced between the two storage layers. In subsequent data audits or traceability queries, the system can quickly locate the original off-chain lineage file through on-chain transaction hashes or file hashes; simultaneously, it can query the corresponding on-chain record through the UID in the off-chain database, realizing a two-way verification mechanism. When a user requests verification, the system calculates the real-time hash of the off-chain file and compares it with the hash of the on-chain evidence. If they match, it proves that the lineage file has not been tampered with; if they do not match, an integrity alarm is issued. In addition, the system supports automatically reconstructing the lineage path based on the entity and activity relationships in the lineage file, enabling visual display and audit traceability. S9 to S10 represent the third stage of the present invention: bloodline query and verification display. This stage mainly realizes the visualization, verifiable query, and integrity audit of bloodline records, ensuring the consistency and credibility of bloodline information on and off the chain.

[0024] S9. The system provides a unified lineage query interface, allowing users or auditors to initiate verification requests in the following three ways: (1) Query based on on-chain hash: The user inputs the on-chain file hash (fileHash or ProvHash), and the system locates the corresponding off-chain file uid based on the blockchain record and reads the original file metadata from the off-chain database. (2) Query based on file name or task identifier: The system retrieves the corresponding record in the off-chain database based on the task ID or file path and automatically compares the hash and signature registered on the chain. (3) Query based on time or uploader identity: The system retrieves historical version records of lineage files by uploader (Uploader) and timestamp (Timestamp), supporting time interval filtering and version backtracking. During the query process, the system first recalculates the hash value of the off-chain file and compares it with the hash stored on the chain. If the two are consistent, it means that the file content has not been tampered with and the verification is successful; if they are inconsistent, the system records the abnormal log and triggers an alert. In addition, the system can also confirm the identity of the signer and the generation time of the file through the digital signature and timestamp verification module, thereby achieving verifiable provenance. S10. After lineage verification is complete, the system invokes the knowledge graph engine to map the entities, activities, and agents in the lineage file (.prov) into a graph structure. The formation path of the data lineage is displayed through a front-end visual interface, including the source entity of the data and its generating activities; the input and output relationships of each data processing step; and the responsible entity and operation time of each node. Users can view detailed information of any node through interactive operations in the interface, such as data file path, hash value, signer, and on-chain time. The system supports path-level backtracking, meaning that clicking on any result node automatically traces back to all upstream input data and processing activities. Furthermore, the system supports exporting a lineage verification report, which includes verification pass / fail results; file hash, signer, and timestamp information; matching block height and transaction hash; and a visual snapshot of the lineage path.

[0025] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A blockchain-based method for tracing kinship ties, characterized in that, Includes the following steps: Step 1: Data Collection and Structured Evidence Storage The system automatically parses data processing relationships from data processing logs or SQL scripts, generates structured lineage description files and lineage model files conforming to the W3C PROV standard, calculates hash values ​​for the original files, generates digital signatures, and stores them on the blockchain, while simultaneously saving complete metadata in an off-chain database. Step 2: Extraction and Uploading of Bloodline Information to the Blockchain The lineage file is uploaded to the knowledge graph database to establish the lineage relationship between entities, activities and subjects, generate a structured lineage record file, and complete on-chain notarization through smart contracts to achieve verifiable traceability of data throughout its entire lifecycle. Step 3: Bloodline Verification and Visual Display A unified verification interface is provided to provide lineage lookup based on file hash, task identifier or time interval. By recalculating off-chain file hashes and comparing them with on-chain evidence, combined with signature verification and timestamp mechanisms, verifiable traceability of file content and responsible parties can be achieved. It also uses a visual interface to display the entire data lineage from source, processing to result.

2. The data lineage tracing method based on blockchain according to claim 1, characterized in that, The first step includes the following steps: S1. Deploy off-chain modules on local servers or data warehouses to handle log parsing, SQL statement analysis, and lineage extraction; on-chain modules implement on-chain storage of data lineage hashes and proofs based on consortium blockchains; deploy query and verification modules on the user end for visualization and integrity verification. S2. Database and on-chain node initialization: assign a unique identity ID to each participant and generate a corresponding key pair; blockchain nodes complete initialization and consensus configuration to ensure the immutability of subsequent lineage information writing; S3. Lineage Model Initialization: Presets a lineage representation model conforming to the W3C PROV standard, defines three types of nodes—entities, activities, and proxies—and their relationships, and establishes mapping rules to automatically convert field dependencies in SQL into the PROV model structure. S4. Data structure conventions define the data structures of three types of files that need to be uploaded to the blockchain during execution: SQL / log files, JSON files, and .prov files.

3. The data lineage tracing method based on blockchain according to claim 1, characterized in that, The second step includes the following steps: S5. Automatically collect execution logs and SQL files during data processing. Identify data operation types through the log monitoring module and SQL parsing module, extract the data source table, target table, and field mapping relationship, and store the parsing results in a structured JSON file. The generated JSON file is the intermediate result file for lineage extraction. S6. The parsed JSON file is converted into a data traceability representation file conforming to the W3C PROV specification by the lineage modeling module. The generated data traceability representation file records the complete path of data from generation, transmission, processing to result generation, and calculates the file hash value ProvHash for subsequent on-chain verification. S7. On-chain evidence storage and hash registration: The on-chain smart contract interface is called to write the generated SON file and the hash digest and metadata of the data traceability representation file into the blockchain; the on-chain data structure includes: file hash, corresponding off-chain identifier, uploader identity, timestamp, file description or version information; the consistency between hash and signature is automatically verified through the on-chain smart contract, and the result is written into the ledger to achieve tamper-proof and traceable evidence storage of lineage records; after the on-chain evidence storage is completed, the transaction hash and block height are returned as the unique and trusted index of the lineage record; S8. After on-chain verification is completed, an on-chain and off-chain correlation index record is generated to ensure that lineage data can be mutually queried between the two storage layers; the original off-chain lineage file can be quickly located through on-chain transaction hash or file hash, and the corresponding on-chain record can be queried through the UID of the off-chain database to realize a two-way verification mechanism; when a user requests verification, the real-time hash of the off-chain file is calculated and compared with the hash of the on-chain evidence. If they match, it proves that the lineage file has not been tampered with; if they do not match, an integrity alarm is issued.

4. The data lineage tracing method based on blockchain according to claim 1, characterized in that, The third step includes the following steps: S9. Users initiate verification requests through a unified lineage query interface; S10. After the lineage verification is completed, the knowledge graph engine is invoked to map the entities, activities and subjects in the lineage file into a graph structure. The formation path of the data lineage is displayed through the front-end visualization interface, including the source entity of the data and its generating activities, the input and output relationship of each data processing step, the responsible subject of each node and the operation time.

5. The data lineage tracing method based on blockchain according to claim 4, characterized in that, In step S9, the user can initiate a verification request in the following three ways: On-chain hash query: The user inputs the hash of the on-chain file, the system locates the corresponding off-chain file uid according to the blockchain record, and reads the original file metadata from the off-chain database; Query based on filename or task identifier: Retrieve the corresponding record in the off-chain database based on the task ID or file path, and automatically compare it with the hash and signature registered on the chain; Search by time or uploader identity: Retrieve historical version records of related files by uploader and timestamp, supporting time range filtering and version backtracking.

Images