A data flow storage and tracing method based on off-chain verifiable and on-chain verification mechanism

Abstract

The application provides a data flow storage and tracing method based on an off-chain verifiable and on-chain verification mechanism, aiming to guarantee the credibility and auditability of the data flow process, encapsulating operation events into structured event objects containing data digest, stage identifier, operation subject signature and timestamp at each circulation stage, constructing a continuous event chain through a hash pointer off-chain to realize complete operation tracing, extracting key information at each stage to generate a compressed stage digest object and writing it into a block chain to form an unalterable on-chain digest chain, thereby significantly reducing storage overhead. During verification, the on-chain digest and the hash value of the complete event data recovered off-chain are compared, and the digital signature is verified to ensure data integrity and operation legality. The application can realize efficient and lightweight storage while taking into account data privacy and traceability requirements, and is suitable for data circulation scenarios involving multiple parties.

Description

Technical Field

[0001] This invention belongs to the field of data security technology, and in particular relates to a data stream evidence storage and traceability method based on off-chain verifiable and on-chain verification mechanisms. Background Technology

[0002] With the booming development of the digital economy, the value of data as a crucial production factor is increasingly prominent, and the data circulation and trading market is rapidly expanding. However, data faces numerous challenges during circulation, such as unclear data sources, data tampering, and improper data use, severely hindering the release of data value. Against this backdrop, a technical solution is needed to design a system capable of reliable evidence storage and traceability throughout the entire data circulation process. This solution aims to meet the data credibility and compliance requirements of multiple stakeholders, including data providers, users, and regulators, thereby promoting the healthy development of the data factor market. The technical requirements for this task include: comprehensively recording key information at each stage of data circulation; preventing data tampering; protecting data privacy; and facilitating subsequent auditing and supervision.

[0003] Research on existing technical solutions revealed the following problems: First, traditional centralized evidence storage methods store data on a single central node, making them vulnerable to attacks and subject to single-point-of-failure risks. If the central node fails, the credibility of the entire evidence storage data will be questioned. Second, some blockchain-based evidence storage solutions directly upload large amounts of data to the blockchain, leading to excessive storage pressure, low transaction processing efficiency, and a complex data upload process that hinders efficient business operations. Third, many solutions struggle to balance data privacy protection and data traceability, either exposing too many data details or failing to meet the requirements for comprehensive traceability. Summary of the Invention

[0004] According to a first aspect of the present invention, the present invention claims protection for a data stream evidence storage and traceability method based on off-chain verifiable and on-chain verification mechanisms, comprising the following steps: Step S1: In the multiple stages of data flow, the operation events of each stage are encapsulated into structured event objects, which include data digests, stage identifiers, operation subject signatures, and timestamps; Step S2: Link each structured event object to the event hash of the previous stage through the event hash to form an off-chain operation event chain; Step S3: For each stage, extract key information to generate a stage summary object. The key information includes event hash, stage number, previous stage hash, and operation subject signature. Step S4: Write each stage summary object into the blockchain to form an on-chain stage summary chain; Step S5: When it is necessary to verify a specific stage, obtain the stage digest object of that stage from the blockchain, and verify the hash pointer structure and stage order; Step S6: Recover the complete event data for this stage from off-chain, recalculate the hash value and compare it with the on-chain digest. If the hash values ​​match, verify the operator's signature to confirm the legitimacy of the operation.

[0005] Furthermore, the data digest is calculated from the stage data using a hash algorithm.

[0006] Furthermore, in step S1, encapsulating the operation events of each stage into structured event objects also includes: Verify the digital certificate of the operator to ensure the validity of the signature, and generate the timestamp using a trusted time source; Generate a unique identifier for the event and associate it with a stage identifier for easy retrieval and tracking later.

[0007] Furthermore, in step S2, when forming the chain of downstream operation events, the following is also included: Store structured event objects in a distributed file system and generate an access index for subsequent retrieval; Chained connections are achieved by hashing the data digest of the current event with the data digest of the previous event, ensuring the continuity and integrity of the chain.

[0008] Furthermore, in step S3, when generating the stage summary object, the method further includes: Compressing critical information reduces on-chain storage space; compression methods include lossless encoding or field merging.

[0009] Furthermore, in step S4, when writing to the blockchain, a smart contract is used to automatically verify the correctness of the format of the phase digest object, including checking the field integrity and hash value format.

[0010] Furthermore, in step S5, when verifying the hash pointer structure, the following steps are also included: Verify the continuity of stage numbers to ensure the correct order, and check that the hash pointer points to the correct previous stage digest.

[0011] Furthermore, in step S6, when recovering complete event data from off-chain, the off-chain storage location is located by using the stage identifier and event hash, and the data is obtained using the query interface of the distributed file system.

[0012] Furthermore, in step S6, when verifying the signature of the operating entity, the validity period and issuing authority of the digital certificate are also checked to ensure that the signature is valid and trustworthy when the event occurs.

[0013] Furthermore, the method also includes step S7: if the verification fails, an audit log is generated and an alarm mechanism is triggered. The audit log records the specific reasons and time of the verification failure, and the alarm mechanism notifies the relevant responsible parties.

[0014] This invention proposes a data flow evidence preservation and traceability method based on off-chain verifiable and on-chain verification mechanisms. It aims to ensure the credibility and auditability of the data circulation process. By encapsulating operational events into structured event objects containing data digests, stage identifiers, operator signatures, and timestamps at each stage of circulation, and constructing a continuous event chain off-chain using hash pointers to achieve complete operation traceability, key information from each stage is extracted to generate compressed stage digest objects, which are then written to the blockchain, forming an immutable on-chain digest chain, significantly reducing storage overhead. During verification, the hash values ​​of the on-chain digests and the fully recovered event data from off-chain are compared, and the digital signature is verified to ensure data integrity and operational legitimacy. This invention achieves efficient and lightweight evidence preservation while also considering data privacy and traceability requirements, making it suitable for data circulation scenarios involving multiple parties. Attached Figure Description

[0015] Figure 1 The flowchart illustrates a data stream evidence storage and traceability method based on off-chain verifiable and on-chain verification mechanisms, as claimed in this embodiment of the invention. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0017] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0018] According to a first embodiment of the present invention, the present invention claims protection for a data stream evidence storage and traceability method based on off-chain verifiable and on-chain verification mechanisms, referring to... Figure 1 This includes the following steps: Step S1: In the multiple stages of data flow, the operation events of each stage are encapsulated into structured event objects, which include data digests, stage identifiers, operation subject signatures, and timestamps; Step S2: Link each structured event object to the event hash of the previous stage through the event hash to form an off-chain operation event chain; Step S3: For each stage, extract key information to generate a stage summary object. The key information includes event hash, stage number, previous stage hash, and operation subject signature. Step S4: Write each stage summary object into the blockchain to form an on-chain stage summary chain; Step S5: When it is necessary to verify a specific stage, obtain the stage digest object of that stage from the blockchain, and verify the hash pointer structure and stage order; Step S6: Recover the complete event data for this stage from off-chain, recalculate the hash value and compare it with the on-chain digest. If the hash values ​​match, verify the operator's signature to confirm the legitimacy of the operation.

[0019] In this embodiment, at each stage of data circulation (such as registration, application, development, transaction, and use), the system captures the operation events for that stage. Operation events include details of data operations (such as data access and modification). The system converts these operation events into structured event objects, which contain the following fields: a data digest (calculated by applying a hash algorithm to the stage data to ensure data integrity), a stage identifier (a string or enumerated value identifying the current stage, such as "registration stage"), an operation subject signature (digitally signing the data digest and stage identifier using the operation subject's private key), and a timestamp (obtained from a trusted time server, recording the exact time the event occurred). The structured event objects are stored in JSON or binary format.

[0020] An event hash is calculated for each structured event object (by applying a hash algorithm to the entire object). For the first stage event, the event hash serves as the starting point of the chain; for subsequent stage events, the system associates the event hash of the current event with the event hash of the previous stage event (e.g., incorporating the previous event hash as a field into the current event object). All event objects are stored in an off-chain storage system (such as a distributed file system or database), forming a chronologically ordered chain of operation events. The chained connection is achieved by storing the previous event hash in each event object, thus ensuring the continuity of the chain.

[0021] For each stage, the system extracts key information from the structured event object: event hash (the hash value of the current event), stage number (a numerical sequence representing the stage order, such as 1, 2, 3), previous stage hash (the event hash of the previous stage's event), and operation subject signature. This information is combined into a stage digest object (JSON or binary format). The stage digest object is designed as a minimal dataset to reduce on-chain storage burden.

[0022] The system submits each stage summary object to the blockchain network. The blockchain network uses a consensus mechanism to package the summary objects into blocks, forming a chain of stage summaries linked sequentially by stage. Each block contains multiple stage summary objects, but the stage summary chain establishes a logical order on the chain using stage numbers and the hash of the previous stage. Blockchain smart contracts may be used to handle the storage and retrieval of summary objects.

[0023] When a stage needs to be verified (e.g., during an audit), the system queries the blockchain for the stage's digest object. The system parses the digest object, checks its hash pointer structure (i.e., whether the hash of the previous stage matches the digest hash of the previous stage on the chain), and verifies whether the stage numbers are consecutive (e.g., stage n should be numbered n-1+1). If a discontinuity or hash mismatch is found, it is marked as an anomaly.

[0024] The complete structured event object is retrieved from the off-chain storage system based on the stage identifier and event hash. The system recalculates the data digest of the event object (using the same hash algorithm) and compares the calculated hash value with the event hash in the stage digest object, which serves as the on-chain digest. If they match, it indicates that the off-chain data has not been tampered with.

[0025] If the hashes match, the system verifies the operator's signature using the operator's public key. Signature verification involves decrypting the signature and comparing it with the hashes of the data digest and the stage identifier. If the signature is valid, the operation is deemed legitimate; otherwise, it is marked as invalid.

[0026] Furthermore, the data digest is calculated from the stage data using a hash algorithm.

[0027] In this embodiment, at each stage of data circulation, the system encapsulates each operation event as a structured object, including a data digest (calculated using a hash algorithm), a stage identifier, an operation subject signature, and a precise timestamp. Events at each stage are chained together with the event hash of the previous stage, forming a complete off-chain "operation event chain." Due to the tamper-resistant nature of the hash structure, this chain possesses strong integrity verification capabilities, enabling local verifiable traceability even when data is not on the chain.

[0028] Furthermore, in step S1, encapsulating the operation events of each stage into structured event objects also includes: Verify the digital certificate of the operator to ensure the validity of the signature, and generate the timestamp using a trusted time source; Generate a unique identifier for the event and associate it with a stage identifier for easy retrieval and tracking later.

[0029] In this embodiment, each stage extracts only the key information of the unique event in that stage (event hash, stage number, previous stage hash, operator signature, etc.), constructs a "stage digest object," and writes this object into the blockchain to form a sequentially connected "stage digest chain." Through stage indexes and forward hash pointers, the on-chain digest chain is ensured to have structural traceability and tamper-resistance, effectively reducing on-chain storage overhead and improving verification efficiency.

[0030] Furthermore, in step S2, when forming the chain of downstream operation events, the following is also included: Store structured event objects in a distributed file system and generate an access index for subsequent retrieval; Chained connections are achieved by hashing the data digest of the current event with the data digest of the previous event, ensuring the continuity and integrity of the chain.

[0031] In this embodiment, when it is necessary to verify an operation at a certain stage, the digest object of that stage is first obtained from the blockchain to verify the correctness of the hash pointer structure and stage order; then, the complete event data is restored from off-chain, the hash value is recalculated and compared with the on-chain digest; if they match, it means that the off-chain data has not been tampered with; then, the legitimacy of the operation subject is verified by combining the event signature, so as to realize the closed-loop confirmation of data authenticity, integrity and behavior attribution.

[0032] Furthermore, in step S3, when generating the stage summary object, the method further includes: Compressing critical information reduces on-chain storage space; compression methods include lossless encoding or field merging.

[0033] In this embodiment, during event encapsulation, at each stage of data circulation (registration, application, development, transaction, use, etc.), the unique event occurring in that stage is encapsulated as a structured data object. This object includes key information such as a data digest (calculated from the stage data using a hash algorithm), a stage identifier (clearly indicating the current circulation stage), an operator's signature (using digital signature technology to ensure the traceability of the responsible party), and a timestamp (recording the specific time the event occurred).

[0034] The operation chain is constructed by linking the hash of the current event with the hash of the previous event, thus gradually forming a verifiable "operation chain" off-chain. This operation chain fully reflects the flow path and operation details of data in each circulation stage. Due to the characteristics of hash values, if the data in any stage is tampered with, the hash value will change, which can be quickly detected.

[0035] Furthermore, in step S4, when writing to the blockchain, a smart contract is used to automatically verify the correctness of the format of the phase digest object, including checking the field integrity and hash value format.

[0036] In this embodiment, when generating the stage summary, for each stage of data circulation, only the hash digest of the core event of that stage and key information such as the stage index (used to identify the sequential position of the current stage in the entire circulation process), the previous stage hash (establishing the association with the previous stage on the chain), and the operator signature are extracted, and then integrated to generate the summary information of that stage.

[0037] When the stage digest chain is formed, the digest information generated in each stage is stored on the chain sequentially, forming a "stage digest chain" connected in the order of stages. Unlike traditional complex structures such as Merkle trees, this method directly simplifies the on-chain evidence storage process, reduces computation and storage costs, and at the same time, by using stage numbering and forward referencing mechanisms (implemented through stage indexes and previous stage hashes), ensures the verifiability and immutability of on-chain digests, laying the foundation for efficient and low-cost data ownership confirmation and auditing.

[0038] Furthermore, in step S5, when verifying the hash pointer structure, the following steps are also included: Verify the continuity of stage numbers to ensure the correct order, and check that the hash pointer points to the correct previous stage digest.

[0039] Furthermore, in step S6, when recovering complete event data from off-chain, the off-chain storage location is located by using the stage identifier and event hash, and the data is obtained using the query interface of the distributed file system.

[0040] Furthermore, in step S6, when verifying the signature of the operating entity, the validity period and issuing authority of the digital certificate are also checked to ensure that the signature is valid and trustworthy when the event occurs.

[0041] Furthermore, the method also includes step S7: if the verification fails, an audit log is generated and an alarm mechanism is triggered. The audit log records the specific reasons and time of the verification failure, and the alarm mechanism notifies the relevant responsible parties.

[0042] The system monitors operational events at each stage of data flow. For example, during the data registration stage, the operator uploads a data file via a web interface. Upon receiving the file, the system immediately calculates the data digest: it reads the file content, applies a hash algorithm (such as SHA-256) to process it block by block, and generates a fixed-length hash value. The stage identifier is predefined according to the business process, such as code 001 corresponding to "registration". The operator's signature process: the system prompts the operator to sign using a Hardware Security Module (HSM) or software key. The signature data includes a concatenated string of the data digest and stage identifier. The timestamp is obtained from an external trusted time service to ensure time authority. Unique identifiers are generated using a random number generator combined with the timestamp to ensure global uniqueness. Structured event objects are serialized into JSON format, including all fields, and stored in a temporary local cache.

[0043] Calculate the event hash for each event object: Normalize the JSON string and then apply a hash algorithm. For the first event, the event hash is stored as the root hash. For subsequent events, the system retrieves the event hash of the previous event and writes it to a field of the current event object. Then, the current event hash is recalculated. All event objects are uploaded to a distributed file system (such as IPFS) using command-line tools or an API, returning the content address (CID). The system updates the local relational database, recording the event hash, stage identifier, storage address, and timestamp. Periodic integrity checks: The system iterates through all events, verifying that each event matches the hash of the previous event, ensuring the chain remains intact.

[0044] When extracting key information from the event object, a parser is used to read the JSON fields. The event hash is directly copied. The stage number is obtained by mapping stage identifiers from the configuration file. The previous stage hash is obtained from the field of the event object. The operation body signature is copied. The digest object is constructed as a minimal JSON object. Compression process: The system uses a custom compressor to shorten field names and uses Base64 encoding for binary data (such as signatures) to reduce string length. After compression, the digest object size is reduced by more than 50%.

[0045] The system connects to blockchain nodes and prepares transaction data. The transaction includes a phase digest object as its payload. A smart contract is deployed on the blockchain; the contract code contains a verification function that executes automatically upon transaction submission. The verification function checks the digest object: ensuring all fields exist, the event hash is a 32-byte hexadecimal string, and the phase number is a positive integer. If verification fails, the transaction is rolled back. Successful transactions are packaged by miners, generating a transaction hash. The system listens for blockchain events, confirms the transaction on-chain, and records the transaction hash and block number for subsequent queries.

[0046] The verification request originates from the auditor. The system queries the phase summary object via the blockchain API, inputting the phase number or transaction hash. After obtaining the summary object, the system parses and extracts the fields. Then, it queries the previous phase summary object.

[0047] Compare the hash values ​​of the current and previous stage digests. Also, check if the stage numbers are consecutive: for example, stage 3 should be numbered 3, and the previous stage should be numbered 2. If the numbers are not consecutive or the hashes do not match, the system flags the verification as failed.

[0048] Using the stage identifier and event hash as the query key, after obtaining the storage address, the system calls the distributed file system's HTTP API to download the event object. After downloading, the system recalculates the data digest: it reads the data fields from the event object and applies the same hash algorithm. The calculated hash is compared with the hash value of the digest object. The comparison uses a string comparison function and is case-sensitive.

[0049] The system extracts the signature and certificate from the event object. Certificate parsing uses library functions to check the validity period: comparing the event timestamp with the certificate's notBefore and notAfter fields. Issuing authority verification: verifying the certificate signature chain up to the root CA certificate, then decrypting the signature using the public key to obtain the raw hash value. The system calculates the hash of the concatenated string of the data digest and stage identifier and compares it with the decrypted value. If they match, the signature is valid.

[0050] When verification fails, the system creates an audit log entry and writes it to a log file or database. The log entry includes detailed error information. The alert mechanism is configured to send an email: using the SMTP protocol, the email content is templated and includes the stage number, error reason, and time. After receiving the email, the administrator logs into the system to view the detailed logs.

[0051] This invention constructs an off-chain "event chain" to support complete traceability and process reconstruction; constructs an on-chain "stage summary chain" with summaries uploaded to the chain and a concise structure; implements a two-layer verification mechanism: hash comparison + signature verification, enabling attribution of operational behaviors; features strong module independence, low coupling, and adaptability to multiple scenarios; has a more systematic structural design: adopting a dual-chain architecture of "event chain + summary chain," it structurally and verifiably connects all data operation behaviors throughout the process, ensuring both privacy and efficiency; has a more comprehensive closed-loop verification mechanism: achieving complete, independent, and reliable data traceability through off-chain restoration + hash comparison + signature verification, unlike relying solely on smart contracts or summary comparison; and focuses on data circulation compliance auditing, especially suitable for scenarios requiring distributed traceability and accountability auditing such as government affairs, industrial internet, and data transactions.

[0052] The specific embodiments of the invention have been described in detail above, but they are only examples, and this application is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications or substitutions to the invention are also within the scope of this application. Therefore, all equivalent changes, modifications, and improvements made without departing from the spirit and principles of this application should be covered within the scope of this application.

Claims

1. A data stream evidence storage and traceability method based on off-chain verifiable and on-chain verification mechanisms, characterized in that, Includes the following steps: Step S1: In the multiple stages of data flow, the operation events of each stage are encapsulated into structured event objects, which include data digests, stage identifiers, operation subject signatures, and timestamps; Step S2: Link each structured event object to the event hash of the previous stage through the event hash to form an off-chain operation event chain; Step S3: For each stage, extract key information to generate a stage summary object. The key information includes event hash, stage number, previous stage hash, and operation subject signature. Step S4: Write each stage summary object into the blockchain to form an on-chain stage summary chain; Step S5: When it is necessary to verify a specific stage, obtain the stage digest object of that stage from the blockchain, and verify the hash pointer structure and stage order; Step S6: Recover the complete event data for this stage from off-chain, recalculate the hash value and compare it with the on-chain digest. If the hash values ​​match, verify the operator's signature to confirm the legitimacy of the operation.

2. The data stream evidence storage and traceability method based on off-chain verifiable and on-chain verification mechanisms as described in claim 1, characterized in that, The data digest is calculated from the stage data using a hash algorithm.

3. The data stream evidence storage and traceability method based on off-chain verifiable and on-chain verification mechanisms as described in claim 1, characterized in that, In step S1, encapsulating the operation events of each stage into structured event objects further includes: Verify the digital certificate of the operator to ensure the validity of the signature, and generate the timestamp using a trusted time source; Generate a unique identifier for the event and associate it with a stage identifier for easy retrieval and tracking later.

4. The data stream evidence storage and traceability method based on off-chain verifiable and on-chain verification mechanisms as described in claim 1, characterized in that, In step S2, when forming the chain of off-chain operation events, the following is also included: Store structured event objects in a distributed file system and generate an access index for subsequent retrieval; Chained connections are achieved by hashing the data digest of the current event with the data digest of the previous event, ensuring the continuity and integrity of the chain.

5. The data stream evidence storage and traceability method based on off-chain verifiable and on-chain verification mechanisms as described in claim 1, characterized in that, In step S3, when generating the stage summary object, the method further includes: Compressing critical information reduces on-chain storage space; compression methods include lossless encoding or field merging.

6. The data stream evidence storage and traceability method based on off-chain verifiable and on-chain verification mechanisms as described in claim 1, characterized in that, In step S4, when writing to the blockchain, a smart contract is used to automatically verify the format correctness of the phase digest object, including checking field integrity and hash value format.

7. The data stream evidence storage and traceability method based on off-chain verifiable and on-chain verification mechanisms as described in claim 1, characterized in that, In step S5, when verifying the hash pointer structure, the following is also included: Verify the continuity of stage numbers to ensure the correct order, and check that the hash pointer points to the correct previous stage digest.

8. The data stream evidence storage and traceability method based on off-chain verifiable and on-chain verification mechanisms as described in claim 1, characterized in that, In step S6, when recovering complete event data from off-chain, the off-chain storage location is located by using the stage identifier and event hash, and the data is obtained using the query interface of the distributed file system.

9. The data stream evidence storage and traceability method based on off-chain verifiable and on-chain verification mechanisms as described in claim 1, characterized in that, In step S6, when verifying the signature of the operating entity, the validity period and issuing authority of the digital certificate are also checked to ensure that the signature is valid and trustworthy when the event occurs.

10. The data stream evidence storage and traceability method based on off-chain verifiable and on-chain verification mechanisms as described in claim 1, characterized in that, It also includes step S7: If the verification fails, an audit log is generated and an alarm mechanism is triggered. The audit log records the specific reasons and time of the verification failure, and the alarm mechanism notifies the relevant responsible parties.

Images