Tamper-proof data bloodline recording method and device

Abstract

The application discloses an anti-tampering data bloodline recording method and equipment, belongs to the technical field of data bloodline recording, and is used for solving the technical problem that traditional data bloodline recording and anti-tampering methods cannot meet the increasingly complex data processing requirements in processing efficiency and anti-tampering ability. The method comprises the following steps: based on a dynamic syntax tree analysis engine, a target SQL statement is analyzed to obtain a dynamic syntax tree; according to node information of the dynamic syntax tree, a corresponding watermark identifier is generated, and the watermark identifier is bound with the node of the dynamic syntax tree; the dynamic syntax tree is subjected to sharding processing to obtain a syntax tree shard; the syntax tree shard is stored in a preset storage unit, and data bloodline relationships contained in the target SQL statement are stored; when the data bloodline relationships are queried in the preset storage unit, the queried data is subjected to watermark identifier verification to verify whether the data is tampered with.

Description

Technical Field

[0001] The present invention relates to the technical field of data lineage recording, and in particular to a tamper-proof data lineage recording method and device. Background Art

[0002] In the digital age, data has become a core asset for businesses and organizations, and the flow and processing of data is becoming increasingly complex. Data is extracted from its original source, transformed, loaded, and processed through a series of operations before ultimately reaching its target application or storage system. During this process, accurately recording data lineage and ensuring data integrity and tamper-proofing are crucial.

[0003] Traditional data lineage tracking methods, mostly based on SQL parsing, can only trace lineage at the field level and cannot track detailed transformation logic. In complex loading operations, especially when multiple functions are nested to process data or when data operations are performed through user-defined functions, nested queries and custom functions can complicate data processing logic. Traditional SQL parsing-based methods are unable to accurately analyze and record the data lineage relationships underlying this complex logic, significantly complicating subsequent data management and auditing, and significantly compromising the accuracy and integrity of data lineage.

[0004] Traditional methods for incremental processing typically require a complete reconstruction of the materialized view. When the data volume is large, this full reconstruction approach is not only time-consuming and computationally intensive, but also inefficient. This approach is insufficient for applications requiring high real-time performance, such as real-time financial auditing.

[0005] Traditional methods rely heavily on log auditing to prevent tampering. However, log information can be bypassed or tampered with, making it impossible to effectively ensure data tamper resistance.

[0006] To sum up, traditional data lineage recording and anti-tampering methods have exposed many shortcomings when faced with increasingly complex data processing needs. There is an urgent need for a more accurate, efficient and reliable technical solution to solve these problems. Summary of the Invention

[0007] Embodiments of the present invention provide a tamper-proof data lineage recording method and device for solving the following technical problem: traditional data lineage recording and tamper-proof methods are unable to meet the increasingly complex data processing requirements in terms of processing efficiency and tamper-proof capabilities.

[0008] The embodiment of the present invention adopts the following technical solutions:

[0009] In one aspect, an embodiment of the present invention provides a tamper-proof data lineage recording method, the method comprising:

[0010] Based on the dynamic syntax tree parsing engine, the target SQL statement is parsed to obtain a dynamic syntax tree;

[0011] Generate a corresponding watermark identifier according to the node information of the dynamic syntax tree, and bind the watermark identifier to the node of the dynamic syntax tree;

[0012] Segmenting the dynamic syntax tree to obtain syntax tree segments;

[0013] Storing the syntax tree fragments in a preset storage unit, and storing the data lineage relationship contained in the target SQL statement;

[0014] When querying the data lineage relationship in the preset storage unit, watermark verification is performed on the queried data to verify whether the data has been tampered with.

[0015] In a feasible implementation, based on a dynamic syntax tree parsing engine, the target SQL statement is parsed to obtain a dynamic syntax tree, specifically including:

[0016] The target SQL is parsed into a plurality of grammatical units by the dynamic grammar tree parsing engine; wherein the grammatical units include at least: aggregation operations, arithmetic operations, logical operations, and field references;

[0017] Each grammar unit is regarded as a node, and the nodes are sequentially connected according to the operation order of the grammar unit to form the dynamic grammar tree;

[0018] The syntax information of each syntax unit is obtained and associated with the corresponding node in the dynamic syntax tree as node information; wherein the syntax information at least includes a syntax type, an operation field, and an operand.

[0019] In a feasible implementation, generating a corresponding watermark identifier according to the node information of the dynamic syntax tree specifically includes:

[0020] According to the node information of the dynamic syntax tree, a matching watermark embedding algorithm and embedding parameters are selected in the watermark generation module; wherein the embedding parameters include at least an embedding position and an embedding strength;

[0021] Randomly selecting a data content from the node information and extracting key information from the data content;

[0022] The extracted key information is hashed using a hash function, and the resulting hash value is determined as a unique watermark identifier.

[0023] In a feasible implementation, binding the watermark identifier to the node of the dynamic syntax tree specifically includes:

[0024] Embedding the watermark identifier into a corresponding syntax tree node according to the watermark embedding algorithm and the embedding parameters;

[0025] At the same time, key information in the node information of the syntax tree node is stored in the watermark generation module and bound to the watermark identifier to implement a two-way binding mechanism between the watermark generation module and the dynamic syntax tree.

[0026] In a feasible implementation, fragmenting the dynamic syntax tree to obtain syntax tree fragments specifically includes:

[0027] Determining a corresponding sharding method based on the data volume, query frequency, business module, and node type of the dynamic syntax tree, including: sharding a dynamic syntax tree whose data volume exceeds a first preset threshold according to a preset number of nodes or a preset data volume; sharding nodes or subtrees whose query frequency exceeds a second preset threshold separately; storing dynamic syntax trees belonging to different business modules separately; and dividing nodes of the same type into the same shard;

[0028] Selecting the most matching fragmentation rule according to the characteristics of the dynamic syntax tree;

[0029] Based on the fragmentation method and fragmentation rules, the dynamic syntax tree is fragmented to obtain syntax tree fragments.

[0030] In a feasible implementation, the sharding rules include at least horizontal sharding, vertical sharding, and mixed sharding;

[0031] The horizontal sharding is divided according to the level or depth of the dynamic syntax tree;

[0032] The vertical sharding is divided according to the subtrees of the dynamic syntax tree;

[0033] The mixed sharding is a flexible division that combines horizontal sharding and vertical sharding.

[0034] In a feasible implementation, the syntax tree fragments are stored in a preset storage unit, and the data lineage relationship contained in the target SQL statement is stored, specifically including:

[0035] The preset storage unit includes a map database, a distributed storage database and a memory cache;

[0036] Storing the relationships between nodes of the dynamic syntax tree through the graph database;

[0037] Distributed storage of the syntax tree fragments and the data lineage relationships included in the target SQL statement through the distributed storage database;

[0038] The memory cache is used to store shards whose query frequency exceeds a second preset threshold.

[0039] In a feasible implementation manner, after storing the syntax tree fragments in a preset storage unit, the method further includes:

[0040] Add a node type label to each node, and locate and query specific type nodes based on the node type label based on Label Scan technology;

[0041] Create query indexes based on the relationships between nodes in the dynamic syntax tree to optimize query efficiency.

[0042] In a feasible implementation, when querying the data lineage relationship in the preset storage unit, watermark verification is performed on the queried data, specifically including:

[0043] Extracting watermark information of the data to be verified;

[0044] Querying the preset storage unit for required syntax tree node data;

[0045] Extracting the watermark identifier bound to the syntax tree node data at the corresponding embedding position and embedding depth according to the embedding parameters and decoding the watermark identifier;

[0046] Performing a two-way comparison between the decoded watermark identifier and the watermark information;

[0047] If the comparison results are consistent, it means that the data has not been tampered with. If the comparison results are inconsistent, an abnormal data risk warning will be triggered.

[0048] On the other hand, an embodiment of the present invention also provides an anti-tampering data lineage recording device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor so that the at least one processor can execute the anti-tampering data lineage recording method.

[0049] Compared with the prior art, the tamper-proof data lineage recording method and device provided by the embodiments of the present invention have the following beneficial effects:

[0050] The present invention brings about significant beneficial effects in many aspects by combining the dynamic syntax tree with the watermark technology.

[0051] In terms of data lineage records, first of all, the precise parsing capability has greatly improved the accuracy of data lineage relationships. When faced with complex SQL statements, traditional methods find it difficult to accurately sort out the source and conversion logic of the data, while the dynamic syntax tree parsing engine of the present invention can carefully decompose SQL statements into various types of syntax units and construct a dynamic syntax tree with a clear structure according to the order of operations. This allows the accurate recording of the origins of data, whether it is a simple data query or a complex multi-table association, multi-layer nested calculation, or other scenarios such as financial risk control systems. It provides a solid and reliable foundation for data tracing, auditing, and other tasks, allowing data managers to clearly understand the entire life cycle of the data and promptly discover potential problems in the data processing process, such as the compliance of the data source and the accuracy of data conversion.

[0052] Secondly, the sharded storage method of the dynamic syntax tree greatly optimizes storage and query efficiency. Reasonable sharding is performed based on factors such as data volume, query frequency, business module, and node type. Combined with horizontal, vertical, and hybrid sharding rules, it can flexibly adapt to data of varying sizes and characteristics. At the same time, different storage units, such as graph databases, distributed storage, and memory caches, are selected to fully leverage their respective strengths. Graph databases store the overall structure to facilitate graph traversal queries, distributed storage handles large-scale data, and memory caches accelerate high-frequency queries. This storage strategy not only effectively saves storage space but also enables rapid data location and retrieval based on diverse query requirements. For example, in high-frequency trading scenarios, rapid query responses to real-time data lineage relationships can help promptly detect transaction anomalies and ensure transaction security and stability.

[0053] In terms of watermark embedding and verification, the application of watermarking technology provides strong assurance for data integrity and tamper resistance. The bidirectional binding mechanism established between the watermark generation module and the syntax tree nodes ensures a close association between the watermark and the data source. By carefully selecting the embedding algorithm and parameters, such as using appropriate LSB or transform domain watermark embedding algorithms for different data types, and adjusting the embedding strength and position appropriately, the watermark is effectively hidden within the data without affecting its normal use, while also being highly robust and difficult to tamper with or remove. During the watermark verification phase, precise watermark extraction and decoding, coupled with a bidirectional comparison with the watermark identifier in the syntax tree node, quickly and accurately determine whether the data has been tampered with. Once a data anomaly is detected, a timely exception handling mechanism is triggered to effectively prevent the further dissemination and use of the tampered data, ensuring data authenticity and reliability. This is crucial in sectors with extremely high data security requirements, such as finance and healthcare, where it can effectively prevent risks such as data leakage and malicious tampering, protecting user privacy and the core interests of institutions.

[0054] Furthermore, the overall solution of the present invention possesses excellent versatility and scalability. The method based on dynamic syntax trees and watermarking technology is applicable to both relational database environments and other data processing scenarios. As data volumes continue to grow and business needs become increasingly complex, further optimization and expansion of sharding storage strategies and watermarking technology can continuously enhance the system's data processing capabilities and security, providing a scalable framework for the future development of data processing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments described in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0056] Figure 1 A flow chart of a tamper-proof data lineage recording method provided by an embodiment of the present invention;

[0057] Figure 2 An example diagram of a dynamic syntax tree construction provided by an embodiment of the present invention;

[0058] Figure 3 An example diagram of dynamic syntax tree fragment storage provided by an embodiment of the present invention;

[0059] Figure 4 A schematic structural diagram of a tamper-proof data lineage recording device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to enable those skilled in the art to better understand the technical solutions of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0061] The embodiment of the present invention provides a tamper-proof data lineage recording method, such as Figure 1 As shown, the tamper-proof data lineage recording method specifically includes steps S101-S104:

[0062] S101. Parse the target SQL statement based on a dynamic syntax tree parsing engine to obtain a dynamic syntax tree.

[0063] Specifically, the dynamic syntax tree parsing engine parses the target SQL into several syntax units, each of which includes at least aggregation operations, arithmetic operations, logical operations, and field references. Each syntax unit is then treated as a node, and the nodes are sequentially connected according to the order of the syntax unit operations to form a dynamic syntax tree.

[0064] Furthermore, the syntax information of each syntax unit is obtained and associated with the corresponding node in the dynamic syntax tree as node information; wherein the syntax information at least includes a syntax type, an operation field, and an operand.

[0065] As a feasible implementation method, when the system receives the target SQL statement, the dynamic syntax tree parsing engine starts to work. The engine performs lexical analysis and grammatical analysis on the target SQL statement according to predefined grammatical rules.

[0066] 1. Lexical analysis: This process divides the target SQL statement into individual word units based on character sequences, for example, identifying words such as SUM, amount, and *. These words can be preliminarily categorized into different grammatical unit types based on their function in the SQL grammar, such as aggregation operators, field references, and operators.

[0067] 2. Syntax analysis: According to SQL grammar rules, the word units obtained from lexical analysis are combined into grammatical units, and the hierarchical relationship and operation order between them are determined. Figure 2 An example diagram of a dynamic syntax tree construction provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, taking risk_score = SUM(amount) * 0.2 as an example, the parsing engine identifies SUM(amount) as an aggregation operation, a separate syntax unit, where SUM is the aggregation function and amount is the operation field. * 0.2 is a multiplication operation, also a syntax unit, whose operands are the result of the aggregation operation and 0.2. These syntax units are then constructed into a dynamic syntax tree according to the order of operations. The aggregation operation node records information such as the aggregation function type (SUM) and the operation field (amount), while the multiplication operation node records information such as the operand (the aggregation result and 0.2). In addition to recording its own relevant information, each node also establishes connections with other nodes, forming a complete dynamic syntax tree that clearly demonstrates the data processing logic and sequence in the SQL statement.

[0068] S102: Generate a corresponding watermark identifier according to the node information of the dynamic syntax tree, and bind the watermark identifier to the node of the dynamic syntax tree.

[0069] Specifically, according to the node information of the dynamic syntax tree, a matching watermark embedding algorithm and embedding parameters are selected in the watermark generation module; wherein the embedding parameters at least include embedding position and embedding strength.

[0070] Randomly select a data content from the node information and extract key information from the data content. Then, use the hash function to perform a hash operation on the extracted key information and determine the resulting hash value as a unique watermark identifier.

[0071] Furthermore, according to the watermark embedding algorithm and embedding parameters, the watermark identifier is embedded into the corresponding syntax tree node. At the same time, the key information in the node information of the syntax tree node is stored in the watermark generation module and bound to the watermark identifier to achieve a two-way binding mechanism between the watermark generation module and the dynamic syntax tree.

[0072] As a feasible implementation method, the specific process of watermark generation is as follows:

[0073] 1) Unique Identifier Determination: Unique identifiers are determined based on the specific business scenario. In financial transactions, a unique identifier can be generated by combining the transaction ID and bank code, for example, "BANK#TX123." This can be generated by applying a hash function to the transaction ID and bank code to produce a fixed-length unique identifier string.

[0074] 2) Embedding Algorithm Selection: Select an appropriate embedding algorithm based on the data type and sensitivity. For the amount field, due to the high accuracy requirements and moderate sensitivity, the least significant bit (LSB) algorithm is selected. This algorithm embeds the watermark information into the least significant bit of the data, minimizing the impact on the original value. For image data, due to the high visual quality requirements, a transform domain watermarking algorithm, such as the discrete wavelet transform (DWT), is selected. This embeds the watermark information into the image's wavelet transform coefficients, ensuring the watermark's invisibility and robustness.

[0075] 3) Embedding Parameter Determination: Embedding strength is adjusted for the LSB algorithm by selecting the number of bits to embed. For example, for less sensitive data, embedding 2-3 least significant bits (LSBs) is recommended, while for sensitive data, embedding 1 LSB is recommended to balance watermark visibility and robustness. Regarding embedding location, in transform domain watermark embedding, specific frequency domain coefficients are selected for embedding based on the data characteristics and watermark security requirements. For example, low-frequency coefficients in an image contain the primary information, so embedding the watermark there improves its robustness.

[0076] As a feasible implementation method, the establishment process of the two-way binding mechanism is as follows:

[0077] 1) Binding Process: When parsing SQL statements and constructing a dynamic syntax tree, a unique identifier is generated for each node. For example, this is generated by hashing the node's type, operation field, operand, and other information. This identifier is then associated with a watermark by adding a watermark identifier field to the node data structure and storing the generated watermark information there. Furthermore, the watermark generation module records the correspondence between the node identifier and the watermark information for subsequent query and verification.

[0078] 2) Unbinding: When data flows to the next stage or undergoes an audit, the watermark information needs to be unbound. The watermark information in the watermark identification field of the syntax tree node is extracted and compared with the watermark in the data. This comparison is performed using a specialized watermark verification algorithm. After verification, if the data has not been tampered with, the watermark information is unbound, which means that the watermark identification field in the node and the corresponding record in the watermark generation module are cleared.

[0079] S103: Slice the dynamic syntax tree to obtain syntax tree slices; store the syntax tree slices in a preset storage unit, and store the data lineage relationship included in the target SQL statement.

[0080] Specifically, the corresponding sharding method is determined according to the data volume, query frequency, business module and node type of the dynamic syntax tree, including: for dynamic syntax trees whose data volume exceeds a first preset threshold, sharding is performed according to the preset number of nodes or preset data volume; for nodes or subtrees whose query frequency exceeds a second preset threshold, separate sharding is performed; dynamic syntax trees belonging to different business modules are stored separately; and nodes of the same type are divided into the same shard.

[0081] Furthermore, according to the characteristics of the dynamic syntax tree, the most matching fragmentation rule is selected; based on the fragmentation method and the fragmentation rule, the dynamic syntax tree is fragmented to obtain syntax tree fragments.

[0082] The sharding rules include at least horizontal sharding, vertical sharding, and mixed sharding. Horizontal sharding is based on the level or depth of the dynamic syntax tree; vertical sharding is based on the subtrees of the dynamic syntax tree; and mixed sharding is a flexible combination of horizontal and vertical sharding.

[0083] As a feasible implementation method, the process of determining the sharding basis is as follows:

[0084] 1) Data volume considerations: If the number of syntax tree nodes is large or the corresponding data volume is large, the system will set a threshold for sharding based on the number of nodes or data volume. For example, if every 1,000 nodes is divided into one shard, sharding will be automatically performed when the number of syntax tree nodes exceeds this threshold.

[0085] 2) Query frequency analysis: Analyze historical query records to identify nodes or subtrees with high-frequency queries. For example, in financial data processing, if an aggregation operation node that calculates account balances is frequently queried, this node and its associated subtrees can be sharded separately.

[0086] 3) Business module division: Based on business logic, clearly define the syntax tree sections corresponding to different business modules. For example, in a banking system, the syntax trees for savings and credit services are stored separately to facilitate management and query by different business departments.

[0087] 4) Node type classification: Classify the nodes in the syntax tree by type, grouping nodes of the same type (such as aggregation nodes and operation nodes) together for sharded storage. This allows for more efficient location of specific type nodes during tag scanning.

[0088] The fragmentation rules are applied as follows:

[0089] 1) Horizontal Sharding: Sharding is performed based on the level or depth of the syntax tree. For example, for a syntax tree with a depth of 5, the root node and child nodes at levels 1-2 can be stored in one shard, child nodes at levels 3-4 can be stored in another shard, and child nodes at level 5 can be stored in a third shard. The benefit of this sharding is that when querying only for summary information at a high level of the syntax tree, the corresponding shard can be queried directly, improving query efficiency.

[0090] 2) Vertical Sharding: Sharding is based on the subtree structure. For example, for a syntax tree containing multiple aggregation operations, each aggregation operation and its associated child nodes are stored in a different shard. For example, to calculate the total transaction amount and total number of transactions, the aggregation operation for calculating the total transaction amount and its associated nodes can be stored in one shard, while the aggregation operation for calculating the total number of transactions and its associated nodes can be stored in another shard.

[0091] 3) Hybrid Sharding: Combining horizontal and vertical sharding rules, flexibly partitioning based on actual business needs. In some complex business scenarios, simple horizontal or vertical sharding cannot meet the storage and query optimization requirements. In these cases, hybrid sharding is used. For example, vertical sharding is first performed by business module, and then horizontal sharding is performed within each business module based on the node hierarchy.

[0092] Furthermore, the syntax tree fragments are stored in a preset storage unit, and the data lineage relationship contained in the target SQL statement is stored at the same time. The preset storage unit includes a graph database, a distributed storage database, and a memory cache.

[0093] Specifically, the relationship between nodes of the dynamic syntax tree is stored through a graph database; the syntax tree shards and the data lineage relationship contained in the target SQL statement are distributedly stored through a distributed storage database; and the shards whose query frequency exceeds a second preset threshold are stored through a memory cache.

[0094] Furthermore, a node type label is added to each node, and Label Scan technology is used to locate and query specific types of nodes based on the node type label. A query index is created based on the relationship between each node in the dynamic syntax tree to optimize query efficiency.

[0095] As a feasible implementation method, the process of selecting and configuring the storage unit is as follows:

[0096] 1) Graph databases (such as Neo4j): Used to store the overall structure of a syntax tree, including the nodes and their relationships. In Neo4j, each node in the syntax tree is created as a graph node, and the connections between nodes are created as graph edges. For example, an aggregation node and a field node are connected by an edge, and the edge attributes can record information such as operational relationships. Neo4j's powerful graph traversal and query capabilities enable efficient searches along the graph structure when querying data lineage relationships. Figure 3 An example diagram of a dynamic syntax tree fragment storage provided by an embodiment of the present invention is shown as follows: Figure 3 As shown, the syntax tree based on the business module and node type is stored in the graph database, and the index of the business module is constructed.

[0097] 2) Distributed storage database (such as HDFS or Cassandra): When the amount of syntax tree data is huge and large-scale data storage is required, a distributed storage system is selected. Taking HDFS as an example, the sharded syntax tree data is stored on multiple data nodes. The redundant storage and data block management mechanism of the distributed file system ensures high data availability and scalability. At the same time, the distributed computing framework of HDFS (such as MapReduce) can be used to perform large-scale parallel processing on the stored syntax tree data. Figure 3 As shown, after the dynamic syntax tree is sharded according to the data volume, shard 1 with a larger data volume is stored in the distributed storage database HDFS.

[0098] 3) Memory cache (such as Redis): For sharded data that is frequently queried, it is stored in Redis. Redis is based on memory storage, which makes the query speed extremely fast. For example, the aggregation operation node of high-frequency query is sharded and stored in Redis. When the relevant data needs to be queried, it can be directly obtained from the Redis cache, which greatly reduces the query response time. Figure 3As shown in Figure 2, the dynamic syntax tree is sharded based on query frequency, resulting in shard 2 being stored in the memory cache for easy querying. Shard 5, obtained through hybrid sharding, is stored collaboratively using both distributed storage and memory caching.

[0099] 4) Labeling and Management: Label each node. Label types are determined by the node's function and attributes, such as "aggregate node," "computation node," or "field node." During storage, label information is stored alongside the node data in the corresponding storage unit. For example, in Neo4j, label attributes are assigned to each node. Using Label Scan technology, when querying for specific node types, the system can quickly locate and filter nodes based on their labels. For example, when querying all aggregate nodes, the system scans for nodes labeled "aggregate node" to quickly retrieve relevant information, improving traversal efficiency.

[0100] As a feasible implementation method, a unique identifier (ID) is generated for each node. The ID can be generated by combining the key information of the node (such as node type, operation field, etc.) through a hash algorithm. When storing, the node ID is stored as an index field in the corresponding storage unit. For example, when storing node information in a relational database, the node ID is set as the primary key, so that when querying a specific node, it can be quickly located by the ID. Indexes are also created for the relationships between nodes. Taking Neo4j as an example, when creating edges between nodes, an index is created for the relationship type of the edge. In this way, when performing graph traversal queries, the system can quickly locate the edges that meet the conditions based on the relationship index, thereby efficiently traversing the syntax tree structure and improving query efficiency.

[0101] S104: When querying the data lineage relationship in the preset storage unit, perform watermark verification on the queried data to verify whether the data has been tampered with.

[0102] Specifically, the watermark information of the data to be verified is extracted; the required syntax tree node data is searched in a preset storage unit; according to the embedding parameters, the watermark identifier bound to the syntax tree node data is extracted and decoded at the corresponding embedding position and embedding depth; the decoded watermark identifier is bidirectionally compared with the watermark information; if the comparison results are consistent, it means that the data has not been tampered with; if the comparison results are inconsistent, an abnormal data risk prompt is triggered.

[0103] As a feasible implementation method, the specific process of watermark verification of data is as follows:

[0104] 1. Watermark extraction:

[0105] 1) Determine the watermark embedding location: Determine the hidden location of the watermark in the data based on the embedding algorithm and embedding location information recorded during the watermark embedding phase. For example, if the LSB algorithm is used to embed the watermark, the watermark is hidden in the lowest binary bit of the data; if the transform domain watermark embedding algorithm is used, the watermark is hidden in a specific transform domain coefficient.

[0106] 2) Applying the extraction algorithm: The watermark is extracted from the data using the same algorithm used in the watermark embedding phase. If the watermark is embedded into the image's DCT coefficients using the discrete cosine transform (DCT), the watermark is also transformed using the DCT algorithm during extraction, extracting the watermark from the transformed coefficients.

[0107] 3) Decoding the watermark: The extracted watermark is usually encoded or encrypted and needs to be restored to its original form using a decoding algorithm. For example, the watermark is encrypted and embedded using an encryption algorithm, and then, after extraction, a corresponding decryption algorithm is used to restore it to a readable watermark string.

[0108] 2. Two-way verification:

[0109] 1) Obtaining watermarks from syntax tree nodes: During the data lineage recording phase, watermarks are stored in the corresponding nodes of the dynamic syntax tree. During the verification phase, the system locates the node corresponding to the data to be verified by querying the syntax tree structure based on the data source and conversion logic, and then obtains the stored watermark from that node.

[0110] 2) Comparing the extracted watermark with the node identifier: After decoding, the extracted watermark information is compared character by character with the watermark identifier obtained from the syntax tree node. If the two are identical, the data has not been tampered with; if they are inconsistent, the data may have been tampered with. This comparison process can be implemented using a string comparison function, and the comparison results are recorded.

[0111] 3) Exception Handling: If verification fails (i.e., watermark inconsistency), the system triggers the exception handling mechanism. In financial compliance audit scenarios, the system generates detailed alert information, including the data source, the SQL statements involved, and the data fields that may have been tampered with, prompting auditors to review the relevant data. Furthermore, abnormal data can be isolated and stored to prevent further circulation in the system and causing further impact.

[0112] In addition, the embodiment of the present invention also provides a tamper-proof data lineage recording device, such as Figure 4 As shown, the tamper-proof data lineage recording device specifically includes:

[0113] at least one processor; and a memory communicatively connected to the at least one processor; wherein,

[0114] The memory stores instructions executable by at least one processor, so as to enable the at least one processor to perform:

[0115] Based on the dynamic syntax tree parsing engine, the target SQL statement is parsed to obtain a dynamic syntax tree;

[0116] Generate a corresponding watermark identifier according to the node information of the dynamic syntax tree, and bind the watermark identifier to the node of the dynamic syntax tree;

[0117] Segmenting the dynamic syntax tree to obtain syntax tree segments;

[0118] Storing the syntax tree fragments in a preset storage unit, and storing the data lineage relationship contained in the target SQL statement;

[0119] When querying the data lineage relationship in the preset storage unit, watermark verification is performed on the queried data to verify whether the data has been tampered with.

[0120] The various embodiments of the present invention are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the device, apparatus, and non-volatile computer storage medium embodiments are generally similar to the method embodiments, so their descriptions are simplified. For relevant details, refer to the descriptions of the method embodiments.

[0121] The above description of specific embodiments of the present invention is provided. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0122] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations may be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A tamper-proof data lineage recording method, characterized in that: The method comprises: Based on the dynamic syntax tree parsing engine, the target SQL statement is parsed to obtain a dynamic syntax tree; Generate a corresponding watermark identifier according to the node information of the dynamic syntax tree, and bind the watermark identifier to the node of the dynamic syntax tree; Segmenting the dynamic syntax tree to obtain syntax tree segments; Storing the syntax tree fragments in a preset storage unit, and storing the data lineage relationship contained in the target SQL statement; When querying the data lineage relationship in the preset storage unit, watermark verification is performed on the queried data to verify whether the data has been tampered with.

2. The tamper-proof data lineage recording method according to claim 1, characterized in that: Based on the dynamic syntax tree parsing engine, the target SQL statement is parsed to obtain a dynamic syntax tree, which specifically includes: The target SQL is parsed into a plurality of grammatical units by the dynamic grammar tree parsing engine; wherein the grammatical units include at least: aggregation operations, arithmetic operations, logical operations, and field references; Each grammar unit is regarded as a node, and the nodes are sequentially connected according to the operation order of the grammar unit to form the dynamic grammar tree; The syntax information of each syntax unit is obtained and associated with the corresponding node in the dynamic syntax tree as node information; wherein the syntax information at least includes a syntax type, an operation field, and an operand.

3. The tamper-proof data lineage recording method according to claim 1, characterized in that: Generate a corresponding watermark identifier according to the node information of the dynamic syntax tree, specifically including: According to the node information of the dynamic syntax tree, a matching watermark embedding algorithm and embedding parameters are selected in the watermark generation module; wherein the embedding parameters include at least an embedding position and an embedding strength; Randomly selecting a data content from the node information and extracting key information from the data content; The extracted key information is hashed using a hash function, and the resulting hash value is determined as a unique watermark identifier.

4. The tamper-proof data lineage recording method according to claim 3, characterized in that: Binding the watermark identifier to the node of the dynamic syntax tree specifically includes: Embedding the watermark identifier into a corresponding syntax tree node according to the watermark embedding algorithm and the embedding parameters; At the same time, key information in the node information of the syntax tree node is stored in the watermark generation module and bound to the watermark identifier to implement a two-way binding mechanism between the watermark generation module and the dynamic syntax tree.

5. The tamper-proof data lineage recording method according to claim 1, characterized in that: The dynamic syntax tree is fragmented to obtain syntax tree fragments, specifically including: Determining a corresponding sharding method based on the data volume, query frequency, business module, and node type of the dynamic syntax tree, including: sharding a dynamic syntax tree whose data volume exceeds a first preset threshold according to a preset number of nodes or a preset data volume; sharding nodes or subtrees whose query frequency exceeds a second preset threshold separately; storing dynamic syntax trees belonging to different business modules separately; and dividing nodes of the same type into the same shard; Selecting the most matching fragmentation rule according to the characteristics of the dynamic syntax tree; Based on the fragmentation method and fragmentation rules, the dynamic syntax tree is fragmented to obtain syntax tree fragments.

6. The tamper-proof data lineage recording method according to claim 5, characterized in that: The sharding rules include at least horizontal sharding, vertical sharding and mixed sharding; The horizontal sharding is divided according to the level or depth of the dynamic syntax tree; The vertical sharding is divided according to the subtrees of the dynamic syntax tree; The mixed sharding is a flexible division that combines horizontal sharding and vertical sharding.

7. The tamper-proof data lineage recording method according to claim 1, characterized in that: Storing the syntax tree fragments in a preset storage unit and storing the data lineage relationship contained in the target SQL statement specifically includes: The preset storage unit includes a map database, a distributed storage database and a memory cache; Storing the relationships between nodes of the dynamic syntax tree through the graph database; Distributed storage of the syntax tree fragments and the data lineage relationships included in the target SQL statement through the distributed storage database; The memory cache is used to store shards whose query frequency exceeds a second preset threshold.

8. The tamper-proof data lineage recording method according to claim 7, characterized in that: After storing the syntax tree fragments in a preset storage unit, the method further includes: Add a node type label to each node, and locate and query specific type nodes based on the node type label based on Label Scan technology; Create query indexes based on the relationships between nodes in the dynamic syntax tree to optimize query efficiency.

9. The tamper-proof data lineage recording method according to claim 1, characterized in that: When querying the data lineage relationship in the preset storage unit, watermark verification is performed on the queried data, specifically including: Extracting watermark information of the data to be verified; Querying the preset storage unit for required syntax tree node data; Extracting the watermark identifier bound to the syntax tree node data at the corresponding embedding position and embedding depth according to the embedding parameters and decoding the watermark identifier; Performing a two-way comparison between the decoded watermark identifier and the watermark information; If the comparison results are consistent, it means that the data has not been tampered with. If the comparison results are inconsistent, an abnormal data risk warning will be triggered.

10. A tamper-proof data lineage recording device, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, so that the at least one processor can execute the tamper-proof data lineage recording method according to any one of claims 1-9.

Images