Automatic data management system based on dynamic metadata and reinforcement learning

Abstract

The invention discloses an automatic data management system based on dynamic metadata and reinforcement learning, and relates to the technical field of data management. In order to solve the problems of metadata management staticization and management strategy stiffness in data management, the method comprises a dynamic metadata sensing layer used for capturing metadata updating information based on an Apache SeaTunnel tool; merkleTree is adopted to realize version storage of metadata, and a metadata version map and a data blood relationship map are generated; a streaming metadata collector is constructed in a Kafka and Spark Streaming integration mode, and the atlas is transmitted to a data governance strategy engine in real time; the intelligent strategy engine layer is used for realizing dynamic optimization of a data governance strategy combination based on a PPO algorithm in a process of executing a task by a data governance strategy engine; the automatic execution layer is used for realizing real-time data acquisition and storage through Kafka, and Airflow acquires data from the storage regularly and executes batch processing workflow; and dynamically optimizing data distribution based on LSH.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data governance, in particular to an automatic data governance system based on dynamic metadata and reinforcement learning. BACKGROUND

[0002] In the current big data era, the global data volume is growing explosively, and the massive heterogeneous data is not only the core asset of enterprises, but also needs to be mined for its value through effective and practical data governance means. At present, although various data governance tools and system products have emerged in the market, these solutions still have significant shortcomings in complex business scenarios and dynamic data environments, which are mainly reflected in the following two points:

[0003] 1. Static metadata management, insufficient governance timeliness and accuracy: Traditional solutions mostly use daily or periodic offline collection mode, and the metadata update cycle is long (such as some enterprise table structure changes need to be presented on the management platform after one day), which cannot capture the dynamic changes of key attributes such as data structure and blood relationship in real time.

[0004] 2. Governance strategy is rigid, lacking of self-adaptation and expandability: Traditional solutions take fixed rule engine as the core, which is essentially a "manually driven rule game", and is completely passive when facing data environment changes. SUMMARY

[0005] The present application provides an automatic data governance system based on dynamic metadata and reinforcement learning to overcome the limitations of metadata collection and execution in previous data governance processes.

[0006] The technical solution adopted by the automatic data governance system based on dynamic metadata and reinforcement learning of the present application to solve the above technical problems is as follows:

[0007] An automatic data governance system based on dynamic metadata and reinforcement learning, comprising:

[0008] A dynamic metadata perception layer, configured to: realize real-time collection and update of database table data based on Apache SeaTunnel tool, and synchronously capture associated metadata update information; realize versioned storage of metadata by using MerkleTree, and generate metadata version graph and data blood relationship graph based on the storage result; and build a streaming metadata collector by integrating Kafka and Spark Streaming, and real-time transfer the above metadata version graph and data blood relationship graph to a data governance strategy engine to support the development of data governance related work;

[0009] The intelligent strategy engine layer is configured to: in the process of executing tasks by the data governance strategy engine, realize dynamic optimization of data governance strategy combination based on a PPO algorithm; and meanwhile, adopt a blockchain smart contract to perform version management on the data governance strategy combination, so as to ensure the tamper-proof nature of version data.

[0010] The automatic execution layer is configured to: realize real-time data ingestion and storage through Kafka, and periodically acquire data from the storage and execute a batch processing workflow through Airflow; for sensitive data involved in the batch processing process, adopt an FPGA-optimized AES-256 encryption algorithm for security protection; and after executing the batch processing, based on an LSH adaptive data partitioning technology, dynamically optimize data distribution through similarity mapping of data features.

[0011] Optionally, the dynamic metadata awareness layer realizes real-time collection and update of database table data based on an Apache SeaTunnel tool, and synchronously captures associated metadata update information, and the specific implementation operations include: through a CDC technology, first synchronously collect historical full data in snapshot reading mode, and synchronously capture full metadata information of this stage; and then, in single-thread mode, listen to database logs such as MySQLBinlog to track incremental data, and in quasi-real-time, extract incremental data corresponding to DML operations, while capturing associated metadata incremental update information, and finally, distribute the data update information and the metadata update information to a downstream system together, to replace a traditional batch loading scheme.

[0012] Further optionally, the dynamic metadata awareness layer realizes versioned storage of metadata by adopting a MerkleTree, and generates a metadata version graph and a data lineage graph based on the storage result, and the specific implementation operations include:

[0013] (1.1) Metadata standardization processing: convert the collected metadata into structured data in the JSON Schema specification; for each metadata field, calculate its feature fingerprint by adopting a SHA256 encryption algorithm, and by means of a FPGA accelerator, accelerate the SHA256 algorithm in hardware to improve the calculation efficiency, while combing the lineage association relationship among the metadata, to lay a data foundation for subsequent graph generation;

[0014] (1.2) MerkleTree construction: adopt a balanced binary tree structure to construct a MerkleTree, and the generated leaf nodes correspond one-to-one to single metadata fields; the hash value stored in the leaf node is a result calculated by combining the feature fingerprint of the field and a version number through a sha3-256 encryption algorithm, while recording the version timestamp and the lineage association identifier of the field in the leaf node, and relying on the hierarchical relationship of the tree structure, preliminarily construct an association framework of metadata version and lineage;

[0015] (1.3) Version incremental storage and graph generation: when the metadata changes, first locate the leaf node corresponding to the changed field, generate a new version of the leaf node and retain the old version node, then recalculate the hash values of all nodes on the entire path from the leaf node to the root node, and complete the version update; based on the hash values of the full amount of MerkleTree nodes, version timestamps and blood relationship identifiers, integrate the change track and blood relationship information of each version node to generate visual metadata version graphs and data bloodline graphs.

[0016] Optionally, in the process of executing tasks by the data governance strategy engine, the intelligent strategy engine layer realizes dynamic optimization of data governance strategy combination based on PPO algorithm, and the specific optimization process is as follows:

[0017] (2.1) Define the mapping relationship between the three elements of reinforcement learning and the data governance scenario: the state is set as the metadata feature vector and the environmental index; the action is set as the data governance strategy combination; the reward is set as the multi-objective function, and the value is the weighted sum of the compliance score, cost saving, performance improvement and operation overhead;

[0018] (2.2) Define the initial value of the weight parameter contained in the reward function, and the specific calculation rules of the four-dimensional indexes of compliance score, cost saving, performance improvement and operation overhead;

[0019] (2.3) Build a PPO algorithm model framework, integrate the state defined in step (2.1), the action and the reward function defined in step (2.2) into the PPO algorithm, and build a policy network and a value network;

[0020] (2.4) When the data governance strategy engine executes tasks, the PPO algorithm model selects the corresponding data governance strategy combination and executes according to the current state and the policy network output;

[0021] (2.5) Collect the actual feedback data of compliance score, cost saving, performance improvement and operation overhead after the execution of the data governance strategy combination, calculate the reward value of this action according to the reward function formula, and record the complete interaction track of "current state - executed action - obtained reward - next state"; and based on the interaction track data, update the policy network parameters through the core logic of the clip strategy gradient of the PPO algorithm, so that the data governance strategy combination is optimized in the direction of maximizing the cumulative reward;

[0022] (2.6) Deploy the optimized data governance strategy combination to the data governance strategy engine, and the data governance strategy engine adopts the strategy combination in subsequent task execution; meanwhile, repeat steps (2.4)-(2.5) to continuously perceive state changes, collect new interaction data and optimize strategy combinations, and realize dynamic iteration.

[0023] Preferably, the calculation formula of the reward function involved is R = w1 * compliance score + w2 * cost savings + w3 * performance improvement - w4 * operation overhead; wherein w1, w2, w3, w4 are weight parameters greater than 0 and less than 1, and w1 + w2 + w3 + w4 = 1.

[0024] Further optionally, the intelligent strategy engine layer involved adopts a blockchain smart contract to perform version management on the data governance strategy combination, so as to ensure the tamper-proof nature of the version data; the specific design of this process is as follows:

[0025] The blockchain smart contract designs a strategy contract channel and an audit contract channel: the strategy contract channel only includes data governance participant nodes, and the audit contract channel realizes transparent supervision by accessing regulatory agency nodes;

[0026] A multi-layer verification mechanism combined with a tamper detection algorithm is adopted to double-protect the tamper-proof nature of the version data: (i) the multi-layer verification mechanism covers the data layer, the contract layer and the network layer: the data layer is based on the hash chain technology to ensure that historical data cannot be tampered with; the contract layer relies on the endorsement strategy to prevent unauthorized changes to the contract content; and the network layer resists malicious node access through TLS two-way authentication and node access control; (ii) the execution process of the tamper detection algorithm is as follows: first, extract the strategy record from the blockchain, verify the Merkle node information, check the continuity of the version hash, and then identify the data tampering behavior.

[0027] Optionally, the automatic execution layer involved adopts the FPGA-optimized AES-256 encryption algorithm to protect the sensitive data involved in the batch processing process, and the specific process is as follows:

[0028] First, input the plaintext data and perform the initial round key addition operation;

[0029] Secondly, perform 10 rounds of standardized encryption loop steps; each round of loop includes four core operations: byte substitution, row shift, column confusion and round key addition;

[0030] Finally, enter the final round processing stage, after completing all encryption operations, output the processed final data in the form of ciphertext, thus completing the entire sensitive data encryption protection process.

[0031] Optionally, after the automatic execution layer involved performs batch processing, based on the adaptive data partitioning technology of LSH (locality-sensitive hashing), the data distribution is dynamically optimized through the similarity mapping of data features, and the specific process is as follows:

[0032] (3.1) Feature vector extraction: For numerical data, logarithmic transformation is first performed to reduce the skewness of data distribution, and then standardization operation is performed to scale the data to a unified dimension range; for text data, the MinHash algorithm is used to generate data signatures to efficiently capture the core features of the text content; finally, all data is converted into 128-dimensional feature vectors to provide standardized input for subsequent similarity mapping;

[0033] (3.2) Constructing LSH hash function family: In order to realize efficient similarity mapping of high-dimensional feature vectors, an LSH hash function family is constructed based on the p-stable distribution of the Euclidean space, which can map high-dimensional feature vectors to low-dimensional hash space while preserving the similarity of data features, providing core algorithm support for subsequent dynamic partitioning;

[0034] (3.3) Dynamic partitioning decision: Considering the total size of the data itself, the current query load pressure of the system and the available storage capacity of each storage node, through real-time evaluation and dynamic trade-off of the above three factors, the optimal number of partitions and data allocation scheme are determined to ensure that the partitioning result can adapt to the system running state and data feature changes;

[0035] (3.4) Sharding storage: According to the allocation scheme obtained by dynamic partitioning decision, the data shards after feature extraction and hash mapping are stored in the corresponding storage nodes or storage areas; the data integrity and feature association of each shard are maintained during storage, laying a foundation for efficient execution of subsequent query operations;

[0036] (3.5) Query routing: When receiving a data query request, the feature vector of the query request is first extracted and the LSH value is calculated, and then according to the mapping relationship between the LSH value and the constructed LSH hash function family, the location of the target data is accurately located, and then the query request is routed to the corresponding storage node, realizing efficient data query response.

[0037] The automatic data governance system based on dynamic metadata and reinforcement learning has the beneficial effects compared with the prior art:

[0038] 1. The dynamic metadata awareness layer of this invention constructs a streaming metadata collector through the integration of Kafka and Spark Streaming. It transmits the metadata version graph and data lineage graph generated by the collaborative operation of Apache SeaTunnel and Merkle Tree to the data governance strategy engine in real time, supporting the implementation of data governance-related work. During the execution of tasks by the data governance strategy engine, the intelligent strategy engine layer realizes dynamic optimization of data governance strategy combination based on the PPO algorithm. This solves the problems of static metadata management in previous data governance, which could not reflect the real-time data status and the rigidity of governance strategies.

[0039] 2. In the process of the data governance strategy engine executing tasks, the intelligent strategy engine layer of this invention uses blockchain smart contracts (Hyperledger Fabric) to manage the version of the data governance strategy combination, thereby ensuring the immutability of the version data;

[0040] 3. The automated execution layer of this invention achieves real-time data ingestion and storage through Kafka, and periodically retrieves data from storage and executes batch processing workflows with the help of Airflow. For sensitive data involved in the batch processing, an FPGA-optimized AES-256 encryption algorithm is used for security protection. After the batch processing is executed, an adaptive data partitioning technology based on LSH (Local Sensitive Hash) is used to dynamically optimize the data distribution through similarity mapping of data features. This enables automated execution of data governance, data security protection during execution, and dynamic partitioning after execution. Attached Figure Description

[0041] Appendix Figure 1 This is a system architecture hierarchy diagram of the present invention;

[0042] Appendix Figure 2 This is a flowchart of the dynamic metadata perception layer of the present invention;

[0043] Appendix Figure 3 This is a flowchart illustrating the encryption of data using the FPGA-optimized AES-256 encryption algorithm of this invention;

[0044] Appendix Figure 4 This is a flowchart of the present invention based on LSH dynamic optimization of data distribution. Detailed Implementation

[0045] To make the technical solution, the technical problem solved, and the technical effect of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments.

[0046] Example 1:

[0047] Combined with appendixFigure 1 The embodiment provides an automatic data governance system based on dynamic metadata and reinforcement learning, which comprises the following:

[0048] The dynamic metadata perception layer is used for: based on an Apache SeaTunnel tool, real-time collection and update of database table data are realized, and associated metadata update information is synchronously captured; a MerkleTree is used to realize versioned storage of the metadata, and a metadata version graph and a data bloodline graph are generated based on the storage result; a stream metadata collector is constructed in a manner of integration of Kafka and Spark Streaming, and the metadata version graph and the data bloodline graph are transmitted to a data governance strategy engine in real time, so that data governance related work is supported.

[0049] Reference is made to the accompanying drawings Figure 2 The dynamic metadata perception layer is described in detail as follows.

[0050] (1A) The dynamic metadata perception layer realizes real-time collection and update of database table data based on an Apache SeaTunnel tool, and synchronously captures associated metadata update information, and the implementation process is as follows: through a CDC technology, historical full data is synchronously captured in a snapshot reading mode, and full metadata information of this stage is synchronously captured; then, a single thread is used to listen to a MySQL Binlog and other database logs to track incremental data, and incremental data corresponding to DML operations is extracted in quasi real time, and associated metadata incremental update information is synchronously captured, and finally, the data update information and the metadata update information are distributed to a downstream system together, so as to replace a traditional batch loading scheme.

[0051] (1B) The dynamic metadata perception layer realizes versioned storage of the metadata by using a MerkleTree, and generates a metadata version graph and a data bloodline graph based on the storage result, and the specific implementation operation comprises the following:

[0052] (1.1) Metadata standardization processing: the collected metadata (such as data table structure, field attribute, data bloodline relationship and the like) is converted into structured data of a JSON Schema specification; for each metadata field, a SHA256 encryption algorithm is used to calculate a feature fingerprint, and a FPGA accelerator is used to accelerate the SHA256 algorithm, so that the calculation efficiency is improved, and the bloodline association relationship among the metadata is combed, so as to lay a data foundation for subsequent graph generation;

[0053] (1.2) MerkleTree construction: a balanced binary tree structure is used to construct the MerkleTree, and the generated leaf nodes correspond one-to-one to individual metadata fields; the hash value stored in the leaf node is the result calculated by combining the feature fingerprint of the field and the version number through the sha3-256 encryption algorithm, and the version timestamp and blood relationship identifier of the field are also recorded in the leaf node, relying on the hierarchical relationship of the tree structure, an initial association framework of metadata version and blood relationship is constructed;

[0054] (1.3) Version increment storage and graph generation: when the metadata changes, first locate the leaf node corresponding to the changed field, generate a new version of the leaf node and retain the old version node, then recalculate the hash values of all nodes on the entire path from the leaf node to the root node, and complete the version update; based on the hash values of the full MerkleTree nodes, the version timestamp and the blood relationship identifier, the change track and blood relationship information of each version node are integrated to generate visual metadata version graphs (clearly showing the version iteration path, timeline and version association of each field) and data blood relationship graphs (presenting the blood relationship between fields, the influence of changes on blood links).

[0055] The intelligent strategy engine layer is used for: in the process of executing tasks by the data governance strategy engine, the dynamic optimization of the data governance strategy combination is realized based on the PPO algorithm; at the same time, the version management of the data governance strategy combination is realized by using the blockchain smart contract (Hyperledger Fabric), so as to ensure the tamper-proof nature of the version data.

[0056] The intelligent strategy engine layer is used for: in the process of executing tasks by the data governance strategy engine, the dynamic optimization of the data governance strategy combination is realized based on the PPO algorithm; at the same time, the version management of the data governance strategy combination is realized by using the blockchain smart contract (Hyperledger Fabric), so as to ensure the tamper-proof nature of the version data.

[0057] (2A) In the process of executing tasks by the data governance strategy engine, the intelligent strategy engine layer realizes the dynamic optimization of the data governance strategy combination based on the PPO algorithm, and the specific optimization process is as follows:

[0058] (2.1) Define the mapping relationship between the three elements of reinforcement learning and the data governance scenario: the state is set as the metadata feature vector (including field sensitivity, storage cost, access frequency, etc.) and the environmental index (including compliance risk score, computing resource margin, etc.); the action is set as the data governance strategy combination (including the selection of encryption algorithm, the adjustment of retention period, the change of partition strategy, the desensitization intensity, etc.); the reward is set as a multi-objective function, and its value is the weighted sum of compliance score, cost saving, performance improvement and operation overhead, and the calculation formula is R = w1 * compliance score + w2 * cost saving + w3 * performance improvement - w4 * operation overhead; wherein w1, w2, w3, w4 are weight parameters greater than 0 and less than 1, and w1 + w2 + w3 + w4 = 1;

[0059] (2.2) Explicitly reward function contains the initial value of the weight parameter, and the specific calculation rule of the four dimension indicators of compliance score, cost saving, performance improvement and operation overhead;

[0060] (2.3) Build PPO algorithm model framework, integrate the state defined in step (2.1), action and reward function defined in step (2.2) to PPO algorithm, build policy network (used to output policy combination decision) and value network (used to evaluate state value);

[0061] (2.4) When the data governance strategy engine executes the task, the PPO algorithm model selects the corresponding data governance strategy combination according to the current state and policy network output, and executes it;

[0062] (2.5) Collect the actual feedback data of compliance score, cost saving, performance improvement and operation overhead after the execution of data governance strategy combination, calculate the reward value of this action according to the reward function formula, record the complete interaction track of "current state-execute action-obtain reward-next state", and update the policy network parameters based on the interaction track data through the core logic of PPO algorithm such as clip policy gradient, so that the data governance strategy combination is optimized towards the direction of maximizing cumulative reward;

[0063] (2.6) Deploy the optimized data governance strategy combination to the data governance strategy engine, and the data governance strategy engine adopts this strategy combination in subsequent task execution; At the same time, repeat steps (2.4)-(2.5), continuously perceive state changes, collect new interaction data, and optimize strategy combination, to realize dynamic iteration.

[0064] (2B) The intelligent strategy engine layer uses blockchain smart contract (Hyperledger Fabric) to manage the version of data governance strategy combination, so as to ensure the tamper resistance of version data; The specific design of this process is as follows:

[0065] The blockchain smart contract designs strategy contract channel and audit contract channel: the strategy contract channel only includes data governance participant nodes, and the audit contract channel realizes transparent supervision by accessing regulatory agency nodes;

[0066] The multi-layer verification mechanism is combined with the tamper detection algorithm to double-protect the version data from tampering: (i) the multi-layer verification mechanism covers the data layer, the contract layer, and the network layer: the data layer is based on the hash chain technology to protect the historical data from being tampered with; the contract layer relies on the endorsement strategy to prevent unauthorized contract content changes; the network layer resists malicious node access through TLS two-way authentication and node access control; (ii) the execution process of the tamper detection algorithm is as follows: first, extract the policy record from the blockchain, verify the Merkle node information, check the continuity of the version hash, and then identify the data tampering behavior.

[0067] (Three) an automated execution layer for: real-time data ingestion and storage through Kafka, Airflow periodically obtains data from the storage and executes batch processing workflows; for sensitive data involved in the batch processing process, the FPGA-optimized AES-256 encryption algorithm is used for security protection; after executing the batch processing, the adaptive data partitioning technology based on LSH (Local Sensitivity Hashing) is used to dynamically optimize the data distribution through the similarity mapping of data features.

[0068] The automated execution layer is described in detail below.

[0069] (3A) The automated execution layer uses the FPGA-optimized AES-256 encryption algorithm to protect sensitive data involved in the batch processing process, as shown in FIG. 10, and the specific process is as follows: Figure 3

[0070] First, input the plaintext data and perform the initial round key addition operation;

[0071] Second, perform 10 rounds of standardized encryption loop steps; each round of loop includes four core operations: byte substitution (SubBytes), row shift (ShiftRows), column confusion (MixColumns), and round key addition (AddRoundKey);

[0072] Finally, enter the final round processing stage, and after completing all encryption operations, output the processed final data in the form of ciphertext, thus completing the entire sensitive data encryption protection process.

[0073] (3B) After the automated execution layer executes the batch processing, the adaptive data partitioning technology based on LSH (Local Sensitivity Hashing) is used to dynamically optimize the data distribution through the similarity mapping of data features, as shown in FIG. 11, and the specific process is as follows: Figure 4

[0074] ​​(3.1) Feature vector extraction: For numerical data, logarithmic transformation is performed to reduce the skewness of data distribution, and then standardization operation is performed to scale the data to a unified dimension range; for text data, the MinHash algorithm is used to generate data signatures to efficiently capture the core features of the text content; finally, all data is converted into 128-dimensional feature vectors to provide standardized input for subsequent similarity mapping;

[0075] (3.2) Constructing LSH hash function family: To realize efficient similarity mapping of high-dimensional feature vectors, an LSH hash function family is constructed based on the p-stable distribution of the Euclidean space, which can map high-dimensional feature vectors to low-dimensional hash space while preserving the similarity of data features, providing core algorithm support for subsequent dynamic partitioning;

[0076] (3.3) Dynamic partitioning decision: Considering the total size of the data itself, the current query load pressure of the system, and the available storage capacity of each storage node, the optimal number of partitions and data allocation scheme are determined through real-time evaluation and dynamic trade-off of the above three factors, ensuring that the partitioning result can adapt to the system running state and data feature changes;

[0077] (3.4) Sharding storage: According to the allocation scheme obtained by dynamic partitioning decision, the data shards after feature extraction and hash mapping are stored in the corresponding storage nodes or storage areas; the data integrity and feature association of each shard are maintained during storage, laying a foundation for efficient execution of subsequent query operations;

[0078] (3.5) Query routing: When receiving a data query request, the feature vector of the query request is first extracted and the LSH value is calculated, and then the LSH value is mapped to the LSH hash function family to accurately locate the shard location of the target data, and then the query request is routed to the corresponding storage node, realizing efficient data query response.

[0079] In summary, the automatic data governance system based on dynamic metadata and reinforcement learning can realize real-time collection of metadata, dynamic optimization of data governance strategy combination, automatic periodic execution of data governance, and dynamic partitioning storage after data governance execution, solving the defects of static metadata management in previous data governance that cannot reflect real-time data state and rigid governance strategy that cannot dynamically change.

[0080] The principles and implementation manners of the present application are described in detail by using the above specific examples, and the examples are only used to help understand the core technical content of the present application. Based on the above specific examples of the present application, any improvement and modification of the present application made by the person skilled in the art without departing from the principles of the present application shall fall within the patent protection scope of the present application.

Claims

1. An automated data governance system based on dynamic metadata and reinforcement learning, characterized in that, It includes: The dynamic metadata awareness layer is used to: realize the real-time collection and updating of database table data based on the Apache SeaTunnel tool, and synchronously capture the associated metadata update information; MerkleTree is used to implement versioned storage of metadata, and metadata version graphs and data lineage graphs are generated based on the storage results. A streaming metadata collector is built by integrating Kafka and Spark Streaming to transmit the above metadata version graphs and data lineage graphs to the data governance strategy engine in real time to support the implementation of data governance-related work. The intelligent strategy engine layer is used to: dynamically optimize the combination of data governance strategies based on the PPO algorithm during the execution of tasks by the data governance strategy engine; at the same time, it uses blockchain smart contracts to manage the version of the data governance strategy combination, thereby ensuring the immutability of version data. The automated execution layer is used for: real-time data ingestion and storage via Kafka; Airflow periodically retrieves data from storage and executes batch processing workflows; for sensitive data involved in batch processing, FPGA-optimized AES-256 encryption algorithm is used for security protection; after batch processing, adaptive data partitioning technology based on LSH dynamically optimizes data distribution through similarity mapping of data features.

2. The automated data governance system based on dynamic metadata and reinforcement learning according to claim 1, characterized in that, The dynamic metadata awareness layer, based on the Apache SeaTunnel tool, enables real-time collection and updating of database table data, and synchronously captures associated metadata update information. Specifically, the operation includes: using CDC technology, firstly, synchronizing historical full data in snapshot read mode, and synchronously capturing full metadata information for that stage; then, single-threaded monitoring of database logs to track incremental data, extracting incremental data corresponding to DML operations in near real-time, and simultaneously capturing associated metadata incremental update information; finally, distributing the data update information and metadata update information together to downstream systems, replacing the traditional batch loading solution.

3. The automated data governance system based on dynamic metadata and reinforcement learning according to claim 2, characterized in that, The dynamic metadata awareness layer uses MerkleTree to implement versioned storage of metadata, and generates metadata version graphs and data lineage graphs based on the storage results. Specific implementation operations include: (1.1) Metadata standardization processing: The collected metadata is converted into structured data in accordance with the JSON Schema specification; for each metadata field, the SHA256 encryption algorithm is used to calculate its feature fingerprint, and the SHA256 algorithm is accelerated by FPGA accelerator to improve the calculation efficiency. At the same time, the lineage relationship between metadata is sorted out to lay the data foundation for subsequent map generation. (1.2) MerkleTree Construction: A balanced binary tree structure is used to construct the MerkleTree. The generated leaf nodes correspond one-to-one with a single metadata field. The hash value stored in the leaf node is the result of combining the feature fingerprint of the field with the version number and then calculating it using the sha3-256 encryption algorithm. At the same time, the version timestamp and lineage association identifier of the field are recorded in the leaf node. Based on the hierarchical relationship of the tree structure, a preliminary framework for the association between metadata version and lineage is constructed. (1.3) Incremental version storage and graph generation: When metadata changes, the leaf node corresponding to the changed field is located first, a new version of the leaf node is generated and the old version node is retained. Then, the hash values ​​of all nodes on the entire path from the leaf node to the root node are recalculated to complete the version update. Based on the hash values, version timestamps and lineage association identifiers of all MerkleTree nodes, the change trajectory and lineage association information of each version node are integrated to generate a visualized metadata version graph and data lineage graph.

4. The automated data governance system based on dynamic metadata and reinforcement learning according to claim 1, characterized in that, During the execution of tasks by the data governance strategy engine, the intelligent strategy engine layer dynamically optimizes the combination of data governance strategies based on the PPO algorithm. The specific optimization process is as follows: (2.1) Define the mapping relationship between the three elements of reinforcement learning and the data governance scenario: the state is set as metadata feature vector and environmental indicators; the action is set as a combination of data governance strategies; the reward is set as a multi-objective function, the value of which is a weighted sum of compliance score, cost saving, performance improvement and operation overhead; (2.2) Clarify the initial values ​​of the weight parameters included in the reward function, as well as the specific calculation rules for the four dimensions of indicators: compliance score, cost savings, performance improvement, and operational overhead; (2.3) Construct the PPO algorithm model framework, integrate the state, action defined in step (2.1) and the reward function defined in step (2.2) into the PPO algorithm, and build the policy network and value network; (2.4) When the data governance strategy engine executes a task, the PPO algorithm model selects and executes the corresponding combination of data governance strategies based on the current state and the output of the strategy network. (2.5) Collect actual feedback data on compliance score, cost savings, performance improvement and operational overhead after the execution of the data governance strategy combination, calculate the reward value of this action according to the reward function calculation formula, and record the complete interaction trajectory of "current state - execution of action - obtaining reward - next state"; and based on this interaction trajectory, update the strategy network parameters through the core logic of the PPO algorithm to optimize the data governance strategy combination in the direction of maximizing cumulative reward. (2.6) Deploy the optimized data governance strategy combination to the data governance strategy engine. The data governance strategy engine will use the strategy combination in subsequent task execution. At the same time, repeat steps (2.4)-(2.5) to continuously sense state changes, collect new interaction data, and optimize the strategy combination to achieve dynamic iteration.

5. The automated data governance system based on dynamic metadata and reinforcement learning according to claim 4, characterized in that, The reward function is calculated as R = w1 * compliance score + w2 * cost savings + w3 * performance improvement - w4 * operation overhead; where w1, w2, w3, and w4 are all weight parameters greater than 0 and less than 1, and w1 + w2 + w3 + w4 = 1.

6. An automated data governance system based on dynamic metadata and reinforcement learning according to claim 4, characterized in that, The intelligent strategy engine layer employs blockchain smart contracts to manage the versioning of data governance strategy combinations, thereby ensuring the immutability of version data; the specific design of this process is as follows: Blockchain smart contract design strategy contract channel and audit contract channel: The strategy contract channel only includes nodes of data governance participants, while the audit contract channel achieves transparent supervision by connecting to regulatory agency nodes; A multi-layered verification mechanism combined with a tamper detection algorithm is adopted to provide dual protection for the immutability of version data: (i) The multi-layered verification mechanism covers the data layer, contract layer, and network layer: the data layer is based on hash chain technology to ensure that historical data cannot be tampered with; the contract layer relies on the endorsement policy to prevent unauthorized changes to the contract content; the network layer uses TLS two-way authentication and node access control to resist malicious node access; (ii) The execution process of the tamper detection algorithm is as follows: first, the policy record is extracted from the blockchain, and the continuity of the version hash is verified by verifying the Merkle node information, thereby identifying data tampering behavior.

7. The automated data governance system based on dynamic metadata and reinforcement learning according to claim 1, characterized in that, The automated execution layer employs an FPGA-optimized AES-256 encryption algorithm for security protection of sensitive data involved in batch processing. The specific process is as follows: First, input the plaintext data and perform the initial round key addition operation; Secondly, 10 standardized encryption cycles are executed; each cycle contains four core operations: byte substitution, row shifting, column obfuscation, and round key addition. Finally, the final processing stage begins. After all encryption operations are completed, the final processed data is output in ciphertext form, thus completing the entire encryption and protection process for sensitive data.

8. The automated data governance system based on dynamic metadata and reinforcement learning according to claim 1, characterized in that, After the automated execution layer performs batch processing, it uses LSH-based adaptive data partitioning technology to dynamically optimize data distribution through similarity mapping of data features. The specific process is as follows: (3.1) Feature vector extraction: For numerical data, logarithmic transformation is first performed to reduce the skewness of the data distribution, and then the data is scaled to a uniform dimension range through standardization. For text data, the MinHash algorithm is used to generate data signatures to efficiently capture the core features of the text content. Finally, all data are transformed into 128-dimensional feature vectors to provide standardized input for subsequent similarity mapping. (3.2) Constructing a family of LSH hash functions: In order to achieve efficient similarity mapping of high-dimensional feature vectors, a family of LSH hash functions is constructed based on the Euclidean space of the p-stable distribution. This family of functions can map high-dimensional feature vectors to low-dimensional hash space while preserving the similarity of data features, providing core algorithm support for subsequent dynamic sharding. (3.3) Dynamic sharding decision: Taking into account the total size of the data itself, the current query load pressure of the system and the available storage capacity of each storage node, the optimal number of shards and data allocation scheme are determined through real-time evaluation and dynamic trade-off of the above three factors, so as to ensure that the partitioning results can adapt to the system operating status and changes in data characteristics. (3.4) Sharded storage: Based on the allocation scheme derived from the dynamic sharding decision, the data after feature extraction and hash mapping is sharded and stored in the corresponding storage nodes or storage areas respectively; during the storage process, the data integrity and feature correlation of each shard are maintained, laying the foundation for the efficient execution of subsequent query operations; (3.5) Query routing: When a data query request is received, the feature vector of the query request is first extracted and its LSH value is calculated. Then, based on the mapping relationship between the LSH value and the constructed LSH hash function family, the shard location of the target data is accurately located, and the query request is routed to the corresponding storage node to achieve efficient data query response.

Images