Government affair data fusion and grading sharing method based on lake and warehouse integrated architecture

Abstract

The invention relates to a government affair data fusion and grading sharing method based on a lake-warehouse integrated framework, and belongs to the technical field of e-government affair and big data. The method comprises the following steps: carding government affair data sources and constructing a government affair data meta-model; a unified storage layer is constructed by adopting a lake-bin integrated framework, and data storage and treatment are carried out; a data fusion assembly line is constructed, and data access, cleaning, standardization, fusion, grading and desensitization processing are carried out; constructing a hierarchical sharing engine, and realizing dynamic authorization and credible sharing by adopting a mixed authorization model, a credible execution environment and block chain evidence storage; business arrangement and performance optimization are carried out by adopting a task arrangement and resource scheduling method, and hybrid task execution and confidential data security sharing are supported; evaluating the data quality and the sharing effect; and carrying out resource elastic expansion and disaster recovery processing. According to the method, the problems of multi-source heterogeneous government affair data, different standards, low quality and difficulty in grading and sharing are solved, and efficient fusion and secure sharing of the data are realized.

Description

Technical Field

[0001] This invention relates to a method for the fusion and hierarchical sharing of government data based on a lake-warehouse integrated architecture, belonging to the field of e-government and big data technology. Background Technology

[0002] 1. Existing technical solutions for the integration and hierarchical sharing of government data include: (1) Traditional data warehouse model: This model uses relational databases such as Oracle, SQL Server, and MySQL as its core, and ETL tools (Informatica, DataStage, Kettle, etc.) to extract, transform, and load data from various business systems into a centralized data warehouse. BI tools are then used for reporting and analysis. This model emphasizes strong data consistency, pre-modeling, and SQL-friendly operation, but it lacks support for semi-structured / unstructured data and has high expansion costs. It is typically used for internal decision analysis within relevant departments.

[0003] (2) Data Lake Model: Based on distributed object storage such as HDFS, S3, and OSS, data lakes preserve original formats (Excel, CSV, JSON, ORC, images, videos, logs, etc.) and perform batch, streaming, and interactive analysis using computing engines such as Spark, Flink, and Presto. Data lakes emphasize storage first, then modeling, making them suitable for exploring massive amounts of heterogeneous data. However, they lack transaction, schema evolution, and data quality management tools, and are prone to forming data swamps. Typical applications include the internet, finance, and telecommunications industries.

[0004] 2. Existing technical deficiencies in methods for integrating and sharing government data include: (1) Severe data silos: For business systems independently built by relevant functional departments, the lack of unified data models, metadata, and coding standards leads to siloed architectures and difficulty in data interoperability. For example, population databases and legal entity databases are maintained by different departments, with inconsistent primary key rules, a missing ID number rate of ≥3.7%, a missing unified social credit code rate of ≥5.1%, and the inability to directly link them.

[0005] (2) Data quality varies: The lack of automated data governance methods has resulted in prominent issues such as dirty, duplicate, and missing data. Taking medical insurance enrollment data as an example, duplicate enrollment records account for approximately 4.8%, and incorrect ID number formats account for approximately 1.9%, leading to difficulties in verifying and managing enrollment data.

[0006] (3) Difficulty in hierarchical sharing: Storing sensitive and non-sensitive data together fails to meet the requirements of the Data Security Law (2021), the Personal Information Protection Law (2021), and the Regulations on the Security Protection of Critical Information Infrastructure (2021) regarding the tiered and categorized opening of core data, important data, and general data. Traditional solutions either physically isolate all data or open it all at once, leading to the risk of data leakage.

[0007] (4) The contradiction between real-time performance and consistency: Data warehouses emphasize structure and strong consistency, while data lakes emphasize originality and flexibility; it's difficult to achieve both simultaneously. For example, after a company's registration information is changed in the provincial market supervision system, the traditional T+1 synchronization results in a 24-hour delay in the tax, social security, and housing provident fund systems obtaining the latest status, impacting the user experience of the "one-stop online service."

[0008] (5) Lack of unified bloodline and influence analysis: Business personnel cannot intuitively understand where a certain field comes from, who uses it, and which reports will be affected if it is changed, making changes difficult. Summary of the Invention

[0009] To address the aforementioned shortcomings, this invention provides a method for the fusion and hierarchical sharing of government data based on a lake-warehouse integrated architecture. This invention solves the problems of multi-source heterogeneity, inconsistent standards, low quality, and difficulty in hierarchical sharing of government data. This invention enables a single set of data to be shared by multiple computing loads, retaining the low-cost elasticity of a data lake while also possessing the high-performance transaction and metadata governance capabilities of a data warehouse.

[0010] The technical solution of this invention is: a method for the fusion and hierarchical sharing of government data based on a lake-warehouse integrated architecture, the method comprising: S1. Sorting out government data sources and constructing government data meta-models; S2. For government data, a unified storage layer is built using a lake-warehouse integrated architecture for data storage and governance; S3. For government data flows, construct a data fusion pipeline to perform data access, cleaning, standardization, fusion, classification and desensitization processing; S4. To address data sharing needs, a hierarchical sharing engine is constructed, employing a hybrid authorization model, a trusted execution environment, and blockchain notarization to achieve dynamic authorization and trusted sharing. S5. For data processing tasks, task orchestration and resource scheduling methods are used for business orchestration and performance optimization, supporting mixed task execution and secure sharing of confidential data. S6. Evaluate the data quality and sharing effectiveness; S7. Perform resource elastic scaling and disaster recovery processing.

[0011] Further, S1 includes: S11. Establish a five-level directory structure of department-system-table-field-record to form a unified government data metadata model (GDMM); S12. Use knowledge graph technology to extract business terms, coding rules, and semantic relationships to construct a government data knowledge graph (KGD). S13. Introduce a data DNA fingerprinting algorithm to generate a 128-bit unique identifier for each record, which is used for subsequent bloodline tracing.

[0012] Further, S2 includes: S21. The storage layer is constructed using a dual-table format of Iceberg and Hudi. Iceberg is an open-source data table format that supports ACID transactions, while Hudi is a table format that supports streaming updates. The combination of the two is used to balance transaction consistency and real-time updates. S22. Using Object Storage Service (OSS) as a unified storage platform, hot data is cached in Alluxio distributed cache; S23. The computing layer adopts a hybrid engine of batch processing SparkSQL, stream processing Flink, and interactive analysis Trino, and realizes the unification of the three computing modes of batch processing, stream processing, and interactive analysis through the same set of metadata services, that is, three computing in one. S24. Introducing mixed row and column storage, Z-Order sorting, and Bloom Filter indexes to improve query performance at high concurrency points.

[0013] Further, S3 includes: S31. Data Access: Utilize Change Data Capture (CDC) technology to capture the Binlog logs of the business database in real time, achieving data access with a minute-level latency. S32. Data Cleaning: The data cleaning module has built-in government data quality inspection rules, including checking the completeness of required fields, the legality of ID cards, the verification of unified social credit codes, and fuzzy matching of addresses. S33. Data Standardization: Automatic mapping of field synonyms, unit conversion, and dictionary unification based on government data knowledge graph (KGD); S34. Data Fusion: The Master Data Management (MDM) algorithm is used to uniquely identify and merge the four major master data categories: natural persons, legal persons, certificates, and matters. S35. Data Classification: Combining the "Guidelines for Classification and Grading of Government Data" (GB / T 35273-2025) with the department's custom sensitive word library, a Natural Language Processing (NLP) model and a rule engine are used to predict the sensitivity level. The NLP model is based on the BERT architecture, which is a pre-trained language model architecture based on a Transformer encoder, pre-trained with a bidirectional masked language model and next-sentence prediction, and can capture the bidirectional semantic dependencies of the text context. The rule engine is used to execute predefined sensitive word matching rules to classify the data into 5 levels: public, internal, restricted, confidential, and top secret. S36. Data anonymization: For fields classified as restricted, confidential, and top secret, the anonymization strategy dynamically selects masking, tokenization, homomorphic encryption, and differential privacy algorithms for data anonymization based on the classification strategy; the anonymization strategy is tied to the data lineage and supports the strategy following the data.

[0014] Further, S4 includes: S41. Dynamic authorization is performed based on a hybrid authorization model of ABAC and PBAC, where ABAC is attribute-based access control and PBAC is policy-based access control. Permission policies are dynamically generated by combining the four-tuple of subject attributes, object attributes, environment attributes and access purpose. S42. Introduce a Trusted Execution Environment (TEE) to store keys and policy indexes to prevent unauthorized access by internal personnel; S43 provides four types of interface gateways: SQL, REST, GraphQL, and ODBC, to support row-level, column-level, and cell-level permissions; S44. Built-in data uses the DUA template of the protocol. After the caller signs it online, a blockchain certificate is automatically generated to ensure traceability of use. S45. When the policy changes, the affected records are located in seconds based on the data DNA fingerprint, triggering logical reclamation or physical destruction. Logical reclamation refers to making the data inaccessible through version control or soft deletion, while physical destruction refers to permanently deleting the data from storage.

[0015] Further, S5 includes: S51. Implement Directed Acyclic Graph (DAG) task orchestration using Airflow and Kubernetes. Airflow is a workflow scheduling platform, and Kubernetes is a container orchestration platform. The combination of the two supports one-click deployment of batch processing, stream processing, and interactive tasks, allowing users to trigger the execution of mixed tasks with a single operation. S52. Based on historical task profiles, use deep reinforcement learning (DQN) to automatically select the computing engine, concurrency, and resource specifications. S53. For high-frequency shared interfaces, materialized views and result caching mechanisms are provided. Materialized views are pre-computed views, and result caching is used to store query results to improve query performance. S54. For confidential data, a federated learning sharing scheme with "data stationary, model dynamic" is adopted. Here, "data stationary, model dynamic" means that the original data is kept locally, and only the model parameters are aggregated and shared through the Trusted Execution Environment (TEE) to ensure that the original data does not leave the domain.

[0016] Further, S6 includes: S61. Establish a five-dimensional evaluation index system that includes completeness, consistency, timeliness, accessibility, and security; S62. Data sharing KPI scores are automatically generated monthly at the departmental level and incorporated into the digital government assessment. S63. Introduce a data asset depreciation model to automatically archive or destroy data that has not been accessed for a long time.

[0017] Further, S7 includes: S71. The storage layer deploys three replicas across availability zones to achieve RPO=0 and RTO<30s. Here, RPO is the recovery point objective, RPO=0 means zero data loss, and RTO is the recovery time objective, RTO<30s means the system recovery time is less than 30 seconds. S72. The compute layer implements elastic scaling based on Spot instances and HPA. Spot instances are low-cost cloud computing instances, while HPA is horizontal Pod auto-scaling, which is used to automatically adjust resources according to the load to achieve peak shaving and valley filling. S73. The metadata service uses the Raft protocol to achieve high availability of three nodes, automatically elect a master, and prevent single points of failure. The Raft protocol is a distributed system consensus algorithm.

[0018] The present invention also provides a government data fusion and hierarchical sharing system based on a lake-warehouse integrated architecture, the system comprising: a module for executing the government data fusion and hierarchical sharing method based on the lake-warehouse integrated architecture.

[0019] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for government data fusion and hierarchical sharing based on the lake-warehouse integrated architecture.

[0020] The beneficial effects of this invention are: 1. The primary objective of this invention is to address the problems of multi-source heterogeneity, inconsistent standards, low quality, and difficulty in hierarchical sharing of government data. It breaks away from the inherent problems of data redundancy, inconsistent definitions, and time delays caused by the traditional "lake-warehouse separation" architecture. By unifying and aggregating structured and unstructured data scattered across hundreds of heterogeneous systems in the government sector into the Iceberg+Hudi lake-warehouse integrated platform, it enables the sharing of multiple computing loads with a single data source. This retains the low-cost elasticity of a data lake while also possessing the high-performance transaction and metadata governance capabilities of a data warehouse. As a result, the previous T+1 batch sharing mode is compressed to the minute or even second level, supporting the implementation of real-time supervision and instant approval scenarios. 2. Addressing the pain points of high security classification requirements, diverse sharing needs, and complex stakeholders in government data, this invention constructs a closed-loop governance system on a unified data foundation, encompassing "automatic sensitive identification—dynamic desensitization—fine-grained authorization—blockchain evidence storage—logical retrieval": Utilizing an NLP model, each record is tagged in real-time with five levels of labels: public, internal, restricted, confidential, and top secret. Row-level and cell-level permission policies are dynamically generated based on stakeholders' attributes, access purposes, and environmental context. A trusted execution environment and consortium blockchain ensure the immutability of keys and audit logs, truly achieving "minimum availability, dynamic desensitization, and full traceability," maximizing data value release while maintaining compliance. 3. This invention uses technological means to assist and promote management innovation. Through tools such as data DNA fingerprinting, knowledge graphs, reinforcement learning scheduling, and federated learning, the effectiveness of data sharing is quantified into five-dimensional KPIs: "completeness, consistency, timeliness, accessibility, and security." These KPIs are directly incorporated into the performance evaluation of digital government, forming a virtuous cycle where "data providers are willing to share, users dare to share, and regulators are confident in sharing." This helps support high-frequency scenarios such as "one-stop birth service" and "one-stop online business deregistration," which are cross-departmental, cross-level, and cross-regional. Within three years, it helps achieve the comprehensive goals of saving more than 40% in provincial government cloud hardware investment, shortening the business launch cycle by more than 50%, and reducing the time for citizens to handle affairs by more than 60%. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the steps in this invention; Figure 2 This is a business logic diagram in the present invention; Figure 3 This is a planning diagram of the integrated lake and warehouse in this invention. Detailed Implementation

[0022] Example 1: The implementation environment of this invention includes: a government cloud (200 Kunpeng 920 ARM servers, 512 TB memory, 10 PB OSS); software versions: Iceberg 1.4.2, Flink 1.17, Trino 426, Kubernetes 1.29; data scale: 120 million legal entity records, 340 million administrative license records, 80 million penalty information records, with a daily increase of 280 GB; Figure 1 This invention relates to a method for the fusion and hierarchical sharing of government data based on a lake-warehouse integrated architecture, the method comprising: S1. Organizing government data sources and constructing a government data meta-model; this stage focuses on resolving the problems of government data being unavailable, incomprehensible, and unmanageable; S1 includes: S11. Establish a five-level directory structure of department-system-database-field-record to form a unified government data meta-model (GDMM); including: establishing the full-link lineage of "department-system-database-field-record"; and assigning business and technical tags to each data element to achieve accurate positioning and impact analysis of data assets. S12. Employ knowledge graph technology to extract business terms, coding rules, and semantic relationships to construct a government data knowledge graph (KGD); including: Deploy lightweight scanning probes (DaemonSet mode) within each government cloud VPC. Employing multi-protocol identification technology, it automatically discovers database instances such as MySQL, Oracle, and PostgreSQL; By implementing configuration-based management, performance impacts on the production system can be avoided. S13. Introduce a data DNA fingerprinting algorithm (which can use MD5 + sensitive word vector identification) to generate a 128-bit unique identifier for each record for subsequent kinship tracing; including: It integrates MD5 summaries and BERT semantic vectors to generate composite fingerprints; supports record-level precision in lineage tracing and change impact analysis; and provides a unique identification basis for subsequent data governance.

[0023] S2. For government data, a unified storage layer is constructed using a lake-warehouse integrated architecture for data storage and governance, achieving unified data management and high-performance access. The core of this stage is to build a unified data foundation for one set of data and multiple computing methods. S2 includes: S21. The storage layer is constructed using a dual-table format of Iceberg and Hudi. Iceberg is an open-source data table format that supports ACID transactions, while Hudi is a table format that supports streaming updates. The combination of the two is used to balance transaction consistency and real-time updates. In the collaborative architecture of Iceberg and Hudi dual-table formats of the present invention, the Iceberg table format guarantees ACID transactions and supports batch processing scenarios; the Hudi table format supports streaming updates and meets the needs of real-time data access; the two table formats share the same set of metadata to avoid data redundancy. S22. Using Object Storage Service (OSS) as a unified storage platform, hot data is cached in Alluxio distributed cache; This invention employs a three-level caching acceleration mechanism: Hot data: Alluxio memory-level cache, response time <100ms; Warm data: SSD local cache, response time <1s; Cold data: OSS object storage, reducing costs by 70%; S23. The computing layer adopts a hybrid engine of batch processing SparkSQL, stream processing Flink, and interactive analysis Trino, and realizes the unification of the three computing modes of batch processing, stream processing, and interactive analysis through the same set of metadata services, that is, three computing in one. Among them, batch processing (SparkSQL), stream processing (Flink), and interactive analysis (Trino) have unified metadata; cross-engine data sharing is supported, avoiding data migration and format conversion; and a unified SQL interface is provided to lower the barrier to entry. S24. Introducing mixed row and column storage, Z-Order sorting, and Bloom Filter indexes to improve query performance at high concurrency points by 3-5 times.

[0024] S3. For government data flows, construct a data fusion pipeline to perform data access, cleaning, standardization, fusion, classification, and de-identification processing to improve data quality and protect sensitive data; this stage realizes the transformation of data from its raw state to a trustworthy and usable state; S3 includes: S31. Data Access: Utilize Change-Data-Capture (CDC) technology to capture the Binlog logs of the business database in real time, achieving data access with minute-level latency; this step includes using Debezium to implement non-intrusive access to the business system. S32. Data Cleaning: The data cleaning module has 200+ built-in government data quality inspection rules, supports custom expansion, and provides real-time data quality monitoring and anomaly alerts; including checking the completeness of required fields, the legality of ID cards, the unified social credit code, and address fuzzy matching; S33. Data Standardization: Automatic mapping of field synonyms, unit conversion, and dictionary unification based on government data knowledge graph KGD; this step includes intelligent standardization driven by knowledge graph, including: Using government knowledge graph KGD to achieve semantic mapping of fields; Automatically identifies synonyms, unit differences, and encoding inconsistencies; Supports batch standardization of historical data and real-time data stream standardization; S34. Data Fusion: The Master Data Management (MDM) algorithm is used to uniquely identify and merge the four major master data categories: natural persons, legal persons, certificates, and matters. S35. Data Classification: Combining the "Guidelines for Classification and Grading of Government Data" (GB / T 35273-2025) with departmental-defined sensitive word libraries, a Natural Language Processing (NLP) model and a rule engine are used to predict the sensitivity level. The NLP model is based on the BERT architecture, a pre-trained language model architecture that captures bidirectional semantic dependencies in text context through a Transformer encoder, bidirectional masked language model, and next-sentence prediction pre-training. The rule engine is used to execute predefined sensitive word matching rules to classify data into five levels: public, internal, restricted, confidential, and top secret. The NLP model (BERT) identifies semantic sensitivity, and the rule engine matches policy and regulatory requirements. S36. Data Desensitization: For fields classified as restricted, confidential, and top secret, masking, tokenization, homomorphic encryption, and differential privacy algorithms are dynamically selected based on the classification strategy for data desensitization; the desensitization strategy is tied to the data lineage, supporting the strategy to follow the data; the desensitization algorithm is dynamically selected to balance security and effectiveness.

[0025] S4. To address data sharing needs, a tiered sharing engine is constructed, employing a hybrid authorization model, a trusted execution environment, and blockchain notarization to achieve dynamic authorization and trusted sharing. This stage addresses the core requirements of secure data sharing and compliant use. S4 includes: S41. Dynamic authorization is performed based on a hybrid authorization model of ABAC (Attribute-Based Access Control) and PBAC (Purpose-Based Access Control). ABAC is attribute-based access control, while PBAC is policy-based access control. It dynamically generates permission policies by combining a four-tuple of subject attributes (who accesses), object attributes (what is accessed), context attributes (under what circumstances), and access purpose (why access is made). ABAC provides fine-grained control, while PBAC ensures policy consistency, enabling real-time permission calculation and dynamic adjustment. S42. Introduce a Trusted Execution Environment (TEE) to store keys and policy indexes to prevent unauthorized access by internal personnel; S43 provides four types of interface gateways: SQL, REST, GraphQL, and ODBC, to support row-level, column-level, and cell-level permissions; S44. Built-in data uses the DUA template of the protocol. After the caller signs it online, a blockchain certificate is automatically generated (Fabric consortium chain) to ensure traceability of use. This invention employs a dual-insurance approach using a Trusted Execution Environment (TEE) and blockchain. The TEE protects core keys and policies, while blockchain evidence storage ensures that operations are tamper-proof and fully traceable, supporting post-event auditing and liability determination. S45. When the policy changes, affected records are located within seconds based on data DNA fingerprinting, triggering logical reclamation or physical destruction. Logical reclamation refers to making the data inaccessible through version control or soft deletion, while physical destruction refers to permanently deleting the data from storage. This invention rapidly locates affected data based on data DNA fingerprinting; version-based management supports data recovery and backtracking, meeting data compliance requirements. S5. For data processing tasks, task orchestration and resource scheduling methods are used for business orchestration and performance optimization, supporting mixed task execution and secure sharing of confidential data; this stage improves the overall system performance and user experience; S5 includes: S51. Implement Directed Acyclic Graph (DAG) task orchestration using Airflow and Kubernetes. Airflow is a workflow scheduling platform that enables visual orchestration of complex workflows. Kubernetes is a container orchestration platform that provides elastic resource guarantees. The combination of the two supports one-click deployment of batch processing, stream processing and interactive tasks, allowing users to trigger the execution of mixed tasks with a single operation; S52. Based on historical task profiles, deep reinforcement learning (DQN) is used to automatically select computing engines, concurrency, and resource specifications, saving an average of 35% of cloud resources. Specifically, this invention predicts resource requirements based on historical task profiles, and the DQN algorithm automatically optimizes task scheduling strategies to achieve a balance between resource utilization and execution efficiency. S53. For high-frequency shared interfaces, materialized views and result caching mechanisms are provided. Materialized views are pre-computed views, and result caching is used to store query results to improve query performance, increasing QPS by 8-10 times. S54. For confidential data, a federated learning sharing scheme with "data-stationary, model-dynamic" is adopted. "Data-stationary, model-dynamic" means that the original data is kept locally, and only model parameters are aggregated and shared through a Trusted Execution Environment (TEE), ensuring that the original data does not leave the domain. This invention's "data-stationary, model-dynamic" approach protects the privacy of the original data, while the TEE ensures the secure aggregation of model parameters, supporting cross-departmental collaborative modeling without exposing the original data. S6. Evaluate data quality and sharing effectiveness; this stage establishes a closed-loop feedback mechanism for data governance; S6 includes: S61. Establish a five-dimensional evaluation index system that includes completeness, consistency, timeliness, accessibility, and security; S62. Data sharing KPI scores are automatically generated monthly at the departmental level and incorporated into the digital government assessment. This invention quantifies the value and effectiveness of data sharing, providing data support for performance evaluation. S63. Introduce a data asset depreciation model to automatically archive or destroy data that has not been accessed for a long time.

[0026] This data asset depreciation model automatically adjusts storage strategies based on data usage frequency and value changes, optimizes storage costs, improves resource utilization efficiency, and supports data lifecycle management; S7. Perform resource elastic scaling and disaster recovery. This stage ensures the stable and reliable operation of the system. S7 includes: S71. The storage layer deploys three replicas across availability zones to achieve RPO=0 and RTO<30s. Here, RPO is the recovery point objective, and RPO=0 means that the amount of data loss is zero. RTO is the recovery time objective, and RTO<30s means that the system recovery time is less than 30 seconds. The cross-availability zone disaster recovery architecture of this invention ensures that the data is guaranteed by three replicas (RPO=0). Rapid failover is achieved with an RTO of <30s; business continuity reaches 99.99%. S72. The computing layer achieves elastic scaling based on Spot instances and HPA. Spot instances are low-cost cloud computing instances, and HPA is a horizontal Pod auto-scaling mechanism used to automatically adjust resources according to the load to achieve peak shaving and valley filling. The cost-optimized elastic scaling of this invention reduces the computing cost of Spot instances by more than 60%, and HPA automatically adjusts the resource scale according to the load to achieve the best balance between performance and cost. S73. The metadata service uses the Raft protocol to achieve high availability of three nodes, automatically elect a master, and prevent single points of failure. The Raft protocol is a distributed system consensus algorithm.

[0027] The government data resource center constructed using the lake-warehouse integrated architecture-based government data fusion and hierarchical sharing method of this invention achieves the following benefits: (1) Governance efficiency: Through the lake warehouse integrated architecture, 200+ heterogeneous government databases, 24,000 tables, and 3PB of data are unified into a single logical entity, reducing redundant copies by 70%; (2) Shared timeliness: CDC + Flink stream processing reduces end-to-end latency from T+1 to minutes; (3) Security and compliance: The accuracy rate of hierarchical de-identification is 99.2%, the abnormal access blocking rate is 100%, and it has passed the big data security assessment of the Cyberspace Administration of China; (4) Economic benefits achieved through assistance: saving 22 million yuan in hardware investment over three years and reducing maintenance manpower by 45%; (5) Social benefits achieved: It supports more than 80 provincial cross-departmental business scenarios, including "one-stop service for birth" and "one-stop online service for enterprise deregistration", reducing the average processing time by 62%.

[0028] Example 2: The present invention also provides a government data fusion and hierarchical sharing system based on a lake-warehouse integrated architecture, the system comprising: a module for executing the government data fusion and hierarchical sharing method based on the lake-warehouse integrated architecture.

[0029] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for government data fusion and hierarchical sharing based on the lake-warehouse integrated architecture.

[0030] The business logic of this invention is as follows: Figure 2 As shown, the business architecture describes the relationships between the various participants in the government data sharing and exchange business, as well as the relationships between each participant and the public data platform.

[0031] Data Management Provider: The data management provider of the public data platform is responsible for overall planning, organization and coordination, standardization and guidance, supervision and management, performance evaluation and management, quality management, system construction and mechanism improvement, etc. It conducts systematic control over the public data platform and supervises the construction effectiveness and service quality of the construction and operation parties. The data management provider supervises and manages the data sharing and exchange of data providers, such as supervising the quality of shared data and the timeliness of response to needs. The data management provider actively responds to the business needs raised by data requesters and conducts responsibility supervision and security supervision of the data requests of data requesters.

[0032] Data providers include various departments, bureaus, commissions, offices, municipalities, public service units, and organizations. Data providers complete preliminary work such as data resource cataloging and data linking according to standards and specifications. They analyze, approve, and implement the sharing requests submitted by data requesters through the public data platform. Implementation work includes data cataloging, data linking, data governance, service development, and service control. Data providers can submit requests and improvement suggestions to data management entities regarding business processes, platforms, and systems. Data providers can also evaluate the service quality of the construction and operation entities and propose optimization and improvement requests.

[0033] Data demanders include various government departments, commissions, offices, municipal and county governments, third-party units, and others. They submit data and service requests through the government data service portal, based on the data resource catalog and supply-demand matching functions. After approval by the data provider, data demanders can utilize the platform for data governance, data fusion, model design, etc., to support their data applications. Data demanders can evaluate the data provider's response to their needs and provide feedback. Regarding the government data sharing and exchange business processes, platforms, and systems, data demanders can submit requests or suggestions to the data management entity and conduct comprehensive evaluations. Data demanders can also evaluate the service quality of the construction and operation entity and propose optimization and improvement requests.

[0034] Construction and Operation Provider: The construction and operation provider is the builder of the public data platform and also the service provider for institutional guarantees, standards and specifications, data governance, operation and maintenance. Under the overall management of the data management provider, it is responsible for platform construction, maintenance and upgrades, data census, auxiliary cataloging, demand management, data governance, data analysis, data operation, and security. The construction and operation provider provides comprehensive and full-lifecycle operation and support services to data suppliers, data demanders, and data managers to ensure the safe, efficient, orderly, and smooth sharing and circulation of government data.

[0035] The integrated lake and warehouse planning diagram of this invention is as follows: Figure 3 As shown, the government data resource center built on the lake-warehouse integration technology mainly constructs a data lake, a cleaning repository, a basic repository, and a thematic repository at the data resource layer, forming a government data resource system.

[0036] (1) In terms of data lake and cleansing repository, the main task is to complete the aggregation of data resources and the construction of data source layer and standard layer of data warehouse.

[0037] (2) In terms of basic databases, the main focus is on building population databases, legal entity databases, credit databases, geographic databases, electronic certificate databases, and economic governance databases needed for the construction of digital government and social governance.

[0038] (3) In terms of subject databases, subject databases will be built based on data applications, mainly including financial subject databases, business environment subject databases, cultural tourism subject databases, and medical and health care subject databases.

[0039] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A method for the fusion and hierarchical sharing of government data based on a lake-warehouse integrated architecture, characterized by: The method includes: S1. Sorting out government data sources and constructing government data meta-models; S2. For government data, a unified storage layer is built using a lake-warehouse integrated architecture for data storage and governance; S3. For government data flows, construct a data fusion pipeline to perform data access, cleaning, standardization, fusion, classification and desensitization processing; S4. To address data sharing needs, a hierarchical sharing engine is constructed, employing a hybrid authorization model, a trusted execution environment, and blockchain notarization to achieve dynamic authorization and trusted sharing. S5. For data processing tasks, task orchestration and resource scheduling methods are used for business orchestration and performance optimization, supporting mixed task execution and secure sharing of confidential data. S6. Evaluate the data quality and sharing effectiveness; S7. Perform resource elastic scaling and disaster recovery processing.

2. The method for government data fusion and hierarchical sharing based on a lake-warehouse integrated architecture as described in claim 1, characterized in that: S1 includes: S11. Establish a five-level directory structure of department-system-table-field-record to form a unified government data metadata model (GDMM); S12. Use knowledge graph technology to extract business terms, coding rules, and semantic relationships to construct a government data knowledge graph (KGD). S13. Introduce a data DNA fingerprinting algorithm to generate a 128-bit unique identifier for each record, which is used for subsequent bloodline tracing.

3. The method for government data fusion and hierarchical sharing based on a lake-warehouse integrated architecture as described in claim 1, characterized in that: S2 includes: S21. The storage layer is constructed using a dual-table format of Iceberg and Hudi. Iceberg is an open-source data table format that supports ACID transactions, while Hudi is a table format that supports streaming updates. The combination of the two is used to balance transaction consistency and real-time updates. S22. Using Object Storage Service (OSS) as a unified storage platform, hot data is cached in Alluxio distributed cache; S23. The computing layer adopts a hybrid engine of batch processing SparkSQL, stream processing Flink, and interactive analysis Trino, and realizes the unification of the three computing modes of batch processing, stream processing, and interactive analysis through the same set of metadata services, that is, three computing in one. S24. Introducing mixed row and column storage, Z-Order sorting, and Bloom Filter indexes to improve query performance at high concurrency points.

4. The method for government data fusion and hierarchical sharing based on a lake-warehouse integrated architecture as described in claim 1, characterized in that: S3 includes: S31. Data Access: Utilize Change Data Capture (CDC) technology to capture the Binlog logs of the business database in real time, achieving data access with a minute-level latency. S32. Data Cleaning: The data cleaning module has built-in government data quality inspection rules, including checking the completeness of required fields, the legality of ID cards, the verification of unified social credit codes, and fuzzy matching of addresses. S33. Data Standardization: Automatic mapping of field synonyms, unit conversion, and dictionary unification based on government data knowledge graph (KGD); S34. Data Fusion: The Master Data Management (MDM) algorithm is used to uniquely identify and merge the four major master data categories: natural persons, legal persons, certificates, and matters. S35. Data Classification: Combining the "Guidelines for Classification and Grading of Government Data" (GB / T 35273-2025) with the department's custom sensitive word library, a Natural Language Processing (NLP) model and a rule engine are used to predict the sensitivity level. The NLP model is based on the BERT architecture, which is a pre-trained language model architecture based on a Transformer encoder, pre-trained with a bidirectional masked language model and next-sentence prediction, and can capture the bidirectional semantic dependencies of the text context. The rule engine is used to execute predefined sensitive word matching rules to classify the data into 5 levels: public, internal, restricted, confidential, and top secret. S36. Data anonymization: For fields classified as restricted, confidential, and top secret, the anonymization strategy dynamically selects masking, tokenization, homomorphic encryption, and differential privacy algorithms for data anonymization based on the classification strategy; the anonymization strategy is tied to the data lineage and supports the strategy following the data.

5. The method for government data fusion and hierarchical sharing based on a lake-warehouse integrated architecture as described in claim 1, characterized in that: S4 includes: S41. Dynamic authorization is performed based on a hybrid authorization model of ABAC and PBAC, where ABAC is attribute-based access control and PBAC is policy-based access control. Permission policies are dynamically generated by combining the four-tuple of subject attributes, object attributes, environment attributes and access purpose. S42. Introduce a Trusted Execution Environment (TEE) to store keys and policy indexes to prevent unauthorized access by internal personnel; S43 provides four types of interface gateways: SQL, REST, GraphQL, and ODBC, to support row-level, column-level, and cell-level permissions; S44. Built-in data uses the DUA template of the protocol. After the caller signs it online, a blockchain certificate is automatically generated to ensure traceability of use. S45. When the policy changes, the affected records are located in seconds based on the data DNA fingerprint, triggering logical reclamation or physical destruction. Logical reclamation refers to making the data inaccessible through version control or soft deletion, while physical destruction refers to permanently deleting the data from storage.

6. The method for government data fusion and hierarchical sharing based on a lake-warehouse integrated architecture as described in claim 1, characterized in that: S5 includes: S51. Implement Directed Acyclic Graph (DAG) task orchestration using Airflow and Kubernetes. Airflow is a workflow scheduling platform, and Kubernetes is a container orchestration platform. The combination of the two supports one-click deployment of batch processing, stream processing, and interactive tasks, allowing users to trigger the execution of mixed tasks with a single operation. S52. Based on historical task profiles, use deep reinforcement learning (DQN) to automatically select the computing engine, concurrency, and resource specifications. S53. For high-frequency shared interfaces, materialized views and result caching mechanisms are provided. Materialized views are pre-computed views, and result caching is used to store query results to improve query performance. S54. For confidential data, a federated learning sharing scheme with "data stationary, model dynamic" is adopted. Here, "data stationary, model dynamic" means that the original data is kept locally, and only the model parameters are aggregated and shared through the Trusted Execution Environment (TEE) to ensure that the original data does not leave the domain.

7. The method for government data fusion and hierarchical sharing based on a lake-warehouse integrated architecture as described in claim 1, characterized in that: S6 includes: S61. Establish a five-dimensional evaluation index system that includes completeness, consistency, timeliness, accessibility, and security; S62. Data sharing KPI scores are automatically generated monthly at the departmental level and incorporated into the digital government assessment. S63. Introduce a data asset depreciation model to automatically archive or destroy data that has not been accessed for a long time.

8. The method for government data fusion and hierarchical sharing based on a lake-warehouse integrated architecture as described in claim 1, characterized in that: S7 includes: S71. The storage layer deploys three replicas across availability zones to achieve RPO=0 and RTO<30s. Here, RPO is the recovery point objective, RPO=0 means zero data loss, and RTO is the recovery time objective, RTO<30s means the system recovery time is less than 30 seconds. S72. The compute layer implements elastic scaling based on Spot instances and HPA. Spot instances are low-cost cloud computing instances, while HPA is horizontal Pod auto-scaling, which is used to automatically adjust resources according to the load to achieve peak shaving and valley filling. S73. The metadata service uses the Raft protocol to achieve high availability of three nodes, automatically elect a master, and prevent single points of failure. The Raft protocol is a distributed system consensus algorithm.

9. A government data fusion and hierarchical sharing system based on a lake-warehouse integrated architecture, characterized in that: The system includes a module for executing the government data fusion and hierarchical sharing method based on the lake-warehouse integrated architecture as described in any one of claims 1 to 8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method for government data fusion and hierarchical sharing based on the lake-warehouse integrated architecture as described in any one of claims 1 to 8.

Images