Government affair data sharing system, method and equipment and storage medium

Abstract

The invention discloses a government affair data sharing system, method and device and a storage medium, and relates to the technical field of big data, and the system comprises a standardization processing module which is used for constructing a target process of original government affair data, carrying out data processing based on the target process, and completing the standardization processing; the real-time synchronous processing module is used for capturing the changed government affair data, sending the changed government affair data to the distributed message queue, transmitting the government affair data in real time by utilizing the distributed message queue, and ensuring the synchronization of the government affair data by utilizing a distributed coordination mechanism; the encryption and authentication module is used for performing fine-grained access control and identity authentication on the data accessor and sending the encrypted government affair data to the data accessor; and the storage and construction module is used for storing the government affair data after standardization processing and synchronization to a distributed storage system, constructing a data query interface and a target retrieval engine, and performing data query and retrieval based on the data query interface and the target retrieval engine so as to realize sharing of the government affair data.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a government data sharing system, method, device and storage medium. Background Art

[0002] With the rapid development of e-government, various government departments have accumulated a large amount of data resources. However, due to historical reasons and technical barriers, these data are often scattered in different systems, forming "data islands". This has led to problems such as repeated data collection, inconsistent information, and difficulty in sharing, which has seriously affected the efficiency and quality of government services. At present, although some places have established data sharing platforms, the following problems still exist: inconsistent data standards increase the difficulty of data integration; insufficient real-time performance makes it difficult to meet immediate business needs; imperfect security protection mechanisms pose a risk of data leakage; inflexible data service interfaces make it difficult to adapt to diverse application needs; lack of effective data quality management mechanisms affects the availability and credibility of data. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a government data sharing system, method, device and storage medium, which can realize real-time update, secure sharing, flexible access and intelligent analysis of government data. It effectively solves the problems of data dispersion, inconsistent format, sharing difficulties and security risks existing in traditional government data management. The specific scheme is as follows:

[0004] In a first aspect, the present application discloses a government data sharing system, including:

[0005] A standardization processing module is used to define the data model of the original government data using the data model definition language of the target data serialization system to generate corresponding serialization codes, and to build a target process of the original government data based on Apache NiFi, so as to process the original government data based on the target process and complete the standardization processing of the original government data;

[0006] A real-time synchronization processing module is used to connect each original government data to a distributed message queue, capture the changed government data through Debezium, send the changed government data to the distributed message queue, use the distributed message queue to transmit the corresponding government data in real time, and use the distributed coordination mechanism of ZooKeeper to ensure the synchronization of the government data;

[0007] An encryption and authentication module is used to encrypt the data transmission of the government data using the transport layer security protocol, perform fine-grained access control and identity authentication on data accessors, and send the corresponding encrypted government data to the data accessors after the identity authentication is passed;

[0008] The storage and construction module is used to store the standardized government data and the synchronized government data in the distributed storage system, build a data query interface through GraphQL technology, and build a target retrieval engine so that the data visitor can perform corresponding data query and data retrieval based on the data query interface, the distributed storage system and the target retrieval engine to realize the sharing of the government data.

[0009] Optionally, the data model includes fields, types and nested relationships of the original government data; the target process includes any one or a combination of extraction, conversion and loading; the conversion includes any one or a combination of format conversion, data cleaning and data integration.

[0010] Optionally, the system further includes:

[0011] The processing and inspection module is used to use Apache Spark to perform target processing and quality inspection on the target quantity of the original government data; the target processing includes statistical analysis and / or data mining processing; the quality inspection is to check whether the field values ​​corresponding to the original government data are within the preset value range, whether the target fields of the original government data are fully filled in, and whether the original government data from different data sources are consistent.

[0012] Optionally, the system further includes:

[0013] The data processing module is used to build a breakpoint-resume function based on the Flink CheckPoint mechanism. When a system failure occurs, the breakpoint-resume function records the processing status of the current government data according to a pre-set checkpoint, so that after the system is restored, the current government data can continue to be processed based on the checkpoint and the processing status.

[0014] Optionally, the system further includes:

[0015] The access behavior monitoring module is used to build a normal access behavior model by collecting and analyzing historical access data, and use the normal access behavior model to monitor the current access behavior of the data visitor in real time, so that the normal access behavior model sends a corresponding alarm when detecting that the current access behavior is a preset abnormal access behavior.

[0016] Optionally, the system further includes:

[0017] The security detection module is used to configure the automated test script and regularly perform security detection on the government data sharing system based on the automated test script to ensure the security of the government data sharing system.

[0018] Optionally, the system further includes:

[0019] The visualization analysis module is used to build a visualization analysis platform based on Apache Superset, so that the data visitor can use the visualization analysis platform to drag and configure the government data based on analysis needs, generate corresponding charts, and perform visualization analysis on the government data through the charts.

[0020] In a second aspect, the present application discloses a government data sharing method, including:

[0021] The data model definition language of the target data serialization system is used to define the data model of the original government data to generate the corresponding serialization code, and the target process of the original government data is constructed based on Apache NiFi, so as to process the original government data based on the target process and complete the standardized processing of the original government data;

[0022] Connect each original government data to a distributed message queue, capture the changed government data through Debezium, send the changed government data to the distributed message queue, use the distributed message queue to transmit the corresponding government data in real time, and use ZooKeeper's distributed coordination mechanism to ensure the synchronization of the government data;

[0023] The government data is encrypted by using the transport layer security protocol, and fine-grained access control and identity authentication are performed on the data accessor. After the identity authentication is passed, the corresponding encrypted government data is sent to the data accessor;

[0024] The standardized government data and the synchronized government data are stored in a distributed storage system, and a data query interface is constructed through GraphQL technology, and a target retrieval engine is constructed so that the data accessor can perform corresponding data query and data retrieval based on the data query interface, the distributed storage system and the target retrieval engine to realize the sharing of the government data.

[0025] In a third aspect, the present application discloses an electronic device, comprising:

[0026] Memory, used to store computer programs;

[0027] A processor is used to execute the computer program to implement the government data sharing method as mentioned above.

[0028] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the government data sharing method as described above.

[0029] The government data sharing system of the present application includes a standardization processing module, which is used to define the data model of the original government data using the data model definition language of the target data serialization system to generate the corresponding serialization code, and based on Apache NiFi constructs the target process of the original government data so as to process the original government data based on the target process and complete the standardization of the original government data; the real-time synchronization processing module is used to connect each original government data to the distributed message queue, capture the changed government data through Debezium, send the changed government data to the distributed message queue, use the distributed message queue to transmit the corresponding government data in real time, and use the distributed coordination mechanism of ZooKeeper to ensure the synchronization of the government data; the encryption and authentication module is used to encrypt the data transmission of the government data using the transport layer security protocol, perform fine-grained access control and identity authentication on the data accessor, and send the corresponding encrypted government data to the data accessor after the identity authentication is passed; the storage and construction module is used to store the standardized government data and the synchronized government data in the distributed storage system, build a data query interface through GraphQL technology, and build a target retrieval engine so that the data accessor can perform corresponding data query and data retrieval based on the data query interface, the distributed storage system and the target retrieval engine to realize the sharing of the government data. It can be seen that this application unifies the data format and structure through data standardization services, so that data from different departments and systems can understand and interact with each other; real-time synchronization services ensure the timeliness of data, and each department can obtain the latest data in real time. It can quickly capture and process changed data in the database, and transmit the latest data to related systems and applications in real time, so that government departments can make timely decisions based on real-time data. The security of government data is guaranteed from various aspects such as data transmission, access control, data storage and calculation, to prevent data from being illegally obtained, tampered with or leaked during the exchange and sharing process, and protect information security. It also solves the problem of efficient exchange and sharing of government data across departments and systems, overcomes the dilemma of government data islands, and significantly improves the efficiency and quality of government services. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0031] Figure 1 A schematic diagram of the structure of a government data sharing system disclosed in this application;

[0032] Figure 2 A flow chart of a government data sharing method disclosed in this application;

[0033] Figure 3 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] At present, although some places have established data sharing platforms, the following problems still exist: data standards are not unified, which increases the difficulty of data integration; lack of real-time performance makes it difficult to meet immediate business needs; imperfect security protection mechanisms pose data leakage risks; inflexible data service interfaces make it difficult to adapt to diverse application needs; lack of effective data quality management mechanisms affects data availability and credibility. In order to solve the above technical problems, this application discloses a government data sharing system, method, device and storage medium that can achieve real-time update, secure sharing, flexible access and intelligent analysis of government data. It effectively solves the problems of data dispersion, inconsistent formats, difficulty in sharing, security risks and other issues existing in traditional government data management.

[0036] See also Figure 1 As shown, the embodiment of the present invention discloses a government data sharing system, including:

[0037] The standardization processing module 11 is used to define the data model of the original government data using the data model definition language of the target data serialization system to generate corresponding serialization codes, and to build a target process of the original government data based on Apache NiFi, so as to process the original government data based on the target process and complete the standardization processing of the original government data.

[0038] In this embodiment, this application uses Apache Avro as a data serialization system. Apache Avro itself provides a way to define data schemas. Developers can use Avro IDL (data schema definition language) to define data structures according to the characteristics and requirements of government data. This data structure is the data schema, including data fields, types, nested relationships and other information, and then automatically generates multi-language compatible serialization code, thereby achieving unified serialization and deserialization of data between different systems and languages. It is convenient for systems written in different programming languages ​​to interact with each other. At the same time, with the help of Apache Atlas to manage metadata, the metadata management here includes the management of information such as the definition, source, purpose, and flow path of government data. Building a data lineage relationship diagram is one of the important functions. Through the data lineage relationship diagram, the origin, conversion process and flow of data in different systems and processes can be clearly traced, which helps to understand the ins and outs of the data and facilitates data governance and problem troubleshooting. Automatic registration and update of metadata is achieved through Atlas REST API (a set of RESTful API (Application Programming Interface) interfaces provided by Apache Atlas). Metadata is data that describes government data, such as the name, type, storage location, creation time, update time, association relationship, and other information of the data. As government data is generated, updated, and circulated, its related metadata also needs to be registered in Atlas in a timely manner for effective management and tracking. When data changes or new data is added, metadata also needs to be updated accordingly. Develop an ETL (Extract-Transform-Load) process based on Apache NiFi (an easy-to-use, powerful and reliable data extraction, data processing and distribution system for automating data flow between systems). Transformation mainly refers to operations such as data format conversion, data cleaning (such as removing duplicate data, correcting erroneous data, supplementing missing data, etc.), and data integration (merging or associating data from different sources according to certain rules) to meet subsequent data processing and analysis needs.

[0039] In addition, this application uses Apache Spark for large-scale data processing and quality inspection. Data processing includes complex calculations (such as statistical analysis, data mining, etc.) and data aggregation (aggregating scattered data into more meaningful information) on massive government data; quality inspection refers to the detection and evaluation of data quality dimensions such as accuracy, completeness, consistency, and timeliness through a series of preset rules and algorithms, such as checking whether the data is within a reasonable range, whether the required fields are complete, and whether the data from different data sources are consistent, so as to achieve real-time monitoring of data quality. Specifically, a data quality inspection system based on a rule engine (such as Drools) is designed to support flexible configuration of complex rules. Various data quality inspection rules can be customized according to the different characteristics and business requirements of government data, such as the value range of specific fields, the logical relationship between data, etc., so as to achieve accurate quality monitoring. In addition, this application innovatively introduces machine learning algorithms to realize intelligent metadata management and adaptive adjustment, automatically discover hidden data associations and business logic, and dynamically update metadata and blood relationship diagrams; at the same time, a deep learning model is used for real-time data quality monitoring and automatic repair, accurately detects quality problems and automatically triggers the repair mechanism. The data quality inspection system based on the rule engine monitors data quality in real time and promptly discovers and corrects data problems; the automated data processing and quality monitoring process reduces errors caused by human intervention, thereby improving the overall quality of government data, making the data more reliable and providing strong support for government decision-making and government services.

[0040] The real-time synchronization processing module 12 is used to connect various original government data to the distributed message queue, capture the changed government data through Debezium, send the changed government data to the distributed message queue, use the distributed message queue to transmit the corresponding government data in real time, and use the distributed coordination mechanism of ZooKeeper to ensure the synchronization of the government data.

[0041] In this embodiment, the present application uses Apache Kafka as a distributed message queue and combines Apache Flink for real-time stream processing. Using the Kafka Connect framework, custom Source and Sink connectors are developed to support access to a variety of government data sources (such as different types of databases, file systems, etc.), so that various government data can be easily accessed to the Kafka message queue. Real-time data transmission with high throughput is achieved, and a large amount of real-time data from various government data sources can be efficiently processed. A real-time data processing pipeline based on Apache Flink is developed to support complex event processing, which can analyze and process the real-time inflow change data in real time, such as identifying event sequences of specific patterns, calculating data indicators in real time, etc. Kafka has powerful large-scale data stream processing capabilities to ensure data real-time and reliability. Flink provides excellent stream processing capabilities, supports complex event processing and state management, and realizes real-time data conversion and analysis. Implement Exactly-Once Semantics based on Kafka transactions to ensure that data will only be accurately processed once during the entire synchronization process from the data source to the final storage or processing, regardless of any failures or retries, and will not be lost or repeatedly processed, thereby ensuring the accuracy of data synchronization.

[0042] This application uses Debezium to capture change data. The change data here refers to the change data generated by the addition, deletion, and modification of data in the government database. Debezium can monitor the database transaction log in real time, detect data changes in a timely manner, and send the captured change data to the Kafka message queue in the form of messages.

[0043] The system of this application also includes a data processing module, which is used to build a breakpoint-resume function based on the Flink CheckPoint mechanism. When a failure occurs during system operation, the breakpoint-resume function records the processing status of the current government data according to the pre-set checkpoint, so that after the system is restored, the current government data can continue to be processed based on the checkpoint and the processing status. The specific design is based on the breakpoint-resume function of the Flink CheckPoint mechanism. During the operation of the system, if a failure occurs (such as node crash, network interruption, etc.), Flink can record the current data processing status according to the previously set checkpoint. When the system is restored, it can continue data processing from the location of the last checkpoint without starting from the beginning, which improves the fault tolerance of the system and ensures the continuity and integrity of data processing.

[0044] This application implements a distributed coordination mechanism based on ZooKeeper, which specifically coordinates the work of each node in a distributed environment. During data synchronization, it ensures that different nodes have consistent knowledge of the data state, ensures the order and reliability of data synchronization, and avoids data conflicts and inconsistencies. For example, when multiple consumers obtain data from the Kafka queue for processing, ZooKeeper can coordinate the consumption progress of consumers to ensure that data is not consumed repeatedly or missed.

[0045] The encryption and authentication module 13 is used to encrypt the data transmission of the government data using the transport layer security protocol, perform fine-grained access control and identity authentication on data accessors, and send the corresponding encrypted government data to the data accessors after the identity authentication is passed.

[0046] In this embodiment, TLS (Transport Layer Security) is used for data transmission encryption to ensure the confidentiality of government data during network transmission and prevent data theft or tampering. Apache Shiro is used to implement fine-grained access control and identity authentication. The specific implementation method is to define a series of permission policies and rules to assign permissions to different users or system roles. Only users or systems that have been authenticated and have corresponding permissions can access specific government data resources. For example, for sensitive data, only administrators of a certain level can perform read and write operations, and ordinary users can only perform read-only access. At the same time, the ELK (Elasticsearch, Logstash, Kibana) stack is integrated to achieve comprehensive log collection and auditing. Logstash collects logs, Elasticsearch stores indexes, and Kibana provides a visual interface for query, analysis and auditing. Logstash collects logs generated by various links of the government system, including data access logs, system operation logs, etc., and stores them in Elasticsearch for indexing and storage. Then Kibana is used to provide a visual interface to facilitate administrators to query, analyze and audit logs, and timely discover potential security issues and abnormal behaviors. This application develops an API gateway based on Apache Knox, which provides a unified secure access portal, unified management and security verification of external requests. Only legitimate requests can access government data services through the gateway to prevent illegal access and malicious attacks. In the process of transmitting data, data desensitization technology based on homomorphic encryption supports direct calculation of encrypted data. On the premise of ensuring data privacy, it allows specific calculation operations on data in an encrypted state, such as statistical analysis, which not only protects data security, but also meets some data analysis needs. At the same time, a hardware security module (HSM) based on PKCS11 is used to manage keys to improve key security. The PKCS11 standard provides a secure way to manage keys. The hardware security module stores keys in dedicated hardware devices, which are isolated from the running system to prevent keys from being illegally obtained or tampered with. At the same time, the hardware security module supports key generation, storage, encryption, decryption and other operations, and provides a secure key backup and recovery mechanism to ensure the security and availability of keys when hardware devices fail or are replaced.

[0047] This application uses an automated security testing process based on OWASP ZAP to perform security vulnerability scans regularly. OWASP ZAP is a widely used open source security testing tool. By configuring automated test scripts, it regularly performs comprehensive security scans on the government integration big data platform, including detecting common Web (World Wide Web, Global Wide Area Network) security vulnerabilities (such as SQL (Structured Query Language, Structured Query Language) injection, cross-site scripting attacks, etc.), timely discovering and repairing potential security risks, and ensuring the security of the platform. At the same time, this application uses an abnormal access detection system based on machine learning to identify potential security threats in real time. The system collects and analyzes a large amount of historical access data, establishes a normal access behavior model, and then monitors the current access behavior in real time. When abnormal access behaviors with large deviations from the normal model are found (such as a user frequently accessing sensitive data in a short period of time, access from abnormal geographical locations, etc.), an alarm is issued in time so that the administrator can take corresponding measures to prevent and deal with it. In this way, the security of government data is guaranteed from various links such as data storage, access control, data storage and calculation, and data is prevented from being illegally obtained, tampered with or leaked during the exchange and sharing process, protecting information security.

[0048] The storage and construction module 14 is used to store the standardized government data and the synchronized government data in the distributed storage system, build a data query interface through GraphQL technology, and build a target retrieval engine, so that the data visitor can perform corresponding data query and data retrieval based on the data query interface, the distributed storage system and the target retrieval engine to realize the sharing of the government data.

[0049] In this embodiment, the application stores standardized and synchronized data in a distributed storage system, which may involve a distributed file system or a distributed database to support large-scale data storage and high concurrent access. In addition, GraphQL technology is used to provide a flexible data query interface, and the client can accurately specify the required data fields and query conditions according to its own needs, avoiding the problem of excessive or insufficient data acquisition that may occur in traditional RESTful APIs, reducing unnecessary data transmission, and improving the efficiency of data acquisition. GraphQL's flexible query interface and Apache Solr's full-text search engine meet the diverse query needs of different users and application scenarios for government data.

[0050] In addition, Apache Solr is used to build a full-text search engine, which can perform efficient full-text search on the text content in government data, such as searching for keywords in policy documents, official documents, etc., quickly locating relevant information, and supporting complex data retrieval requirements such as keyword search, fuzzy query, semantic query, etc. This application integrates TensorFlowServing, deploys machine learning models, and provides intelligent data analysis services, such as data prediction (such as predicting government service demand trends, social and economic development trends, etc.), data classification (such as classifying government affairs), and intelligent recommendation (such as recommending suitable government service projects to citizens), etc., to help government departments make more scientific decisions. This application system also includes: a visual analysis module, which is used to build a visual analysis platform based on Apache Superset, so that data visitors can use the visual analysis platform to drag and configure government data based on analysis needs, generate corresponding charts, and perform visual analysis of government data through charts. Specifically, this application develops a visual analysis platform based on Apache Superset, which supports custom dashboards and reports. Users can display government data in the form of intuitive charts (such as bar charts, line charts, maps, etc.) through simple drag and drop and configuration operations according to their own analysis needs, which facilitates data visualization analysis, quickly understands the information behind the data, and provides strong support for decision-making.

[0051] This application also uses a Redis-based cache layer to optimize the access performance of hot data. For frequently accessed government data (such as popular policy documents, commonly used statistical data, etc.), they are cached in Redis. When the client requests these data again, they can be directly obtained from the cache, which greatly improves the data access speed and reduces the pressure on back-end data storage and processing. In addition, Apache Airflow is used to build ETL (Extract, transform, load) and machine learning workflows to automate data processing and model training. Through Airflow, a series of tasks and their execution order and dependencies can be defined, such as regularly extracting data from government data sources, performing data cleaning and conversion, training machine learning models, etc., to achieve automated scheduling and management of the entire data processing and model training process, improve work efficiency, and reduce errors that may be caused by manual intervention. An intelligent question-and-answer system based on knowledge graphs has also been developed to provide a natural language interactive interface. By constructing a government knowledge graph, entities, relationships and other information in government data are represented in a structured manner. The intelligent question-answering system uses natural language processing technology to understand the user's questions, and then performs queries and reasoning in the knowledge graph to provide users with accurate answers, thus achieving a more convenient and intelligent way of interacting with government data services. In this way, the intelligent question-answering system based on the knowledge graph provides a natural language interaction method, making it easier for citizens to obtain government information. These flexible data services have inspired innovative applications of government data, such as intelligent government assistants and personalized government service recommendations.

[0052] It can be seen that this application unifies the data format and structure through data standardization services, so that data from different departments and systems can understand and interact with each other; real-time synchronization services ensure the timeliness of data, and each department can obtain the latest data in real time. It can quickly capture and process changed data in the database, and transmit the latest data to related systems and applications in real time, so that government departments can make timely decisions based on real-time data. The security of government data is guaranteed from various aspects such as data transmission, access control, data storage and calculation, to prevent data from being illegally obtained, tampered with or leaked during the exchange and sharing process, and protect information security. It also solves the problem of efficient exchange and sharing of government data across departments and systems, overcomes the dilemma of government data islands, and significantly improves the efficiency and quality of government services.

[0053] See also Figure 2 As shown, an embodiment of the present invention discloses a government data sharing method, including:

[0054] Step S11: Use the data model definition language of the target data serialization system to define the data model of the original government data to generate the corresponding serialization code, and build the target process of the original government data based on Apache NiFi, so as to process the original government data based on the target process and complete the standardized processing of the original government data.

[0055] Step S12: connect each original government data to a distributed message queue, capture the changed government data through Debezium, send the changed government data to the distributed message queue, use the distributed message queue to transmit the corresponding government data in real time, and use ZooKeeper's distributed coordination mechanism to ensure the synchronization of the government data.

[0056] Step S13: encrypt the government data using the transport layer security protocol, perform fine-grained access control and identity authentication on data accessors, and send the corresponding encrypted government data to the data accessors after the identity authentication is passed.

[0057] Step S14: Store the standardized government data and the synchronized government data in a distributed storage system, build a data query interface through GraphQL technology, and build a target retrieval engine, so that the data accessor can perform corresponding data query and data retrieval based on the data query interface, the distributed storage system and the target retrieval engine to realize the sharing of the government data.

[0058] It can be seen that this application unifies the data format and structure through data standardization services, so that data from different departments and systems can understand and interact with each other; real-time synchronization services ensure the timeliness of data, and each department can obtain the latest data in real time. It can quickly capture and process changed data in the database, and transmit the latest data to related systems and applications in real time, so that government departments can make timely decisions based on real-time data. The security of government data is guaranteed from various aspects such as data transmission, access control, data storage and calculation, to prevent data from being illegally obtained, tampered with or leaked during the exchange and sharing process, and protect information security. It also solves the problem of efficient exchange and sharing of government data across departments and systems, overcomes the dilemma of government data islands, and significantly improves the efficiency and quality of government services.

[0059] Furthermore, the present application also discloses an electronic device. Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0060] Figure 3A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the government data sharing method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0061] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0062] In addition, the memory 22 as a carrier for resource storage may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon may include an operating system 221, a computer program 222, etc., and the storage method may be temporary storage or permanent storage.

[0063] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, which can be Windows Server, Netware, Unix, Linux, etc. In addition to computer programs that can be used to complete the government data sharing method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include computer programs that can be used to complete other specific tasks.

[0064] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned disclosed government data sharing method. The specific steps of the method can refer to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.

[0065] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0066] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0067] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0068] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0069] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A government data sharing system, characterized in that: include: A standardization processing module is used to define the data model of the original government data using the data model definition language of the target data serialization system to generate corresponding serialization codes, and to build a target process of the original government data based on Apache NiFi, so as to process the original government data based on the target process and complete the standardization processing of the original government data; A real-time synchronization processing module is used to connect each original government data to a distributed message queue, capture the changed government data through Debezium, send the changed government data to the distributed message queue, use the distributed message queue to transmit the corresponding government data in real time, and use the distributed coordination mechanism of ZooKeeper to ensure the synchronization of the government data; An encryption and authentication module is used to encrypt the data transmission of the government data using the transport layer security protocol, perform fine-grained access control and identity authentication on data accessors, and send the corresponding encrypted government data to the data accessors after the identity authentication is passed; The storage and construction module is used to store the standardized government data and the synchronized government data in the distributed storage system, build a data query interface through GraphQL technology, and build a target retrieval engine so that the data visitor can perform corresponding data query and data retrieval based on the data query interface, the distributed storage system and the target retrieval engine to realize the sharing of the government data.

2. The government data sharing system according to claim 1 is characterized in that: The data model includes the fields, types and nested relationships of the original government data; the target process includes any one or a combination of extraction, conversion and loading; the conversion includes any one or a combination of format conversion, data cleaning and data integration.

3. The government data sharing system according to claim 1 is characterized in that: Also includes: The processing and inspection module is used to use Apache Spark to perform target processing and quality inspection on the target quantity of the original government data; the target processing includes statistical analysis and / or data mining processing; the quality inspection is to check whether the field values ​​corresponding to the original government data are within the preset value range, whether the target fields of the original government data are fully filled in, and whether the original government data from different data sources are consistent.

4. The government data sharing system according to claim 1 is characterized in that: Also includes: The data processing module is used to build a breakpoint-resume function based on the Flink CheckPoint mechanism. When a system failure occurs, the breakpoint-resume function records the processing status of the current government data according to a pre-set checkpoint, so that after the system is restored, the current government data can continue to be processed based on the checkpoint and the processing status.

5. The government data sharing system according to claim 1 is characterized in that: Also includes: The access behavior monitoring module is used to build a normal access behavior model by collecting and analyzing historical access data, and use the normal access behavior model to monitor the current access behavior of the data visitor in real time, so that the normal access behavior model sends a corresponding alarm when detecting that the current access behavior is a preset abnormal access behavior.

6. The government data sharing system according to claim 1 is characterized in that: Also includes: The security detection module is used to configure the automated test script and regularly perform security detection on the government data sharing system based on the automated test script to ensure the security of the government data sharing system.

7. The government data sharing system according to any one of claims 1 to 6, characterized in that: Also includes: The visualization analysis module is used to build a visualization analysis platform based on Apache Superset, so that the data visitor can use the visualization analysis platform to drag and configure the government data based on analysis needs, generate corresponding charts, and perform visualization analysis on the government data through the charts.

8. A government data sharing method, characterized in that: include: The data model definition language of the target data serialization system is used to define the data model of the original government data to generate the corresponding serialization code, and the target process of the original government data is constructed based on Apache NiFi, so as to process the original government data based on the target process and complete the standardized processing of the original government data; Connect each original government data to a distributed message queue, capture the changed government data through Debezium, send the changed government data to the distributed message queue, use the distributed message queue to transmit the corresponding government data in real time, and use ZooKeeper's distributed coordination mechanism to ensure the synchronization of the government data; The government data is encrypted by using the transport layer security protocol, and fine-grained access control and identity authentication are performed on the data accessor. After the identity authentication is passed, the corresponding encrypted government data is sent to the data accessor; The standardized government data and the synchronized government data are stored in a distributed storage system, and a data query interface is constructed through GraphQL technology, and a target retrieval engine is constructed so that the data accessor can perform corresponding data query and data retrieval based on the data query interface, the distributed storage system and the target retrieval engine to realize the sharing of the government data.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor is used to execute the computer program to implement the government data sharing method as described in claim 8.

10. A computer-readable storage medium, characterized in that: Used to store computer programs; wherein, when the computer program is executed by a processor, it implements the government data sharing method as described in claim 8.

Images