Government affair business data management method and system

Abstract

The invention provides a government affair business data governance method and system, and relates to the technical field of data governance, and the method comprises the steps: obtaining to-be-governed government affair business data; performing data preprocessing on the government affair business data to obtain preprocessed data; the data preprocessing comprises data cleaning, data conversion and data integration; performing data standardization on the preprocessed data to obtain standard data; performing text classification based on a natural language processing technology on the standard data to obtain various types of classified business data; and performing partition storage on the classified service data according to types. According to the method, the quality and availability of the data are improved, and powerful support is provided for intelligent decision making and efficient management of government affair departments.

Description

Technical Field

[0001] The present invention relates to the field of data governance technology, and in particular to a government business data governance method and system. Background Art

[0002] In the modern information society, government departments have accumulated a large amount of business data, which covers many fields such as population, legal persons, economy, transportation, education, and medical care. However, due to the diverse sources of data, non-uniform formats, and uneven quality, the management and use of data face many challenges.

[0003] At present, government data often have problems such as duplication, omissions, and errors, which affect the accuracy and reliability of the data. Due to imperfect data governance, the potential value of the data is difficult to fully explore and utilize, affecting the efficiency of government decision-making and public services.

[0004] In order to address the above-mentioned issues, government departments urgently need a systematic data governance approach to improve data quality, achieve data standardization, promote data sharing, development and utilization, and secure management, so as to support the construction of digital government and digital economy and improve new quality productivity. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a government business data governance method and system, which not only improves the quality and availability of data, but also provides strong support for intelligent decision-making and efficient management of government departments.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A government business data governance method, comprising:

[0008] Obtain government business data to be governed;

[0009] Preprocessing the government business data to obtain preprocessed data; the data preprocessing includes data cleaning, data conversion and data integration;

[0010] Performing data standardization on the preprocessed data to obtain standard data;

[0011] Performing text classification on the standard data based on natural language processing technology to obtain various types of classified business data;

[0012] The classified business data is partitioned and stored according to type.

[0013] Preferably, the government business data includes: population data, legal person data, economic data, traffic data, education data, medical data, social service data, environmental data, legal and administrative data, etc.

[0014] Preferably, obtaining the government affairs business data to be governed includes:

[0015] Obtain access ports in various government business systems;

[0016] Using ETL tools, the government affairs business data is obtained from the database, file, and API channels of each government affairs business system through the access port;

[0017] Record the time, source and quantity of obtaining the government business data, and package the time, source and quantity information into the corresponding government business data.

[0018] Preferably, the government service data is preprocessed to obtain preprocessed data, including:

[0019] Deleting duplicate data, filling missing values, and deleting outliers on the government business data in turn to obtain cleaned data;

[0020] Performing format unification and code conversion on the cleaned data to obtain converted data;

[0021] The converted data are merged according to a preset unique identifier to obtain the preprocessed data.

[0022] Preferably, the preprocessed data is subjected to data standardization to obtain standard data, including:

[0023] The field name, type, length and meaning of the preprocessed data are unified according to a preset standard, and the codes and enumeration values ​​in the preprocessed data are standardized to obtain the standard data.

[0024] Preferably, the standard data is subjected to text classification based on natural language processing technology to obtain various types of classified business data, including:

[0025] Input the sample government affairs standard data into the preset text context relationship feature extraction layer to extract the context relationship feature information;

[0026] Inputting the standard data into the global feature extraction layer to extract global feature information;

[0027] Fusing the contextual feature information and the global feature information to obtain a fused text feature;

[0028] Constructing a loss function using the fused text features;

[0029] Continuously optimizing the loss function to obtain a text classification model;

[0030] The text classification model is used to perform text classification on the standard data to obtain various types of classified business data.

[0031] Preferably, the sample government affairs standard data is input into a preset text context feature extraction layer to extract context feature information, including:

[0032] Using a pre-trained language model to extract initial feature information of the sample government affairs standard data;

[0033] Inputting the initial feature information into a forward gated recurrent unit and a reverse gated recurrent unit;

[0034] The outputs of the forward gated recurrent unit and the reverse gated recurrent unit are concatenated to obtain contextual relationship feature information; the formula of the contextual relationship feature information is: Among them, H t ={H1,H2,…H l} represents the initial feature information input at time t, represents the output of the positive gated recurrent unit at time t, Represents the output of the reverse gated recurrent unit at time t, GRU represents the gated recurrent unit, and G represents the contextual relationship feature information.

[0035] Preferably, the standard data is input into the global feature extraction layer to extract global feature information, including:

[0036] The initial feature information of the standard data is sequentially input into the convolution layer and the pooling layer to obtain the global feature information; wherein the global feature information extraction formula is:

[0037] c i =f(ω·H+b)

[0038]

[0039]

[0040] In the formula, f is the activation function, ω is the convolution kernel, h is the convolution kernel size, b is the bias value, c is i is the feature vector extracted by the i-th convolutional layer, It represents the value of the features extracted by three different convolution kernels after the maximum pooling layer, and C represents the extracted global feature information.

[0041] A government affairs business data governance system, comprising:

[0042] Data acquisition module, used to obtain government business data to be governed;

[0043] A data preprocessing module, used to perform data preprocessing on the government business data to obtain preprocessed data; the data preprocessing includes data cleaning, data conversion and data integration;

[0044] A standardization module, used for performing data standardization on the preprocessed data to obtain standard data;

[0045] A classification module is used to perform text classification on the standard data based on natural language processing technology to obtain various types of classified business data;

[0046] The storage module is used to partition and store the classified business data according to type.

[0047] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0048] The present invention provides a method and system for government business data governance, the method comprising: obtaining government business data to be governed; performing data preprocessing on the government business data to obtain preprocessed data; the data preprocessing includes data cleaning, data conversion and data integration; performing data standardization on the preprocessed data to obtain standard data; performing text classification based on natural language processing technology on the standard data to obtain various types of classified business data; partitioning and storing the classified business data according to type. The present invention not only improves the quality and availability of data, but also provides strong support for intelligent decision-making and efficient management of government departments. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0050] Figure 1 A flow chart of a method provided by an embodiment of the present invention;

[0051] Figure 2 A schematic diagram of the system structure provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] The purpose of the present invention is to provide a government business data governance method and system, which not only improves the quality and availability of data, but also provides strong support for intelligent decision-making and efficient management of government departments.

[0054] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] Figure 1 A flow chart of a method provided by an embodiment of the present invention, such as Figure 1 As shown, the present invention provides a government business data governance method, including:

[0056] Step 100: Obtain government business data to be managed;

[0057] Step 200: preprocessing the government business data to obtain preprocessed data; the data preprocessing includes data cleaning, data conversion and data integration;

[0058] Step 300: performing data standardization on the preprocessed data to obtain standard data;

[0059] Step 400: performing text classification on the standard data based on natural language processing technology to obtain various types of classified business data;

[0060] Step 500: The classified business data is partitioned and stored according to type.

[0061] Preferably, the government business data includes: population data, economic data, traffic data, education data, medical data, social service data, environmental data and legal and administrative data.

[0062] Specifically, government business data includes the following types of information:

[0063] 1. Population data:

[0064] Basic demographic information (name, gender, age, ID number, etc.);

[0065] Household registration information (place of residence, family members, etc.);

[0066] Social security information (social security number, medical insurance information, etc.).

[0067] 2. Legal person data

[0068] Enterprise registration information (enterprise name, registration number, legal representative, etc.);

[0069] Enterprise qualification information (qualification name, level, issuance time, etc.);

[0070] 3. Economic data:

[0071] Tax information (taxpayer identification number, tax registration, tax records, etc.);

[0072] Financial data (budget, expenditure, income, etc.).

[0073] 4. Traffic data:

[0074] Transportation information (vehicle registration, driver’s license information, traffic accident records, etc.);

[0075] Public transportation data (bus and subway operating hours, passenger flow, etc.).

[0076] 5. Educational data:

[0077] School information (school name, type, address, etc.);

[0078] Student information (name, student number, grades, student status, etc.);

[0079] Teacher information (teacher's name, title, courses taught, etc.).

[0080] 6. Medical data:

[0081] Medical institution information (hospital name, address, department, etc.);

[0082] Patient information (medical records, medical records, medical insurance information, etc.);

[0083] Epidemic data (epidemic reports, vaccination status, etc.).

[0084] 7. Social service data:

[0085] Social assistance information (minimum living allowance, assistance application, assistance record, etc.);

[0086] Public service information (community services, cultural activities, volunteer services, etc.).

[0087] 8. Environmental data:

[0088] Environmental monitoring data (air quality, water quality monitoring, etc.);

[0089] Natural resource information (land use, mineral resources, etc.).

[0090] 9. Legal and administrative data:

[0091] Legal and regulatory information (policy documents, legal clauses, etc.);

[0092] Administrative approval information (approval process, approval results, etc.).

[0093] Specifically, in addition to the data listed above, other relevant government data may also be included, which will not be elaborated in this embodiment. The above data in this embodiment not only involves the basic information of individuals and enterprises, but also covers all aspects of social and economic activities. It is an important basis for government departments to make decisions, management and services.

[0094] Preferably, obtaining the government affairs business data to be governed includes:

[0095] Obtain access ports in various government business systems;

[0096] Using ETL tools, the government affairs business data is obtained from the database, file, and API channels of each government affairs business system through the access port;

[0097] Record the time, source and quantity of obtaining the government business data, and package the time, source and quantity information into the corresponding government business data.

[0098] Specifically, this embodiment first identifies and obtains the access port information of each government business system, including database connection port, file access path and API endpoint, etc. This information can be obtained through system configuration files, network scanning or communication with system administrators. Then, select a suitable ETL (Extract, Transform, Load) tool, such as Apache Nifi, Talend or Informatica, and configure it to connect to the access port of each business system.

[0099] In the ETL tool, define the data extraction process. For the database, use JDBC connection and write SQL query statements to extract the required data; for the file system, use the file reading module to grab files in specific formats (such as CSV, Excel, JSON); for the API, obtain data through HTTP requests and parse the data content in the response.

[0100] During the data extraction process, record detailed information for each data acquisition, including acquisition time, data source (such as system name or IP address), and data volume (such as number of records or file size). This information can be automatically recorded through the logging function of the ETL tool, or manually added to the data extraction script.

[0101] Finally, package these metadata (time, source, quantity) together with the extracted data and store them as part of the data set. This can be achieved by adding additional fields to the data records, or storing the metadata in a separate log table for subsequent data governance and traceability.

[0102] Through the above steps, this embodiment can achieve systematic and automated acquisition of government business data, while ensuring the traceability and integrity of data sources, laying the foundation for subsequent data governance processes.

[0103] Preferably, the government service data is preprocessed to obtain preprocessed data, including:

[0104] Deleting duplicate data, filling missing values, and deleting outliers on the government business data in turn to obtain cleaned data;

[0105] Performing format unification and code conversion on the cleaned data to obtain converted data;

[0106] The converted data are merged according to a preset unique identifier to obtain the preprocessed data.

[0107] Preferably, the preprocessed data is subjected to data standardization to obtain standard data, including:

[0108] The field name, type, length and meaning of the preprocessed data are unified according to a preset standard, and the codes and enumeration values ​​in the preprocessed data are standardized to obtain the standard data.

[0109] Furthermore, this embodiment first cleans the government business data. Use data cleaning tools or write scripts to perform the following operations in sequence: delete duplicate data, remove redundant information by identifying the unique identifier (such as ID) of duplicate records; fill missing values, use the mean, median or interpolation method to fill missing data, or make reasonable inferences based on business rules; detect and delete outliers, identify and eliminate abnormal data points by setting a reasonable value range or using statistical methods (such as standard deviation method).

[0110] Next, the cleaned data is formatted and encoded. Date, number, text and other fields are converted to a unified format (such as ISO date format, unified currency unit, etc.), and all text data is ensured to use a consistent character encoding (such as UTF-8) to avoid garbled characters.

[0111] The transformed data is then merged based on preset unique identifiers, such as social security numbers, business registration numbers, etc. By matching these identifiers, data from different sources are integrated into a complete record to form preprocessed data.

[0112] In the data standardization stage, the field names, types, lengths, and meanings of the preprocessed data are unified according to the preset standards. For example, all "name" fields are uniformly named "Name" and their types are ensured to be strings with a length of no more than 50 characters. At the same time, the codes and enumeration values ​​in the data are standardized to ensure that they comply with industry or national standards (such as administrative division codes, gender codes, etc.).

[0113] Through the above steps, this embodiment can effectively improve the consistency and availability of data, and provide high-quality standardized data support for subsequent data analysis and application.

[0114] Preferably, the standard data is subjected to text classification based on natural language processing technology to obtain various types of classified business data, including:

[0115] Input the sample government affairs standard data into the preset text context relationship feature extraction layer to extract the context relationship feature information;

[0116] Inputting the standard data into the global feature extraction layer to extract global feature information;

[0117] Fusing the contextual feature information and the global feature information to obtain a fused text feature;

[0118] Constructing a loss function using the fused text features;

[0119] Continuously optimizing the loss function to obtain a text classification model;

[0120] The text classification model is used to perform text classification on the standard data to obtain various types of classified business data.

[0121] Preferably, the sample government affairs standard data is input into a preset text context feature extraction layer to extract context feature information, including:

[0122] Using a pre-trained language model to extract initial feature information of the sample government affairs standard data;

[0123] Inputting the initial feature information into a forward gated recurrent unit and a reverse gated recurrent unit;

[0124] The outputs of the forward gated recurrent unit and the reverse gated recurrent unit are concatenated to obtain contextual relationship feature information; the formula of the contextual relationship feature information is: Among them, H t ={H1,H2,…H l} represents the initial feature information input at time t, represents the output of the positive gated recurrent unit at time t, Represents the output of the reverse gated recurrent unit at time t, GRU represents the gated recurrent unit, and G represents the contextual relationship feature information.

[0125] Specifically, in practical applications, this embodiment can use the XLNet model to complete the preliminary feature extraction step. The XLNet model is a pre-trained language model based on the combination of autoregression and autoencoding. Its core part is the Transformer-XL architecture, which solves the defect of the context input length constraint of the BERT model (Bidirectional Encoder Representations from Transformers), so that the pre-trained model can learn more context information.

[0126] Furthermore, the present invention can more fully learn the context relationship of the text and obtain the context information by splicing the outputs of the forward gated recurrent unit and the reverse gated recurrent unit.

[0127] Preferably, the standard data is input into the global feature extraction layer to extract global feature information, including:

[0128] The initial feature information of the standard data is sequentially input into the convolution layer and the pooling layer to obtain the global feature information; wherein the global feature information extraction formula is:

[0129] c i =f(ω·H+b)

[0130]

[0131]

[0132] In the formula, f is the activation function, ω is the convolution kernel, h is the convolution kernel size, b is the bias value, c is i is the feature vector extracted by the i-th convolutional layer, It represents the value of the features extracted by three different convolution kernels after the maximum pooling layer, and C represents the extracted global feature information.

[0133] Specifically, this embodiment determines the type of data (such as population data, economic data, traffic data, etc.) according to the classification results of business data. Create corresponding storage partitions for each type of data, which can be achieved by creating a partition table in the database or through a directory structure in a distributed file system. Next, design an intelligent storage strategy to select appropriate storage media and storage methods based on factors such as data access frequency, importance, and life cycle. For example, high-frequency access data is stored on high-speed storage media (such as SSD), while historical data or low-frequency access data is stored on large-capacity storage media (such as HDD or cloud storage). During the data storage process, the data storage location and storage strategy are dynamically adjusted using the automation function of data management tools or storage systems. For example, using hot data and cold data tiered storage technology, regularly analyze data access patterns, and migrate hot data to faster storage media to improve access efficiency. At the same time, use data compression, deduplication, and encryption technologies to optimize storage space and improve data security.

[0134] Through the above steps, this embodiment can achieve efficient partitioned storage of classified business data, and optimize the use of storage resources through intelligent storage strategies, improve data access performance and security, and meet the diverse needs of government affairs.

[0135] Corresponding to the above method, this embodiment also provides a government business data management system, such as Figure 2 As shown, the system includes:

[0136] Data acquisition module, used to obtain government business data to be governed;

[0137] A data preprocessing module, used to perform data preprocessing on the government business data to obtain preprocessed data; the data preprocessing includes data cleaning, data conversion and data integration;

[0138] A standardization module, used for performing data standardization on the preprocessed data to obtain standard data;

[0139] A classification module is used to perform text classification on the standard data based on natural language processing technology to obtain various types of classified business data;

[0140] The storage module is used to partition and store the classified business data according to type.

[0141] The beneficial effects of the present invention are as follows:

[0142] (1) The present invention can effectively eliminate noise, duplication and erroneous information in data through data cleaning, conversion and integration, thereby improving the accuracy and reliability of data.

[0143] (2) The present invention standardizes data to ensure that data from different sources and formats can be unified, which facilitates subsequent analysis and application and reduces the difficulty of data integration caused by inconsistent standards.

[0144] (3) The present invention uses natural language processing technology for text classification, which can automatically identify and classify large amounts of unstructured data, improve data availability and retrieval efficiency, and enable government departments to obtain required information more quickly.

[0145] (4) The classified business data of the present invention can provide more accurate information support for decision-making, helping government departments to make more scientific decisions in policy formulation, resource allocation, etc.

[0146] (5) The present invention performs partition storage according to type, which can improve the efficiency of data access, reduce query time, enhance the overall performance of the system, and facilitate subsequent data management and maintenance.

[0147] (6) Through standardization and classification, the present invention can more conveniently realize cross-departmental and cross-system data sharing, break the "data island", and promote the coordination and integration of government information.

[0148] (7) During the process of classified storage, the present invention can formulate corresponding security policies according to the data type and sensitivity, enhance the security management of data, and reduce the risk of data leakage.

[0149] (8) The present invention can flexibly adapt to different types of government business data, has strong scalability and adaptability, and can meet the ever-changing government needs.

[0150] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0151] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for government business data governance, characterized in that: include: Obtain government business data to be governed; Performing data preprocessing on the government affairs business data to obtain preprocessed data; The data preprocessing includes data cleaning, data conversion and data integration; Performing data standardization on the preprocessed data to obtain standard data; Performing text classification on the standard data based on natural language processing technology to obtain various types of classified business data; The classified business data is partitioned and stored according to type.

2. The government affairs business data management method according to claim 1 is characterized in that: The government business data includes: population data, legal person data, economic data, transportation data, education data, medical data, social service data, environmental data and legal and administrative data.

3. The government affairs business data management method according to claim 1 is characterized in that: Obtain the government business data to be governed, including: Obtain access ports in various government business systems; Using ETL tools, the government affairs business data is obtained from the database, file, and API channels of each government affairs business system through the access port; Record the time, source and quantity of obtaining the government business data, and package the time, source and quantity information into the corresponding government business data.

4. The government affairs business data management method according to claim 1 is characterized in that: The government affairs business data is preprocessed to obtain preprocessed data, including: Deleting duplicate data, filling missing values, and deleting outliers on the government business data in turn to obtain cleaned data; Performing format unification and code conversion on the cleaned data to obtain converted data; The converted data are merged according to a preset unique identifier to obtain the preprocessed data.

5. The government affairs business data management method according to claim 1 is characterized in that: The preprocessed data is subjected to data standardization to obtain standard data, including: The field name, type, length and meaning of the preprocessed data are unified according to a preset standard, and the codes and enumeration values ​​in the preprocessed data are standardized to obtain the standard data.

6. The government affairs business data management method according to claim 1 is characterized in that: The standard data is subjected to text classification based on natural language processing technology to obtain various types of classified business data, including: Input the sample government affairs standard data into the preset text context relationship feature extraction layer to extract the context relationship feature information; Inputting the standard data into the global feature extraction layer to extract global feature information; Fusing the contextual feature information and the global feature information to obtain a fused text feature; Constructing a loss function using the fused text features; Continuously optimizing the loss function to obtain a text classification model; The text classification model is used to perform text classification on the standard data to obtain various types of classified business data.

7. The government affairs business data management method according to claim 6 is characterized in that: The sample government affairs standard data is input into the preset text context feature extraction layer to extract context feature information, including: Using a pre-trained language model to extract initial feature information of the sample government affairs standard data; Inputting the initial feature information into a forward gated recurrent unit and a reverse gated recurrent unit; The outputs of the forward gated recurrent unit and the reverse gated recurrent unit are concatenated to obtain contextual relationship feature information; the formula of the contextual relationship feature information is: Among them, H t ={H1,H2,…H l } represents the initial feature information input at time t, represents the output of the positive gated recurrent unit at time t, Represents the output of the reverse gated recurrent unit at time t, GRU represents the gated recurrent unit, and G represents the contextual relationship feature information.

8. The government affairs business data management method according to claim 7 is characterized in that: The standard data is input into the global feature extraction layer to extract global feature information, including: The initial feature information of the standard data is sequentially input into the convolution layer and the pooling layer to obtain the global feature information; wherein the global feature information extraction formula is: c i =f(ω·H+b) In the formula, f is the activation function, ω is the convolution kernel, h is the convolution kernel size, b is the bias value, c is i is the feature vector extracted by the i-th convolutional layer, It represents the value of the features extracted by three different convolution kernels after the maximum pooling layer, and C represents the extracted global feature information.

9. A government affairs business data management system, characterized in that: include: Data acquisition module, used to obtain government business data to be governed; A data preprocessing module, used to perform data preprocessing on the government business data to obtain preprocessed data; The data preprocessing includes data cleaning, data conversion and data integration; A standardization module, used for performing data standardization on the preprocessed data to obtain standard data; A classification module is used to perform text classification on the standard data based on natural language processing technology to obtain various types of classified business data; The storage module is used to partition and store the classified business data according to type.

Images