A classification management method and device for government data

Through data cleaning, standardized processing and clustering algorithm classification, combined with quality analysis generation and adjustment strategies, the problems of low classification efficiency and insufficient stability in government data governance are solved, and efficient and reliable governance of government data is achieved.

CN118796954BActive Publication Date: 2024-12-06FOSHAN RUIKE INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410781272.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-18
Publication Date
2024-12-06
Estimated Expiration
2044-06-18

AI Technical Summary

Technical Problem

The classification efficiency and insufficient stability in the existing government data governance have resulted in low reliability of data processing adjustment strategies, making it difficult to achieve reliable management of government data.

Method used

Through data cleaning, standardized processing, event feature extraction and hot keyword extraction, clustering algorithms are used for classification, and data processing adjustment strategies are generated based on quality analysis results, and data integration is combined with distributed storage technology.

Benefits of technology

It improves the reliability and efficiency of government data classification governance, enhances the accuracy of data adjustment, reduces errors in the governance process, and enables government data classification governance to achieve more ideal results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118796954B_ABST
    Figure CN118796954B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and device for classified governance of government data, wherein the method includes: performing data cleaning processing on government data in several data sources; performing standardization processing on the government data; performing matter feature extraction and hot keyword extraction on the standardized government data to obtain matter feature information and hot keyword; using a clustering algorithm to classify the standardized government data based on the matter feature information and hot keyword to obtain standardized government data of various categories; performing quality analysis on standardized government data of various categories based on a preset data quality analysis strategy, and using a secondary decision tree to generate a data processing adjustment strategy to adjust standardized government data of various categories; performing data integration on the standardized government data of various categories, and storing the data integration results. The present invention improves the reliability of classified governance of government data, so that the classified governance of government data can achieve a more ideal effect.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and device for classifying and managing government data. Background Art

[0002] With the advent of the big data era, e-government systems have been widely used. Government systems contain a large amount of government data. In order to achieve effective management and application of government data and improve the management and application level of government data, it is necessary to carry out government data governance related work. In the existing government data governance, the classification of government data is usually done by relevant personnel who manage government data, but this has low efficiency in the classification of government data, and the stability of government data classification cannot be fully guaranteed. In the governance of government data, it is necessary to determine the data processing adjustment strategy based on the quality analysis results of government data, but currently, a simple strategy matching is performed based on the quality analysis results to determine the data processing adjustment strategy, but this will undoubtedly lead to the low reliability of the determined data processing adjustment strategy, making the accuracy of the adjustment of government data insufficient. As a result, the governance of government data fails to achieve the expected effect, making it difficult to achieve reliable management of government data. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the prior art. The present invention provides a method and device for the classification and management of government data, which improves the reliability of the classification and management of government data and enables the classification and management of government data to achieve a more ideal effect.

[0004] In order to solve the above technical problems, the present invention provides a classification management method for government data, the method comprising:

[0005] Acquire government data from several data sources, and perform data cleaning on the government data from several data sources to obtain the government data after data cleaning;

[0006] Standardize the government data after data cleaning to obtain standardized government data;

[0007] Extracting item features from the standardized government affairs data to obtain corresponding item feature information, and extracting hot keywords from the standardized government affairs data to obtain corresponding target hot keywords;

[0008] Based on the feature information of the items and the hot keywords, the standardized government affairs data are classified by using a clustering algorithm to obtain standardized government affairs data of each category;

[0009] Based on the preset data quality analysis strategy, quality analysis is performed on each category of standardized government data to obtain corresponding quality analysis results, and based on the corresponding quality analysis results, a data processing adjustment strategy is generated using a secondary decision tree, and based on the data processing adjustment strategy, each category of standardized government data is adjusted and processed to obtain adjusted standardized government data of each category;

[0010] The adjusted standardized government affairs data of various categories are integrated to obtain data integration results, and the data integration results are stored based on distributed storage technology.

[0011] Optionally, performing data cleaning on the government data in the plurality of data sources to obtain the government data after data cleaning includes:

[0012] Filtering the government data in several data sources to obtain filtered government data;

[0013] The filtered government data is deduplicated to obtain the government data after data cleaning.

[0014] Optionally, the step of standardizing the government data after data cleaning to obtain standardized government data includes:

[0015] Detecting key data information in the government data after data cleaning, and acquiring a target mapping relationship corresponding to the government data after data cleaning based on the key data information;

[0016] Based on the target mapping relationship, the government data after data cleaning is mapped into standardized government data.

[0017] Optionally, extracting item features from the standardized government affairs data to obtain corresponding item feature information includes:

[0018] Performing word segmentation processing on the standardized government affairs data to obtain a corresponding number of target word segmentations;

[0019] Perform sensitive word removal processing on a number of target segmented words to obtain a number of target segmented words after the sensitive word removal processing;

[0020] A number of target word segmentations after sensitive words are removed are used to determine a number of keywords using a word frequency-inverse document frequency algorithm, and corresponding event feature information is generated based on the number of keywords.

[0021] Optionally, extracting hot keywords from the standardized government affairs data to obtain corresponding target hot keywords includes:

[0022] Dividing the standardized government data into a plurality of government data subsets, calculating the amount of interactive information of the standardized government data in each government data subset, and calculating a heat evaluation index based on the amount of interactive information;

[0023] Extracting a number of hot keywords of the standardized government data based on the hot evaluation index;

[0024] Obtaining a recommended node of a preset target ternary tree, and obtaining corresponding branch nodes and branch node frequencies based on the recommended node;

[0025] Based on the branch nodes and branch node frequencies, a number of hot keywords are used to determine corresponding target hot keywords.

[0026] Optionally, the step of classifying the standardized government affairs data by using a clustering algorithm based on the event feature information and hot keywords to obtain standardized government affairs data of various categories includes:

[0027] Extracting samples from the standardized government affairs data to obtain a sample subset, and generating a plurality of cluster centers based on the sample subset;

[0028] Establishing association mapping of each cluster center based on the event feature information and hot keywords, and generating linearly related cluster centers based on the association mapping of each cluster center;

[0029] The standardized government affairs data are classified based on linear correlation cluster centers to obtain standardized government affairs data of each category.

[0030] Optionally, the quality analysis of standardized government data of various categories based on a preset data quality analysis strategy to obtain corresponding quality analysis results includes:

[0031] Select corresponding data quality analysis rules from the preset data quality analysis strategy for each category of standardized government data;

[0032] Based on the data quality analysis rules, quality analysis is performed on each category of standardized government data to obtain quality analysis results corresponding to each category of standardized government data.

[0033] Optionally, generating a data processing adjustment strategy based on the corresponding quality analysis result using a secondary decision tree includes:

[0034] Inputting the quality analysis result into a preset secondary decision tree, wherein the first level decision tree in the preset secondary decision tree is used to preprocess the quality analysis result to obtain the preprocessed quality analysis result;

[0035] The second level decision tree in the preset secondary decision tree is used to perform evaluation based on the quality analysis result after preprocessing, obtain evaluation results, and generate a data processing adjustment strategy based on the evaluation results.

[0036] Optionally, the step of integrating the adjusted standardized government affairs data of various categories to obtain data integration results, and storing the data integration results based on distributed storage technology includes:

[0037] Determine the data integration scope and data integration target, and integrate the adjusted standardized government data of various categories using the data integration scope and data integration target based on ETL technology to obtain data integration results;

[0038] Calculate a hash index based on a hash algorithm, calculate a target key value based on the hash index, perform a conversion operation on the target key value to obtain a target hash value, and determine a split partition based on the target hash value using an InnoDB engine and a mongos function;

[0039] Based on the split partitions, the data integration result is horizontally split to obtain the data integration result after the horizontal split;

[0040] The data integration results after horizontal segmentation are stored in the corresponding storage nodes in the distributed database.

[0041] In addition, the present invention also provides a classification management device for government data, the device comprising:

[0042] Data cleaning module: used to obtain government data from several data sources, and perform data cleaning on the government data from several data sources to obtain the government data after data cleaning;

[0043] Data standardization module: used to standardize the government data after data cleaning to obtain standardized government data;

[0044] Item feature information and hot keyword extraction module: used to extract item features from the standardized government affairs data to obtain corresponding item feature information, and to extract hot keywords from the standardized government affairs data to obtain corresponding target hot keywords;

[0045] Data classification module: used to classify the standardized government data using a clustering algorithm based on the feature information of the matter and hot keywords to obtain standardized government data of various categories;

[0046] Quality analysis and adjustment module: used to perform quality analysis on standardized government data of various categories based on a preset data quality analysis strategy, obtain corresponding quality analysis results, generate data processing adjustment strategies based on the corresponding quality analysis results using a secondary decision tree, perform adjustment processing on standardized government data of various categories based on the data processing adjustment strategies, and obtain standardized government data of various categories after adjustment;

[0047] Data integration and storage module: used to integrate the adjusted standardized government data of various categories, obtain data integration results, and store the data integration results based on distributed storage technology.

[0048] In an embodiment of the present invention, the standardized government data is subjected to item feature extraction to obtain corresponding item feature information, and the standardized government data is subjected to hot keyword extraction in combination with a preset ternary tree and a heat evaluation index to obtain corresponding hot keywords, so that the obtained item feature information and hot keywords are more reliable. The standardized government data is classified using a clustering algorithm based on the item feature information and hot keywords, thereby avoiding the insufficient reliability of the classification of government data by single item feature information, improving the classification efficiency of government data, and ensuring the stability of the classification of government data. Based on the quality analysis results, a data processing adjustment strategy is generated using a secondary decision tree to adjust and process standardized government data of various categories, further improving the reliability of government data governance, improving the accuracy of government data adjustment, and greatly reducing errors in the process of government data governance. The reliability of government data classification governance is improved, so that the classification governance of government data can achieve more ideal results. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 It is a flowchart of a classification management method for government data in an embodiment of the present invention;

[0051] Figure 2 It is a schematic diagram of the structural composition of the classification and management device for government data in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] Embodiment 1

[0054] See also Figure 1 , Figure 1 It is a flow chart of the classification management method of government data in an embodiment of the present invention.

[0055] like Figure 1 As shown, a classification management method for government data includes:

[0056] S11: obtaining government data from a plurality of data sources, and performing data cleaning on the government data from the plurality of data sources to obtain government data after data cleaning;

[0057] In the specific implementation process of the present invention, the government data in several data sources are cleansed to obtain the government data after data cleaning, including: filtering the government data in several data sources to obtain the filtered government data; deduplicating the filtered government data to obtain the government data after data cleaning.

[0058] Specifically, the corresponding government data is obtained from different data platforms and systems, such as the comprehensive management system of urban management cases and the monitoring and analysis platform of urban operation status and vital signs. The government data in several data sources are filtered and processed. The filtering rules are defined in the configuration file through rule definition. The multi-threaded use period calculates the operating capacity of the hardware system to obtain the optimal number of threads. According to the optimal number of threads, the government data of each data source is sent to the filtering queue for filtering and processing using the filtering rules to obtain the filtered government data. The filtered government data is deduplicated, and the filtered government data is generated by using similar hash functions and hash functions to generate binary strings, and the arrangement order of the binary strings is adjusted. The corresponding sets are generated with different sub-binary strings as the front-end binary strings and stored in the preset sample library. The sample library is matched with the front-end binary string of each set to obtain the candidate results of each set returned by the sample library. The Hamming distance of any two government data is calculated according to each candidate result. If the Hamming distance is less than or equal to the threshold, deduplication is performed. After the deduplication is completed, the government data after data cleaning can be obtained.

[0059] S12: Standardizing the government data after data cleaning to obtain standardized government data;

[0060] In the specific implementation process of the present invention, the government data after data cleaning is standardized to obtain standardized government data, including: detecting key data information in the government data after data cleaning, and obtaining a target mapping relationship corresponding to the government data after data cleaning based on the key data information; mapping the government data after data cleaning to standardized government data based on the target mapping relationship.

[0061] Specifically, the government data after data cleaning is input into a pre-trained detection model so that the detection model can detect key information. The detection model is a convergence model obtained by inputting a sample data set into a deep neural network for training. The key data information in the government data is output through the detection model. Based on the key data information, a target mapping relationship corresponding to the government data after data cleaning is obtained, data features of the key data information are extracted, feature levels corresponding to the data features are determined, and several alternative mapping relationships corresponding to the government data are obtained based on the data features and feature levels. The confidence corresponding to each alternative mapping relationship is calculated based on a probability distribution function, and the alternative mapping relationship with the highest confidence is used as the target mapping relationship corresponding to the government data after data cleaning. Based on the target mapping relationship, the government data after data cleaning is mapped to standardized government data, and the government data after data cleaning is mapped to standardized government data in the corresponding mapping space through the target mapping relationship.

[0062] S13: extracting item features from the standardized government affairs data to obtain corresponding item feature information, and extracting hot keywords from the standardized government affairs data to obtain corresponding target hot keywords;

[0063] In the specific implementation process of the present invention, the matter feature extraction of the standardized government data to obtain corresponding matter feature information includes: performing word segmentation processing on the standardized government data to obtain a corresponding number of target word segmentations; performing sensitive word removal processing on the several target word segmentations to obtain a number of target word segmentations after the sensitive word removal processing; determining a number of keywords based on the several target word segmentations after the sensitive word removal processing using the word frequency-reverse file frequency algorithm, and generating corresponding matter feature information based on the several keywords.

[0064] Furthermore, the hot keyword extraction of the standardized government data to obtain corresponding target hot keywords includes: dividing the standardized government data into several government data subsets, calculating the amount of interactive information of the standardized government data in each government data subset, and calculating a heat evaluation index based on the amount of interactive information; extracting several heat keywords of the standardized government data based on the heat evaluation index, obtaining recommended nodes of a preset target ternary tree, and obtaining corresponding branch nodes and branch node frequencies based on the recommended nodes; and determining corresponding target hot keywords using several heat keywords based on the branch nodes and branch node frequencies.

[0065] Specifically, the standardized government data is segmented, the frequency of adjacent characters is used to reflect the reliability of the word, the frequency of the combination of adjacent characters in the corpus is counted, and when the combination frequency is higher than the critical value, it can be considered that the combination of characters will constitute a word, thereby obtaining a corresponding number of target segmented words. Sensitive word removal processing is performed on a number of target segmented words, a dynamic sensitive word library is constructed, an update cycle is set, and the dynamic sensitive word library is updated regularly. The first pinyin letter of the first character of the target segmented word is selected from the dynamic sensitive word library. Starting from the first character of the segmented word, the sensitive word matching the first character of the target segmented word is selected from the corresponding sensitive word category, and then the sensitive word matching the next character of the target segmented word is continuously selected from the selected sensitive words until the sensitive word matching the last character of the target segmented word is selected. It is detected whether there is a sensitive word with the same number of segmented characters among the selected sensitive words matching the last character of the target segmented word. If there is, it means that the target segmented word is a sensitive word, and a number of target segmented words after sensitive word removal processing are obtained. Based on several target segmented words after sensitive words are removed, several keywords are determined by using the word frequency-reverse file frequency algorithm, and the frequency and reverse file frequency value of each target segmented word in the standardized government data are calculated by the word frequency-reverse file frequency algorithm. The frequency and reverse file frequency value of each target segmented word in the standardized government data constitute an importance value, and the segmented words whose importance values ​​meet the preset threshold are used as keywords, and corresponding matter feature information is generated based on several keywords, and each keyword is composed of a keyword group, and the keyword group is used as the corresponding matter feature information. The standardized government data is divided into several government data subsets, and the number of interactive information of the standardized government data in each government data subset is calculated. The interactive information of the standardized government data can be comment information and forwarding information in the standardized government data, etc., the number of interactive information is calculated, and the heat evaluation index is calculated based on the number of interactive information. The heat evaluation index is calculated by combining the number of interactive information with a preset weight ratio. The preset weight ratio can be learned from the database, and the database stores the weight ratio corresponding to the number of interactive information. Based on the heat evaluation index, several heat keywords of the standardized government data are extracted, and a subset of government data whose heat evaluation index meets the preset threshold is screened, and the keywords of the subset of government data are used as heat keywords. The recommended nodes of the preset target ternary tree are obtained, and the preset ternary tree includes a root node and several branches, each branch includes several branch nodes, and each node stores at least one character. In the adjacent two-level branch nodes, the path between the upper-level branch node and the lower-level branch node represents the frequency of the lower-level branch node. The recommended nodes of the preset ternary tree are obtained through heat keywords; based on the recommended nodes, the corresponding branch nodes and branch node frequencies are obtained. After obtaining the recommended nodes, the corresponding branch nodes and branch node frequencies can be directly known through the ternary tree.Based on the branch nodes and branch node frequencies, several hot keywords are used to determine corresponding target hot keywords, and hot branch nodes in the corresponding branch nodes are obtained. The hot branch nodes are branch nodes whose branch node frequencies exceed a preset threshold within the most recent preset time period. The corresponding target hot keywords are determined from the several hot keywords based on the hot branch nodes.

[0066] S14: Classifying the standardized government affairs data using a clustering algorithm based on the event feature information and hot keywords to obtain standardized government affairs data of each category;

[0067] In the specific implementation process of the present invention, the standardized government data is classified by using a clustering algorithm based on the matter feature information and hot keywords to obtain standardized government data of various categories, including: extracting samples from the standardized government data to obtain a sample subset, and generating a number of cluster centers based on the sample subset; establishing an association mapping of each cluster center based on the matter feature information and hot keywords, and generating linearly correlated cluster centers based on the association mapping of each cluster center; classifying the standardized government data based on the linearly correlated cluster centers to obtain standardized government data of various categories.

[0068] Specifically, samples are extracted from the standardized government data to obtain a sample subset, and several cluster centers are generated based on the sample subset. The Chernoff boundary is used to determine the minimum amount of sampling data, and several cluster centers are obtained based on the minimum amount of sampling data combined with a hierarchical clustering algorithm. An association mapping of each cluster center is established based on the event feature information and hot keywords, a cluster center space vector is generated based on the event feature information and hot keywords, the distance between the cluster center space vectors is calculated, the average value is calculated based on the distance between the cluster center space vectors, a mapping relationship is established based on the average value using a preset rule mapping table, and an association mapping of each cluster center is established based on the established mapping relationship. Linearly related cluster centers are generated based on the association mapping of each cluster center, a density matrix and a distance matrix are calculated based on the association mapping of each cluster center, and linearly related cluster centers are generated based on the density matrix and the distance matrix combined with a multiple linear regression analysis method. The standardized government data is classified based on the linear correlation cluster center, and the standardized government data of the same category is classified into the corresponding category through the linear correlation cluster center to obtain standardized government data of each category. The standardized government data of each category may include data of ecological environment, urban events and public services, and the data can also be classified into heat categories in the divided categories through hot keywords. The standardized government data is classified by clustering algorithm through the feature information of the matter and the hot keywords, so as to avoid the lack of reliability of the classification of government data by single feature information, improve the classification efficiency of government data, and ensure the stability of the classification of government data.

[0069] S15: performing quality analysis on standardized government data of each category based on a preset data quality analysis strategy to obtain corresponding quality analysis results, generating a data processing adjustment strategy based on the corresponding quality analysis results using a secondary decision tree, and performing adjustment processing on standardized government data of each category based on the data processing adjustment strategy to obtain adjusted standardized government data of each category;

[0070] In the specific implementation process of the present invention, quality analysis is performed on each category of standardized government data based on a preset data quality analysis strategy to obtain corresponding quality analysis results, including: selecting corresponding data quality analysis rules in the preset data quality analysis strategy for each category of standardized government data; quality analysis is performed on each category of standardized government data based on the data quality analysis rules to obtain quality analysis results corresponding to each category of standardized government data.

[0071] Furthermore, the method of generating a data processing adjustment strategy based on the corresponding quality analysis results using a secondary decision tree includes: inputting the quality analysis results into a preset secondary decision tree, wherein the first-level decision tree in the preset secondary decision tree is used to preprocess the quality analysis results to obtain preprocessed quality analysis results; the second-level decision tree in the preset secondary decision tree is used to evaluate based on the preprocessed quality analysis results to obtain evaluation results, and generating a data processing adjustment strategy based on the evaluation results.

[0072] Specifically, corresponding data quality analysis rules are selected from the preset data quality analysis strategy for each category of standardized government data, and the preset data quality analysis strategy includes quality analysis rules corresponding to each category of standardized government data. Based on the data quality analysis rules, quality analysis is performed on each category of standardized government data to obtain quality analysis results corresponding to each category of standardized government data. The quality analysis results are input into a preset secondary decision tree, wherein the first level decision tree in the preset secondary decision tree is used to preprocess the quality analysis results to obtain the preprocessed quality analysis results, and the preprocessing includes data cleaning and data reduction processing. The second-level decision tree in the preset secondary decision tree is used to evaluate based on the quality analysis result after preprocessing. The corresponding branch is selected in the second-level decision tree for the quality analysis result after preprocessing, and the entropy of the corresponding branch is calculated. The arithmetic mean entropy is calculated according to the entropy of the corresponding branch. The attribute that produces the maximum entropy reduction is selected according to the arithmetic mean entropy. The recommended node in the branch is obtained according to the attribute of the maximum entropy reduction. The recommended node contains corresponding evaluation information, that is, the evaluation result is obtained, and a data processing adjustment strategy is generated based on the evaluation result. The corresponding data processing adjustment strategy is obtained in the local database according to the evaluation information. The data processing adjustment strategy may include a data cleaning threshold and a data standardization threshold. Based on the data processing adjustment strategy, the standardized government data of each category is adjusted, and the standardized government data of each category is cleaned and standardized again through the data processing adjustment strategy, and outlier filling and other processes are performed to obtain the adjusted standardized government data of each category.

[0073] S16: Integrate the adjusted standardized government data of various categories to obtain data integration results, and store the data integration results based on distributed storage technology.

[0074] In the specific implementation process of the present invention, the data integration of the adjusted standardized government data of various categories is performed to obtain data integration results, and the data integration results are stored based on distributed storage technology, including: determining the data integration scope and data integration target, and integrating the adjusted standardized government data of various categories based on the data integration scope and data integration target based on ETL technology to obtain data integration results; calculating a hash index based on a hash algorithm, calculating a target key value based on the hash index, converting the target key value to obtain a target hash value, and determining a split partition based on the target hash value using the InnoDB engine and the mongos function; horizontally splitting the data integration result based on the split partition to obtain a data integration result after horizontal splitting; and storing the data integration result after horizontal splitting in the corresponding storage node in the distributed database.

[0075] Specifically, determine the scope and target of data integration, clarify the data tables and fields that need to be integrated in the government data, define the integration rules, and use the data integration scope and data integration target based on the Extract-Transform-Load (ETL) technology to integrate the standardized government data of various categories after adjustment, select the target component, the target component includes a collection component, an aggregation component, a sorting component, and a connection component, etc., and use the target component through the ETL technology to integrate the standardized government data of various categories into a unified data warehouse to obtain the data integration result. Calculate the hash index based on the hash algorithm, calculate the target key value based on the hash index, perform a conversion operation on the target key value, that is, convert the key value into a new hash value, obtain the target hash value, and determine the split partition based on the target hash value using the InnoDB engine and the mongos function, and use the InnoDB engine to perform adaptive index optimization and creation to obtain the hash index. The hash index can be used to quickly and accurately determine the split partition. After determining the split partition, use the mongos function to directly locate the split partition, and then determine the split partition. The data integration result is horizontally segmented based on the segmentation and partitioning. The data storage can be elastically expanded horizontally through data segmentation to improve storage performance and obtain the data integration result after horizontal segmentation. The data integration result after horizontal segmentation is stored in the corresponding storage node in the distributed database, and the insert function is used to add elements to the hash map. After the elements are added to the hash map, the specific value of the key is obtained, that is, the volume identifier is obtained. The storage node can be accurately and quickly determined by the volume identifier, and the data integration result after horizontal segmentation is stored in the corresponding storage node in the distributed database. At this point, the classified governance of government data is completed.

[0076] In an embodiment of the present invention, the standardized government data is subjected to item feature extraction to obtain corresponding item feature information, and the standardized government data is subjected to hot keyword extraction in combination with a preset ternary tree and a heat evaluation index to obtain corresponding hot keywords, so that the obtained item feature information and hot keywords are more reliable. The standardized government data is classified using a clustering algorithm based on the item feature information and hot keywords, thereby avoiding the insufficient reliability of the classification of government data by single item feature information, improving the classification efficiency of government data, and ensuring the stability of the classification of government data. Based on the quality analysis results, a data processing adjustment strategy is generated using a secondary decision tree to adjust and process standardized government data of various categories, further improving the reliability of government data governance, improving the accuracy of government data adjustment, and greatly reducing errors in the process of government data governance. The reliability of government data classification governance is improved, so that the classification governance of government data can achieve more ideal results.

[0077] Embodiment 2

[0078] See also Figure 2 , Figure 2 It is a schematic diagram of the structural composition of the classification and management device for government data in an embodiment of the present invention.

[0079] like Figure 2 As shown, a classification management device for government data includes:

[0080] Data cleaning module 21: used to obtain government data from several data sources, and perform data cleaning on the government data from several data sources to obtain the government data after data cleaning;

[0081] In the specific implementation process of the present invention, the government data in several data sources are cleansed to obtain the government data after data cleaning, including: filtering the government data in several data sources to obtain the filtered government data; deduplicating the filtered government data to obtain the government data after data cleaning.

[0082] Specifically, the corresponding government data is obtained from different data platforms and systems, such as the comprehensive management system of urban management cases and the monitoring and analysis platform of urban operation status and vital signs. The government data in several data sources are filtered and processed. The filtering rules are defined in the configuration file through rule definition. The multi-threaded use period calculates the operating capacity of the hardware system to obtain the optimal number of threads. According to the optimal number of threads, the government data of each data source is sent to the filtering queue for filtering and processing using the filtering rules to obtain the filtered government data. The filtered government data is deduplicated, and the filtered government data is generated by using similar hash functions and hash functions to generate binary strings, and the arrangement order of the binary strings is adjusted. The corresponding sets are generated with different sub-binary strings as the front-end binary strings and stored in the preset sample library. The sample library is matched with the front-end binary string of each set to obtain the candidate results of each set returned by the sample library. The Hamming distance of any two government data is calculated according to each candidate result. If the Hamming distance is less than or equal to the threshold, deduplication is performed. After the deduplication is completed, the government data after data cleaning can be obtained.

[0083] Data standardization module 22: used to standardize the government data after data cleaning to obtain standardized government data;

[0084] In the specific implementation process of the present invention, the government data after data cleaning is standardized to obtain standardized government data, including: detecting key data information in the government data after data cleaning, and obtaining a target mapping relationship corresponding to the government data after data cleaning based on the key data information; mapping the government data after data cleaning to standardized government data based on the target mapping relationship.

[0085] Specifically, the government data after data cleaning is input into a pre-trained detection model so that the detection model can detect key information. The detection model is a convergence model obtained by inputting a sample data set into a deep neural network for training. The key data information in the government data is output through the detection model. Based on the key data information, a target mapping relationship corresponding to the government data after data cleaning is obtained, data features of the key data information are extracted, feature levels corresponding to the data features are determined, and several alternative mapping relationships corresponding to the government data are obtained based on the data features and feature levels. The confidence corresponding to each alternative mapping relationship is calculated based on a probability distribution function, and the alternative mapping relationship with the highest confidence is used as the target mapping relationship corresponding to the government data after data cleaning. Based on the target mapping relationship, the government data after data cleaning is mapped to standardized government data, and the government data after data cleaning is mapped to standardized government data in the corresponding mapping space through the target mapping relationship.

[0086] The item feature information and hot keyword extraction module 23 is used to extract item features from the standardized government affairs data to obtain corresponding item feature information, and to extract hot keywords from the standardized government affairs data to obtain corresponding target hot keywords;

[0087] In the specific implementation process of the present invention, the matter feature extraction of the standardized government data to obtain corresponding matter feature information includes: performing word segmentation processing on the standardized government data to obtain a corresponding number of target word segmentations; performing sensitive word removal processing on the several target word segmentations to obtain a number of target word segmentations after the sensitive word removal processing; determining a number of keywords based on the several target word segmentations after the sensitive word removal processing using the word frequency-reverse file frequency algorithm, and generating corresponding matter feature information based on the several keywords.

[0088] Furthermore, the hot keyword extraction of the standardized government data to obtain corresponding target hot keywords includes: dividing the standardized government data into several government data subsets, calculating the amount of interactive information of the standardized government data in each government data subset, and calculating a heat evaluation index based on the amount of interactive information; extracting several heat keywords of the standardized government data based on the heat evaluation index, obtaining recommended nodes of a preset target ternary tree, and obtaining corresponding branch nodes and branch node frequencies based on the recommended nodes; and determining corresponding target hot keywords using several heat keywords based on the branch nodes and branch node frequencies.

[0089] Specifically, the standardized government data is segmented, the frequency of adjacent characters is used to reflect the reliability of the word, the frequency of the combination of adjacent characters in the corpus is counted, and when the combination frequency is higher than the critical value, it can be considered that the combination of characters will constitute a word, thereby obtaining a corresponding number of target segmented words. Sensitive word removal processing is performed on a number of target segmented words, a dynamic sensitive word library is constructed, an update cycle is set, and the dynamic sensitive word library is updated regularly. The first pinyin letter of the first character of the target segmented word is selected from the dynamic sensitive word library. Starting from the first character of the segmented word, the sensitive word matching the first character of the target segmented word is selected from the corresponding sensitive word category, and then the sensitive word matching the next character of the target segmented word is continuously selected from the selected sensitive words until the sensitive word matching the last character of the target segmented word is selected. It is detected whether there is a sensitive word with the same number of segmented characters among the selected sensitive words matching the last character of the target segmented word. If there is, it means that the target segmented word is a sensitive word, and a number of target segmented words after sensitive word removal processing are obtained. Based on several target segmented words after sensitive words are removed, several keywords are determined by using the word frequency-reverse file frequency algorithm, and the frequency and reverse file frequency value of each target segmented word in the standardized government data are calculated by the word frequency-reverse file frequency algorithm. The frequency and reverse file frequency value of each target segmented word in the standardized government data constitute an importance value, and the segmented words whose importance values ​​meet the preset threshold are used as keywords, and corresponding matter feature information is generated based on several keywords, and each keyword is composed of a keyword group, and the keyword group is used as the corresponding matter feature information. The standardized government data is divided into several government data subsets, and the number of interactive information of the standardized government data in each government data subset is calculated. The interactive information of the standardized government data can be comment information and forwarding information in the standardized government data, etc., the number of interactive information is calculated, and the heat evaluation index is calculated based on the number of interactive information. The heat evaluation index is calculated by combining the number of interactive information with a preset weight ratio. The preset weight ratio can be learned from the database, and the database stores the weight ratio corresponding to the number of interactive information. Based on the heat evaluation index, several heat keywords of the standardized government data are extracted, and a subset of government data whose heat evaluation index meets the preset threshold is screened, and the keywords of the subset of government data are used as heat keywords. The recommended nodes of the preset target ternary tree are obtained, and the preset ternary tree includes a root node and several branches, each branch includes several branch nodes, and each node stores at least one character. In the adjacent two-level branch nodes, the path between the upper-level branch node and the lower-level branch node represents the frequency of the lower-level branch node. The recommended nodes of the preset ternary tree are obtained through heat keywords; based on the recommended nodes, the corresponding branch nodes and branch node frequencies are obtained. After obtaining the recommended nodes, the corresponding branch nodes and branch node frequencies can be directly known through the ternary tree.Based on the branch nodes and branch node frequencies, several hot keywords are used to determine corresponding target hot keywords, and hot branch nodes in the corresponding branch nodes are obtained. The hot branch nodes are branch nodes whose branch node frequencies exceed a preset threshold within the most recent preset time period. The corresponding target hot keywords are determined from the several hot keywords based on the hot branch nodes.

[0090] Data classification module 24: used to classify the standardized government data using a clustering algorithm based on the event feature information and hot keywords to obtain standardized government data of various categories;

[0091] In the specific implementation process of the present invention, the standardized government data is classified by using a clustering algorithm based on the matter feature information and hot keywords to obtain standardized government data of various categories, including: extracting samples from the standardized government data to obtain a sample subset, and generating a number of cluster centers based on the sample subset; establishing an association mapping of each cluster center based on the matter feature information and hot keywords, and generating linearly correlated cluster centers based on the association mapping of each cluster center; classifying the standardized government data based on the linearly correlated cluster centers to obtain standardized government data of various categories.

[0092] Specifically, samples are extracted from the standardized government data to obtain a sample subset, and several cluster centers are generated based on the sample subset. The Chernoff boundary is used to determine the minimum amount of sampling data, and several cluster centers are obtained based on the minimum amount of sampling data combined with a hierarchical clustering algorithm. An association mapping of each cluster center is established based on the event feature information and hot keywords, a cluster center space vector is generated based on the event feature information and hot keywords, the distance between the cluster center space vectors is calculated, the average value is calculated based on the distance between the cluster center space vectors, a mapping relationship is established based on the average value using a preset rule mapping table, and an association mapping of each cluster center is established based on the established mapping relationship. Linearly related cluster centers are generated based on the association mapping of each cluster center, a density matrix and a distance matrix are calculated based on the association mapping of each cluster center, and linearly related cluster centers are generated based on the density matrix and the distance matrix combined with a multiple linear regression analysis method. The standardized government data is classified based on the linear correlation cluster center, and the standardized government data of the same category is classified into the corresponding category through the linear correlation cluster center to obtain standardized government data of each category. The standardized government data of each category may include data of ecological environment, urban events and public services, and the data can also be classified into heat categories in the divided categories through hot keywords. The standardized government data is classified by clustering algorithm through the feature information of the matter and the hot keywords, so as to avoid the lack of reliability of the classification of government data by single feature information, improve the classification efficiency of government data, and ensure the stability of the classification of government data.

[0093] The quality analysis and adjustment module 25 is used to perform quality analysis on standardized government data of various categories based on a preset data quality analysis strategy, obtain corresponding quality analysis results, generate data processing adjustment strategies based on the corresponding quality analysis results using a secondary decision tree, and perform adjustment processing on standardized government data of various categories based on the data processing adjustment strategies to obtain adjusted standardized government data of various categories;

[0094] In the specific implementation process of the present invention, quality analysis is performed on each category of standardized government data based on a preset data quality analysis strategy to obtain corresponding quality analysis results, including: selecting corresponding data quality analysis rules in the preset data quality analysis strategy for each category of standardized government data; quality analysis is performed on each category of standardized government data based on the data quality analysis rules to obtain quality analysis results corresponding to each category of standardized government data.

[0095] Furthermore, the method of generating a data processing adjustment strategy based on the corresponding quality analysis results using a secondary decision tree includes: inputting the quality analysis results into a preset secondary decision tree, wherein the first-level decision tree in the preset secondary decision tree is used to preprocess the quality analysis results to obtain preprocessed quality analysis results; the second-level decision tree in the preset secondary decision tree is used to evaluate based on the preprocessed quality analysis results to obtain evaluation results, and generating a data processing adjustment strategy based on the evaluation results.

[0096] Specifically, corresponding data quality analysis rules are selected from the preset data quality analysis strategy for each category of standardized government data, and the preset data quality analysis strategy includes quality analysis rules corresponding to each category of standardized government data. Based on the data quality analysis rules, quality analysis is performed on each category of standardized government data to obtain quality analysis results corresponding to each category of standardized government data. The quality analysis results are input into a preset secondary decision tree, wherein the first level decision tree in the preset secondary decision tree is used to preprocess the quality analysis results to obtain the preprocessed quality analysis results, and the preprocessing includes data cleaning and data reduction processing. The second-level decision tree in the preset secondary decision tree is used to evaluate based on the quality analysis result after preprocessing. The corresponding branch is selected in the second-level decision tree for the quality analysis result after preprocessing, and the entropy of the corresponding branch is calculated. The arithmetic mean entropy is calculated according to the entropy of the corresponding branch. The attribute that produces the maximum entropy reduction is selected according to the arithmetic mean entropy. The recommended node in the branch is obtained according to the attribute of the maximum entropy reduction. The recommended node contains corresponding evaluation information, that is, the evaluation result is obtained, and a data processing adjustment strategy is generated based on the evaluation result. The corresponding data processing adjustment strategy is obtained in the local database according to the evaluation information. The data processing adjustment strategy may include a data cleaning threshold and a data standardization threshold. Based on the data processing adjustment strategy, the standardized government data of each category is adjusted, and the standardized government data of each category is cleaned and standardized again through the data processing adjustment strategy, and outlier filling and other processes are performed to obtain the adjusted standardized government data of each category.

[0097] Data integration and storage module 26: used to integrate the adjusted standardized government data of various categories, obtain data integration results, and store the data integration results based on distributed storage technology.

[0098] In the specific implementation process of the present invention, the data integration of the adjusted standardized government data of various categories is performed to obtain data integration results, and the data integration results are stored based on distributed storage technology, including: determining the data integration scope and data integration target, and integrating the adjusted standardized government data of various categories based on the data integration scope and data integration target based on ETL technology to obtain data integration results; calculating a hash index based on a hash algorithm, calculating a target key value based on the hash index, converting the target key value to obtain a target hash value, and determining a split partition based on the target hash value using the InnoDB engine and the mongos function; horizontally splitting the data integration result based on the split partition to obtain a data integration result after horizontal splitting; and storing the data integration result after horizontal splitting in the corresponding storage node in the distributed database.

[0099] Specifically, determine the scope and target of data integration, clarify the data tables and fields that need to be integrated in the government data, define the integration rules, and use the data integration scope and data integration target based on the Extract-Transform-Load (ETL) technology to integrate the standardized government data of various categories after adjustment, select the target component, the target component includes a collection component, an aggregation component, a sorting component, and a connection component, etc., and use the target component through the ETL technology to integrate the standardized government data of various categories into a unified data warehouse to obtain the data integration result. Calculate the hash index based on the hash algorithm, calculate the target key value based on the hash index, perform a conversion operation on the target key value, that is, convert the key value into a new hash value, obtain the target hash value, and determine the split partition based on the target hash value using the InnoDB engine and the mongos function, and use the InnoDB engine to perform adaptive index optimization and creation to obtain the hash index. The hash index can be used to quickly and accurately determine the split partition. After determining the split partition, use the mongos function to directly locate the split partition, and then determine the split partition. The data integration result is horizontally segmented based on the segmentation and partitioning. The data storage can be elastically expanded horizontally through data segmentation to improve storage performance and obtain the data integration result after horizontal segmentation. The data integration result after horizontal segmentation is stored in the corresponding storage node in the distributed database, and the insert function is used to add elements to the hash map. After the elements are added to the hash map, the specific value of the key is obtained, that is, the volume identifier is obtained. The storage node can be accurately and quickly determined by the volume identifier, and the data integration result after horizontal segmentation is stored in the corresponding storage node in the distributed database. At this point, the classified governance of government data is completed.

[0100] In an embodiment of the present invention, the standardized government data is subjected to item feature extraction to obtain corresponding item feature information, and the standardized government data is subjected to hot keyword extraction in combination with a preset ternary tree and a heat evaluation index to obtain corresponding hot keywords, so that the obtained item feature information and hot keywords are more reliable. The standardized government data is classified using a clustering algorithm based on the item feature information and hot keywords, thereby avoiding the insufficient reliability of the classification of government data by single item feature information, improving the classification efficiency of government data, and ensuring the stability of the classification of government data. Based on the quality analysis results, a data processing adjustment strategy is generated using a secondary decision tree to adjust and process standardized government data of various categories, further improving the reliability of government data governance, improving the accuracy of government data adjustment, and greatly reducing errors in the process of government data governance. The reliability of government data classification governance is improved, so that the classification governance of government data can achieve more ideal results.

[0101] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0102] In addition, the above is a detailed introduction to a method and device for classifying and managing government data provided by an embodiment of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A classification management method for government data, characterized in that: The method comprises: Acquire government data from several data sources, and perform data cleaning on the government data from several data sources to obtain the government data after data cleaning; Standardize the government data after data cleaning to obtain standardized government data; Extracting item features from the standardized government affairs data to obtain corresponding item feature information, and extracting hot keywords from the standardized government affairs data to obtain corresponding target hot keywords; Based on the feature information of the items and the hot keywords, the standardized government affairs data are classified by using a clustering algorithm to obtain standardized government affairs data of each category; Based on the preset data quality analysis strategy, quality analysis is performed on each category of standardized government data to obtain corresponding quality analysis results, and based on the corresponding quality analysis results, a data processing adjustment strategy is generated using a secondary decision tree, and based on the data processing adjustment strategy, each category of standardized government data is adjusted and processed to obtain adjusted standardized government data of each category; Integrate the adjusted standardized government data of various categories to obtain data integration results, and store the data integration results based on distributed storage technology; The step of extracting item features from the standardized government data to obtain corresponding item feature information includes: performing word segmentation processing on the standardized government data to obtain a number of corresponding target word segments; performing sensitive word removal processing on the target word segments to obtain a number of target word segments after the sensitive word removal processing; determining a number of keywords based on the target word segments after the sensitive word removal processing using a word frequency-inverse file frequency algorithm, and generating corresponding item feature information based on the number of keywords; The step of extracting hot keywords from the standardized government data to obtain corresponding target hot keywords includes: dividing the standardized government data into a plurality of government data subsets, calculating the amount of interactive information of the standardized government data in each government data subset, and calculating a heat evaluation index based on the amount of interactive information; extracting a plurality of heat keywords of the standardized government data based on the heat evaluation index; obtaining a recommended node of a preset target ternary tree, and obtaining a corresponding branch node and branch node frequency based on the recommended node; and determining a corresponding target hot keyword using a plurality of heat keywords based on the branch node and the branch node frequency; The method of classifying the standardized government affairs data by using a clustering algorithm based on the feature information of the matters and hot keywords to obtain standardized government affairs data of various categories includes: extracting samples from the standardized government affairs data to obtain a sample subset, and generating a plurality of cluster centers based on the sample subset; establishing an association mapping of each cluster center based on the feature information of the matters and hot keywords, and generating linearly correlated cluster centers based on the association mapping of each cluster center; and classifying the standardized government affairs data based on the linearly correlated cluster centers to obtain standardized government affairs data of various categories.

2. The classification management method of government data according to claim 1 is characterized in that: The step of performing data cleaning on the government data in the plurality of data sources to obtain the government data after data cleaning includes: Filtering the government data in several data sources to obtain filtered government data; The filtered government data is deduplicated to obtain the government data after data cleaning.

3. The classification management method of government data according to claim 1 is characterized in that: The step of standardizing the government data after data cleaning to obtain standardized government data includes: Detecting key data information in the government data after data cleaning, and acquiring a target mapping relationship corresponding to the government data after data cleaning based on the key data information; Based on the target mapping relationship, the government data after data cleaning is mapped into standardized government data.

4. The classification management method of government data according to claim 1 is characterized in that: The quality analysis of standardized government data of various categories is performed based on the preset data quality analysis strategy to obtain corresponding quality analysis results, including: Select corresponding data quality analysis rules from the preset data quality analysis strategy for each category of standardized government data; Based on the data quality analysis rules, quality analysis is performed on each category of standardized government data to obtain quality analysis results corresponding to each category of standardized government data.

5. The classification management method of government data according to claim 1 is characterized in that: The method of generating a data processing adjustment strategy based on the corresponding quality analysis result using a secondary decision tree includes: Inputting the quality analysis result into a preset secondary decision tree, wherein the first level decision tree in the preset secondary decision tree is used to preprocess the quality analysis result to obtain the preprocessed quality analysis result; The second level decision tree in the preset secondary decision tree is used to perform evaluation based on the quality analysis result after preprocessing, obtain evaluation results, and generate a data processing adjustment strategy based on the evaluation results.

6. The classification management method of government data according to claim 1 is characterized in that: The step of integrating the adjusted standardized government affairs data of various categories to obtain data integration results, and storing the data integration results based on distributed storage technology, includes: Determine the data integration scope and data integration target, and integrate the adjusted standardized government data of various categories using the data integration scope and data integration target based on ETL technology to obtain data integration results; Calculate a hash index based on a hash algorithm, calculate a target key value based on the hash index, perform a conversion operation on the target key value to obtain a target hash value, and determine a split partition based on the target hash value using an InnoDB engine and a mongos function; Based on the split partitions, the data integration result is horizontally split to obtain the data integration result after the horizontal split; The data integration results after horizontal segmentation are stored in the corresponding storage nodes in the distributed database.

7. A classification management device for government data, characterized in that: The device comprises: Data cleaning module: used to obtain government data from several data sources, and perform data cleaning on the government data from several data sources to obtain the government data after data cleaning; Data standardization module: used to standardize the government data after data cleaning to obtain standardized government data; Item feature information and hot keyword extraction module: used to extract item features from the standardized government affairs data to obtain corresponding item feature information, and to extract hot keywords from the standardized government affairs data to obtain corresponding target hot keywords; Data classification module: used to classify the standardized government data using a clustering algorithm based on the feature information of the matter and hot keywords to obtain standardized government data of various categories; Quality analysis and adjustment module: used to perform quality analysis on standardized government data of various categories based on a preset data quality analysis strategy, obtain corresponding quality analysis results, generate data processing adjustment strategies based on the corresponding quality analysis results using a secondary decision tree, perform adjustment processing on standardized government data of various categories based on the data processing adjustment strategies, and obtain standardized government data of various categories after adjustment; Data integration and storage module: used to integrate the adjusted standardized government data of various categories, obtain data integration results, and store the data integration results based on distributed storage technology; The step of extracting item features from the standardized government data to obtain corresponding item feature information includes: performing word segmentation processing on the standardized government data to obtain a number of corresponding target word segments; performing sensitive word removal processing on the target word segments to obtain a number of target word segments after the sensitive word removal processing; determining a number of keywords based on the target word segments after the sensitive word removal processing using a word frequency-inverse file frequency algorithm, and generating corresponding item feature information based on the number of keywords; The step of extracting hot keywords from the standardized government data to obtain corresponding target hot keywords includes: dividing the standardized government data into a plurality of government data subsets, calculating the amount of interactive information of the standardized government data in each government data subset, and calculating a heat evaluation index based on the amount of interactive information; extracting a plurality of heat keywords of the standardized government data based on the heat evaluation index; obtaining a recommended node of a preset target ternary tree, and obtaining a corresponding branch node and branch node frequency based on the recommended node; and determining a corresponding target hot keyword using a plurality of heat keywords based on the branch node and the branch node frequency; The method of classifying the standardized government affairs data by using a clustering algorithm based on the feature information of the matters and hot keywords to obtain standardized government affairs data of various categories includes: extracting samples from the standardized government affairs data to obtain a sample subset, and generating a plurality of cluster centers based on the sample subset; establishing an association mapping of each cluster center based on the feature information of the matters and hot keywords, and generating linearly correlated cluster centers based on the association mapping of each cluster center; and classifying the standardized government affairs data based on the linearly correlated cluster centers to obtain standardized government affairs data of various categories.