Government affair data processing intelligent analysis decision-making method
By collecting, preprocessing and integrating government data, and using Text Rank algorithm and decision tree model to achieve automated decision support, it solves the problem that traditional methods are difficult to efficiently process government data, and improves the scientificity and accuracy of decision-making.
Patent Information
- Application Number
- CN202411774571.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-05
- Publication Date
- 2025-03-14
AI Technical Summary
Traditional data processing methods are difficult to efficiently process huge government data and cannot provide fast and accurate decision-making support.
By collecting government data from different sources, preprocessing and integrating them, using the Text Rank algorithm to extract key information, build a decision tree model for training and evaluation, and realize automated decision support.
It realizes efficient processing and intelligent analysis of government data, improves the scientificity and accuracy of intelligent analysis decisions, and promotes collaboration and data sharing among various departments.
Smart Images
Figure CN119622536A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent analysis technology, and in particular to a method for intelligent analysis and decision-making in government data processing. Background Art
[0002] In today's information age, the processing and analysis of government data has become an indispensable part of the government's decision-making process. With the development and application of big data technology, how to efficiently extract valuable information from massive government data has become the key to improving government governance capabilities and service levels. Traditional data processing methods are difficult to efficiently process huge amounts of government data and cannot provide fast and accurate decision support. These problems urgently require the intervention of new technologies to improve the efficiency and intelligence of government data processing.
[0003] The present invention aims to achieve efficient processing and intelligent analysis of government data through artificial intelligence and big data technology, thereby improving the scientificity and accuracy of intelligent analysis and decision-making. The method includes data preprocessing, feature engineering, model training and evaluation, and uses advanced algorithms and technical means to solve the deficiencies in existing technologies and promote the development of government data processing towards intelligence. Summary of the invention
[0004] The purpose of the present invention is to provide a method for intelligent analysis and decision-making of government data processing to overcome the deficiencies in the background technology.
[0005] The purpose of the present invention can be achieved through the following technical solutions: A method for intelligent analysis and decision-making of government data processing, comprising:
[0006] Step 1: Collect government data, pre-process the government data, and integrate and obtain government processing data;
[0007] Step 2: Use the Text Rank algorithm to identify government affairs processing data, extract key information and combine it into a government affairs information summary;
[0008] Step 3: Divide the government affairs processing data into training data sets and test data sets randomly according to the proportion, build a decision tree model for training and evaluation, until the model reaches the preset indicators;
[0009] Step 4: Integrate the trained model into the existing system, design the user interface, and implement automated decision support.
[0010] Preferably, in step one, government data is collected from different types of data sources, the collected government data is preprocessed, and the preprocessed government data is integrated to obtain government processing data; wherein the data sources include different government department databases, public service platforms, social media, and Internet of Things devices;
[0011] The method for preprocessing the collected government data is as follows:
[0012] The collected government data is preliminarily cleaned, duplicate information in the data is deleted, errors or inconsistencies in the data are checked and corrected, the government data is split into words or phrases, and converted into numerical form using word embedding technology, and integrated to generate government processing data.
[0013] Preferably, in step 2, the specific process of identifying the government affairs processing data includes:
[0014] The words or phrases in the government processing data are marked as nodes, a node connection graph is constructed, the cosine similarity between sentences is used as the edge weight, the node weight is calculated through a loop iteration, the weight of each node is obtained and compared with the preset weight, and the nodes with a weight greater than or equal to the preset weight are marked as key information and combined into a government information summary.
[0015] Preferably, the weight of node i is expressed as:
[0016]
[0017] Where W (t+1) (i) represents the weight of node i at the t+1th iteration, d is the damping coefficient; In(i) represents all nodes connected to node i, Out(j) represents all nodes that can be reached from node j; w ji Represents the correlation between node j and node i.
[0018] Preferably, in step three, the method for constructing a decision tree model is as follows:
[0019] Select the feature with the largest information gain from the features of the training data as the current split node, and recursively generate child nodes by evaluating the importance of the selected feature until the stopping condition is met and a leaf node is obtained;
[0020] Among them, the importance of features is obtained based on information gain. The specific formula is as follows:
[0021] Information gain: IG(D,A)=H(D)-H(DA);
[0022] Where H(D) is the information entropy of the training dataset D, H(D|A) is the conditional entropy of the training dataset D under the condition of feature A; IG(D,A) is the importance of feature A;
[0023] For each leaf node, analyze each decision path from the root node to the leaf node, extract the decision rules represented by the decision path, including backtracking along the decision path, recording the features selected by each internal node, and generating decision recommendations based on the decision rules.
[0024] Preferably, the F1 value is calculated and used as an evaluation index to analyze the decision tree model:
[0025] Count the number of samples TP with correct decisions as positive class, the number FP with wrong decisions as positive class, and the number FN with wrong decisions as negative class to obtain the precision and recall rate of the model; calculate the evaluation index F1 value of the model based on the obtained precision and recall rate of the model;
[0026] The obtained model evaluation index F1 is compared with the preset index. If F1 ≥ the preset index, it means that the model has reached the preset level; if F1 < the preset index, it means that the model parameters need to be adjusted using the labeled data in the training set until the performance of the model on the validation set reaches the preset level.
[0027] Preferably, the calculation formulas for the precision and recall of the model are:
[0028]
[0029] In the formula, P represents the precision of the model, and R represents the recall of the model.
[0030] Preferably, the calculation formula of the evaluation index F1 value of the model is:
[0031] The calculation formula is:
[0032]
[0033] In the formula, F1 is the evaluation index of the model.
[0034] In order to solve the problem, the present invention also discloses a government data processing intelligent analysis and decision-making system, including:
[0035] The data collection and processing module collects government data, pre-processes government data, and integrates and obtains government processing data;
[0036] The data identification and integration module uses the Text Rank algorithm to identify government processing data, extract key information and combine it into a government information summary;
[0037] The model building module randomly divides the government processing data into training data sets and test data sets according to the proportion, builds a decision tree model for training and evaluation, until the model reaches the preset indicators;
[0038] The automatic decision-making module integrates the trained model into the existing system, designs the user interface, and realizes automated decision support.
[0039] Compared with the existing solutions, the present invention achieves the following beneficial effects:
[0040] The present invention collects government data from different sources, integrates data from different channels, ensures the comprehensiveness and representativeness of the data, realizes data sharing and circulation, and promotes collaboration among departments; the data is strictly cleaned and standardized to provide a basis for subsequent analysis and decision support.
[0041] The present invention uses the Text Rank algorithm to extract keywords from government processing data, accurately identifies important information in the text, and generates concise and clear government information summaries based on the extracted keywords, making it convenient for decision makers to quickly understand key content.
[0042] The present invention ensures the representativeness and generalization ability of the data set by randomly dividing the government processing data into training sets and test sets in a certain proportion; the model is evaluated through multiple tests to ensure that its performance on the test set meets the preset indicator requirements.
[0043] The present invention seamlessly integrates the trained decision tree model into the existing system, thereby realizing automated decision support without additional development work, thereby enhancing the flexibility of the system and user experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] The present invention will be further described below in conjunction with the accompanying drawings.
[0045] Figure 1 A flowchart of a method for intelligent analysis and decision-making of government data processing according to the present invention;
[0046] Figure 2 This is a module block diagram of a government data processing intelligent analysis and decision-making system of the present invention. DETAILED DESCRIPTION
[0047] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.
[0048] Embodiment 1, as Figure 1 As shown, a method for intelligent analysis and decision-making of government data processing includes:
[0049] Step 1: Collect government data, pre-process the government data, and integrate and obtain government processing data;
[0050] Step 2: Use the Text Rank algorithm to identify government affairs processing data, extract key information and combine it into a government affairs information summary;
[0051] Step 3: Divide the government affairs processing data into training data sets and test data sets randomly according to the proportion, build a decision tree model for training and evaluation, until the model reaches the preset indicators;
[0052] Step 4: Integrate the trained model into the existing system, design the user interface, and implement automated decision support.
[0053] In step one, government data is collected from different types of data sources, pre-processed, and integrated to obtain government processing data; data sources include different government department databases, public service platforms, social media, and IoT devices;
[0054] Furthermore, the method for preprocessing the collected government data is as follows:
[0055] Using data cleaning tools such as Open Refine, the collected government data is initially cleaned to remove duplicate information, check for errors or inconsistencies in the data, and make corrections. The government data is split into words or phrases, and converted into numerical form using word embedding technology, and integrated to generate government processing data.
[0056] In this embodiment, preprocessing is performed on the collected government data to improve data quality and ensure data consistency and accuracy, which provides a basis for subsequent analysis and decision support. Integrating government data from different data sources helps break down information silos, achieve data sharing and circulation, and promote collaboration among departments.
[0057] In step 2, the specific process of identifying government affairs processing data includes:
[0058] Mark the words or phrases in the government affairs processing data as nodes, construct a node connection graph, use the cosine similarity between sentences as the weight of the edge, calculate the node weight through loop iteration, obtain the weight of each node and compare it with the preset weight, mark the nodes with a weight greater than or equal to the preset weight as key information and combine them into a government affairs information summary; wherein the preset weight is set according to the ranking of all node weights;
[0059] Furthermore, the weight of node i is expressed as:
[0060]
[0061] Where W (t+1) (i) represents the weight of node i at the t+1th iteration, d is the damping coefficient, which is usually 0.85; In(i) represents all nodes connected to node i, Out(j) represents all nodes that can be reached from node j; w ji Represents the correlation between node j and node i;
[0062] In this embodiment, by applying the Text Rank algorithm to government data processing, key information can be automatically extracted from large amounts of text data, improving the information processing rate, helping decision makers to quickly grasp key issues and trends, thereby making more informed decisions and providing valuable reference for government decision-making.
[0063] In step three, the method of building a decision tree model is as follows:
[0064] Select the feature with the largest information gain from the features of the training data as the current split node, and recursively generate child nodes by evaluating the importance of the selected feature until the stopping condition is met and a leaf node is obtained;
[0065] It should be noted that the stopping condition may be: the number of samples in the node reaches a preset minimum number of samples or the information gain is less than a preset threshold;
[0066] Among them, the importance of features is obtained based on information gain. The specific formula is as follows:
[0067] Information gain: IG(D,A)=H(D)-H(DA);
[0068] Where H(D) is the information entropy of the training dataset D, H(D|A) is the conditional entropy of the training dataset D under the condition of feature A; IG(D,A) is the importance of feature A;
[0069] Furthermore, in order to prevent overfitting, a pruning strategy can be used to remove some unnecessary branches and improve the generalization ability of the model;
[0070] For each leaf node, analyze each decision path from the root node to the leaf node, extract the decision rule represented by the decision path, including backtracking along the decision path, recording the features selected by each internal node, and generating decision recommendations based on the decision rules;
[0071] Calculate the F1 value and use it as an evaluation index to analyze the decision tree model: count the number of samples with correct decisions as positive class TP, the number of samples with wrong decisions as positive class FP, and the number of samples with wrong decisions as negative class FN to obtain the precision and recall rate of the model;
[0072] The calculation formula is:
[0073]
[0074] In the formula, P represents the precision of the model, and R represents the recall rate of the model;
[0075] According to the obtained model precision and recall rate, calculate the model evaluation index F1 value;
[0076] The calculation formula is:
[0077]
[0078] The obtained model evaluation index F1 is compared with the preset index. If F1 ≥ the preset index, it means that the model has reached the preset level; if F1 < the preset index, it means that the model parameters need to be adjusted using the labeled data in the training set until the performance of the model on the validation set reaches the preset level.
[0079] In this embodiment, by selecting the feature with the largest information gain for data division, the uncertainty of the data can be minimized, thereby improving the accuracy of classification; by calculating the F1 value and analyzing the decision tree model as an evaluation indicator, it is helpful to improve the accuracy and generalization ability of the model.
[0080] Embodiment 2, as Figure 2 As shown, a government data processing intelligent analysis and decision-making system includes:
[0081] The data collection and processing module collects government data, pre-processes government data, and integrates and obtains government processing data;
[0082] The data identification and integration module uses the Text Rank algorithm to identify government processing data, extract key information and combine it into a government information summary;
[0083] The model building module randomly divides the government processing data into training data sets and test data sets according to the proportion, builds a decision tree model for training and evaluation, until the model reaches the preset indicators;
[0084] The automatic decision-making module integrates the trained model into the existing system, designs the user interface, and realizes automated decision support.
[0085] In the several embodiments provided by the present invention, it should be understood that the disclosed system can be implemented in other ways. For example, the above-described embodiments of the invention are only illustrative, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation.
[0086] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0087] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of hardware plus software functional modules.
[0088] It is obvious to a person skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the essential characteristics of the present invention.
[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.
Claims
1. A method for intelligent analysis and decision-making of government data processing, characterized in that: include: Step 1: Collect government data, pre-process the government data, and integrate and obtain government processing data; Step 2: Use the Text Rank algorithm to identify government affairs processing data, extract key information and combine it into a government affairs information summary; Step 3: Divide the government affairs processing data into training data sets and test data sets randomly according to the proportion, build a decision tree model for training and evaluation, until the model reaches the preset indicators; Step 4: Integrate the trained model into the existing system, design the user interface, and implement automated decision support.
2. According to claim 1, a method for intelligent analysis and decision-making of government data processing is characterized in that: In step one, government data is collected from different types of data sources, pre-processed, and integrated to obtain government processing data; data sources include different government department databases, public service platforms, social media, and IoT devices; The method for preprocessing the collected government data is as follows: The collected government data is preliminarily cleaned, duplicate information in the data is deleted, errors or inconsistencies in the data are checked and corrected, the government data is split into words or phrases, and converted into numerical form using word embedding technology, and integrated to generate government processing data.
3. The method for intelligent analysis and decision-making of government data processing according to claim 2 is characterized in that: In step 2, the specific process of identifying government affairs processing data includes: The words or phrases in the government processing data are marked as nodes, a node connection graph is constructed, the cosine similarity between sentences is used as the edge weight, the node weight is calculated through a loop iteration, the weight of each node is obtained and compared with the preset weight, and the nodes with a weight greater than or equal to the preset weight are marked as key information and combined into a government information summary.
4. The method for intelligent analysis and decision-making of government data processing according to claim 3 is characterized in that: The weight of node i is expressed as: Where W (t+1) (i) represents the weight of node i at the t+1th iteration, d is the damping coefficient; In(i) represents all nodes connected to node i, Out(j) represents all nodes that can be reached from node j; w ji Represents the correlation between node j and node i.
5. The method for intelligent analysis and decision-making of government data processing according to claim 1 is characterized in that: In step three, the method of building a decision tree model is as follows: Select the feature with the largest information gain from the features of the training data as the current split node, and recursively generate child nodes by evaluating the importance of the selected feature until the stopping condition is met and a leaf node is obtained; Among them, the importance of features is obtained based on information gain. The specific formula is as follows: Information gain: IG(D,A)=H(D)-H(D|A); Where H(D) is the information entropy of the training dataset D, H(D|A) is the conditional entropy of the training dataset D under the condition of feature A; IG(D,A) is the importance of feature A; For each leaf node, analyze each decision path from the root node to the leaf node, extract the decision rules represented by the decision path, including backtracking along the decision path, recording the features selected by each internal node, and generating decision recommendations based on the decision rules.
6. The method for intelligent analysis and decision-making of government data processing according to claim 5 is characterized in that: Calculate the F1 value and use it as an evaluation indicator to analyze the decision tree model: Count the number of samples TP with correct decisions as positive class, the number FP with wrong decisions as positive class, and the number FN with wrong decisions as negative class to obtain the precision and recall rate of the model; calculate the evaluation index F1 value of the model based on the obtained precision and recall rate of the model; Compare the evaluation index F1 of the obtained model with the preset index. If F1 ≥ the preset index, it means that the model has reached the preset level. If F1 is less than the preset indicator, it means that the model parameters need to be adjusted using the labeled data in the training set until the performance of the model on the validation set reaches the preset level.
7. The method for intelligent analysis and decision-making of government data processing according to claim 6 is characterized in that: The calculation formula for the model's precision and recall is: In the formula, P represents the precision of the model, and R represents the recall of the model.
8. The method for intelligent analysis and decision-making of government data processing according to claim 7 is characterized in that: The calculation formula of the model's evaluation index F1 value is: The calculation formula is: In the formula, F1 is the evaluation index of the model.
9. An intelligent analysis and decision-making system for government data processing, characterized in that: include: The data collection and processing module collects government data, pre-processes government data, and integrates and obtains government processing data; The data identification and integration module uses the Text Rank algorithm to identify government processing data, extract key information and combine it into a government information summary; The model building module randomly divides the government processing data into training data sets and test data sets according to the proportion, builds a decision tree model for training and evaluation, until the model reaches the preset indicators; The automatic decision-making module integrates the trained model into the existing system, designs the user interface, and realizes automated decision support.