Government affair data storage method and system based on Internet of Things
By extracting the feature vectors and data source information of government data, combined with the Kmeans clustering algorithm, the problem of inaccurate classification of government data is solved, and the accurate classification and storage of government data is realized, and the accuracy and efficiency of data storage are improved.
Patent Information
- Application Number
- CN202411471234.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-22
- Publication Date
- 2024-11-19
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing technology ignores the classification information during the collection process in the classification storage of government data, resulting in inaccurate classification of government data and the inability to effectively realize the classification storage of government data.
By extracting the embedding vectors of multiple fields of government data, calculating the feature vectors, and predicting the category probability of government data based on the data source, combining with the Kmeans clustering algorithm, the data distance between government data is calculated to achieve accurate classification and storage of government data.
Ensure the accuracy of the classification results of government data, realize the effective classification storage of government data, and improve the accuracy and efficiency of data storage.
Smart Images

Figure CN118981489A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a government data storage method and system based on the Internet of Things. Background Art
[0002] With the advent of the big data era, e-government systems have been widely used. E-government is the use of modern computer technology and network technology to transfer the management and service functions of various departments to the network, while realizing the reorganization and optimization of the work processes between departments, transcending the constraints of time and departmental separation, and providing efficient, high-quality, standardized, transparent and comprehensive management and services to the society. In the process of realizing e-government, it is necessary to deploy multiple collection terminals to collect a large amount of government data. How to store and manage a large amount of government data is an urgent problem to be solved.
[0003] At present, a patent application document with application publication number CN114722801A discloses a government data classification storage method and related devices, the method comprising: obtaining matter description information of a target matter; performing feature extraction on the matter description information to obtain matter feature information of the target matter; classifying and identifying the target matter based on the matter feature information to obtain the category to which the target matter belongs; and associating and storing the matter description information of the target matter with the category to which the target matter belongs.
[0004] The above method obtains the category of government data based on the matter description information of the government data, and realizes the classified storage of government data. It only classifies the government data according to the information reflected in the collected government data, and ignores the information that is effective for classification during the government data collection process, resulting in inaccurate classification of government data and inability to effectively realize the classified storage of government data. Summary of the invention
[0005] In order to solve the technical problem of inaccurate classification of government data and inability to effectively implement classified storage of government data, the present application provides a government data storage method and system based on the Internet of Things, which can accurately obtain the classification results of government data and realize classified storage of government data.
[0006] In a first aspect, the present application provides a method for storing government data based on the Internet of Things, the storage method comprising: extracting multiple fields of government data, and taking the sum of the embedded vectors of each field as the feature vector of the government data; predicting the probability that any two government data belong to the same category based on the data source, the data source including the ID of the collection terminal called when collecting government data; defining the data distance between government data; clustering the government data using the Kmeans clustering algorithm to obtain clustering results, and classifying and storing the government data based on the clustering results; and government data Data distance for: , and Government data and government data The characteristic vector of For government data and government data The probability of belonging to the same category, is the adjustment coefficient.
[0007] In the process of government data collection, one government data needs to call multiple specific collection terminals. Similarly, one collection terminal also needs to be called by multiple government data. Therefore, the data source in the process of government data collection can reflect the category information of government data. First, multiple fields of government data are extracted, and the feature vector of government data is obtained according to the embedded vector of each field. Secondly, the probability of any two government data belonging to the same category is predicted according to the data source. The data distance between government data is calculated by combining the probability of belonging to the same category and the Euclidean distance of the feature vector, so that the data distance can accurately reflect the category information between government data, ensuring that the data distance between government data of the same category is small and the data distance between government data of different categories is large. Finally, the data distance between government data is used as the distance calculation formula in the Kmeans clustering algorithm to realize the classification of multiple government data, accurately obtain the classification results of government data, and realize the classified storage of government data.
[0008] Preferably, the method for obtaining the feature vector of government data includes: for any field of government data, counting the number of samples of the field in each category in the government data sample; taking the variance of the number of samples in each category as the discrimination of the field, marking the field whose discrimination is greater than the discrimination threshold as a key field; and weightedly summing the embedding vectors of the key fields according to the discrimination to obtain the feature vector.
[0009] Since a government data contains a large number of fields, some fields are helpful for the classification of government data, while some fields do not play a significant role in the classification of government data. In order to ensure that the feature vector of the government data can obtain accurate classification results, multiple fields of the government data are screened; in the government data samples, if the variance of the number of samples of a field in each category is large, then the field can effectively distinguish different categories. The feature vector is constructed based on the field that can effectively distinguish different categories, so that the feature vector can reflect the category information of the storage category to which the government data belongs.
[0010] Preferably, predicting the probability that any two government data belong to the same category based on the data source includes: for a collection terminal, the statistical data source includes the proportion of the number of government data samples of the collection terminal in each category, and obtaining the category vector of the collection terminal; calculating the sum of the category vectors of each collection terminal in the data source of the government data, and obtaining the probability vector of the government data; the probability that any two government data belong to the same category is negatively correlated with the Euclidean distance between the probability vectors of the two government data.
[0011] Preferably, government data and government data Probability of belonging to the same category satisfy: , and Government data and government data probability vector, for and The Euclidean distance between .
[0012] Preferably, predicting the probability that any two government data belong to the same category based on the data source includes: marking the collection terminal in the data source as 1 and marking other collection terminals as 0 to obtain the source vector of the government data; performing an XOR operation on the source vectors of any two government data to obtain a difference vector, and inputting the difference vector into the BP neural network to output the probability that the two government data belong to the same category.
[0013] The difference vector can reflect the difference in data sources between two government data. Using the difference vector as the input of the BP neural network enables the BP neural network to accurately output the probability that the two government data belong to the same category based on the difference in data sources.
[0014] Preferably, the training method of the BP neural network includes: randomly selecting two government data samples as training pairs, in response to the two government data samples having the same category, the label of the training pair is 1, otherwise, the label of the training pair is 0; inputting the difference vector of the training pair into the BP neural network to obtain an output result, using the cross entropy function value between the output result and the training pair label as the loss function, and training the BP neural network until the loss function is less than the preset loss.
[0015] Preferably, the adjustment coefficient is negatively correlated with the prediction error of the probability that any two government data belong to the same category.
[0016] The smaller the prediction error, the better the government data and government data Probability of belonging to the same category Accurately calculate government data and government data At this time, the adjustment coefficient should be larger, so that the data distance between government data of the same category is smaller, and the data distance between government data of different categories is larger.
[0017] Preferably, the adjustment coefficient is equal to the difference between 1 and the prediction error; the prediction error for: , is the number of government data sample pairs, is the probability that the government data samples belong to the same category, is the label of the government data sample pair; in response to the fact that the categories of the government data samples in the government data sample pair are the same, the label of the government data sample pair is 1, otherwise, the label of the government data sample pair is 0.
[0018] Using government data samples of known categories, multiple government data sample pairs are constructed, and the probability that the government data sample pairs belong to the same category is statistically analyzed, as well as the difference between the labels of the government data sample pairs to accurately quantify the prediction error.
[0019] In a second aspect of the present application, there is also provided a government data storage system based on the Internet of Things, comprising a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the government data storage method based on the Internet of Things described in the first aspect of the present application is implemented.
[0020] The technical solution of this application has the following beneficial technical effects: In the process of government data collection, one government data needs to call multiple specific collection terminals. Similarly, one collection terminal also needs to be called by multiple government data. Therefore, the data source in the process of government data collection can reflect the category information of government data. First, multiple fields of government data are extracted, and the feature vector of government data is obtained according to the embedded vector of each field. Secondly, the probability of any two government data belonging to the same category is predicted according to the data source. The data distance between government data is calculated by combining the probability of belonging to the same category and the Euclidean distance of the feature vector, so that the data distance can accurately reflect the category information between government data, ensuring that the data distance between government data of the same category is small and the data distance between government data of different categories is large. Finally, the data distance between government data is used as the distance calculation formula in the Kmeans clustering algorithm to realize the classification of multiple government data, accurately obtain the classification results of government data, and realize the classified storage of government data. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become easy to understand. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein: Figure 1 is a flow chart of a method for storing government data based on the Internet of Things according to an embodiment of the present application; Figure 2 It is a structural block diagram of the government data storage system based on the Internet of Things according to an embodiment of the present application. DETAILED DESCRIPTION
[0022] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.
[0023] It should be understood that when the terms "first", "second", etc. are used in the claims, specification and drawings of the present application, they are only used to distinguish different objects, rather than to describe a specific order. The terms "include" and "comprise" used in the specification and claims of the present application indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections.
[0024] According to the first aspect of the present application, the present application provides a government data storage method based on the Internet of Things. Figure 1 1 is a flow chart of a method for storing government data based on the Internet of Things according to an embodiment of the present application. Figure 1 As shown, the Internet of Things-based government data storage method includes steps S101 to S104, which are described in detail below.
[0025] S101, extract multiple fields of government data, and use the sum of the embedded vectors of each field as the feature vector of the government data.
[0026] In one embodiment, the data sources of government data are various, and one government data corresponds to at least one data source; for example, when collecting government data "urban population statistics", its data sources include multiple collection terminals that affect urban population, such as births, employment, and deaths. And the government data contains multiple fields, for example, the fields in the government data "urban population statistics" include multiple fields such as births, deaths, and population migration.
[0027] Extract multiple fields of government data, use word2vec or Bert model to obtain the embedding vector of each field, and use the sum of the embedding vectors of each field as the feature vector of the government data. Among them, the Bert model uses the Transformer encoder architecture, which can extract deep semantic information of the field and is a common means of obtaining embedding vectors in natural language processing.
[0028] In another embodiment, since a government data contains a large number of fields, some fields are helpful for the classification of the government data, while some fields do not play a significant role in the classification of the government data. Therefore, the embedded vectors of all fields are involved in the calculation of the feature vector, which will inevitably introduce information that is irrelevant to the classification of the government data. In order to ensure that the feature vector of the government data can obtain accurate classification results, it is necessary to screen multiple fields of the government data.
[0029] Specifically, the method for obtaining the feature vector of government data includes: for any field of government data, counting the number of samples of the field in each category in the government data sample; taking the variance of the number of samples in each category as the discrimination of the field, marking the field whose discrimination is greater than the discrimination threshold as a key field; and weightedly summing the embedding vectors of the key fields according to the discrimination to obtain the feature vector.
[0030] Among them, government data The eigenvector of for: , For government data The number of key fields in For key fields The degree of differentiation, For key fields The embedding vector of For government data The sum of the discriminations of all key fields in .
[0031] Among them, the government data samples are government data that have been classified and stored in historical time. One government data sample corresponds to a category. The category can be divided according to the processing department of the government data or according to the processing results of the government data. There is no restriction here. The government data can be classified and stored according to different categories later.
[0032] In this way, a feature vector of the government data is obtained according to the embedding vector of each field in the government data, and the feature vector can reflect the category information of the storage category to which the government data belongs.
[0033] S102, predicting the probability that any two government data belong to the same category based on the data source, where the data source includes the ID of the collection terminal called when collecting the government data.
[0034] In one embodiment, in the process of collecting government data, it is necessary to collect government data based on at least one collection terminal, that is, one piece of government data comes from multiple collection terminals, and multiple collection terminals are used as data sources of the government data.
[0035] Specifically, the data source includes the ID of the collection terminal called when collecting government data. For example, 100 collection terminals are deployed to collect government data, and each collection terminal corresponds to an ID. When the collection terminal 23, the collection terminal 34 and the collection terminal 56 are called, the government data The data sources are 23, 34 and 56.
[0036] The probability that two government data belong to the same category is determined based on their data sources. The greater the probability, the greater the possibility that the two government data belong to the same category, and the smaller the data distance between the two government data should be. Therefore, whether the probability of two government data belonging to the same category can be accurately determined directly affects the data distance, and thus affects the accuracy of the clustering results.
[0037] Specifically, predicting the probability that any two government data belong to the same category based on the data source includes: for a collection terminal, the statistical data source includes the proportion of the number of government data samples of the collection terminal in each category, and obtaining the category vector of the collection terminal; calculating the sum of the category vectors of each collection terminal in the data source of the government data, and obtaining the probability vector of the government data; the probability that any two government data belong to the same category is negatively correlated with the Euclidean distance between the probability vectors of the two government data.
[0038] Among them, government data and government data Probability of belonging to the same category satisfy: , and Government data and government data probability vector, for and Euclidean distance between two entities, government data The probability vector Used to reflect government data The probability distribution belonging to each category.
[0039] Exemplarily, all government data are classified and stored according to four categories; for the collection terminal 23, the data source includes 1500 government data samples from the collection terminal 23, of which 300 belong to category 1, 450 belong to category 2, 300 belong to category 3, and the rest belong to category 4; then the category vector of the collection terminal 23 is If government data The data source only includes the collection terminal 23, then the government data The probability vector of is the same as the category vector of the acquisition terminal 23, which is also , which can reflect government data The probability distribution belonging to each category.
[0040] In this way, when the Euclidean distance between the probability vectors of two government data is smaller, it means that the probability distribution of the two government data belonging to different categories is consistent, and the probability that the two government data belong to the same category is greater.
[0041] In another embodiment, a neural network method can be used to predict the probability that two government data belong to the same category. Specifically, predicting the probability that any two government data belong to the same category based on the data source includes: marking the collection terminal in the data source as 1, marking other collection terminals as 0, and obtaining the source vector of the government data; performing an XOR operation on the source vectors of any two government data to obtain a difference vector, and inputting the difference vector into the BP neural network to output the probability that the two government data belong to the same category.
[0042] For example, collecting government data When the collection terminal 23, the collection terminal 34 and the collection terminal 56 are called, the government data In the source vector of , the values corresponding to the collection terminals 23, 34 and 56 are 1, and the values corresponding to the other collection terminals are 0; When the collection terminal 23, the collection terminal 34 and the collection terminal 60 are called, the government data In the source vector, the values corresponding to collection terminal 23, collection terminal 34 and collection terminal 60 are 1, and the values corresponding to other collection terminals are 0; an exclusive OR operation is performed on the source vectors of any two government data, that is, an exclusive OR operation is performed on the values of corresponding positions in the source vector. The values of corresponding positions in the two source vectors are the same as 0 and different as 1. In the obtained difference vector, the values corresponding to collection terminal 56 and collection terminal 60 are 1, and the values corresponding to other collection terminals are 0.
[0043] It can be understood that the sizes of the source vector and the difference vector are related to the number of collection terminals, and the difference vector can reflect the difference in data sources between two government data.
[0044] In order to ensure that the BP neural network can accurately output the probability that two government data belong to the same category, it is necessary to train the BP neural network. The training method of the BP neural network includes: randomly selecting two government data samples as training pairs, in response to the two government data samples being of the same category, the label of the training pair is 1, otherwise, the label of the training pair is 0; inputting the difference vector of the training pair into the BP neural network to obtain the output result, using the cross entropy function value between the output result and the training pair label as the loss function, and training the BP neural network until the loss function is less than the preset loss.
[0045] Among them, the preset loss is 0.001.
[0046] In this way, the BP neural network is used to accurately predict the probability that two government data belong to the same category.
[0047] S103, define the data distance between government data.
[0048] In one embodiment, data distance is used to measure the distance between government data. Data distance directly affects the clustering result. The smaller the data distance, the more likely the two government data are to belong to the same category. The larger the data distance, the more likely the two government data are to belong to different categories. In the process of calculating the data distance between government data, the feature vector of the government data is taken into account, and the data distance between the government data is adjusted according to the probability that the two government data belong to the same category obtained in step S102.
[0049] Specifically, government data and government data Data distance for: , and Government data and government data The characteristic vector of For government data and government data The probability of belonging to the same category, is the adjustment coefficient.
[0050] Among them, the adjustment coefficient The value of is 0.5, and the adjustment coefficient is used to adjust the influence of the probability that two government data belong to the same category on the calculated data distance.
[0051] In another embodiment, the adjustment coefficient is negatively correlated with the prediction error of the probability that any two government data belong to the same category. The smaller the prediction error, the more likely the government data are to be in the same category. and government data Probability of belonging to the same category Accurately calculate government data and government data The larger the data distance, the larger the adjustment coefficient.
[0052] Optionally, the prediction error is: , is the number of government data sample pairs, is the probability that the government data samples belong to the same category, is the label of the government data sample pair; in response to the fact that the categories of the government data samples in the government data sample pair are the same, the label of the government data sample pair is 1, otherwise, the label of the government data sample pair is 0.
[0053] Optionally, the prediction error may also be: , is the number of government data sample pairs, is the probability that the government data samples belong to the same category, is the label of the government data sample pair; in response to the fact that the categories of the government data samples in the government data sample pair are the same, the label of the government data sample pair is 1, otherwise, the label of the government data sample pair is 0.
[0054] The adjustment factor Equal to the difference between 1 and the prediction error.
[0055] In this way, the data distance between any government data is obtained.
[0056] S104, clustering the government data using the Kmeans clustering algorithm to obtain clustering results, and classifying and storing them according to the clustering results.
[0057] In one embodiment, since the number of categories in which the government data is classified and stored is known in advance, the K value in the Kmeans clustering algorithm is set to the number of categories, and the data distance between the government data is used as the distance calculation formula in the Kmeans clustering algorithm to cluster the government data to obtain a clustering result. The clustering result includes clusters with the same number of categories, and a cluster includes at least one government data. A cluster corresponds to one category, and a large amount of government data is classified and stored.
[0058] The above introduces the technical principles and implementation details of the Internet of Things-based government data storage method of the present application through specific embodiments. In the process of government data collection, one government data needs to call multiple specific collection terminals. Similarly, one collection terminal also needs to be called by multiple government data. Therefore, the data source in the process of government data collection can reflect the category information of government data; first, extract multiple fields of government data, and obtain the feature vector of government data according to the embedded vector of each field. Secondly, predict the probability that any two government data belong to the same category based on the data source, and calculate the data distance between government data based on the probability of belonging to the same category and the Euclidean distance of the feature vector, so that the data distance can accurately reflect the category information between government data, and ensure that the data distance between government data of the same category is small, and the data distance between government data of different categories is large; finally, the data distance between government data is used as the distance calculation formula in the Kmeans clustering algorithm to realize the classification of multiple government data, accurately obtain the classification results of government data, and realize the classified storage of government data.
[0059] According to the second aspect of the present application, the present application also provides a government data storage system based on the Internet of Things. Figure 2 is a structural block diagram of a government data storage system based on the Internet of Things according to an embodiment of the present application. Figure 2 As shown, the system 50 includes a processor and a memory, and the memory stores computer program instructions. When the computer program instructions are executed by the processor, the government data storage method based on the Internet of Things according to the first aspect of the present application is implemented. The system also includes other components well known to those skilled in the art, such as a communication bus and a communication interface, whose settings and functions are known in the art, so they are not described here.
[0060] In the present application, the aforementioned memory may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. For example, a computer-readable storage medium may be any suitable magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc., or any other medium that can be used to store the required information and can be accessed by an application, a module, or both. Any such computer storage medium may be part of a device or accessible or connectable to a device. Any application or module described in this application may be implemented using computer-readable / executable instructions that may be stored or otherwise maintained by such a computer-readable medium.
[0061] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0062] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.
Claims
1. Government data storage method based on the Internet of Things, It is characterized in that The storage method comprises: Extract multiple fields of government data, and use the sum of the embedding vectors of each field as the feature vector of the government data; Predict the probability that any two government data belong to the same category based on the data source, where the data source includes the ID of the collection terminal called when collecting government data; Define the data distance between government data; use Kmeans clustering algorithm to cluster government data, obtain clustering results, and classify and store them according to the clustering results; Government data and government data Data distance for: , and Government data and government data The characteristic vector of For government data and government data The probability of belonging to the same category, is the adjustment coefficient.
2. According to the method for storing government data based on the Internet of Things in claim 1, It is characterized in that Methods for obtaining feature vectors of government data include: For any field of government data, count the number of samples of the field in each category in the government data sample; The variance of the number of samples in each category is used as the discrimination of the field, and the field whose discrimination is greater than the discrimination threshold is marked as a key field; The embedding vectors of the key fields are weighted and summed according to the discrimination to obtain the feature vector.
3. According to the Internet of Things-based government data storage method of claim 1, It is characterized in that The probability of predicting that any two government data belong to the same category based on the data source includes: For a collection terminal, the statistical data source includes the proportion of the government data samples of the collection terminal in each category, and the category vector of the collection terminal is obtained; Calculate the sum of the category vectors of each collection terminal in the data source of the government data to obtain the probability vector of the government data; The probability that any two government data belong to the same category is negatively correlated with the Euclidean distance of the probability vectors between the two government data.
4. According to the Internet of Things-based government data storage method of claim 3, It is characterized in that Government data and government data Probability of belonging to the same category satisfy: , and Government data and government data probability vector, for and The Euclidean distance between .
5. According to the Internet of Things-based government data storage method of claim 1, It is characterized in that The probability of predicting that any two government data belong to the same category based on the data source includes: Mark the collection terminal in the data source as 1, and mark other collection terminals as 0, and obtain the source vector of government data; An XOR operation is performed on the source vectors of any two government data to obtain a difference vector, and the difference vector is input into the BP neural network to output the probability that the two government data belong to the same category.
6. The method for storing government data based on the Internet of Things according to claim 5, It is characterized in that The training methods of BP neural network include: Randomly select two government data samples as training pairs, and in response to the two government data samples being of the same category, the label of the training pair is 1, otherwise, the label of the training pair is 0; The difference vector of the training pair is input into the BP neural network to obtain an output result, and the cross entropy function value between the output result and the training pair label is used as the loss function to train the BP neural network until the loss function is less than the preset loss.
7. The method for storing government data based on the Internet of Things according to claim 1, It is characterized in that The adjustment coefficient is negatively correlated with the prediction error of the probability that any two government data belong to the same category.
8. The method for storing government data based on the Internet of Things according to claim 7, It is characterized in that The adjustment factor is equal to the difference between 1 and the prediction error; the prediction error for: , is the number of government data sample pairs, is the probability that the government data samples belong to the same category, is the label of the government data sample pair; In response to the categories of the government data samples in the government data sample pair being the same, the label of the government data sample pair is 1, otherwise, the label of the government data sample pair is 0.
9. A government data storage system based on the Internet of Things, It is characterized in that It comprises a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the government affairs data storage method based on the Internet of Things according to any one of claims 1 to 8 is implemented.
Citation Information
Cited By
Government affair process intelligent optimization method, system and equipment based on large model and medium
CN119849900A
