Distributed data management system applied to urban pipeline detection

Abstract

The invention discloses a distributed data management system applied to urban pipeline detection, which relates to the field of data management and comprises a data acquisition module, a slice making module, a database module, an association rule module and a query module. The data acquisition module collects urban pipeline operation data and historical query data; the slice making module obtains a slice data set according to historical query data, and then slices operation data; the association rule module utilizes an Apriori algorithm to mine a value association rule set; the query module outputs query and recommendation results according to the user real-time query and value association rule set; according to the method, the fragmentation rule can be reasonably designed, the high-frequency query data can be centrally stored, and the data query efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of data management, and more specifically, to a distributed data management system applied to urban pipeline detection. Background Art

[0002] In the current accelerating process of urban modernization, the urban pipeline system, as the "lifeblood" of the city, its safe and stable operation is crucial. Urban pipelines cover various types such as water supply, drainage, gas, and heating. Their operation data not only has a large scale but also has multi-dimensional characteristics.

[0003] In recent years, distributed data management technology has gradually emerged. By dispersing data storage on multiple nodes, it greatly improves the storage and processing capabilities of the system. However, the existing distributed data management technology still has the following defects:

[0004] Most of them adopt static sharding rules (such as dividing by region or time), and cannot reasonably design sharding rules according to historical query records, which easily leads to load imbalance of storage nodes and significantly increases query response latency.

[0005] Reasonably designing sharding rules and improving query efficiency are the problems we need to solve. For this purpose, a distributed data management system applied to urban pipeline detection is provided. Summary of the Invention

[0006] The purpose of the present invention is to provide a distributed data management system applied to urban pipeline detection.

[0007] The purpose of the present invention can be achieved through the following technical solutions: A distributed data management system applied to urban pipeline detection includes:

[0008] A data acquisition module: used to collect operation data and historical query data during the operation of urban pipelines;

[0009] A slice formulation module: processes historical query data to obtain a transaction data set, obtains a slice data set according to the transaction data set, and slices the operation data according to the slice data set;

[0010] A database module: includes multiple storage nodes, and is used to store the sliced operation data into each storage node;

[0011] An association rule module: used to perform association rule mining on the slice data set according to the Apriori algorithm to obtain a set of value association rules;

[0012] A query module: used for users to input query statements in real time as real-time queries, and output query results and recommended results according to the real-time queries and the set of value association rules.

[0013] Preferably, the operation data includes: pressure, temperature, flow rate, flow velocity, vibration, stress, strain, humidity, pH value, region, groundwater level, time, and population density; the historical query data is all the query statements input by the user to the data management system within a set time interval.

[0014] Preferably, the process of processing the historical query data to obtain a transaction data set, obtaining a slice data set according to the transaction data set, and slicing the operation data according to the slice data set is as follows:

[0015] Perform word segmentation on the query statement to obtain a word segmentation set of the query statement;

[0016] Match the elements in the word segmentation set with the elements in the operation data, and use the successfully matched elements as the query set corresponding to the query statement;

[0017] Obtain a transaction data set according to the query set, and obtain a slice data set according to the transaction data set;

[0018] Slice the operation data according to the slice data set.

[0019] Preferably, the process of obtaining a slice data set according to the transaction data set is as follows:

[0020] D1: Select a certain query set and mark it as the selection set, and mark the query sets in the transaction data set other than the selection set as the comparison data set;

[0021] D2: Obtain the non-empty proper subsets of the selection set, and obtain the number of corresponding non-empty proper subsets according to the number of occurrences of the selection set;

[0022] D3: Match the non-empty proper subsets with the query sets in the comparison data set, and determine the update times of the query sets in the comparison data set according to the matching results;

[0023] D4: Perform the operations of steps D1 - D3 on all query sets in the transaction data set with the number of elements greater than or equal to 2, so as to obtain the update times of each query set in the transaction data set;

[0024] D5: Sort the query sets in the transaction data set according to the update times to obtain a permutation set, and intercept the permutation set to obtain a slice data set.

[0025] Preferably, the process of slicing the operation data according to the slice data set is as follows:

[0026] W1: Calculate the number of occurrences of each element in the operation data according to the slice data set;

[0027] W2: Sort the elements in the operation data according to the number of occurrences of each element in the operation data to obtain an operation permutation;

[0028] W3: Obtain a policy data set based on the operation permutation and the slice data set, and classify the policy data set to obtain various category sets;

[0029] W4: Obtain a total category based on the update times of the query set under each category set;

[0030] W5: Extract the query set with the largest update times in the total category as the storage set corresponding to the first element in the operation permutation, and slice the operation data according to the elements in the storage set; at the same time, remove the corresponding elements in the storage set from the operation permutation to obtain an updated permutation;

[0031] W6: Extract the first element in the updated permutation to perform the analysis of the above steps W1 - W5 until there is no data in the updated permutation.

[0032] Preferably, the process for mining association rules from the slice data set according to the Apriori algorithm to obtain a value association rule set is as follows:

[0033] According to the slice data set, obtain the query rate of each element in the operation data, preset a single - item query threshold, and extract the elements in the operation data with a query rate greater than the single - item query threshold as the frequent 1 - item set;

[0034] Generate frequent i - item sets based on the frequent 1 - item set, where i = 1, 2, 3...k, and k represents the number of data types in the operation data

[0035] Generate various association rules based on the frequent i - item sets, and screen the various association rules to obtain a value association rule set.

[0036] Preferably, the process for processing real - time queries and outputting query results and recommendation results is as follows:

[0037] Obtain a corresponding real - time query set according to the real - time query, and output a corresponding query result according to the real - time query set;

[0038] Couple the real - time query set with the value association rule set. If the coupling is successful, output the corresponding recommendation result; otherwise, do not output the recommendation result.

[0039] Compared with the prior art, the beneficial effects of the present invention are:

[0040] The present invention processes historical query data to obtain a query set, filters the query set to obtain a transaction data set, analyzes the transaction data set again to obtain a slice data set, and finally slices the running data according to the slice data set, so that the slicing basis is associated with historical queries, enabling high-frequency query data to be stored centrally, reducing cross-node retrieval, and thus improving query efficiency.

[0041] The present invention analyzes historical query data to obtain a value association rule set. When a user inputs a query statement to the data management system, it can not only output the data corresponding to the query statement but also intelligently recommend relevant data in combination with the value association rule set, improving the intelligence level of the data management system. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments recorded in the present invention, and those of ordinary skill in the art can also obtain other drawings based on these drawings.

[0043] Figure 1 is the schematic diagram of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] As Figure 1 shown, a distributed data management system applied to urban pipeline detection includes:

[0045] A data acquisition module: used to acquire the running data and historical query data during the operation of urban pipelines;

[0046] The running data includes: pressure, temperature, flow rate, flow velocity, vibration, stress, strain, humidity, pH value, region, groundwater level, time, and population density;

[0047] Specifically, the pressure, temperature, flow rate, flow velocity, vibration, stress, strain, humidity, and pH value are obtained by corresponding sensors, where the humidity and pH value refer to the humidity and pH value of the soil around the pipeline; the region refers to which area of the city the pipeline is located in, accurate to the town where it is located; the population density is the population density of the area where the pipeline is located.

[0048] The historical query data is all the query statements input by the user to the data management system within a set time interval.

[0049] A slice formulation module: processes the historical query data to obtain a transaction data set, obtains a slice data set according to the transaction data set, and slices the running data according to the slice data set;

[0050] Use the jieba tokenization software to tokenize the query statement to obtain the token set of the query statement;

[0051] Match the elements in the token set with the elements in the running data, and use the successfully matched elements as the query set corresponding to the corresponding query statement;

[0052] Specifically, if the user enters a query statement "Find the pipeline flow rate and temperature in xx district in xx month of xx year", the corresponding token set is {Find, xx year, xx month, xx district, of, pipeline flow rate, and, temperature}, and the elements in the token set refer to "Find, xx year, xx month, xx district, of, pipeline flow rate, and, temperature"; then both xx year and xx month can be successfully matched with "time" in the running words, xx district can be successfully matched with "area", pipeline flow rate is successfully matched with "flow rate", temperature is successfully matched with "temperature", and the others are not successfully matched. Therefore, the query set corresponding to this statement is {time, area, flow rate, temperature};

[0053] Specifically, the elements in the running data refer to pressure, temperature, flow rate, flow velocity, vibration, stress, strain, humidity, pH value, area, groundwater level, time, and population density;

[0054] Specifically, the cosine similarity method can be used for matching. Convert the elements in the token set and the elements in the running data into semantic vectors, and then calculate the cosine similarity between the semantic vectors corresponding to the elements in the token set and the semantic vectors converted from each running data. And set a threshold. Only when the calculated cosine similarity is greater than the set threshold does it represent a successful match;

[0055] Obtain the transaction data set according to the query set;

[0056] Specifically, calculate the frequency of each query set. Frequency = the number of times the query set appears / the total number of query sets; preset a frequency threshold, extract the query sets with frequencies greater than the frequency threshold, and form the transaction data set;

[0057] Obtain the slice data set according to the transaction data set;

[0058] D1: Select a certain query set and mark it as the selection set, and mark the query sets in the transaction data set other than the selection set as the comparison data set;

[0059] D2: Obtain the non-empty proper subsets of the selection set. If the number of times the selection set appears is b times, then record the number of all non-empty proper subsets corresponding to the selection set as b;

[0060] D3: Match the non - empty proper subsets with the query sets in the comparison dataset. If the match is successful, add the number of occurrences of the non - empty proper subset to the number of times the successfully - matched query set appears. The result obtained is used as the updated number of times for the successfully - matched query set.

[0061] Specifically, if the elements in a non - empty proper subset of a query set are exactly the same as the elements in another query set other than this query set, it represents a successful match.

[0062] D4: Perform the operations of steps D1 - D3 on all query sets in the transaction dataset whose number of elements is greater than or equal to 2, so as to obtain the updated number of times for each query set in the transaction dataset.

[0063] In detail, if there are two query sets M and N, and M is a proper subset of N, then the data queried through query set M contains the data queried through query set N. Therefore, compared with the number of occurrences of the query set, the updated number of times can better reflect the actual number of queries by users for a certain query set.

[0064] D5: Preset an updated quantile threshold. Arrange the updated number of times of each query set in the transaction dataset from largest to smallest to obtain an arranged set. Intercept the arranged set according to the updated quantile threshold to obtain a sliced dataset.

[0065] Specifically, if the updated quantile threshold is 30%, then intercept the first 30% of the query datasets in the arranged set as the sliced dataset.

[0066] Slice the running data according to the sliced dataset.

[0067] W1: Calculate the number of times each element appears in the running data according to the sliced dataset.

[0068] In detail, if there are three sliced datasets, namely {temperature, humidity}, {temperature, pH}, and {humidity, pH}, and the corresponding updated numbers of times are 3, 4, and 5 respectively; then the number of times corresponding to temperature = 3 + 4 = 7, the number of times corresponding to humidity = 3 + 5 = 8, and the number of times corresponding to pH = 4 + 5 = 9.

[0069] W2: Sort the elements in the running data according to the number of times each element appears in the running data to obtain a running arrangement. The more times an element appears, the higher its corresponding ranking.

[0070] In detail, sorting the elements to obtain a running arrangement is to quantify the importance of the elements in the running data and prepare for subsequent slicing of the data.

[0071] W3: Select all query sets containing the first element of the running permutation in the sliced dataset as the policy dataset corresponding to the data, and classify the policy dataset to obtain category sets;

[0072] Specifically, for example, there are four query sets, namely A = {temperature, pH}, B = {temperature, humidity}, C = {temperature, pH, groundwater level}, and D = {temperature, humidity, groundwater level}. Since both A and B contain only two elements, A and B are classified into one category, and since both C and D contain three elements, C and D are classified into one category;

[0073] W4: Sum up the update times of the query sets under each category to obtain the sum value of each category, and take the category with the largest sum value as the aggregated category;

[0074] W5: Extract the query set with the largest update time in the aggregated category as the storage set corresponding to the first element in the running permutation, slice the running data according to the elements in the storage set; at the same time, remove the corresponding elements in the storage set from the running permutation to obtain the updated permutation;

[0075] Specifically, slicing refers to a data classification method, which is divided into single-dimensional slicing and multi-dimensional slicing. Precise data slicing can improve the query efficiency of data;

[0076] Specifically, if the storage set is {time, region, temperature}, then the pipeline data will first be classified according to time intervals, then classified according to regions under each time interval, and then classified according to temperature under each region;

[0077] W6: Extract the first element in the updated permutation for the analysis of the above steps W1 - W5 until there are no elements in the updated permutation.

[0078] Database module: Contains multiple storage nodes for storing the sliced running data into each storage node.

[0079] Association rule module: Used to mine association rules for the sliced dataset according to the Apriori algorithm to obtain a set of value association rules;

[0080] Based on the sliced dataset, obtain the query rate of each element in the running data, preset a single-item query threshold, and extract the elements in the running data with a query rate greater than the single-item query threshold as the frequent 1-item set;

[0081] Based on the frequent 1-item set, generate frequent i-item sets, where i = 1, 2, 3... k, and k represents the number of data types in the running data;

[0082] Specifically, pair the data in the frequent 1-itemsets to form each 2-itemset, calculate the query rate of each 2-itemset, preset a 2-item query threshold, and extract the 2-itemsets in which the query rate is greater than the 2-item query threshold as the frequent 2-itemsets;

[0083] Specifically, generate each 3-itemset based on the frequent 2-itemsets, calculate the query rate of each 3-itemset, preset a 3-item query threshold, and extract the 3-itemsets in which the query rate is greater than the 3-item query threshold as the frequent 3-itemsets, and so on until the frequent k-itemsets are generated;

[0084] Specifically, if there are four elements A, B, C, and D in the frequent 1-itemsets, then 6 2-itemsets can be generated, namely {A, B}, {A, C}, {A, D}, {B, C}, {B, D}, and {C, D}; if after threshold screening, {A, B}, {A, C}, and {A, D} become the frequent 2-itemsets, then 4 frequent 3-itemsets can be generated, namely {A, B, C}, {A, B, D}, {B, C, D}, and {A, C, D};

[0085] Generate each association rule based on the frequent i-itemsets, screen each association rule, and obtain a set of valuable association rules;

[0086] Specifically, for a frequent m-itemset, first decompose it into two query data sets U and V, and satisfy that the sum of the number of elements in U and V is m, the intersection of the two sets is an empty set, and the elements in both sets belong to the frequent m-itemset; calculate the support of set U, V, and the frequent m-itemset respectively, where,

[0087] Support = the number of updates corresponding to the query data set / the sum of the number of updates of all query sets;

[0088] Calculate the confidence ZXD(U→V) of the association rule U→V and the confidence ZXD(V→U) of the association rule V→U respectively, where,

[0089]

[0090] Preset a confidence threshold. If the confidence of a certain association rule is greater than the confidence threshold, then mark this association rule as a valuable association rule;

[0091] Specifically, based on a frequent itemset, many groups of association rules can be obtained, and the more the number of elements in the frequent itemset, the more association rules are decomposed;

[0092] Specifically, decompose all the frequent itemsets, summarize all the obtained valuable association rules, and thus obtain a set of valuable association rules.

[0093] Query module: It is used for users to input query statements in real time. As a real-time query, according to the real-time query and the value association rule set, it outputs query results and recommended results;

[0094] Obtain the corresponding real-time query set according to the real-time query, and output the corresponding query results according to the real-time query set;

[0095] Couple the real-time query set with the value association rule set. If the coupling is successful, output the corresponding recommended results; otherwise, do not output recommended results;

[0096] Specifically, regard the value association rule L→P as two parts, L as the input and P as the output. Couple the real-time query set with the inputs of all value association rules. If the elements in the real-time query set are exactly the same as the elements in the input of a certain value association rule, it means the coupling is successful. Take the output of this value association rule as the recommended query, and find the corresponding results in the database according to the elements in the recommended query as the recommended results.

[0097] The above are only the preferred embodiments of the present invention, and do not impose any form of limitation on the present invention. Although the present invention has been disclosed above with preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some changes or modifications to equivalent embodiments by using the disclosed technical content within the scope of the technical solution of the present invention. However, as long as it does not depart from the content of the technical solution of the present invention, any modification or equivalent replacement made to the above embodiments based on the technical essence of the present invention still falls within the scope of the technical solution of the present invention.

Claims

1. A distributed data management system applied to urban pipeline detection, characterized in that including: Data acquisition module: used to acquire the operation data and historical query data during the operation of urban pipelines; Slice formulation module: processes the historical query data to obtain a transaction dataset, obtains a slice dataset based on the transaction dataset, and slices the operation data according to the slice dataset; Database module: contains multiple storage nodes, and is used to store the sliced operation data into each storage node; Association rule module: used to perform association rule mining on the slice dataset according to the Apriori algorithm to obtain a set of value association rules; Query module: used for users to input query statements in real time as real-time queries, and outputs query results and recommended results according to the real-time queries and the set of value association rules.

2. The distributed data management system applied to urban pipeline detection according to claim 1, characterized in that, The operation data includes: pressure, temperature, flow rate, flow velocity, vibration, stress, strain, humidity, pH value, region, groundwater level, time, and population density; the historical query data is: all query statements input by users to the data management system within a set time interval.

3. The distributed data management system applied to urban pipeline detection according to claim 1, characterized in that, The process of processing the historical query data to obtain a transaction dataset, obtaining a slice dataset based on the transaction dataset, and slicing the operation data according to the slice dataset is as follows: Perform word segmentation processing on the query statements to obtain a word segmentation set of the query statements; Match the elements in the word segmentation set with the elements in the operation data, and use the successfully matched elements as the query set corresponding to the query statement; Obtain a transaction dataset based on the query set, and obtain a slice dataset based on the transaction dataset; Slice the operation data according to the slice dataset.

4. The distributed data management system applied to urban pipeline detection according to claim 3, wherein The process of obtaining a slice dataset based on the transaction dataset is as follows: D1: Select a certain query set and mark it as the selection set, and mark the query sets in the transaction dataset other than the selection set as the comparison dataset; D2: Obtain the non-empty proper subsets of the selection set, and obtain the number of corresponding non-empty proper subsets according to the number of times the selection set appears; D3: Match the non-empty proper subsets with the query sets in the comparison dataset, and determine the update times of the query sets in the comparison dataset according to the matching results; D4: Perform the operations of steps D1 - D3 on all query sets in the transaction dataset with the number of elements greater than or equal to 2, so as to obtain the update times of each query set in the transaction dataset; D5: Sort the query sets in the transaction dataset according to the update times to obtain a permutation set, and intercept the permutation set to obtain a slice dataset.

5. The distributed data management system applied to urban pipeline detection according to claim 3, characterized in that, The process of slicing the operation data according to the slice dataset is as follows: W1: Calculate the number of times each element appears in the operation data according to the slice dataset; W2: Sort the elements in the operation data according to the number of times each element appears in the operation data to obtain an operation permutation; W3: Obtain a strategy dataset according to the operation permutation and the slice dataset, and classify the strategy dataset to obtain various category sets; W4: Obtain the total category according to the update times of the query sets under each category set; W5: Extract the query set with the largest number of update times in the aggregation category as the storage set corresponding to the first element in the running permutation, and slice the running data according to the elements in the storage set; at the same time, remove the corresponding elements in the storage set from the running permutation to obtain the updated permutation. W6: Extract the first element in the updated permutation and perform the analysis of the above steps W1-W5 until there is no data in the updated permutation.

6. The distributed data management system applied to urban pipeline detection according to claim 1, characterized in that The process for performing association rule mining on the sliced data set according to the Apriori algorithm to obtain the value association rule set is as follows: Based on the sliced data set, obtain the query rate of each element in the running data, preset a single-item query threshold, and extract the elements in the running data whose query rate is greater than the single-item query threshold as the frequent 1-item set. Generate frequent i-item sets based on the frequent 1-item set, where i = 1, 2, 3... k, and k represents the number of data types in the running data. Generate each association rule based on the frequent i-item set, and screen each association rule to obtain the value association rule set.

7. The distributed data management system applied to urban pipeline detection according to claim 1, wherein The process for processing the real-time query and outputting the query result and the recommended result is as follows: Obtain the corresponding real-time query set according to the real-time query, and output the corresponding query result according to the real-time query set. Couple the real-time query set with the value association rule set. If the coupling is successful, output the corresponding recommended result; otherwise, do not output the recommended result.

Images