Urban drainage pipe network data cleaning method and cleaning device based on graph database

Abstract

The embodiment of the application discloses a kind of based on graph database's city drainage pipe network data cleaning method and cleaning device, it is related to data processing technical field, the method includes: obtaining original drainage pipe network data, original drainage pipe network data is preprocessed, and obtain after preprocessing data;After preprocessing data is imported into preset graph database, topological model for drainage pipe network is generated, and topological model includes node information and pipe information;Based on preset graph database, node information is carried out data cleaning processing, and first cleaned information is obtained;Based on preset graph database, pipe information is carried out data cleaning processing, and second cleaned information is obtained;First cleaned information and second cleaned information are based on and generate cleaned data after information.The graph database is used as the carrier of drainage pipe network data, and the data cleaning of drainage pipe network data is carried out by the graph database, so that the cleaning efficiency of data is greatly improved, and the data accuracy is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method for cleaning urban drainage network data based on a graph database and a device for cleaning urban drainage network data based on a graph database. Background Technology

[0002] As cities continue to grow, so do urban buildings and the population, leading to ever-increasing living demands. For example, the drainage needs are also increasing due to the expanding urban area, necessitating the construction of a large-scale underground drainage network.

[0003] To achieve better urban management, precise and effective management of the vast underground drainage network is required. The drainage network can be viewed as a network composed of nodes and pipes, corresponding to spatial data and attribute data. Spatial data is characterized by describing the location and shape of entities, while attribute data is characterized by describing the characteristics of entities. In urban management, the vast underground drainage network generates a large amount of management data. However, there are many errors and deviations in the storage and management of the above management data, such as data anomalies caused by improper operation during data collection, data anomalies caused by different standards of different data collection entities, data anomalies caused by manual import, editing and data repair, and data anomalies caused by improper use by database operators.

[0004] To address the aforementioned technical issues, databases are used to manage the management data, such as GIS systems. However, existing drainage network databases are relational databases, which suffer from slow data depth queries and difficulties in changing data structures. These limitations directly lead to inefficient data management and cleaning. Summary of the Invention

[0005] To overcome the aforementioned technical problems in the prior art, this invention provides a method and apparatus for cleaning urban drainage network data based on a graph database. By using a graph database as the carrier of drainage network data and cleaning the data through the graph database, the cleaning efficiency and accuracy of the data are greatly improved.

[0006] To achieve the above objectives, embodiments of the present invention provide a method for cleaning urban drainage network data based on a graph database. The method includes: acquiring raw drainage network data; preprocessing the raw drainage network data to obtain preprocessed data; importing the preprocessed data into a preset graph database to generate a topology model for the drainage network, the topology model including node information and pipe information; performing data cleaning processing on the node information based on the preset graph database to obtain first cleaned information; performing data cleaning processing on the pipe information based on the preset graph database to obtain second cleaned information; and generating cleaned data based on the first cleaned information and the second cleaned information.

[0007] Preferably, the preprocessing of the original drainage network data to obtain preprocessed data includes: correcting the original drainage network data based on a preset database system to obtain corrected data, wherein the corrected data includes corrected pipeline data and corrected node data; extracting the first node number corresponding to the pipeline endpoint in the corrected pipeline data and the second node number in the corrected node data; performing default verification on the first node number and deduplication on the second node number to obtain preprocessed data.

[0008] Preferably, the step of importing the preprocessed data into a preset map database to generate a topology model for the drainage network includes: converting the format of the preprocessed data to generate converted data, wherein the format of the converted data corresponds to the preset map database; and importing the converted data into the preset map database to generate a topology model for the drainage network.

[0009] Preferably, the step of performing data cleaning processing on the node information based on the preset graph database to obtain first cleaned information includes: sequentially taking each node in the node information as the current node; performing a first data analysis operation on the node information based on the connected nodes of the current node to obtain first missing data; performing a second data analysis operation on the node information based on the upstream and downstream nodes of the current node to obtain second missing data; repairing the first missing data and the second missing data to obtain repaired data; obtaining a preset node range, extracting nodes with coordinate errors from the node information based on the preset node range, performing a coordinate correction operation on the nodes corresponding to the nodes with coordinate errors to obtain corrected coordinates; extracting duplicate node data based on the node information, performing a node deduplication operation on the duplicate node data to obtain deduplicated node data; and updating the node information based on the repaired data, the corrected coordinates, and the deduplicated node data to obtain the first cleaned information.

[0010] Preferably, the first data analysis operation performed on the node information based on the connected nodes of the current node to obtain the first missing data includes: obtaining the current bottom elevation value of the current node and obtaining the connected bottom elevation value of the connected nodes; determining whether the current bottom elevation value and the connected bottom elevation value meet a preset connection condition; if the current bottom elevation value and the connected bottom elevation value do not meet the preset connection condition, determining the data corresponding to the current node as the first missing data.

[0011] Preferably, the second data analysis operation performed on the node information based on the upstream and downstream nodes of the current node to obtain the second missing data includes: obtaining the upstream and downstream information of the upstream and downstream nodes; obtaining the first upstream and downstream node constraints and the second upstream and downstream node constraints; and analyzing the node information based on the current bottom elevation value, the upstream and downstream information, the first upstream and downstream node constraints, and the second upstream and downstream node constraints to extract the second missing data.

[0012] Preferably, the step of repairing the first missing data and the second missing data to obtain repaired data includes: extracting first data to be repaired from the first missing data and the second missing data that lacks ground elevation values, performing a first repair operation, and obtaining first repaired data; extracting second data to be repaired from the first missing data and the second missing data that lacks burial depth values, performing a second repair operation, and obtaining second repaired data; extracting third data to be repaired from the first missing data and the second missing data that lacks both burial depth values ​​and bottom elevation values, performing a third repair operation, and obtaining third repaired data; and obtaining repaired data based on the first repaired data, the second repaired data, and the third repaired data.

[0013] Preferably, the step of performing data cleaning processing on the pipeline information based on the preset graph database to obtain the second cleaned information includes: sequentially taking each pipeline in the above pipeline information as the current pipeline and determining the upstream and downstream pipelines of the current pipeline; obtaining the current pipeline information of the current pipeline and the upstream and downstream pipeline information of the upstream and downstream pipelines; obtaining preset upstream and downstream pipeline constraint information; determining missing pipeline data based on the current pipeline information, the upstream and downstream pipeline information, and the preset upstream and downstream pipeline constraint information; repairing the missing pipeline data to obtain repaired pipeline data; and updating the pipeline information based on the repaired pipeline data to obtain the second cleaned information.

[0014] Preferably, the step of repairing the missing pipeline data to obtain repaired pipeline data includes: repairing the diameter value of the current pipeline based on the upstream and downstream pipeline information to obtain repaired pipe diameter data; repairing the bottom elevation value of the current pipeline based on the upstream and downstream pipeline information to obtain repaired bottom elevation data; and generating repaired pipeline data based on the repaired pipe diameter data and the repaired bottom elevation value.

[0015] Accordingly, the present invention also provides a data cleaning device for urban drainage pipe network based on a graph database. The device includes: a data acquisition unit for acquiring raw drainage pipe network data and preprocessing the raw drainage pipe network data to obtain preprocessed data; a model generation unit for importing the preprocessed data into a preset graph database to generate a topology model for the drainage pipe network, the topology model including node information and pipe information; a first cleaning unit for performing data cleaning processing on the node information based on the preset graph database to obtain first cleaned information; a second cleaning unit for performing data cleaning processing on the pipe information based on the preset graph database to obtain second cleaned information; and a data determination unit for generating cleaned data based on the first cleaned information and the second cleaned information.

[0016] The present invention has at least the following technical effects through the technical solution provided by the present invention:

[0017] By improving existing data management methods and using graph databases as the carrier of drainage network data, it is possible to effectively improve the processing efficiency of massive amounts of drainage network data and achieve more efficient data cleaning.

[0018] On the other hand, by combining the actual problems existing in the actual collection and recording process of drainage network data, corresponding data cleaning was carried out on node data and pipeline data respectively, thereby ensuring the accuracy of the cleaned data and meeting the actual needs of enterprises.

[0019] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0020] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings:

[0021] Figure 1 This is a flowchart illustrating the specific implementation of the urban drainage network data cleaning method based on graph database provided in this embodiment of the invention.

[0022] Figure 2This is a schematic diagram illustrating the import of original drainage network data into a GIS system in the urban drainage network data cleaning method based on graph database provided in this embodiment of the invention.

[0023] Figure 3 This is a flowchart illustrating the specific implementation of the method for generating a topology model in the urban drainage network data cleaning method based on a graph database provided in this embodiment of the invention.

[0024] Figure 4 This is a schematic diagram of the generation of a topology model in the urban drainage network data cleaning method based on graph database provided in an embodiment of the present invention;

[0025] Figure 5 This is a flowchart illustrating the specific implementation of the data cleaning operation for node information in the urban drainage network data cleaning method based on graph database provided in this embodiment of the invention.

[0026] Figure 6 This is a schematic diagram illustrating the data analysis performed on node information based on the first upstream and downstream node constraints in the urban drainage network data cleaning method based on graph database provided in this embodiment of the invention.

[0027] Figure 7 This is a schematic diagram illustrating the data analysis performed on node information based on the second upstream and downstream node constraints in the urban drainage network data cleaning method based on graph database provided in this embodiment of the invention.

[0028] Figure 8 This is a schematic diagram of duplicate node data in the urban drainage network data cleaning method based on graph database provided in the embodiments of the present invention;

[0029] Figure 9 This is a flowchart illustrating the specific implementation of the data cleaning operation for pipeline information in the urban drainage network data cleaning method based on graph database provided in this embodiment of the invention.

[0030] Figure 10 This is a schematic diagram of a questionable pipe in the urban drainage network data cleaning method based on graph database provided in an embodiment of the present invention;

[0031] Figure 11 This is a schematic diagram illustrating the repair of missing pipeline data in the urban drainage network data cleaning method based on graph database provided in this embodiment of the invention.

[0032] Figure 12 This is a schematic diagram of the structure of the urban drainage network data cleaning device based on graph database provided in an embodiment of the present invention. Detailed Implementation

[0033] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0034] In this invention, the terms "system" and "network" are used interchangeably. "Multiple" refers to two or more; therefore, in this invention, "multiple" can also be understood as "at least two." "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / ", unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, it should be understood that in the description of this invention, terms such as "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or order.

[0035] Please see Figure 1 This invention provides a method for cleaning urban drainage network data based on a graph database, the method comprising:

[0036] S10) Obtain the original drainage network data, preprocess the original drainage network data, and obtain the preprocessed data;

[0037] S20) Import the preprocessed data into a preset graph database to generate a topology model for the drainage network, the topology model including node information and pipe information;

[0038] S30) Based on the preset graph database, perform data cleaning processing on the node information to obtain the first cleaned information;

[0039] S40) Based on the preset map database, perform data cleaning processing on the pipeline information to obtain second cleaned information;

[0040] S50) Generate post-cleaning data based on the first post-cleaning information and the second post-cleaning information.

[0041] In one possible implementation, raw drainage network data is first acquired. For example, in this embodiment, the raw drainage network data can be obtained through on-site surveys by technicians. In another embodiment, a corresponding sensor network can be set up in the drainage network, and the raw drainage network data can be automatically collected in real time through the sensor network. However, the amount of raw drainage network data acquired is large. For example, in this embodiment, the raw drainage network data includes nodes and pipes, as well as corresponding attribute data. Both nodes and pipes contain more than 5000 nodes. Node attribute data includes, but is not limited to, node number, node coordinates, node bottom elevation, node ground elevation, and node burial depth. Pipe attribute data includes, but is not limited to, the number of upstream and downstream connecting nodes, pipe diameter, and upstream and downstream bottom elevation data of the pipe. To ensure the integrity of the raw drainage network data, before data cleaning, it is necessary to ensure that the missing values ​​of each attribute do not exceed 20%.

[0042] It is easy for those skilled in the art to know that there will inevitably be data errors or data deviations in the aforementioned massive amount of data. Therefore, in order to improve the accuracy of the subsequent data cleaning process, it is necessary to preprocess the original drainage network data after obtaining it.

[0043] In this embodiment of the invention, the preprocessing of the original drainage network data to obtain preprocessed data includes: correcting the original drainage network data based on a preset database system to obtain corrected data, wherein the corrected data includes corrected pipeline data and corrected node data; extracting the first node number corresponding to the pipeline endpoint in the corrected pipeline data and the second node number in the corrected node data; performing default verification processing on the first node number and deduplication processing on the second node number to obtain preprocessed data.

[0044] In one possible implementation, the original drainage network data is first corrected based on a pre-defined database system, for example, by importing the original drainage network data into a GIS system, as described in [reference needed]. Figure 2This is a schematic diagram for importing the original drainage network data into the GIS system. The GIS system is used to delete the rainwater grates on both sides of the main pipe, retaining only the main pipe and the rainwater pipelines on it. It also checks whether the rainwater pipeline outlets are located beside rivers and whether the sewage pipelines ultimately lead to the sewage treatment plant. If any anomalies are found, corresponding corrections are made, and the corrected data is obtained. This corrected data includes corrected pipeline data and corrected node data. Then, the first node number corresponding to the pipeline endpoint is extracted from the corrected pipeline data. For example, a pre-written Python script can be used to number each node in the corrected data and generate a current node number set. Then, the current node number set is analyzed based on a pre-stored preset node number set to find default data that exists in the preset node number set but not in the current node number set, and then the corresponding empty attribute nodes are automatically added.

[0045] The process involves extracting the second node number from the corrected node data, and then using a Python script to perform a duplicate check on the second node number based on the node type of the drainage network data (including but not limited to manholes, storm drain grates, drainage outlets, supplementary nodes, septic tanks, and attributeless nodes), deleting duplicate nodes to obtain the preprocessed data. At this point, data cleaning of the drainage network data can be performed.

[0046] Please see Figure 3 In this embodiment of the invention, importing the preprocessed data into a preset graph database to generate a topology model for the drainage pipe network includes:

[0047] S21) The preprocessed data is converted to generate converted data, the format of which corresponds to the preset graph database;

[0048] S21) Import the converted data into the preset graph database to generate a topology model for the drainage pipe network.

[0049] In one possible implementation, after obtaining the preprocessed drainage network data, a graph database is used to analyze the data. For example, in this embodiment, the Neo4j graph database is used for analysis. Specifically, the preprocessed data is first converted using the Python script, for example, by converting it to the CSV format specified by the Neo4j graph database to obtain the converted data. Then, the CSV data is imported into the Neo4j graph database, and a topology model for the drainage network is generated. For example, please refer to [link to relevant documentation]. Figure 4In one embodiment, after generating the corresponding topology model, the model contains 236,124 nodes and 203,172 pipes. In this embodiment, to ensure the accuracy of the topology model, it can be limited to situations where no node or pipe in a pipe network segment with a depth exceeding 5 has a continuously missing attribute value. At this point, data cleaning operations are performed based on the aforementioned graph database and topology model.

[0050] In this embodiment of the invention, by using a graph database to store, analyze, and process drainage network data, the efficient storage, query, and association characteristics of the graph database can be effectively utilized. This enables efficient and accurate analysis and processing of massive amounts of drainage network data, greatly improving work efficiency and the accuracy of urban drainage network management.

[0051] Please see Figure 5 In this embodiment of the invention, the step of performing data cleaning processing on the node information based on the preset graph database to obtain first cleaned information includes:

[0052] S31) Sequentially select each node in the node information as the current node;

[0053] S32) Based on the connected nodes of the current node, perform a first data analysis operation on the node information to obtain the first missing data;

[0054] S33) Based on the upstream and downstream nodes of the current node, perform a second data analysis operation on the node information to obtain the second missing data;

[0055] S34) Repair the first missing data and the second missing data to obtain the repaired data;

[0056] S35) Obtain a preset node range, extract coordinate error nodes from the node information based on the preset node range, perform coordinate correction operation on the nodes corresponding to the coordinate error nodes, and obtain the corrected coordinates;

[0057] S36) Extract duplicate node data based on the node information, perform node deduplication on the duplicate node data, and obtain deduplicated node data;

[0058] S37) Update the node information based on the repaired data, the corrected coordinates, and the deduplicated node data to obtain the first cleaned information.

[0059] In one possible implementation, the node information in the topology model is first cleaned. This node information includes multiple nodes, and data cleaning can be performed on each node sequentially, thus treating each node as the current node for data cleaning. Specifically, a first data analysis operation is first performed on the node information based on the connected nodes of the current node to obtain the corresponding first missing data. For example, in this embodiment of the invention, the first data analysis operation on the node information based on the connected nodes of the current node to obtain the first missing data includes: obtaining the current bottom elevation value of the current node and obtaining the connected bottom elevation values ​​of the connected nodes; determining whether the current bottom elevation value and the connected bottom elevation value satisfy a preset connection condition; if the current bottom elevation value and the connected bottom elevation value do not satisfy the preset connection condition, the data corresponding to the current node is determined as the first missing data.

[0060] Specifically, by obtaining the current bottom elevation value of the current node and the connected bottom elevation values ​​of the connected nodes, the graph database queries for nodes whose current bottom elevation value is greater than the minimum value among the connected bottom elevation values ​​based on the connection conditions between nodes. This identifies nodes where pipelines are not connected to the building represented by the node and identifies them as suspicious nodes. At this time, the attribute values ​​of the suspicious nodes (such as including but not limited to bottom elevation value, burial depth value, and ground elevation value) can be set to empty, and the node data after setting can be marked as the first missing data.

[0061] Then, based on the upstream and downstream nodes of the current node, a second data analysis operation is performed on the node information to obtain the corresponding second missing data. In this embodiment of the invention, the second data analysis operation on the node information based on the upstream and downstream nodes of the current node to obtain the second missing data includes: obtaining the upstream and downstream information of the upstream and downstream nodes; obtaining the first upstream and downstream node constraints and the second upstream and downstream node constraints; and analyzing the node information based on the current bottom elevation value, the upstream and downstream information, the first upstream and downstream node constraints, and the second upstream and downstream node constraints to extract the second missing data.

[0062] In one possible implementation, upstream and downstream information of the current node's upstream and downstream nodes is obtained. Then, a second data analysis is performed on the node information based on the first and second upstream and downstream node constraints. Specifically, nodes satisfying the first upstream and downstream node constraints are queried through a graph database. These first upstream and downstream node constraints can be: 1. The bottom elevation values ​​of the current node and the two nodes with a depth of 1 in the upstream and downstream nodes are not empty; 2. The absolute value of the difference between the current bottom elevation value and the average of the upstream and downstream bottom elevation values ​​in the upstream and downstream information is greater than 3 meters; 3. The absolute values ​​of the differences between the current bottom elevation value and the upstream bottom elevation value, and the differences between the current bottom elevation value and the downstream bottom elevation value, are both greater than 3 meters. If the above three conditions are met, it can be determined that the bottom elevation of the current node satisfying the above constraints deviates significantly from the bottom elevation of the upstream and downstream nodes. Therefore, the current node is considered a questionable node. Please refer to [link to relevant documentation]. Figure 6 This is a schematic diagram illustrating the data analysis of node information based on the first upstream and downstream node constraints provided in an embodiment of the present invention.

[0063] The aforementioned second set of upstream and downstream node constraints can be expressed as follows: 1. The burial depth values ​​of the current node and the two upstream and downstream nodes with a depth of 1 are not empty; 2. The current bottom elevation value is not empty; 3. The absolute value of the difference between the current node's burial depth and the average upstream and downstream burial depths in the upstream and downstream node information is greater than 2 meters; 4. The absolute values ​​of the current node's burial depth and the upstream burial depth, as well as the absolute values ​​of the difference between the current node's burial depth and the downstream burial depth, are both greater than 2 meters. If the above four conditions are met, it can be determined that the bottom elevation of the current node is not significantly different from that of the upstream and downstream nodes, but the ground elevation and burial depth values ​​deviate significantly from those of the upstream and downstream nodes. Therefore, the current node is considered a questionable node. Please refer to [link to relevant documentation]. Figure 7 This is a schematic diagram illustrating the data analysis of node information based on the second upstream and downstream node constraints provided in an embodiment of the present invention.

[0064] At this point, the questionable node attribute values ​​(such as, but not limited to, bottom elevation, burial depth, and ground elevation) queried based on the first and second upstream and downstream node constraints are set to empty, and the set node data is marked as the second missing data. Then, the first and second missing data are repaired to obtain the corresponding repaired data. In this embodiment of the invention, the repair of the first and second missing data to obtain the repaired data includes: extracting the first data to be repaired from the first and second missing data that lacks ground elevation values, performing a first repair operation, and obtaining the first repaired data; extracting the second data to be repaired from the first and second missing data that lacks burial depth values, performing a second repair operation, and obtaining the second repaired data; extracting the third data to be repaired from the first and second missing data that lacks both burial depth and bottom elevation values, performing a third repair operation, and obtaining the third repaired data; and obtaining the repaired data based on the first repaired data, the second repaired data, and the third repaired data.

[0065] For example, firstly, the graph database is used to query nodes with missing ground elevation data in the first and second missing data. Then, the first repair operation is performed. Specifically, if there are nodes with non-empty ground elevations upstream and downstream of a node within depth 5, the ground elevation value of the current node is linearly interpolated based on the ground elevations of the upstream and downstream nodes with ground elevations, and the fill value is set to the ground elevation value of the current node through the graph database. If there are nodes with non-empty ground elevations upstream of a node within depth 5, but not downstream, the ground elevation value of the current node is forward interpolated based on the ground elevation value of the upstream node, and the fill value is set to the ground elevation value of the current node through the graph database. If there are nodes with non-empty ground elevations downstream of a node within depth 5, but not upstream, the ground elevation value of the current node is backward interpolated based on the ground elevation value of the downstream node, and the fill value is set to the ground elevation value of the current node through the graph database. This is how the first repaired data is obtained.

[0066] Then, the graph database is used to query nodes that have ground elevation and bottom elevation values ​​but lack burial depth values ​​in the first and second missing data. At this time, the second repair operation is performed. Specifically, the difference between the ground elevation value and bottom elevation value of the current node is used as the fill value and set as the burial depth value of the current node, thereby obtaining the second repair data.

[0067] At this point, the graph database is further used to query nodes in the first and second missing data that have ground elevations but lack burial depth and bottom elevation values. Then, the third repair operation is performed. Specifically, if there are nodes with non-empty bottom elevation values ​​upstream and downstream of a node within depth 5, linear interpolation is performed based on the bottom elevation values ​​of the upstream and downstream nodes. If there are nodes with non-empty bottom elevations upstream of a node within depth 5, but not downstream, the bottom elevation value of the current node is forward interpolated based on the bottom elevation value of the upstream node. If there are nodes with non-empty bottom elevations downstream of a node within depth 5, but not upstream, the bottom elevation value of the current node is backward interpolated based on the bottom elevation value of the downstream node. This process is used to obtain the third repair data.

[0068] It should be noted that, in order to further improve the accuracy of the repair, after repairing the above data and obtaining the corresponding repaired data, the first and second analysis operations mentioned above can be performed again on the repaired data to determine whether the repaired data meets the actual requirements. If the actual requirements are met, the corresponding repaired data is obtained based on the first, second, and third repaired data, and subsequent data cleaning operations are performed. If the actual requirements are not met, the attribute values ​​of the connected nodes of the current node are used as the attribute values ​​of the current node, and the final repaired data is obtained.

[0069] Since the data of the drainage network corresponds to the actual geographical area, the data may be recorded beyond its actual range during the data acquisition process, which may further cause data deviation and errors.

[0070] To address the aforementioned technical issues, further data cleaning is performed on the node information based on the actual range of the data. Specifically, a preset node range is first obtained, which can be determined manually by technicians. Then, based on this preset range, nodes with coordinate errors are extracted from the node information. For example, the node information also includes the geographic coordinates (x, y) of each node. The graph database is used to query whether the x and y coordinate values ​​of the current node are within a reasonable range. Nodes with x or y coordinates outside the reasonable range are considered as coordinate error nodes. Then, data correction is performed on the node data corresponding to the aforementioned coordinate error nodes. Specifically, a Python script can be used to extract the x or y coordinates of the aforementioned coordinate error nodes and change their decimal point position so that the integer part of the corrected coordinates meets the aforementioned reasonable range. At this point, it is further determined whether the overall corrected coordinates meet the aforementioned reasonable range. If so, the adjusted coordinate value replaces the original incorrect x or y coordinate, and the corrected coordinates are obtained.

[0071] Furthermore, the aforementioned node information inevitably contains a large amount of duplicate data due to repeated measurements or records. Therefore, it is necessary to deduplicatize this duplicate data. Specifically, the graph database can be used to query node pairs with the same x and y coordinate values. The number of upstream and downstream nodes within a depth of 5 for each node can be checked. If one node has fewer than 5 upstream and downstream nodes within a depth of 5, and the other node has more than 5, then the node with fewer than 5 upstream and downstream nodes within a depth of 5, along with its upstream and downstream nodes, is deleted, and the pipes connected to the deleted node are also deleted. If the number of upstream and downstream nodes within a depth of 5 for both nodes is less than 5, then both nodes and their upstream and downstream nodes are deleted, and the pipes connected to the deleted node are also deleted. If the number of upstream and downstream nodes within a depth of 5 for both nodes is greater than 5, then the graph database is used to find nodes with a depth of 1 upstream and downstream of the node with the smaller number of upstream and downstream nodes. Pipes connecting these nodes to another node with the same coordinate are created, inheriting the pipe attributes between these nodes and the node with the smaller number of upstream and downstream nodes. Then, the node with the smaller number of upstream and downstream nodes and its connected pipes are deleted. Please refer to [link to relevant documentation]. Figure 8 This is a schematic diagram of node duplicate data provided in an embodiment of the present invention;

[0072] Furthermore, by querying the graph database, two nodes that meet the following three conditions are identified: 1. There is a pipe connecting the two nodes; 2. There are 2-5 pipes connecting the two nodes; 3. In condition 2, all nodes on the link connecting the two nodes with 2-5 pipes have only one upstream node and one downstream node within a depth range of 1. Two nodes that meet the above three conditions may be due to repeated measurements of the pipeline. Therefore, the above points and the pipes connected to them are deleted, and the deduplicated data of the nodes is obtained, thus completing the deduplication operation for duplicate nodes.

[0073] After completing the above data analysis and processing operations, the node information is updated based on the repaired data, corrected coordinates, and deduplicated node data to obtain the first cleaned information. This completes the data cleaning operation for the node information; at this point, data cleaning can be performed on the pipeline information.

[0074] Please see Figure 9 In this embodiment of the invention, the step of performing data cleaning processing on the pipeline information based on the preset map database to obtain second cleaned information includes:

[0075] S41) Sequentially take each pipeline in the pipeline information as the current pipeline and determine the upstream and downstream pipelines of the current pipeline;

[0076] S42) Obtain the current pipeline information of the current pipeline and the upstream and downstream pipeline information of the upstream and downstream pipelines;

[0077] S43) Obtain preset upstream and downstream pipeline constraint information;

[0078] S44) Determine missing pipeline data based on the current pipeline information, the upstream and downstream pipeline information, and the preset upstream and downstream pipeline constraint information;

[0079] S45) Repair the missing pipeline data to obtain repaired pipeline data;

[0080] S46) Update the pipeline information based on the repaired pipeline data to obtain the second post-cleaning information.

[0081] In one possible implementation, data cleaning is performed sequentially on each pipeline in the pipeline information. Specifically, first, the current pipeline information and the upstream and downstream pipeline information of the current pipeline are obtained. At this time, preset upstream and downstream pipeline constraint information is obtained. For example, the preset upstream and downstream pipeline constraint information can be expressed as: 1. The pipe diameter values ​​of the two pipelines with a depth of 1 in the current pipeline and the upstream and downstream pipelines are not empty; 2. The absolute value of the difference between the pipe diameter of the current pipeline and the average diameter value of the upstream and downstream pipelines is greater than 0.2 meters; 3. The absolute values ​​of the difference between the pipe diameter value of the current pipeline and the pipe diameter value of the upstream pipeline, and the absolute values ​​of the difference between the pipe diameter value of the current pipeline and the pipe diameter value of the downstream pipeline are both greater than 0.2 meters. If the above three conditions are met, the current pipeline is considered a questionable pipeline. Please refer to [link to relevant documentation]. Figure 10 This is a schematic diagram of a questionable pipeline provided in an embodiment of the present invention. At this time, the pipeline data of the current pipeline can be reset, and the reset pipeline data can be determined as missing pipeline data.

[0082] On the other hand, the difference between the bottom elevation of the current pipeline's front end and the bottom elevation of the upstream pipeline connection node is defined as the inflow head, and the difference between the bottom elevation of the current pipeline's rear end and the bottom elevation of the downstream pipeline connection node is defined as the outflow head. Pipelines with an inflow head or outflow head greater than 2 meters are queried from the graph database and considered as pipelines with questionable front and rear end bottom elevation data. Their pipeline data is then reset, and the reset pipeline data is identified as missing pipeline data. This missing pipeline data is then repaired.

[0083] In this embodiment of the invention, the step of repairing the missing pipeline data to obtain repaired pipeline data includes: repairing the diameter value of the current pipeline based on the upstream and downstream pipeline information to obtain repaired pipe diameter data; repairing the bottom elevation value of the current pipeline based on the upstream and downstream pipeline information to obtain repaired bottom elevation data; and generating repaired pipeline data based on the repaired pipe diameter data and the repaired bottom elevation value.

[0084] For example, firstly, the graph database is used to find pipes with missing diameter values. If a pipe with a diameter value exists within a depth of 5 downstream of the current pipe, then the diameter value of the downstream pipe with the smallest depth and a diameter value is set as the diameter value of the current pipe. If there is no pipe with a diameter value within a depth of 5 downstream of the current pipe, but a pipe with a diameter value exists within a depth of 5 upstream of the current pipe, then the diameter value of the upstream pipe with the smallest depth and a diameter value is set as the diameter value of the current pipe.

[0085] Then, the graph database is used to find the pipeline with missing upstream bottom elevation values, and the bottom elevation values ​​of the nodes connected to the upstream of that pipeline are used as the bottom elevation values ​​of the upstream pipeline. Next, the graph database is used to find the pipeline with missing downstream bottom elevation values, and the bottom elevation values ​​of the nodes connected to the downstream of that pipeline are used as the bottom elevation values ​​of the downstream pipeline. This process repairs all missing pipeline data and obtains the corresponding repaired pipeline data (modified pipelines). For example, please refer to [link to relevant documentation]. Figure 11 This is a schematic diagram of repairing missing pipeline data provided in an embodiment of the present invention. The pipeline information is updated based on the repaired pipeline data, and the corresponding second cleaning information is obtained.

[0086] After obtaining the first and second post-cleaning information, post-cleaning data for urban drainage network data is obtained.

[0087] In this embodiment of the invention, by using a graph database as the data carrier for drainage network data and cleaning the drainage network data through the graph database, the cleaning efficiency of data cleaning can be greatly improved, thus increasing work efficiency. At the same time, by accurately cleaning the drainage network data based on the actual influencing factors existing in the process of data collection and recording, the accuracy of the drainage network data can be greatly improved, providing strong data support for subsequent management, analysis and maintenance, and meeting the actual needs of enterprises.

[0088] The following description, in conjunction with the accompanying drawings, describes the urban drainage network data cleaning device based on a graph database provided in an embodiment of the present invention.

[0089] Please see Figure 12Based on the same inventive concept, this invention provides a data cleaning device for urban drainage pipe networks based on a graph database. The device includes: a data acquisition unit for acquiring raw drainage pipe network data and preprocessing the raw drainage pipe network data to obtain preprocessed data; a model generation unit for importing the preprocessed data into a preset graph database to generate a topology model for the drainage pipe network, the topology model including node information and pipe information; a first cleaning unit for performing data cleaning processing on the node information based on the preset graph database to obtain first cleaned information; a second cleaning unit for performing data cleaning processing on the pipe information based on the preset graph database to obtain second cleaned information; and a data determination unit for generating cleaned data based on the first cleaned information and the second cleaned information.

[0090] The optional embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details in the above embodiments. Within the scope of the technical concept of the embodiments of the present invention, various simple modifications can be made to the technical solutions of the embodiments of the present invention, and these simple modifications all fall within the protection scope of the embodiments of the present invention.

[0091] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the embodiments of the present invention will not describe the various possible combinations separately.

[0092] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a microcontroller, chip, or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0093] Furthermore, various different implementations of the present invention can be combined arbitrarily, as long as they do not violate the spirit of the present invention, they should also be regarded as the content disclosed in the present invention.

Claims

1. A method for cleaning urban drainage network data based on graph databases, characterized in that, The method includes: Obtain raw drainage network data, preprocess the raw drainage network data, and obtain preprocessed data; The preprocessed data is imported into a preset graph database to generate a topology model for the drainage network, the topology model including node information and pipe information; Based on the preset graph database, the node information is cleaned to obtain first cleaned information: each node in the node information is taken as the current node; a first data analysis operation is performed on the node information based on the connected nodes of the current node to obtain first missing data; a second data analysis operation is performed on the node information based on the upstream and downstream nodes of the current node to obtain second missing data; the first missing data and the second missing data are repaired to obtain repaired data; a preset node range is obtained, and nodes with coordinate errors are extracted from the node information based on the preset node range, and coordinate correction operations are performed on the nodes corresponding to the nodes with coordinate errors to obtain corrected coordinates; duplicate node data is extracted based on the node information, and node deduplication operations are performed on the duplicate node data to obtain deduplicated node data; the node information is updated based on the repaired data, the corrected coordinates, and the deduplicated node data to obtain the first cleaned information. Based on the preset graph database, the pipeline information is cleaned to obtain second cleaned information: Each pipeline in the above pipeline information is sequentially taken as the current pipeline, and its upstream and downstream pipelines are determined; the current pipeline information and the upstream and downstream pipeline information of the current pipeline are obtained; preset upstream and downstream pipeline constraint information is obtained; missing pipeline data is determined based on the current pipeline information, the upstream and downstream pipeline information, and the preset upstream and downstream pipeline constraint information; the missing pipeline data is repaired to obtain repaired pipeline data; the pipeline information is updated based on the repaired pipeline data to obtain second cleaned information. Post-cleaning data is generated based on the first post-cleaning information and the second post-cleaning information.

2. The method according to claim 1, characterized in that, The preprocessing of the original drainage network data to obtain preprocessed data includes: The original drainage network data is corrected based on a preset database system to obtain corrected data, which includes corrected pipeline data and corrected node data. Extract the first node number corresponding to the pipe endpoint in the corrected pipe data, and the second node number in the corrected node data; The first node number is subjected to default verification and the second node number is subjected to deduplication to obtain preprocessed data.

3. The method according to claim 1, characterized in that, The step of importing the preprocessed data into a preset map database to generate a topology model for the drainage pipe network includes: The preprocessed data is converted to generate converted data, and the format of the converted data corresponds to the preset graph database. The transformed data is imported into the preset graph database to generate a topology model for the drainage pipe network.

4. The method according to claim 3, characterized in that, The first data analysis operation performed on the node information based on the connected nodes of the current node to obtain the first missing data includes: Obtain the current bottom elevation value of the current node and the connected bottom elevation value of the connected nodes; Determine whether the current bottom elevation value and the connected bottom elevation value meet the preset connection conditions; If the current bottom elevation value and the connected bottom elevation value do not meet the preset connection conditions, the data corresponding to the current node will be determined as the first missing data.

5. The method according to claim 4, characterized in that, The second data analysis operation performed on the node information based on the upstream and downstream nodes of the current node to obtain the second missing data includes: Obtain the upstream and downstream information of the upstream and downstream nodes; Obtain the constraints of the first upstream and downstream nodes and the second upstream and downstream nodes; The node information is analyzed based on the current bottom elevation value, the upstream and downstream information, the first upstream and downstream node constraints, and the second upstream and downstream node constraints to extract the second missing data.

6. The method according to claim 3, characterized in that, The step of repairing the first missing data and the second missing data to obtain repaired data includes: Extract the first data to be repaired from the first missing data and the second missing data that lack ground elevation values, perform the first repair operation, and obtain the first repaired data; Extract the second data to be repaired from the first missing data and the second missing data that lack burial depth values, perform the second repair operation, and obtain the second repaired data; Extract the third data to be repaired from the first missing data and the second missing data that lack burial depth and bottom elevation values, perform the third repair operation, and obtain the third repaired data; The repaired data is obtained based on the first repair data, the second repair data, and the third repair data.

7. The method according to claim 6, characterized in that, The process of repairing the missing pipeline data to obtain repaired pipeline data includes: Based on the upstream and downstream pipeline information, the current pipeline diameter value is repaired to obtain the repaired pipeline diameter data; Based on the upstream and downstream pipeline information, the bottom elevation value of the current pipeline is repaired to obtain the repaired bottom elevation data; Repaired pipe data is generated based on the repaired pipe diameter data and the repaired bottom elevation data.

8. A data cleaning device for urban drainage pipe networks based on graph databases, characterized in that, The device includes: The data acquisition unit is used to acquire raw drainage network data, preprocess the raw drainage network data, and obtain preprocessed data. The model generation unit is used to import the preprocessed data into a preset graph database to generate a topology model for the drainage network, the topology model including node information and pipe information; The first cleaning unit is configured to perform data cleaning processing on the node information based on the preset graph database to obtain first cleaned information: Each node in the node information is sequentially taken as the current node; a first data analysis operation is performed on the node information based on the connected nodes of the current node to obtain first missing data; a second data analysis operation is performed on the node information based on the upstream and downstream nodes of the current node to obtain second missing data; the first missing data and the second missing data are repaired to obtain repaired data; a preset node range is obtained; nodes with coordinate errors are extracted from the node information based on the preset node range; coordinate correction operations are performed on the nodes corresponding to the nodes with coordinate errors to obtain corrected coordinates; duplicate node data is extracted based on the node information; node deduplication operations are performed on the duplicate node data to obtain deduplicated node data; the node information is updated based on the repaired data, the corrected coordinates, and the deduplicated node data to obtain the first cleaned information. The second cleaning unit is used to perform data cleaning processing on the pipeline information based on the preset map database to obtain second cleaned information: Each pipeline in the above pipeline information is sequentially taken as the current pipeline, and the upstream and downstream pipelines of the current pipeline are determined; the current pipeline information of the current pipeline and the upstream and downstream pipeline information of the upstream and downstream pipelines are obtained; preset upstream and downstream pipeline constraint information is obtained; missing pipeline data is determined based on the current pipeline information, the upstream and downstream pipeline information, and the preset upstream and downstream pipeline constraint information; the missing pipeline data is repaired to obtain repaired pipeline data; and the pipeline information is updated based on the repaired pipeline data to obtain second cleaned information. The data determination unit is used to generate post-cleaning data based on the first post-cleaning information and the second post-cleaning information.

Images