Map data preprocessing method and device, electronic equipment and readable storage medium

Abstract

The present disclosure relates to the technical field of map data processing, and provides a map data preprocessing method and device, an electronic device and a readable storage medium. The method comprises: receiving uploaded map data; analyzing the map data to obtain geometric data, attribute data and spatial reference information of the map data, automatically identifying an original coordinate system and a projection mode according to the spatial reference information, and converting the geometric data to a target coordinate system through a coordinate system conversion algorithm; storing the converted geometric data and corresponding attribute data to a distributed database, calculating the outer bounding latitude and longitude range of all the geometric data, and storing the map sheet number and corresponding geometric data and attribute data to the distributed database; calling the geometric data and corresponding attribute data in batches according to the map sheet number, detecting topological abnormalities and / or geometric abnormalities of the called data based on a preset rule, automatically repairing the abnormalities through a preset algorithm, and repeating the detection and repair steps until the processing result meets the standard.

Description

Technical Field

[0001] This disclosure relates to the field of map data processing technology, and in particular to a map data preprocessing method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] With the rapid development of digital surveying and mapping technology, satellite remote sensing technology, and Internet of Things technology, the sources of map data are becoming increasingly diversified, including collection by professional surveying and mapping equipment, provision by third-party data providers, and crowdsourced collection. As the foundation for core applications such as navigation and positioning, intelligent traffic scheduling, and urban planning decisions, the quality of map data directly determines the accuracy and reliability of the upper-level systems.

[0003] Traditional map data preprocessing relies on professional technicians to manually check coordinate deviations, fix topological errors, and correct road morphology. For large-scale city-level road network data, it often takes several technicians several weeks to complete, which is difficult to meet the needs of rapid processing of massive amounts of data.

[0004] Furthermore, map data from different data sources have different formats and use different coordinate systems and projection methods, making it difficult for traditional processing tools to achieve unified compatibility across multiple formats and automatic coordinate conversion; the time and economic costs of manual preprocessing are difficult to control; and the algorithms and rules of traditional processing tools are fixed and cannot be customized according to the needs of different application scenarios, resulting in poor adaptability. Summary of the Invention

[0005] In view of this, the present disclosure provides a map data preprocessing method, apparatus, electronic device, and computer-readable storage medium to solve the technical problems of low efficiency and poor compatibility in the prior art of map data preprocessing.

[0006] A first aspect of this disclosure provides a map data preprocessing method, comprising: receiving uploaded map data, the map data including at least two formats; parsing the map data to obtain geometric data, attribute data, and spatial reference information of the map data; automatically identifying the original coordinate system and projection method based on the spatial reference information; and converting the geometric data to a target coordinate system using a coordinate system transformation algorithm; storing the converted geometric data and corresponding attribute data in a distributed database; calculating the bounding latitude and longitude range of all geometric data; generating a map sheet number based on the bounding latitude and longitude range; and associating the map sheet number with the corresponding geometric data and attribute data and storing it in the distributed database; retrieving geometric data and corresponding attribute data in batches according to the map sheet number; detecting topological anomalies and / or geometric anomalies in the retrieved data based on preset rules; automatically repairing the anomalies using a preset algorithm; and repeating the detection and repair steps until the processing result meets the standard.

[0007] A second aspect of this disclosure provides a map data preprocessing apparatus, comprising: a data receiving module for receiving uploaded map data, the map data including at least two formats; a parsing and coordinate unification module for parsing the map data to obtain geometric data, attribute data, and spatial reference information of the map data, automatically identifying the original coordinate system and projection method based on the spatial reference information, and converting the geometric data to a target coordinate system through a coordinate system transformation algorithm; a storage and map sheet calculation module for storing the converted geometric data and corresponding attribute data in a distributed database, calculating the bounding latitude and longitude range of all geometric data, generating a map sheet number based on the bounding latitude and longitude range, and associating the map sheet number with the corresponding geometric data and attribute data and storing it in the distributed database; and an automated processing module for retrieving geometric data and corresponding attribute data in batches according to the map sheet number, detecting topological anomalies and / or geometric anomalies in the retrieved data based on preset rules, automatically repairing the anomalies through a preset algorithm, and repeating the detection and repair steps until the processing result meets the standard.

[0008] A third aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0009] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0010] Compared with the prior art, the beneficial effects of this disclosed embodiment are as follows: by parsing and converting map data of different formats and coordinate systems, generating map sheet numbers based on the bounding latitude and longitude range of geometric data, storing the processed data in a distributed database, and detecting and automatically repairing the stored data, the entire process of map data from import to export is automated, which improves processing efficiency, reduces labor costs, and can simultaneously adapt to multi-format, multi-coordinate system data and different application scenarios. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic flowchart of a map data preprocessing method provided in an embodiment of this disclosure; Figure 2This is a flowchart illustrating another map data preprocessing method provided in this embodiment of the disclosure; Figure 3 This is a flowchart illustrating the detection and automated repair method provided in the embodiments of this disclosure; Figure 4 This is a schematic diagram of the structure of a map data preprocessing device provided in an embodiment of this disclosure; Figure 5 This is a schematic diagram of another map data preprocessing apparatus provided in an embodiment of this disclosure; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0013] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, so as to provide a thorough understanding of the embodiments of this disclosure. However, those skilled in the art will understand that this disclosure may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this disclosure with unnecessary detail.

[0014] The map data preprocessing method and apparatus according to embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings.

[0015] Figure 1 This is a schematic flowchart of a map data preprocessing method provided in an embodiment of this disclosure. The method provided in this embodiment can be executed by any electronic device with computer processing capabilities, such as a terminal or server. Figure 1 As shown, the map data preprocessing method includes: Step S101: Receive the uploaded map data, which includes at least two formats.

[0016] Step S102: Parse the map data to obtain the geometric data, attribute data and spatial reference information of the map data. Automatically identify the original coordinate system and projection method based on the spatial reference information, and transform the geometric data to the target coordinate system through the coordinate system transformation algorithm.

[0017] Specifically, parsing map data can yield multiple layers such as road networks, nodes, relationships, and background surfaces. From the road network layer, node layer, relationship layer, and background surface layer, each specific element of the layer can be extracted, such as the geometric data, attribute data, and spatial reference information of each road and each node.

[0018] Step S103: Store the converted geometric data and corresponding attribute data in a distributed database, calculate the bounding latitude and longitude range of all geometric data, generate map sheet number based on the bounding latitude and longitude range, and associate the map sheet number with the corresponding geometric data and attribute data and store it in the distributed database.

[0019] Step S104: Retrieve geometric data and corresponding attribute data in batches according to map sheet number; detect topological anomalies and / or geometric anomalies in the retrieved data based on preset rules; automatically repair the anomalies using preset algorithms; and repeat the detection and repair steps until the processing result meets the standard.

[0020] This solution achieves end-to-end automated processing of map data, from uploading, parsing, coordinate unification, storage, batch-based automated repair to achieving satisfactory results, reducing manual intervention. It supports multi-format data compatibility and automatic conversion between multiple coordinate systems, resolving compatibility issues with multi-source data. Distributed database storage and batch processing by map sheet improve the efficiency of processing massive amounts of data. A cyclical detection and repair mechanism ensures the quality of the processed results meets standards, providing high-precision, highly consistent map data for subsequent applications.

[0021] In step S101, map data is received via the HTTP protocol. The map data supports multiple formats such as Shapefile, GeoJSON, and MID / MIF. The system automatically matches the corresponding parser through the format recognition module to achieve compatible import of multi-format data. After step S101, the map data can also be validated for format validity and / or content integrity.

[0022] By enabling convenient data uploads via the HTTP protocol and performing format validity and content integrity checks on the data before processing, invalid or corrupted data can be filtered out in advance, preventing invalid data from entering the subsequent processing flow and improving the reliability and stability of the entire preprocessing process.

[0023] During data parsing in step S102, parsers corresponding to the GDAL, OGR, JTS, GeoTools, and Proj4J libraries can be called to read the road network layer, node layer, relationship layer, and attribute data corresponding to each layer from the uploaded data. These attribute data can include road names, lengths, grades, node numbers, connection relationships, etc., and are not limited to these.

[0024] Spatial metadata parsing technology can automatically identify the original coordinate system and projection method of data. The original coordinate system can be WGS84 or GCJ02, and the projection method can be Gauss-Kruger or Mercator projection. Using coordinate system transformation algorithms, data from different coordinate systems and projection methods can be uniformly converted to a preset target coordinate system, ensuring data consistency in subsequent processing. These transformation algorithms can include Gauss-Kruger projection transformation, Mercator projection transformation, and WGS84 to GCJ02 coordinate system transformation. The target coordinate system can be configured by the user or use WGS84 as the default.

[0025] In this embodiment, the distributed database includes a spatial index table and an attribute relationship table. Step S103 may include the following steps: First, traverse the coordinates of the spatial features of all geometric data and calculate the bounding latitude and longitude range of the geometric data, which includes minimum longitude, maximum longitude, minimum latitude, and maximum latitude. Then, according to a preset map sheet division standard, calculate the set of map sheet numbers covered based on the bounding latitude and longitude range. Finally, associate the map sheet numbers in the set of map sheet numbers with the corresponding geometric data and store them in the spatial index table, and store the attribute data corresponding to the geometric data in the attribute relationship table.

[0026] By separating the spatial index table and the attribute relationship table, the spatial data and attribute data can be stored in a decoupled manner: the spatial index table is partitioned based on map sheet number, supporting efficient spatial range query and batch retrieval by map sheet; the attribute relationship table is stored in JSONB format, supporting flexibly expandable attribute fields and efficient query; the two are linked by feature identifiers, which can significantly improve storage and query performance while ensuring data integrity, and adapt to the distributed processing needs of large-scale map data.

[0027] Citus is a distributed relational database based on PostgreSQL. It is compatible with the PostgreSQL ecosystem, such as PostGIS spatial extension and native support for JSONB data types, making it naturally suitable for mixed storage scenarios of "spatial data + attribute data". It supports distributed partitioned tables, which can be horizontally expanded based on business dimensions such as map sheet number to match the partitioning requirements of spatial data divided by map sheet. It retains the ACID properties of PostgreSQL to ensure the consistency of core data such as road network / nodes, while supporting high-performance parallel queries to meet the efficiency requirements of spatial range queries.

[0028] In Citus's storage structure, the spatial index table is a spatial partition table based on map sheet numbers, supporting efficient spatial range queries. It closely matches the geographical boundary characteristics of spatial data and quickly filters out non-target map sheet fragments during queries, reducing the scanning range. The attribute relationship table stores attribute data for road networks, nodes, and relationship layers. It uses JSONB format to achieve mixed storage of structured and unstructured data. Unlike ordinary JSON, JSONB supports field-level indexing / querying / updating, accommodating both structured and unstructured data, avoiding the "field bloat" or "multi-table joins" problems of traditional tables, and adapting to business scenarios with flexible expansion of road network attributes.

[0029] Specifically, the parsed and coordinate-unified layer feature data and attribute data can be stored in the corresponding data tables of the Citus distributed database, using spatial indexing technology to optimize data query efficiency. After data storage, the background automatically traverses all spatial features, calculates the maximum longitude, maximum latitude, minimum longitude, and minimum latitude of the data's bounding area, forming the spatial boundary. Based on the preset map sheet division rules, the map sheet number covered by the above spatial boundary is calculated, and the map sheet number is associated with the corresponding data and stored in the Citus distributed database, providing a basis for subsequent batch processing. The preset map sheet division rules can be based on the 1:25,000 scale map sheet division standard.

[0030] Next, step S104 is executed. Based on the map sheet number calculated in step S103, the system quickly locates the layer features of the target map sheet from the Citus distributed database through the spatial index table. Data is retrieved in batches according to map sheet, and an automated "rule detection-algorithm repair" process is executed for each batch of data. Specifically, when retrieving data in batches, the parallel query capability of the Citus distributed database can be used to batch load feature data such as road networks, nodes, relationships, and background surfaces within the current map sheet. The attribute information of features such as road level and surface feature type is parsed by combining the attribute relationship table, providing a data foundation for subsequent rule matching.

[0031] In step S104, the preset rules used for anomaly detection based on preset rules include hanging node processing rules, pseudo node processing rules, road breakage processing rules, background and road overlap processing rules, self-intersecting road processing rules, self-intersecting surface processing rules, redundant node removal rules, surface feature gap processing rules, duplicate road deduplication rules, duplicate background surface deduplication rules, and surface feature removal rules.

[0032] These preset rules form a rule base covering a variety of common map data anomalies, standardizing and systematizing anomaly detection, ensuring that all kinds of topological errors and geometric defects can be accurately identified, providing a comprehensive basis for subsequent automated repair, thereby improving the logical consistency and geometric integrity of the processed data.

[0033] Specifically, the system calls a preset rule base to perform item-by-item checks on the loaded feature data, identifying abnormal features that need to be processed. The content includes: According to the "Road Endpoint Repair Rules", the distance between the road endpoint and the adjacent road is detected, and the endpoints that exceed the set threshold connection tolerance are marked as needing repair. The threshold can be 0.5 meters.

[0034] Based on the "Hanging Node Processing Rules", unconnected line feature endpoints with a length greater than or equal to a set threshold are selected and marked as hanging nodes. The threshold can be 0.5 meters.

[0035] Based on the "pseudo-node processing rules", pseudo-nodes are identified by calculating the vector angle. When calculating the vector angle, three consecutive nodes are collinear and the angle between them is ≤10°.

[0036] Based on the "Road Fracture Repair Rules", the directional deviation between adjacent roads is analyzed, and line segments with a deviation ≥45° are marked as fractures.

[0037] According to the "Rules for Processing Background and Road Overlay Elements", the intersection area ratio of surface elements and road line elements is calculated, and areas with an intersection area greater than or equal to the set threshold are marked as overlays. The threshold can be 10%.

[0038] Based on the "self-intersecting road / surface processing rules", self-intersecting line features and surface features are marked through topological relationship detection.

[0039] Based on the "redundant node removal rule", redundant vertices are identified by analyzing the included angle and length ratio of three consecutive node vectors, where the included angle is less than or equal to 5° and the length ratio is greater than or equal to 1:10.

[0040] According to the "Surface Feature Gap Processing Rules", the gaps between adjacent surface boundaries are detected, and areas with gaps greater than or equal to a set threshold are marked as gaps. The threshold can be 0.3 meters.

[0041] Based on the "duplicate road / background surface deduplication rule", duplicate elements are identified by comparing geometric similarity and attribute consistency. Elements with a geometric similarity of 95% or higher and an attribute consistency of 90% or higher are considered duplicate elements.

[0042] Based on the "surface feature removal rules", surface features with an area smaller than the set threshold are filtered out and marked as useless surfaces. The threshold can be 0.5 square meters.

[0043] In step S104, the preset algorithms used for automated repair based on preset algorithms include suspended node processing algorithm, pseudo node processing algorithm, road fracture repair algorithm, background and road overlay processing algorithm, self-intersecting road processing algorithm, self-intersecting surface processing algorithm, redundant node removal algorithm, surface feature gap processing algorithm, and road endpoint repair algorithm.

[0044] These preset algorithms provide dedicated automated repair algorithms for various anomalies identified by the rule base, such as node snapping based on KD tree index, pseudo-node removal by vector collinearity analysis, and self-intersection processing of polyline segmentation, ensuring that the repair process is accurate and efficient, and that the repaired data meets the preset quality threshold, significantly improving the topological correctness and geometric accuracy of map data.

[0045] Specifically, for the detected abnormal elements, the system calls the corresponding preset algorithm for automated repair, including: The "dangling node processing algorithm" is used for dangling nodes. It quickly matches neighboring target nodes through KD tree indexing and attaches the dangling endpoints to the target nodes based on coordinate interpolation, thus eliminating isolated endpoints.

[0046] The pseudo-node processing algorithm is used to handle pseudo-nodes. Based on vector collinearity analysis (angle threshold ≤ 5°, distance threshold ≤ 0.3 meters), redundant intermediate nodes are deleted or the topology of line elements is reconstructed, and continuous collinear line segments are merged. The angle threshold and distance threshold of vector collinearity analysis are less than or equal to 5° and 0.3 meters, respectively.

[0047] The "self-intersecting road processing algorithm" is used for self-intersecting roads. The intersection point is identified with a threshold of 0.1 meters as the tolerance. The self-intersecting line elements are split into independent non-intersecting sub-segments through a multi-segment line segmentation algorithm, and the connection relationship between the sub-segments is adjusted. The "self-intersecting surface processing algorithm" is used for self-intersecting surfaces. When the proportion of the length of the self-intersecting loop is greater than or equal to the threshold, the effective outer loop and inner loop are extracted by the loop boundary splitting algorithm, and the simple surface features without self-intersecting loops are reconstructed. The threshold can be 5%.

[0048] The "background and road overlap processing algorithm" is used to handle background and road overlap. For areas where the overlap area ratio is greater than or equal to a threshold, the boundary of the surface feature is adjusted by the minimum common edge fusion algorithm to eliminate overlapping areas. The threshold can be 10%.

[0049] The gap between opposite features is handled by the "surface feature gap processing algorithm". For adjacent faces with a gap distance of less than or equal to the threshold of 0.3 meters, the boundary is extended or shortened by the boundary alignment algorithm to achieve seamless splicing. The threshold can be 0.3.

[0050] The "road breakage repair algorithm" is used to repair road breaks. Combined with directional consistency analysis, the broken road is restored to connectivity by inserting transition segments or adjusting the endpoint positions. Road breaks are defined as those with a deviation of 45° or greater and a distance of 2 meters or greater.

[0051] A "redundant node elimination algorithm" is used for redundant nodes. Based on vector collinearity analysis, iteratively eliminates vertices that have minimal impact on the shape of line features according to a preset precision threshold, simplifying the geometric structure. Among them, features with an included angle of less than or equal to 5° and a length ratio of greater than or equal to 1:10 are identified as redundant nodes.

[0052] The "road endpoint repair algorithm" is used for road endpoints. Within a tolerance range of 0.5 meters to 2 meters, the algorithm automatically matches the endpoints to the nearest valid connection point through dynamic connection, ensuring the integrity of the road network topology.

[0053] After each batch of data is processed, the system automatically performs a secondary verification on the repaired elements, such as re-checking topological relationships and geometric integrity, to ensure that the repair results meet the preset rules. If there are any elements that do not meet the standards, a secondary repair process is triggered until the requirements are met. After step S104, the repaired geometric data and corresponding attribute data obtained after all batch data processing are stored in the target data table of the distributed database according to the layer classification. The attribute fields of the attribute data are updated, and a spatial index is created for the repaired geometric data to optimize the efficiency of subsequent queries.

[0054] The repaired data is categorized and stored by layer, which facilitates data management and subsequent applications. Updating attribute fields such as processing status and map sheet number enables the tracking of the processing process and status marking. A new spatial index is established for the repaired geometric data, which optimizes the efficiency of subsequent queries, exports and incremental updates, and ensures the high-performance response of the system.

[0055] In this embodiment of the disclosure, the distributed database provides a download service for the processed data. The download service supports exporting data in Shapefile, GeoJSON and MID / MIF formats, and supports batch exporting data compressed packages in ZIP or TAR formats.

[0056] This solution offers flexible data export capabilities, allowing users to select from various commonly used map formats for download as needed, ensuring compatibility with the requirements of different downstream application systems. It also supports batch compression and export, facilitating the transmission and storage of large-scale data and improving user experience and data delivery efficiency.

[0057] In one embodiment of this disclosure, such as Figure 2 As shown, the method for map data preprocessing includes the following steps: Step S201: Upload map data.

[0058] Step S202: Data verification. If it passes, proceed to step S203. If it fails, end the process.

[0059] Step S203: Data parsing and coordinate unification.

[0060] Step S204: Parse the layer and attribute data.

[0061] Step S205: Identify the original coordinate system and projection method.

[0062] Step S206: Transform the coordinates to a unified target coordinate system.

[0063] Step S207: Store in a distributed database.

[0064] Step S208: Calculate the data outsourcing range.

[0065] Step S209: Calculate the map sheet number and update the map sheet field value for each data entry.

[0066] Step S210: Retrieve data in batches according to map sheet number.

[0067] Step S211: Detect abnormal data based on preset rules.

[0068] Step S212: Automated repair based on a preset algorithm.

[0069] Step S213: Check if it is qualified. If yes, proceed to step S214; if no, proceed to step S211.

[0070] Step S214: After processing, store the data in the database.

[0071] Step S215: Data download.

[0072] In another embodiment of this disclosure, the user selects three original map data in different formats, such as Shapefile, GeoJSON, and MID / MIF, through the system's front-end interface and uploads them to the device of this invention via the HTTP protocol. The system automatically matches the corresponding parser, such as a Shapefile parser, a GeoJSON parser, and a MID / MIF parser, through the format recognition module, and performs data integrity verification.

[0073] During format validity validation, the system checks whether file extensions and metadata formats conform to specifications. For example, it checks whether Shapefile contains complete .shp, .shx, and .dbf files, whether GeoJSON conforms to JSON syntax specifications, and whether MID / MIF has paired .mid and .mif files. During content integrity validation, the system checks the layer structure, attribute field integrity, and geometric feature closure, and detects whether the data is corrupted, such as NULL coordinate values ​​or bounded data for polygon features. After the validation passes, the system automatically matches the corresponding parser. Subsequently, libraries such as GDAL, OGR, JTS, GeoTools, and Proj4J were used to parse and obtain three types of layers and attribute data: road network, nodes, and relationships. After identifying the coordinate systems of each data set, all data were uniformly converted to the WGS84 coordinate system using the GCJ02→WGS84 conversion algorithm and the projection transformation algorithm. Specifically, the Gauss-Kruger projection conversion is suitable for coordinate unification of large-scale topographic maps; the Mercator projection conversion is suitable for coordinate unification of nautical or aeronautical maps; and the GCJ02→WGS84 conversion algorithm, i.e., the conversion between the GCJ02 coordinate system and the WGS84 coordinate system, is used to address the discrepancy between the State Bureau of Surveying and Mapping's 02 map and the WGS84 coordinate system.

[0074] Furthermore, using GDAL / OGR, GeoTools, and FileUtils to read Shapefile, GeoJSON, or MID / MIF data, road network layers, node layers, relationship layers, and background surface layers are extracted. The road network layer consists of line features containing attributes such as road grade, length, name, and road coordinates. The node layer consists of point features containing attributes such as node number, connecting road ID, and node coordinates. The relationship layer records the associations between nodes and roads. The background surface layer includes land parcels and green areas. JTS is used to convert the parsed geometric data into standardized geometric objects such as LineString, Point, and Polygon.

[0075] The system parses the Shapefile's .prj file, GeoJSON's metadata, and MID / MIF header files to identify the coordinate system. It then uses Proj4J's GCJ02→WGS84 conversion algorithm to convert the data to the WGS84 coordinate system. The parsed data is stored in the Citus distributed database, and spatial indexing technology (R-tree index) is used to optimize query efficiency.

[0076] Standardized road networks, nodes, background surfaces, and other spatial elements are stored in a spatial index table of the Citus distributed database. This spatial index table is partitioned by map sheet and supports spatial range queries.

[0077] The system iterates through the coordinates of all spatial elements, calculating the data's bounded range as: 114.200°-116.500° E, 30.500°-33.800° N. Based on the 1:25,000 scale map sheet division standard (longitude difference 15′, latitude difference 10′), this range covers 960 map sheets, with 400 sheets in H50 and 560 sheets in I50. The map sheet numbers are then associated with the corresponding element data and stored in the Citus database for subsequent batch processing.

[0078] The system processes data in 240 batches according to map sheet number, with 4 map sheets in each batch. The process is as follows: First, the spatial index query of Citus is used to load the feature data of 4 map sheets, namely H50F028017, H50F028018, H50F029017 and H50F029018. These feature data contain 6420 roads, 15350 nodes and 3106 background surfaces.

[0079] The rule-based detection results are as follows: 156 suspended nodes conforming to the rule of "unconnected and length ≥ 3 meters" were detected; 43 pseudo-nodes conforming to the rule of "three consecutive collinear nodes with an included angle ≤ 10°" were detected, such as a line segment with three nodes having an included angle of 8°; 6 self-intersecting roads where line features intersect each other were detected; 3 self-intersecting surfaces where background surfaces intersect each other at their boundaries were detected; 8 instances of background and road overlap where the area of ​​surface features intersecting with the road is ≥ 10% were detected, such as a green space where the area of ​​overlap with the road is 12%; 12 adjacent face boundary gaps ≥ 0.3 meters, such as face feature gaps of 0.4 meters between two plots; 2 duplicate roads with 96% similarity and 92% attribute consistency were detected, such as 1 duplicate background face with 97% similarity and 91% attribute consistency; 358 redundant vertices with consecutive three-node vectors with an included angle ≤ 5° and a length ratio ≥ 1:10 were detected; 12 abnormal road endpoints with a distance > 0.5 meters from adjacent roads were detected; 2 invalid face features with an area < 0.5 square meters were detected.

[0080] The results of the algorithm-based automated repair are as follows: For suspended nodes: KD-tree indexing was used to match the nearest node, and coordinate interpolation was used to snap 153 suspended endpoints to the target node. After repair, the distance was ≤0.3 meters, and the remaining 3 nodes were marked as abnormal due to distance >5 meters and await manual confirmation; For pseudo nodes: Based on vector collinearity analysis, 43 pseudo nodes were deleted and the line segments were reconstructed. After merging, the line segments were continuous and there were no redundant nodes; For self-intersecting roads: The 6 roads were split into 15 sub-segments with an intersection tolerance of 0.1 meters. After adjusting the connection relationship, there were no self-intersections; For self-intersecting surfaces: For the 3 self-intersecting surfaces, the effective outer and inner loops were extracted and reconstructed, and after repair, they were simple polygons; For overlays: 8 overlay areas were repaired. The background surface boundary was adjusted using the least common edge fusion algorithm to eliminate overlaps. For gaps in surface features, the boundary alignment algorithm was used to extend / shorten the boundary at 12 locations, resulting in gaps ≤0.1 meters after repair. For duplicate features, one of the two duplicate roads and one duplicate background surface were deleted, with the one with complete attributes retained among the two duplicate roads. For redundant nodes, 358 redundant vertices were removed, simplifying the line segments without changing their shape and reducing the data volume by 2%. For road endpoints, a dynamic connection algorithm was used to match the 12 endpoints to the nearest road (tolerance 0.5-2 meters), resulting in connection distances ≤0.4 meters after repair. For invalid surface features, two surface features with an area <0.5 square meters were automatically removed.

[0081] The second inspection showed that 98% of the abnormal features were repaired to the required standard, with only 3 hanging nodes marked as abnormal, and these will proceed to the next batch for processing. After all batches of data have been processed, the system will store the repaired geometric data, attribute data, and other feature data in the results table of the Citus distributed database, and archive them in association with the original data.

[0082] Users select the export format, such as Shapefile, and the compression method, ZIP, through the front end. The system then batch packages the processed 960 map sheets and generates a download link, which users can click to obtain the pre-processed map data.

[0083] like Figure 3 As shown, this disclosure provides a detection and automated repair method, including the following steps: Step S301: Load the map sheet data for the current batch.

[0084] Step S302, rule base call, call Figure 3 Data monitoring is performed using preset rules.

[0085] Step S303, algorithm library call, call Figure 3 The preset algorithm in the software is used for data repair.

[0086] Step S304: Integrate the repair results.

[0087] Step S305: Check if the standard is met. If it is met, end the process. If it is not met, proceed to step S306.

[0088] Step S306: Determine whether to repair again. If yes, proceed to step S303; otherwise, proceed to step S307.

[0089] Step S307: Mark as manual confirmation.

[0090] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0091] According to the map data preprocessing method of this disclosure, map data of different formats and coordinate systems are parsed and coordinate system transformed. After generating map sheet numbers based on the bounding latitude and longitude range of geometric data, the processed data is stored in a distributed database. The stored data is then detected and automatically repaired. This achieves fully automated processing of map data from import to export, improves processing efficiency, reduces labor costs, and can simultaneously adapt to multi-format, multi-coordinate system data and different application scenarios.

[0092] The map data preprocessing scheme of this disclosure can automate the entire process of map data from uploading, parsing, processing to downloading without manual intervention. Compared to traditional manual processing, it improves efficiency by more than 10 times, reducing city-level data processing time from weeks to hours. It supports multiple map data formats such as Shapefile, GeoJSON, and MID / MIF, and is compatible with mainstream coordinate systems such as WGS84 and GCJ02, as well as various projection methods, resolving the incompatibility issues of multi-source data formats. Furthermore, parameters such as target coordinate system, repair threshold, and export format can be flexibly configured to adapt to different application scenarios. Based on the Citus distributed database, it enables rapid storage and retrieval of massive amounts of data, meeting the processing needs of large-scale map data. It employs refined anomaly handling technology, with a rule base that clearly quantifies thresholds, such as hanging nodes. With point lengths ≥3 meters and surface gaps ≥0.3 meters, the algorithm is specifically designed, employing techniques such as KD-tree indexing and vector collinearity analysis to remove false nodes, ensuring repair accuracy and data consistency. A comprehensive quality control mechanism combining algorithm processing and rule verification ensures that the processed data achieves an attribute accuracy pass rate ≥98.6% and a logical consistency pass rate ≥99.5%, with no dirty data generated. The topology rule base supports dynamic loading, modification, and customization, and the algorithm model can adaptively switch according to terrain complexity, adapting to different scenarios such as plains, mountains, and cities, as well as different needs such as surveying data import and navigation data optimization. No manual intervention by professional technicians is required, reducing labor costs by over 80%, and the cost of processing a single batch of data can be reduced from tens of thousands of yuan to thousands of yuan. It also supports multi-format export and incremental updates to adapt to data usage needs in different scenarios.

[0093] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. The map data preprocessing apparatus described below corresponds to the map data preprocessing method described above. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the method embodiments of this disclosure.

[0094] Figure 4 This is a schematic diagram of a map data preprocessing apparatus provided in an embodiment of this disclosure. Figure 4 As shown, the map data preprocessing device includes: The data receiving module 401 is used to receive uploaded map data, which includes at least two formats.

[0095] The parsing and coordinate unification module 402 is used to parse map data to obtain geometric data, attribute data and spatial reference information of map data, automatically identify the original coordinate system and projection method based on spatial reference information, and transform the geometric data to the target coordinate system through coordinate system transformation algorithm.

[0096] The storage and map sheet calculation module 403 is used to store the converted geometric data and corresponding attribute data in a distributed database, calculate the bounding latitude and longitude range of all geometric data, generate a map sheet number based on the bounding latitude and longitude range, and associate the map sheet number with the corresponding geometric data and attribute data and store it in the distributed database.

[0097] The automated processing module 404 is used to retrieve geometric data and corresponding attribute data in batches according to map sheet number, detect topological and / or geometric anomalies in the retrieved data based on preset rules, automatically repair the anomalies through preset algorithms, and repeat the detection and repair steps until the processing result meets the standard.

[0098] By integrating the core functions of map data preprocessing into a modular design, a device that can be deployed independently is formed. The modules work together to achieve full automation of the entire process from data reception, parsing, coordinate unification, storage, batch repair to result output. The device features multi-format compatibility, multi-coordinate system adaptation, distributed and efficient processing, and configurable repair rules, which can significantly reduce labor costs, improve processing efficiency and data quality, and is suitable for various map data preprocessing scenarios.

[0099] like Figure 5 As shown, a map data preprocessing apparatus according to an embodiment of this disclosure includes an input module 501, a control module 502, a processing module 503, and an output module 504. The direction of the arrows in the figure indicates the direction of data flow.

[0100] In input module 501, data is processed by data upload unit, HTTP protocol receiving module, format recognition and parsing unit and data integrity verification unit.

[0101] In the control module 502, the main control unit includes a rule management unit, an algorithm management unit, and a log monitoring unit.

[0102] In the processing module 503, the parsing and coordinate transformation unit includes a layer attribute parsing subunit, a coordinate system projection recognition subunit, a coordinate transformation subunit, a storage and map sheet calculation subunit, a batch retrieval subunit, a rule detection subunit, an automated algorithm repair subunit, and a result verification subunit.

[0103] In the output module 504, the data storage unit includes a processing result storage subunit, a data download subunit, a format conversion subunit, and a compression and packaging subunit.

[0104] Since the functional modules of the map data preprocessing apparatus in the example embodiments of this disclosure correspond to the steps of the example embodiments of the map data preprocessing method described above, for details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the map data preprocessing method described above.

[0105] The map data preprocessing apparatus according to the embodiments of this disclosure parses and transforms map data of different formats and coordinate systems, generates map sheet numbers based on the bounding latitude and longitude range of geometric data, stores the processed data in a distributed database, and performs detection and automated repair on the stored data. This realizes fully automated processing of map data from import to export, improves processing efficiency, reduces labor costs, and can simultaneously adapt to multi-format, multi-coordinate system data and different application scenarios.

[0106] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.

[0107] Figure 6 This is a schematic diagram of the electronic device 6 provided in an embodiment of this disclosure. Figure 6 As shown, the electronic device 6 of this embodiment includes a processor 601, a memory 602, and a computer program 603 stored in the memory 602 and executable on the processor 601. When the processor 601 executes the computer program 603, it implements the steps in the various method embodiments described above. Alternatively, when the processor 601 executes the computer program 603, it implements the functions of each module in the various device embodiments described above.

[0108] Electronic device 6 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 6 may include, but is not limited to, processor 601 and memory 602. Those skilled in the art will understand that... Figure 6This is merely an example of electronic device 6 and does not constitute a limitation on electronic device 6. It may include more or fewer components than shown, or different components.

[0109] The processor 601 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0110] The memory 602 can be an internal storage unit of the electronic device 6, such as a hard disk or RAM of the electronic device 6. The memory 602 can also be an external storage device of the electronic device 6, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 6. The memory 602 can also include both internal and external storage units of the electronic device 6. The memory 602 is used to store computer programs and other programs and data required by the electronic device.

[0111] The electronic device in this embodiment can be deployed using a distributed architecture, including a data receiving server, a processing server cluster, and a database server. The data receiving server runs a data upload module, receiving user-uploaded data and forwarding it to the processing server cluster. The processing server cluster includes multiple processing nodes that execute parsing, coordinate transformation, and data processing steps in parallel to improve processing efficiency. The database server deploys a Citus distributed database, providing data storage and query services. The database server is not a single physical node but a logical service unit composed of the Citus distributed cluster. It receives the "preprocessed map data" output by the processing server cluster, such as parsed road network geometry data and standardized node attribute data. It provides highly scalable, high-performance, and highly available distributed storage, index building, data querying, and transaction guarantee capabilities, serving as the core data carrier for map data from preprocessing to application querying. During deployment, a master node can occupy one server, and then three worker nodes can be configured to store fragmented data. Users can also configure multiple worker nodes according to their actual needs.

[0112] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0113] If the integrated module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0114] The above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit it. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be included within the protection scope of this disclosure.

Claims

1. A map data preprocessing method, characterized in that, include: Receive uploaded map data, which includes at least two formats; The map data is parsed to obtain geometric data, attribute data and spatial reference information. The original coordinate system and projection method are automatically identified based on the spatial reference information, and the geometric data is transformed to the target coordinate system through a coordinate system transformation algorithm. The converted geometric data and corresponding attribute data are stored in a distributed database. The bounding latitude and longitude range of all the geometric data is calculated. A map sheet number is generated based on the bounding latitude and longitude range. The map sheet number is associated with the corresponding geometric data and attribute data and stored in the distributed database. Geometric data and corresponding attribute data are retrieved in batches according to map sheet number. Topological and / or geometric anomalies in the retrieved data are detected based on preset rules. Anomalies are automatically repaired using preset algorithms. The detection and repair steps are repeated until the processing results meet the standards.

2. The method according to claim 1, characterized in that, The map data is received via the HTTP protocol. After receiving the uploaded map data, the method further includes: performing format validity verification and / or content integrity verification on the map data.

3. The method according to claim 1, characterized in that, After the processing results meet the standards, the method further includes: storing the repaired geometric data and corresponding attribute data obtained after processing all batches of data into the distributed database according to the layer classification, updating the attribute fields of the attribute data, and establishing a spatial index for the repaired geometric data.

4. The method according to claim 1, characterized in that, The distributed database includes a spatial index table and an attribute relationship table. It calculates the bounding latitude and longitude range of all the geometric data, generates map sheet numbers based on the bounding latitude and longitude ranges, and associates and stores the map sheet numbers with the corresponding geometric data and attribute data in the distributed database, including: Traverse the coordinates of all spatial elements of the geometric data, and calculate the bounding latitude and longitude range of the geometric data, which includes minimum longitude, maximum longitude, minimum latitude, and maximum latitude; Based on the preset map sheet division standard, the set of map sheet numbers covered by the outer latitude and longitude range is calculated. The map sheet numbers in the map sheet number set are associated with the corresponding geometric data and stored in the spatial index table, and the attribute data corresponding to the geometric data are stored in the attribute relationship table.

5. The method according to claim 1, characterized in that, The preset rules include rules for handling suspended nodes, pseudo-nodes, road breaks, background and road overlap, self-intersecting roads, self-intersecting surfaces, redundant nodes, surface feature gaps, duplicate roads, duplicate background surfaces, and surface feature removal.

6. The method according to claim 5, characterized in that, The preset algorithms include a suspended node processing algorithm, a pseudo node processing algorithm, a road fracture repair algorithm, a background and road overlap processing algorithm, a self-intersecting road processing algorithm, a self-intersecting surface processing algorithm, a redundant node removal algorithm, a surface feature gap processing algorithm, and a road endpoint repair algorithm.

7. The method according to claim 1, characterized in that, The distributed database provides a download service for processed data, which supports exporting data in Shapefile, GeoJSON, and MID / MIF formats, and supports batch exporting data compressed packages in ZIP or TAR formats.

8. A map data preprocessing apparatus, characterized in that, include: A data receiving module is used to receive uploaded map data, which includes at least two formats; The parsing and coordinate unification module is used to parse the map data to obtain the geometric data, attribute data and spatial reference information of the map data, automatically identify the original coordinate system and projection method according to the spatial reference information, and transform the geometric data to the target coordinate system through a coordinate system transformation algorithm; The storage and map sheet calculation module is used to store the converted geometric data and corresponding attribute data in a distributed database, calculate the bounding latitude and longitude range of all the geometric data, generate a map sheet number based on the bounding latitude and longitude range, and associate the map sheet number with the corresponding geometric data and attribute data and store it in the distributed database. The automated processing module is used to retrieve geometric data and corresponding attribute data in batches according to map sheet number, detect topological and / or geometric anomalies in the retrieved data based on preset rules, automatically repair the anomalies through preset algorithms, and repeat the detection and repair steps until the processing result meets the standard.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images