A method for updating multi-source heterogeneous pipeline data

Abstract

This invention discloses a method for updating multi-source heterogeneous pipeline data, belonging to the field of data update technology. The method includes: creating a data download task, recording the download range and pipeline type information, and determining whether there is an associated data entry task; if so, waiting for the associated task to execute; creating a data entry task, determining whether there is an associated data download task, and waiting for the associated task to execute; reading multi-source pipeline data within a specified range through a download queue concurrency control mechanism, and assembling and downloading the data format; performing a quality check before data entry through an entry queue concurrency control mechanism, reading the data to be entered into the database and the data in the current database; performing repair and testing on the read data, and comparing the differences between additions, deletions, and modifications; updating the compared addition, deletion, and modification information to the current database, and synchronizing the deletion and modification information to the historical database.

Description

Technical Field

[0001] This invention belongs to the field of information technology, and in particular relates to a data integration and updating method, specifically a method for updating multi-source heterogeneous pipeline data for urban underground space management. Background Technology

[0002] Underground pipeline data management is a crucial foundation for urban infrastructure planning, construction, and emergency response. Its accuracy and timeliness directly impact urban safety and efficient decision-making. Multi-source pipeline data involves integrating information from different departments, formats, and update cycles. Ensuring the timely and consistent synchronization of this data into a unified database is of paramount importance.

[0003] In the current pipeline data synchronization process, data from different sources often need to be downloaded, standardized, quality checked, and stored sequentially, with clear dependencies between tasks. However, in practice, tasks from multiple sources are frequently executed concurrently, causing subsequent tasks to operate on older versions of data that are not yet completed or are still being processed, resulting in data conflicts and inconsistencies. Especially when data from one source is being stored while related data from another source has already begun difference comparisons, situations arise where some data has been updated while related data remains in the old state, disrupting the overall logical consistency of the pipeline network.

[0004] A more complex situation arises when processing repair and testing data, requiring the rapid identification of differences in additions, deletions, and modifications, and simultaneous updates to both the current and historical databases to support traceability. If the difference comparison phase is directly based on data from the previous phase that is not yet fully stable, the identified additions, deletions, and modifications may include false changes or omit genuine modifications, leading to incorrect topological relationships in the current database and the historical database failing to accurately record the actual change process. This comparison bias caused by improper handling of task dependencies is particularly prominent in business scenarios where pipeline data is frequently repaired and tested. For example, if the coordinates of connected pipe points are not updated synchronously due to a delay in another task after the attributes of a pipeline segment are modified, the comparison results will incorrectly classify normal associations as abnormal disconnections.

[0005] How to effectively handle data dependencies and potential conflicts between tasks in the entire process of concurrent downloading, standardization, quality inspection and storage of multi-source pipeline data, and ensure that the difference comparison stage is always based on stable and consistent previous results, so as to achieve accurate synchronous updates between the current database and the historical database, has become a key issue in ensuring the integrity and traceability of pipeline data. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a method and system for updating multi-source heterogeneous pipeline data in order to address the shortcomings of the prior art, thereby solving problems such as chaotic task dependencies, frequent data conflicts, inaccurate comparisons, and difficulties in synchronizing dual databases in the process of updating multi-source pipeline data.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A collaborative update method for multi-source heterogeneous pipeline data includes the following steps: Step 1: Create a data download task, record the download range and pipeline type information, and determine whether there is a related data import task. If so, wait for the related task to be executed. Step 2: Create a data import task, check if there is a related data download task, and if so, wait for the related task to be executed; Step 3: Through the download queue concurrency control mechanism, read multi-source pipeline data within a specified range, and assemble and download the data format; Step 4: Through the concurrency control mechanism of the inbound queue, perform quality checks before data is inbound, and read the data of the file to be inbound and the data in the current database; Step 5: Repair and test the read data, and compare the differences between added, deleted, and modified data; Step 6: Update the current database with the compared addition, deletion and modification information, and synchronize the deletion and modification information to the historical database.

[0008] As a further preferred embodiment of the collaborative update method for multi-source heterogeneous pipeline data of the present invention, the multi-source pipeline data includes DWG format and GDB format, and lossless reading and writing and format conversion are achieved by integrating ODA and GDAL technologies.

[0009] As a further preferred embodiment of the collaborative updating method for multi-source heterogeneous pipeline data of the present invention, the method realizes the standardized mapping and management of multi-source heterogeneous data based on a four-table comparison system, wherein the four tables include: a layer comparison table, an attribute comparison table, an element code comparison table, and an enumeration value comparison table.

[0010] As a further preferred embodiment of the collaborative update method for multi-source heterogeneous pipeline data of the present invention, the quality inspection includes integrity verification, logical consistency verification and accuracy review, and adopts a hierarchical inspection mechanism to support fully parameterized and partially parameterized inspection item configuration.

[0011] As a further preferred embodiment of the collaborative updating method for multi-source heterogeneous pipeline data of the present invention, the logical consistency verification includes elevation and burial depth consistency checks and large-pipe-to-small-pipe connection checks, and the accuracy review includes cross-sectional size checks, elevation checks, and burial depth checks.

[0012] As a further preferred embodiment of the collaborative update method for multi-source heterogeneous pipeline data of the present invention, the method adopts an intelligent caching mechanism, including multi-level hierarchical design, hot spot preloading and consistency guarantee, to improve data access speed.

[0013] As a further preferred embodiment of the collaborative update method for multi-source heterogeneous pipeline data of the present invention, the difference comparison is based on an incremental update protocol and achieves second-level identification through log parsing and difference comparison technology.

[0014] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects: This invention ensures proper handling of dependencies between download and database entry tasks through a mutual waiting mechanism during task creation, avoiding data conflicts; it achieves lossless reading and writing of multiple formats by integrating ODA and GDAL, and completes standardized mapping of heterogeneous data based on a four-table comparison system; it adopts a parameterized and configurable hierarchical quality inspection mechanism to improve data quality controllability; it achieves second-level difference identification based on incremental update protocol and log parsing, ensuring accurate synchronization between the current and historical databases; and it significantly improves data processing efficiency and system stability by combining intelligent caching and concurrent queue mechanisms. This invention realizes a full-process collaborative update from data download, format conversion, quality inspection to difference comparison and dual-database synchronization, and is suitable for the continuous governance and dynamic maintenance of urban pipeline "one map". Attached Figure Description

[0015] Figure 1 This is a flowchart of the data update and entry process for this invention; Figure 2 This is a flowchart of a collaborative update method for multi-source heterogeneous pipeline data according to the present invention. Detailed Implementation

[0016] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings: The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention. The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments. The purpose and effects of the present invention will become clearer. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] like Figures 1-2The collaborative update method for multi-source heterogeneous pipeline data in this embodiment may specifically include: S101 includes the following steps: create a data download task, record the download range and pipeline type information, and determine whether there is an associated data import task. If so, wait for the associated task to be executed.

[0019] S102. Create a data import task, determine whether there is an associated data download task, and if so, wait for the associated task to be executed; through the download queue concurrency control mechanism, read multi-source pipeline data within a specified range, and perform data format assembly and download.

[0020] Create a data import task, check if there is a related data download task, and if so, wait for the related task to be executed.

[0021] The system uses a download queue concurrency control mechanism to read multi-source pipeline data within a specified range and then assembles and downloads the data.

[0022] S1: Create a data import task, check if there is a related data download task, and if so, wait for the related task to be executed.

[0023] S11, when creating a data import task, first query the task list in the task management system to check whether the current import task is associated with any download task, for example by matching task IDs or comparing dependency chain identifiers.

[0024] S12, if there is an associated download task, monitor the status of the download task and check the task completion flag at fixed time intervals, such as 5 seconds, through a polling mechanism until the download task is completed.

[0025] S13. After the download task is completed, extract the temporary data path generated by the download task and pass it as the input parameter of the database insertion task to ensure the continuity of data from download to database insertion.

[0026] In one embodiment, when determining associated tasks, if the download task involves multi-source data such as DWG and GDB formats, the task log is updated in real time during the waiting process to record delay time and potential conflicts.

[0027] S2, through a download queue concurrency control mechanism, reads multi-source pipeline data within a specified range and performs data format assembly and download.

[0028] S21 reads multi-source pipeline data within a specified range through a download queue concurrency control mechanism.

[0029] S211, initialize the download queue, and use semaphores to limit the number of concurrent threads, for example, set the maximum concurrency to 10 to prevent resource overload.

[0030] S212, for a specified range such as a region defined by geographic coordinates, scan the list of data sources, including DWG files and GDB databases, and put read requests into the queue one by one.

[0031] S213, each request in the queue is handled by an independent thread. When reading data, the ODA library is used to parse the DWG format and the GDAL library is used to parse the GDB format to achieve lossless data extraction.

[0032] S214 After reading, the extracted data is cached using a multi-level caching mechanism. It is first stored in the memory cache, and if the memory is insufficient, it overflows to the disk cache to ensure efficient access for subsequent assembly.

[0033] S22, perform data format assembly and download.

[0034] S221. Retrieve the read data from the cache and map the DWG data to the GDB standard structure according to the four-table comparison system, namely the layer comparison table and the attribute comparison table. For example, convert the layer names in the DWG to the feature class names in the GDB.

[0035] S222, Incremental updates are performed during the assembly process. The differences between the old and new data are compared through log parsing, and only the changed parts are assembled, such as only updating the modified pipeline attribute values.

[0036] S223, after assembly, generates a file package in a unified format and pushes it for download via HTTP protocol, achieving seamless data transmission from source to target.

[0037] For example, in a pipeline data processing scenario, assuming the specified range is a certain district in a city, when reading data from multiple sources, the concurrency control mechanism can handle 10 threads reading files from different sources simultaneously, avoiding the latency caused by single-thread bottlenecks and improving overall efficiency by more than 30%.

[0038] In one possible implementation, for the step of determining related tasks, if the download task is not completed, the waiting mechanism can prevent the use of incomplete data for data entry, thereby ensuring the integrity and consistency of the final data entered into the database, such as avoiding missing pipeline segments during the repair and testing phase.

[0039] It should be noted that when reading multi-source pipeline data, the integration of ODA and GDAL can handle compatibility issues of heterogeneous formats. For example, two-dimensional coordinates in DWG can be directly converted into three-dimensional features in GDB, ensuring data accuracy and no loss.

[0040] For example, in the data format assembly process, the application of the four-table comparison system enables data from different production stages, such as the DWG in the aggregation stage, to be seamlessly integrated with the GDB in the shared governance stage, reducing manual intervention and improving the level of automation.

[0041] In one embodiment, the concurrency control mechanism dynamically adjusts the semaphore under high load, for example, by reducing it from 10 to 5, to adapt to fluctuations in server resources and maintain system stability.

[0042] It should be noted that the role of incremental updates during assembly is to process only differential data, such as updating only pipeline points with changes in burial depth, thereby saving computing resources and accelerating the download process.

[0043] For example, DWG and GDB files generated after offline operations can be read and assembled through this mechanism to quickly form data packages that can be stored in the database, which is suitable for real-time pipeline management needs.

[0044] In one possible implementation, the entire process, from task creation to download, forms a closed loop, ensuring efficient management of multi-source heterogeneous data within a single pipeline diagram.

[0045] S103. Through the concurrency control mechanism of the data entry queue, perform quality checks before data entry and read the data of the file to be entered and the data in the current database.

[0046] In one embodiment, a data entry queue concurrency control mechanism is used to perform quality checks before data entry, and to read the data of the file to be entered and the data in the current database, specifically including: S1 uses a concurrent control mechanism for the inbound queue.

[0047] S11 monitors the number of tasks to be added to the database. When the number of tasks exceeds the threshold, the concurrent queue is activated, and multiple data files are distributed to different processing threads for processing.

[0048] S12 dynamically adjusts the thread pool size based on the queue length. For example, when there are more than 10 tasks in the queue, the number of threads is increased to twice the original number to ensure that processing is not blocked.

[0049] S13 utilizes priority sorting to prioritize urgent pipeline data, such as repair test files, to avoid low-priority tasks consuming resources.

[0050] For example, during the pipeline sharing and governance phase, when multiple sources of data are submitted simultaneously, this mechanism can handle peak concurrency, ensure no data loss, and improve data entry efficiency.

[0051] In one possible implementation, this mechanism has beneficial effects, such as reducing waiting time and increasing system throughput.

[0052] S2 performs quality checks before data is entered into the database.

[0053] S21, based on the four-table comparison system, loads the comparison table of layers, attributes, feature codes and enumeration values, and performs preliminary format verification on the files to be imported.

[0054] S211, check if the field is not empty and verify if the format is compliant, such as whether the coordinate value in the attribute field is in numeric format.

[0055] S212 performs logic consistency verification, including elevation and burial depth consistency checks and large-pipe-to-small-pipe connection checks. The elevation and burial depth consistency check compares the difference between the elevation and burial depth values ​​to see if it is within the specified range. The large-pipe-to-small-pipe connection check verifies whether the pipe diameter at the connection point transitions from large to small.

[0056] S22 adopts a layered inspection mechanism, first verifying integrity, then logically verifying, and finally reviewing accuracy. For example, the cross-sectional dimension inspection is to compare the measured pipeline cross-sectional parameters with the standard values, the elevation inspection is to check that the elevation of the point is consistent with the terrain data, and the burial depth inspection is to confirm that the burial depth value does not exceed the safety threshold.

[0057] S221. For fully parameterizable check items, such as attribute checks, abstract rule templates. Users can define custom rules such as "pipe diameter is greater than 0". Point-line logical relationship checks include whether a point lands on a line.

[0058] S222, For partially parameterizable inspection items, the variables are partially parameterized. For example, the burial depth inspection threshold can be set to 1-5 meters and modified according to business needs.

[0059] S23 generates a visual inspection report, marking the location of non-compliant items and suggesting corrections.

[0060] Specifically, the inspection uses a unified algorithm to ensure that the data complies with industry standards, which is beneficial for subsequent governance decisions.

[0061] For example, during the submission phase, if inconsistencies in elevation are found during the pre-entry inspection of DWG files, they can be automatically marked and reported, reducing manual intervention and improving accuracy.

[0062] In one embodiment, for repair test data, the logic of connecting a large pipe to a small pipe is checked. If the pipe diameter is changed from 100mm to 50mm, the connection rationality is verified to avoid potential engineering problems.

[0063] It should be noted that this tiered mechanism improves inspection efficiency. Basic verification can quickly filter out simple errors, while subsequent in-depth verification ensures comprehensiveness.

[0064] S3 reads the data from the file to be imported into the database and the data in the current database.

[0065] By integrating ODA and GDAL, the contents of DWG and GDB files are read, and corresponding regional data is extracted from the current database for comparison.

[0066] S104. Repair and test the read data, and compare the differences between additions, deletions and modifications; update the comparison information of additions, deletions and modifications to the current database, and synchronize the deletion and modification information to the historical database.

[0067] S105. The multi-source pipeline data includes DWG and GDB formats, and lossless reading, writing and format conversion are achieved by integrating ODA and GDAL technologies.

[0068] S106. The method is based on a four-table comparison system to achieve standardized mapping and management of multi-source heterogeneous data. The four tables include: layer comparison table, attribute comparison table, feature code comparison table, and enumeration value comparison table.

[0069] In one embodiment, the method realizes standardized mapping and management of multi-source heterogeneous data based on a four-table comparison system, wherein the four tables include: a layer comparison table, an attribute comparison table, an element code comparison table, and an enumeration value comparison table.

[0070] Specifically, the four-table comparison system ensures the compatibility of multi-source data, such as DWG and GDB formats, during pipeline production by defining the correspondence between different data formats.

[0071] Step S1: Construct a layer mapping table to map layer information in different formats to a unified standard.

[0072] In one possible implementation, the process of constructing the layer mapping table in step S1 first involves collecting layer names from DWG data from the aggregation phase and GDB data from the remediation phase, and then establishing a one-to-one or one-to-many mapping relationship. For example, mapping the "water pipe layer" in the DWG to the "water supply pipeline layer" in the GDB achieves seamless integration. Through this mapping, the data can be directly used for standardized management during the shared governance phase, avoiding reading errors caused by format differences.

[0073] Step S11: Extract layer information based on the data source and compare it with the standard layer template to generate an initial comparison table.

[0074] For example, in pipeline data processing, a list of all layers is extracted from the DWG file, and a comparison algorithm is used to match the layer definitions in the standard template. If the matching degree exceeds a preset threshold such as 0.8, the layers are automatically associated; otherwise, they are marked as needing manual adjustment. However, this method prioritizes automation to improve efficiency.

[0075] Step S12: Verify the integrity of the layer lookup table to ensure that all input formats have corresponding layers.

[0076] It should be noted that this verification works by iterating through all layer elements of the input data and checking whether a corresponding item exists in the lookup table. If a missing item is found, a default mapping is added to ensure lossless data conversion.

[0077] In one embodiment, the application of this layer lookup table has beneficial effects, such as accelerating data read and write speeds in multi-stage pipeline production, because the predefined mapping reduces real-time parsing overhead.

[0078] Step S2: Construct an attribute mapping table to map attribute fields from heterogeneous sources to a unified attribute structure.

[0079] Specifically, step S2 involves a detailed expansion of the attribute lookup table because attribute processing is complex in multi-source data, requiring handling differences in field types, lengths, and values.

[0080] For example, the integration of ODA and GDAL technologies mentioned in the technical briefing is expanded here to use ODA to read DWG attributes, GDAL to process GDB attributes, and then unify them through a lookup table, such as mapping the "diameter" attribute of DWG to the "pipe diameter" field of GDB, and ensuring data type compatibility, such as converting from string to number.

[0081] Step S21: Identify the attribute fields in each format and define mapping rules including type conversion and value normalization.

[0082] For example, in a pipeline sharing governance scenario, for the attribute "burial depth", DWG may represent it as a string such as "1.5m", while GDB may represent it as a floating-point number. The string is first parsed to extract the value through mapping rules, and then converted into a unified floating-point format to ensure the accuracy of subsequent checks such as burial depth consistency verification.

[0083] Step S22: Apply the attribute lookup table to perform data conversion, achieving lossless attribute transfer.

[0084] In one possible implementation, this transformation uses logs to record differences and triggers an alert if the attribute value exceeds the specification range, such as a negative burial depth value. However, this method focuses on the mapping itself, which is beneficial for improving data quality assurance.

[0085] For example, the construction of this attribute mapping table can support the standardization of pipeline data from multiple perspectives. When dealing with a large number of attributes during the aggregation phase, parameterized rules allow users to adjust the mapping thresholds to adapt to different project needs, thereby reducing manual intervention in data governance.

[0086] Step S3: Construct an element code comparison table to unify element codes from different sources into a standard coding system.

[0087] It should be noted that step S3 is an expansion of the innovation point, because the element coding in pipeline data involves unique business scenarios such as the logic of connecting large pipes to small pipes, which needs to be described in detail. Based on the four-table system of technical disclosure, the reasoning here is to first list all element types such as "valve" and "elbow", and then map the codes, such as "001" in DWG corresponding to "VL01" in GDB, to ensure that the unified coding is used for logical verification during the pre-entry inspection.

[0088] Step S31: Enumerate common elements and establish corresponding codes to ensure coverage of all pipeline types.

[0089] In one embodiment, the coding lookup table can be parameterized for data from the patching phase, such as by allowing users to define additional coding rules to handle custom elements, thereby improving versatility.

[0090] Step S32: Test the effectiveness of the encoding lookup table in the conversion, and verify that there is no loss by simulating data.

[0091] For example, in shared governance, using this table improves data download speed because unified encoding reduces parsing time and benefits the integration of high-performance computing architectures.

[0092] Step S4: Construct an enumeration value mapping table to map enumeration values ​​from heterogeneous data to the standard enumeration set.

[0093] Specifically, step S4 is relatively conventional and can be summarized as collecting enumerations of various formats, such as "steel" and "plastic" in the "material" field, and mapping them to a unified set such as "1: steel" and "2: plastic" to support attribute checking.

[0094] In one possible implementation, the table ensures that the enumerated values ​​are compliant when integrity checks are performed before the data is stored in the database.

[0095] For example, in pipeline processes, the benefit of an enumeration value lookup table is that it avoids logical errors caused by material inconsistencies and supports the operation of automated inspection engines.

[0096] In one embodiment, the above four tables are used as a whole in the data processing flow. Starting from the download, DWG and GDB files are generated through offline operations, and then the four tables are mapped to perform pre-entry checks before finally entering the database, thus ensuring the standardized mapping and management of multi-source heterogeneous data.

[0097] It should be noted that the integration of the four-table comparison system brings technical benefits, such as seamless connection at different production stages, and improved data access and conversion efficiency.

[0098] For example, in a pipeline project, there are 10 layers for data aggregation and 12 layers for repair testing. After mapping through the lookup table, they are unified into 8 standard layers, reducing redundancy and improving management efficiency.

[0099] In one possible implementation, the application of attribute lookup tables, for example when processing 1000 pipeline records, maps attribute fields from 15 variants to 10 standard ones, reducing conversion time from minutes to seconds, which is beneficial for parallel computing.

[0100] For example, when checking the connection between large and small pipes, the element coding lookup table uses a unified code to identify diameter differences. If the diameter corresponding to the large pipe code is larger than that of the small pipe, it is marked as abnormal, which supports quality assessment.

[0101] Specifically, in the burial depth check, the enumeration value lookup table maps "shallow burial" and "deep burial" to numerical ranges, such as less than 1.0 for shallow burial, to ensure accurate verification of elevation consistency.

[0102] In one embodiment, the combined use of these tables, in a shared governance scenario, enables the parametric adjustment of different urban pipeline standards to jointly support lossless data conversion and standardized management.

[0103] S107. The quality inspection includes integrity verification, logical consistency verification, and accuracy review, and adopts a layered inspection mechanism, supporting fully parameterized and partially parameterized inspection item configuration. Furthermore, the logical consistency verification includes elevation and burial depth consistency checks and large-pipe-to-small-pipe connection checks, and the accuracy review includes cross-sectional dimension checks, elevation checks, and burial depth checks.

[0104] The quality checks include integrity verification, logical consistency verification, and accuracy review, and adopt a layered inspection mechanism that supports fully parameterized and partially parameterized inspection item configuration.

[0105] In one embodiment, the integrity check first performs a non-empty validation on the data fields to ensure that the required attributes of each feature exist, and then checks the format compliance, such as whether the coordinate values ​​are numeric types.

[0106] Logical consistency verification includes elevation and burial depth consistency checks and large-pipe-to-small-pipe connection checks.

[0107] In one embodiment, the elevation and burial depth consistency check ensures that the sum of the elevation and burial depth values ​​of pipeline points equals the surface elevation by comparing them. Specifically, this includes: S1, extracting the elevation and burial depth values ​​for each point in the pipeline data; S11, calculating the expected sum based on the surface elevation benchmark; and S12, marking any discrepancies as inconsistent if the actual sum deviates from the expected sum by more than 0.1 meters. This comparison avoids logical errors in the data aggregation stage and improves the reliability of subsequent data sharing and governance.

[0108] The large-to-small pipe check is used to verify the diameter matching at the pipe connection.

[0109] For example, during the repair testing phase, the upstream and downstream pipe diameters of the connection nodes are extracted. If the upstream pipe diameter is larger than the downstream diameter, it is checked whether it conforms to the design specifications. For example, if the upstream diameter is 300 mm and the downstream diameter is 200 mm, the presence of a transition piece is verified. If none is found, the abnormality is marked. This inspection ensures smooth pipeline flow and reduces the risk of potential leaks.

[0110] Accuracy review includes checking cross-sectional dimensions, elevation, and burial depth.

[0111] In one embodiment, the cross-sectional dimension check verifies the dimensions of the pipeline cross-section, specifically including: S2, extracting the width and height attributes of the pipeline elements; S21, comparing with a standard size range, such as the width should be between 0.5 and 2 meters; S22, if the deviation exceeds 5%, performing a correction prompt. This approach improves the accurate management of data in GDB format.

[0112] Elevation checks verify the rationality of elevation values.

[0113] For example, for elevation points in DWG data, check whether they are within the known terrain range. If the elevation is 0 meters below sea level or higher than the highest point in the local area, mark it as needing to be verified, thereby ensuring the authenticity of the data entering the database.

[0114] Burial depth checks ensure that the burial depth value meets safety standards.

[0115] In one embodiment, the burial depth data is range-verified. For example, the burial depth of urban pipelines is at least 1 meter. If it is less than this value, it is adjusted in conjunction with the surrounding environmental data to ensure that ground interference is avoided.

[0116] Furthermore, the logical consistency verification includes elevation and burial depth consistency checks and large-pipe-to-small-pipe connection checks, while the accuracy review includes cross-sectional dimension checks, elevation checks, and burial depth checks.

[0117] In one embodiment, the elevation and burial depth consistency check is extended to multi-point pipeline segments, specifically including: S3, extracting the elevation and burial depth of the starting and ending points of the pipeline segment; S31, calculating the average deviation within the segment; S32, if the average deviation is greater than 0.05 meters, then the entire segment is marked as inconsistent.

[0118] For example, in the shared governance phase, for a 10-meter pipeline segment, if the starting elevation is 50 meters and the burial depth is 2 meters, and the ending elevation is 50.1 meters and the burial depth is 1.9 meters, with an average deviation of 0.05 meters, then the inspection demonstrates the guarantee of data continuity and helps to improve the stability of the overall pipeline network.

[0119] The large-pipe-to-small-pipe inspection can be applied to branch connections.

[0120] For example, for a T-type connection, the main pipe diameter is extracted to be 400 mm and the branch diameter to be 200 mm. It is verified that the branch diameter is less than 70% of the main pipe diameter. If it meets this requirement, the connection passes; otherwise, the design basis needs to be checked. This method reduces connection failures and improves data governance efficiency during repair testing.

[0121] Cross-sectional dimensions are checked in conjunction with enumerated values.

[0122] In one embodiment, for a circular cross-section, S4, the diameter value is obtained; S41, it is matched with the enumerated values ​​in four tables, such as standard diameters of 100, 150, and 200 mm; S42, if there is no match, a report is generated prompting adjustment. This comparison ensures seamless integration of multi-source data such as DWG and GDB.

[0123] Elevation checks integrate terrain data.

[0124] For example, for an elevation value of 50 meters, the model is used to verify whether it falls within the range of 45 to 55 meters. If it exceeds this range, it is marked, which effectively prevents the accuracy deviation after the data is downloaded.

[0125] The soil type should be considered when checking the burial depth.

[0126] In one embodiment, for a burial depth of 1.5 meters, the depth is verified in a sandy environment to be greater than 1.2 meters. If it is, the pipeline passes; otherwise, adjustments are made. This inspection is performed before the pipeline is stored, improving the long-term safety of the pipeline.

[0127] A layered inspection mechanism is adopted, which first performs integrity verification before data is entered into the database, then performs logical consistency verification, and finally completes accuracy review, ensuring that each layer is progressively improved.

[0128] It supports fully parameterized and partially parameterized check item configuration. Fully parameterized check items allow users to define check rules, such as customizing the field list for non-empty attribute rules. Partially parameterized check items allow for the opening of variable parts, such as setting the deviation threshold to 0.1 or 0.2 meters.

[0129] In one embodiment, fully parameterized check items such as point-line logical relationships are performed: S5, user-defined rule template; S51, applied to data to compare point locations with line segment connections; S52, output the number of disconnected points.

[0130] For example, in the data aggregation process, rules are set to check whether the points are online. If five points are off-line, a report is issued, thus achieving flexible and standardized management.

[0131] Some parameterized checks, such as elevation checks, have an open threshold parameter that users can modify to 0.05 meters, making them applicable to different production stages and improving versatility.

[0132] S108. The method adopts an intelligent caching mechanism, including multi-level layered design, hot spot preloading and consistency guarantee, to improve data access speed.

[0133] S1. The method adopts an intelligent caching mechanism, including multi-level layered design, hot spot preloading and consistency guarantee, to improve data access speed.

[0134] In one embodiment, step S1 employs an intelligent caching mechanism that processes access requests by organizing data into different levels of storage structures. The multi-level hierarchical design divides the cache into memory level, disk level, and remote level. Each level allocates resources based on access frequency and data size. For example, the memory level stores recently accessed small datasets, the disk level stores data with medium frequency, and the remote level is used for large historical data, thus ensuring a fast response to common queries.

[0135] S11, multi-level hierarchical design.

[0136] In one embodiment, in step S11, the multi-level hierarchical design first identifies the data access pattern and determines the capacity of each layer by statistically analyzing historical query logs. For example, pipeline data such as DWG format files are placed in memory to support real-time read and write, which has the benefit of reducing the latency of reading from the underlying storage.

[0137] S111, Identify data access patterns.

[0138] Specifically, in step S111, query records from the past week are collected, the number of accesses for each data item is calculated, and a frequency list is formed for subsequent layer allocation.

[0139] S112. Determine the capacity of each layer.

[0140] For example, in step S112, based on the frequency list, data accessed more than 50 times is moved into the memory level, with the capacity set to 20% of the system memory, thus optimizing resource utilization.

[0141] S12, Hotspot preloading.

[0142] In one embodiment, in step S12, hotspot preloading is performed periodically during system startup or idle time to predict the data to be accessed and load it to the upper layer. For example, for GDB data in the pipeline sharing governance phase, hotspot element codes are preloaded to improve download efficiency. Its beneficial effect is to prepare data in advance and shorten user waiting time.

[0143] S121, Predict the data to be accessed.

[0144] Specifically, in step S121, based on the frequency list and time pattern, such as weekday peak hours, the top 10% of popular items are selected.

[0145] S122, Load to the upper layer.

[0146] For example, in step S122, the selected items are moved from the disk level to the memory level, and the index is updated for faster location.

[0147] In one possible implementation, for the repair testing phase, relevant cross-sectional dimension data is preloaded to ensure a smooth inspection process.

[0148] S13, Consistency Guarantee.

[0149] In one embodiment, in step S13, consistency assurance is achieved through periodic verification and synchronization mechanisms, such as comparing the cached version with the source data version. If they are inconsistent, the data is refreshed to ensure lossless conversion of multi-format data such as DWG and GDB. Its beneficial effect is to maintain data accuracy and avoid error propagation.

[0150] S131. Periodic verification.

[0151] Specifically, in step S131, hot items are scanned every 5 minutes, and the hash value is calculated and compared with the source.

[0152] S132, Synchronization Mechanism.

[0153] For example, in step S132, if the hash does not match, an incremental update is triggered, and the difference is pulled from the source to replace the cache.

[0154] In one possible implementation, for large amounts of data during the aggregation phase, the mechanism prioritizes the consistency of attribute fields and supports parallel validation to accelerate the process.

[0155] For example, in the pipeline data ingestion scenario, the multi-level layered design stores attribute and feature codes in layers, hotspot preloading targets the enumeration value table and preloads it into memory, and consistency guarantee checks the update log before ingestion to ensure that all layers of data are synchronized, thus improving the overall access speed by up to 30%.

[0156] In one embodiment, considering the shared governance phase, hotspot preloading is combined with user role prediction, such as when an administrator loads a governance report. Consistency is ensured by using log parsing to identify differences and refreshing the cache in seconds. The beneficial effect is that it supports real-time collaboration without data conflicts.

[0157] S109. The difference comparison is based on the incremental update protocol and achieves second-level identification through log parsing and difference comparison technology.

[0158] The difference comparison is based on an incremental update protocol and achieves second-level identification through log parsing and difference comparison technology.

[0159] 1. Difference comparison based on incremental update protocol.

[0160] In one embodiment, the incremental update protocol defines rules for capturing and transmitting data changes, ensuring that only modified portions of the data are processed, rather than the entire dataset.

[0161] Specifically, the protocol first captures data change events, then encapsulates these events into standardized packets for subsequent parsing and comparison. This approach is suitable for the pipeline data transition from aggregation to shared governance, avoiding the latency caused by full data transmission.

[0162] In one possible implementation, when pipeline data is modified during the production phase, such as when new elements are added during the repair and testing phase, the protocol automatically marks the changed areas to achieve efficient updates.

[0163] 2. Through log parsing.

[0164] For example, log parsing involves reading data operation records and breaking them down into atomic changes. The log records the timestamp, operation type, and scope of each data modification. By scanning the log line by line, pipeline elements that have been added, deleted, or modified, such as point coordinates or attribute values, can be extracted.

[0165] Specifically, the log parsing process involves filtering out irrelevant records, retaining only changes related to the pipeline format, such as layer adjustments in DWG files or feature encoding updates in GDB. This parsing ensures the accuracy of subsequent comparisons.

[0166] It should be noted that the output of log parsing is directly used for difference comparison, forming a continuous processing chain.

[0167] 21. Reading and decomposing logs.

[0168] In one embodiment, the log file is first opened, and each line of record is scanned. For pipeline data, the operation type is identified, such as inserting a new pipeline segment or updating the burial depth value. After decomposition, a change list is generated, with each item containing both the old and new values.

[0169] For example, in a pipeline repair test in DWG format, the log records the migration of the layer from temporary to formal, and the specific attribute changes are extracted by parsing, such as the adjustment of enumeration values.

[0170] 22. Filter and extract changes.

[0171] For example, filtering rules are applied to exclude non-pipeline-related logs, such as system logs, retaining only data content changes. After extraction, the changes are stored as key-value pairs for quick access.

[0172] In one possible implementation, for GDB format, modifications to the feature codes are extracted to ensure consistency with the four-table comparison system. This filtering improves parsing efficiency, achieving a response time within seconds.

[0173] 3. Comparison techniques with differences.

[0174] In one embodiment, the difference comparison technique uses hash comparison and structured matching to compare the parsed changes with the original data one by one. Hash comparison calculates a check value for each feature to quickly detect inconsistencies; structured matching checks for differences in attributes and geometric relationships, such as the logical consistency of pipeline connections.

[0175] Specifically, this technology is applied before pipeline data is entered into the database to ensure that the updated data meets the standards.

[0176] It should be noted that the comparison results identify specific differences, such as elevation mismatch or cross-sectional dimension deviation.

[0177] 31. Calculation and detection of hash comparisons.

[0178] For example, an MD5 hash value is calculated for each pipeline element and compared with the original value. If the hashes differ, it is marked as a change. This method is suitable for scenarios involving large-to-small pipe connections, quickly identifying connection anomalies.

[0179] In one possible implementation, during the shared governance phase, the differences between the DWG and GDB dual formats are compared to ensure seamless integration. The benefit of hash comparison is reduced computational overhead, enabling efficient detection.

[0180] 32. Structured matching check.

[0181] In one embodiment, the matching process iterates through the change list, verifying the consistency of attributes such as burial depth and elevation. If a discrepancy is found, such as burial depth exceeding a threshold, a report is generated.

[0182] For example, in repairing test data, the logical relationships between points and lines are matched to ensure there are no breaks. This matching improves data accuracy and reduces the workload of subsequent remediation.

[0183] 4. Achieve recognition within seconds.

[0184] For example, through the combination of log parsing and difference comparison described above, the entire process completes difference identification within seconds. The rapid scanning of log parsing and the instantaneous calculation of hash comparison ensure rapid response.

[0185] In one possible implementation, a high-performance computing architecture is used in conjunction with parallel computing to further accelerate identification for real-time pipeline data download and storage.

[0186] In one embodiment, the difference comparison based on the incremental update protocol in step 1, when applied during the pipeline junction stage, captures change events of newly added pipeline elements, such as the addition of a new drainage pipe, encapsulates them into packets, and transmits them, avoiding the time consumption caused by retransmitting all data. The application of this protocol makes data updates more efficient, especially in multi-source heterogeneous environments, reducing the overhead of format conversion.

[0187] For example, in step 2, log parsing is used to analyze the changes in pipeline point coordinates during the repair and testing phase. These changes, from (100, 200) to (100, 210), are extracted from the logs, and irrelevant system time records are filtered out. The benefit of this parsing is the precise location of the modification points, providing foundational data for subsequent comparisons and ensuring targeted data governance.

[0188] In one possible implementation, step 21 involves reading and decomposing the log in GDB format. This decomposition breaks the log down into atomic items, such as changes in feature codes from 001 to 002, generating a list for comparison. This decomposition improves the granularity of the processing, facilitating sub-second response times.

[0189] It should be noted that in step 22, filtering and extracting change items, the filtering rules in shared governance are based on a four-table comparison system. The extraction of attribute changes, such as updates to enumerated values, moves from "underground" to "overhead." This extraction ensures the realization of data standardization.

[0190] Specifically, step 3, the discrepancy comparison technique, identifies inconsistencies in elevation and burial depth during the pre-warehousing inspection phase. For example, if the burial depth is 2.5 meters while the calculated elevation should be 2.0 meters, this is marked as a discrepancy. The benefit of this technique is that it automates problem detection and generates visual reports to support decision-making.

[0191] For example, in step 31, the hash comparison calculation and detection involves calculating a hash for the pipeline segment in the DWG file. If the length is modified from 10 meters to 15 meters, the hash change is detected. This comparison is fast enough to achieve identification within seconds, making it suitable for real-time download scenarios.

[0192] In one possible implementation, step 32, the structured matching check, in a large-pipe-to-small-pipe scenario, verifies whether the connection between pipes with diameters ranging from 200mm to 100mm is compliant. If it is non-compliant, the discrepancy is flagged. The benefit of this matching is maintaining logical consistency and preventing incorrect data entry into the database.

[0193] It should be noted that step 4 achieves second-level recognition. In the entire process, from downloading to data entry, the recognition time is controlled within 1-3 seconds. This speed improves the update efficiency of a pipeline map and ensures data real-time performance.

[0194] Multi-source heterogeneous data read / write compatibility: Pipeline data reception adopted a dual-format approach, using both DWG and GDB, at different production stages (convergence, repair survey, and shared management). To accommodate different data formats and content, a four-table comparison system (layer, attribute, element code, and enumeration value) was developed based on data standards to achieve seamless integration and standardized management of multi-format data. By integrating ODA and GDAL technologies, the system enables reading and writing of commonly used pipeline data formats such as DWG and GDB, achieving lossless conversion of pipeline data.

[0195] High-performance computing architecture: Employing intelligent caching, parallel computing, distributed architecture, and incremental update protocols, it efficiently achieves data ingestion and download. Intelligent caching uses a multi-level layered design, combined with hotspot preloading and consistency guarantee mechanisms, to improve data access speed; parallel computing improves computational efficiency through task pipeline decomposition; and the incremental update protocol achieves second-level difference identification through log parsing and difference comparison technologies.

[0196] Data Quality Assurance Technology: A unified data quality inspection algorithm engine has been developed. Through a four-table comparison system and quality assessment model, this engine enables automated inspection of multi-source heterogeneous data (such as DWG and GDB). The engine employs a layered inspection mechanism, performing integrity checks (e.g., field non-empty, format compliance), logical consistency checks (e.g., elevation and depth consistency checks, large-pipe-to-small-pipe connection checks), and accuracy reviews (e.g., cross-sectional dimension checks, elevation checks, depth checks) before data is entered into the database. Inspection items are categorized into two types based on scalability: fully parameterizable and partially parameterizable. Fully parameterizable items, such as attribute checks and point-line logical relationship checks, abstract various inspection rules, allowing users to freely design inspection content based on the provided rules. Partially parameterizable items expose the variable parts of the inspection function in parameter form, allowing users to modify them according to specific requirements, maximizing the versatility of the inspection function. Through a unified inspection algorithm, the engine ensures that the entered data complies with industry standards and business requirements, while generating visual inspection reports to provide decision-making support for data governance.

[0197] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of the embodiments of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

[0198] It will be understood by those skilled in the art that the above descriptions are merely preferred examples of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention. All technical features in this embodiment can be freely combined according to actual needs.

[0199] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for updating multi-source heterogeneous pipeline data, characterized in that: Specifically, it includes the following steps; Step 1: Create a data download task, record the download range and pipeline type information, and determine whether there is a related data import task. If so, wait for the related task to be executed. Step 2: Create a data import task, check if there is a related data download task, and if so, wait for the related task to be executed; Step 3: Through the download queue concurrency control mechanism, read multi-source pipeline data within a specified range, and assemble and download the data format; Step 4: Through the concurrency control mechanism of the inbound queue, perform quality checks before data is inbound, and read the data of the file to be inbound and the data in the current database; Step 5: Repair and test the read data, and compare the differences between added, deleted, and modified data; Step 6: Update the current database with the compared addition, deletion and modification information, and synchronize the deletion and modification information to the historical database.

2. The method for updating multi-source heterogeneous pipeline data according to claim 1, characterized in that: The multi-source pipeline data includes DWG and GDB formats, and lossless reading, writing and format conversion are achieved by integrating ODA and GDAL technologies.

3. The method for updating multi-source heterogeneous pipeline data according to claim 1, characterized in that: The method is based on a four-table comparison system to achieve standardized mapping and management of multi-source heterogeneous data. The four tables include: a layer comparison table, an attribute comparison table, an element code comparison table, and an enumeration value comparison table.

4. The method for updating multi-source heterogeneous pipeline data according to claim 1, characterized in that: The quality checks include integrity verification, logical consistency verification, and accuracy review, and adopt a layered inspection mechanism that supports fully parameterized and partially parameterized inspection item configuration.

5. The method for updating multi-source heterogeneous pipeline data according to claim 1, characterized in that: The logical consistency verification includes elevation and burial depth consistency checks and large-pipe-to-small-pipe connection checks, while the accuracy review includes cross-sectional dimension checks, elevation checks, and burial depth checks.

6. The method for updating multi-source heterogeneous pipeline data according to claim 1, characterized in that: The method employs an intelligent caching mechanism, including multi-level layered design, hotspot preloading, and consistency guarantee, to improve data access speed.

7. The method for updating multi-source heterogeneous pipeline data according to claim 1, characterized in that: The difference comparison is based on an incremental update protocol and achieves second-level identification through log parsing and difference comparison technology.

8. The method for updating multi-source heterogeneous pipeline data according to claim 1, characterized in that: The download queue concurrency control mechanism and the database entry queue concurrency control mechanism respectively adopt task priority scheduling and dynamic thread pool management.

Images