A dynamic association method of waterway models based on multi-source data indexing

Abstract

The present application discloses a method for dynamic association of waterway models based on multi-source data indexing, which involves digital management of waterway engineering: constructing component codes of waterway engineering to identify the same physical entity; establishing a three-layer index table structure based on the component codes; constructing a dynamic partition management mechanism of a three-level cache architecture, the three-level cache architecture includes a first-level cache stored in memory, a second-level cache stored on local disks, and a third-level cache stored in distributed storage; performing data migration between caches at all levels based on data access information in the three-layer index table structure; establishing a data version management mechanism based on a timestamp concurrency control protocol based on the dynamic partition management mechanism, the data version management mechanism generates incremental data by comparing original data and updated data, and selects incremental updates or full updates based on thresholds. The present application solves the problem of difficulty in tracking data evolution caused by fragmentation of multi-source heterogeneous data and uneven access efficiency in waterway engineering.

Description

Technical Field

[0001] The present application relates to digital management of waterway engineering, and more specifically, to a dynamic association method of waterway models based on multi-source data indexing. Background Art

[0002] With the advancement of informatization, digitization, and intelligent development in waterway engineering, waterway infrastructure management has gradually evolved from traditional static two-dimensional representation to a multi-dimensional, all-time and all-space data-driven model. Modern waterway engineering encompasses multiple disciplines, including hydraulic structures, waterway regulation projects, dredging, and communication and navigation facilities. This generates heterogeneous datasets from multiple sources, including BIM 3D models (IFC / RVT formats), GIS geospatial information (GeoJSON / Shapefile formats), CAD engineering drawings (DWG / DXF formats), and IoT sensor monitoring data (structured / time-series data). Under current technology, this data is generally stored in different business systems, each with its own independent data storage structure and access interface, resulting in a decentralized and heterogeneous data silo architecture.

[0003] From the perspective of system architecture and data structure, the existing waterway engineering data management technology has the following technical bottlenecks:

[0004] In existing systems, data from different sources uses independent encoding rules, such as GUIDs in BIM models, spatial indexes in GIS systems, and device IDs in monitoring systems. This results in the same physical entity having different identifiers in different systems. This identifier heterogeneity requires data mapping relationships between systems to be implemented through complex intermediate tables, resulting in a query complexity of O(n × m), and making it difficult to ensure the consistency and integrity of the mapping.

[0005] Existing technologies use a single storage strategy, failing to optimize the access characteristics of different data types. For example, BIM model data is stored as a single file, requiring the entire model to be loaded for fine-grained access. Sensor data is stored in relational databases, resulting in significant performance degradation in high-frequency time-series queries. This structural mismatch leads to an unbalanced system I / O load and latency of up to several seconds for accessing hotspot data.

[0006] Multi-source data formats vary significantly, lacking a unified preprocessing pipeline and data adaptation layer. Existing technologies typically employ a point-to-point conversion approach. When there are n data source types, n converters are required. These converters are unable to adaptively process undefined data formats, severely limiting system scalability.

[0007] Existing version control systems often rely on file-level snapshots, which lack efficient support for incremental updates. When project data volumes reach terabytes, full backups lead to exponentially increasing storage overhead and inefficient historical version retrieval. Furthermore, the lack of a timestamp-based data consistency mechanism makes it impossible to accurately trace the project status at a specific point in time. Summary of the Invention

[0008] This application aims to solve the problem of data evolution tracking difficulties caused by fragmentation of multi-source heterogeneous data and uneven access efficiency in waterway engineering. Specifically:

[0009] (1) Waterway engineering involves multiple professional fields such as hydraulic structures, waterway regulation projects, and dredging projects, generating multi-source heterogeneous data such as BIM models, GIS geographic information, CAD drawings, and sensor monitoring data. Each of these uses independent coding rules and data structures, forming scattered data islands.

[0010] (2) Traditional storage strategies cannot optimize the access characteristics of different types of data, resulting in unbalanced system load and access delays;

[0011] (3) Existing version control mostly uses file-level snapshots, which do not provide sufficient support for incremental updates and make it difficult to efficiently trace the project status at a specific point in time.

[0012] One aspect of the present application provides a dynamic association method for a waterway model based on multi-source data indexing, including: constructing a component code for a waterway project, where the component code is used to identify the same physical entity; establishing a three-layer index table structure based on the component code; constructing a dynamic partition management mechanism for a three-level cache architecture, where the three-level cache architecture includes a first-level cache for memory storage, a second-level cache for local disk storage, and a third-level cache for distributed storage; migrating data between caches at each level based on data access information in the three-layer index table structure; establishing a data version management mechanism based on a timestamp concurrency control protocol based on the dynamic partition management mechanism, where the data version management mechanism generates incremental data by comparing original data and updated data, and selects incremental update or full update based on a threshold.

[0013] Among them, in waterway engineering, physical entities refer to specific components, structures or equipment.

[0014] Furthermore, component coding for waterway projects is constructed, including: establishing multi-level component coding rules for waterway projects, the multi-level component coding rules using a hierarchical structure to represent project, section, structure and component information; establishing standardized specifications for component attribute fields based on the multi-level component coding rules, the standardized specifications defining the geometric properties, material properties, construction properties and operation and maintenance properties of components; establishing a relationship mapping table between component codes and external data sources based on the multi-level component coding rules and the standardized specifications for component attribute fields, the relationship mapping table being used to establish a unified index between BIM model element IDs, CAD drawing numbers, sensor equipment IDs and operation and maintenance record numbers and component codes; generating component codes based on the multi-level component coding rules, the standardized specifications and the relationship mapping table.

[0015] Among them, in this application, the hierarchical structure represents a top-down multi-level classification organizational structure. In this scheme, it specifically refers to the coding structure that refines the waterway engineering information layer by layer according to the engineering level, section level, structure level and component level. This structure can clearly express the subordinate relationship between components at all levels, which is convenient for systematic management and rapid positioning of specific components. Standardization specifications represent the unified definition and classification standards of the properties of waterway engineering components, including geometric properties (size, position), material properties (material, strength), construction properties (construction methods, quality standards) and operation and maintenance properties (maintenance cycle, usage status). This specification ensures the consistency of data description in different professional fields. External data sources represent various types of heterogeneous data generated in waterway engineering and their source systems, mainly including BIM modeling systems, CAD design systems, GIS geographic information systems, sensor monitoring systems and operation and maintenance management systems.

[0016] Furthermore, the component coding adopts the following component coding format: project code - section number_construction type_professional code_component ID_description field; among them, the project code is a 3-digit letter used to identify the project to which it belongs; the section number is a 2-digit number corresponding to the actual construction section; the construction type is a 3-digit letter indicating the type of structure to which the component belongs; the professional code is a 3-digit letter indicating the professional category to which the component belongs; the component ID is a 6-digit number indicating the specific component serial number; the description field is optional and indicates component characteristics or version information.

[0017] In particular, as a core component of water transport infrastructure construction, waterway projects have significant characteristics such as long linear projects, diverse structures, complex environmental conditions, multiple professional intersections, and long life cycles. In addition, waterway projects extend tens or even hundreds of kilometers along the river, spanning multiple administrative regions, and are often constructed in sections. The hierarchical coding structure proposed in this application (project code - section number_construction type_professional code_component ID_description field) is designed to adapt to the spatial distribution characteristics of waterway projects:

[0018] The project code (3 letters) solves the identification and distinction between different waterway projects and meets the needs of waterway network management;

[0019] The section number (2 digits) corresponds to the common sub-contracting model for waterway projects, which facilitates the division of responsibilities;

[0020] The structure type (3 letters) can distinguish various structures in waterway engineering, such as revetments, diversion dikes, groynes, underwater submerged weirs, and navigation marks;

[0021] The professional code (3 letters) reflects the cross-disciplinary characteristics of waterway engineering, including hydrology, geology, structure, and mechanical and electrical engineering.

[0022] On the other hand, waterway engineering data sources are highly complex. This application addresses the consistency issue between waterway 3D modeling and actual components by mapping BIM model element IDs to physical components. It also establishes a correlation between CAD drawing numbers and components, adapting to the need to transform 2D design into 3D implementation of waterway engineering. It also associates sensor device IDs with components, meeting the need for real-time data correlation for waterway hydrology and structural health monitoring. It also integrates operation and maintenance record numbers, resolving the issue of historical data tracing in the long-term operation and maintenance management of waterway structures.

[0023] Furthermore, a three-layer index table structure is established, including: setting up an information resource layer to classify and store multi-source data according to the engineering code and section number of the component code; setting up a fusion index layer to establish a data index table according to the construction type, professional code and component ID in the component code; setting up a service support layer to provide data analysis and query according to the description field in the component code; building a data type recognition engine according to the multi-level component coding rules, the engine matches the component identifier in the data file with the component code, and classifies and archives the data according to the construction type and professional code field; setting up a data access mechanism of the adapter mode according to the recognition result of the data type recognition engine, and standardizing and converting data from different sources according to the field structure of the component code, so that data from different sources correspond one-to-one to each component of the component code; establishing a data index table based on the component ID in the fusion index layer, the data index table uses the component code as the primary key, associates the data type, data path, timestamp and version number, and performs multi-level indexing and query through the engineering code, section number, construction type and professional code in the component code.

[0024] The data access mechanism of the adapter pattern represents a heterogeneous data processing architecture based on design patterns. In this application, it is used to address the issue of inconsistent data formats from multiple sources in waterway engineering. This mechanism configures dedicated adapter components for each data source (BIM, CAD, sensor data, etc.). These adapters are responsible for converting the specific structure of the raw data into a unified system data structure, decoupling the data source from the processing logic. The adapters receive the raw data, extract key identification information, and then map it to the corresponding fields in the component code. This enables the system to process various heterogeneous data types in a unified manner, without having to understand the internal structure of each data source.

[0025] Standardization conversion: This process converts raw data from various sources and formats within waterway engineering projects into standardized data that conforms to unified coding rules. This process includes steps such as data format unification, field mapping, structural reorganization, and metadata extraction, ensuring that all data types accurately correspond to fields such as the project code, section number, and construction type used in component coding. Standardization conversion not only processes structured data (such as database records), but also parses semi-structured data (such as XML files) and unstructured data (such as documents and images), extracting and normalizing component information, laying the foundation for subsequent data indexing and correlation analysis.

[0026] In particular, on the one hand, the data organization of waterway projects has the characteristics of "linear extension, point distribution, and multi-dimensional association". In this application, the information resource layer is classified and stored based on the project code and section number, which adapts to the cross-regional distribution characteristics of waterway projects and realizes the geographic spatial organization of data; it also solves the storage and management problems caused by the huge amount of data in long-distance waterway projects. The fusion index layer establishes indexes based on construction type, professional code and component ID to solve the data management differences of different structures in waterway projects (such as revetments, anchors, and navigation facilities), and supports cross-indexing and correlation analysis of multi-disciplinary data such as hydrology, geology, and structure. The service support layer uses description fields to provide data analysis and query, adapting to the multi-objective management needs of waterway projects, such as flood control standards, navigation levels, ecological requirements, etc.; it supports thematic analysis unique to waterway projects, such as riverbed evolution, navigation condition assessment, and stability analysis of hydraulic structures.

[0027] On the other hand, waterway engineering data types are highly heterogeneous, ranging from traditional CAD drawings to modern BIM models and real-time hydrological sensor data. In this application, a data type recognition engine overcomes the barriers to identifying different data formats, such as AutoCAD, Revit, Excel, and database records, by matching component identifiers and component codes within data files. This addresses the data format incompatibilities that arise from using different software during waterway survey, design, construction, and monitoring.

[0028] Furthermore, a dynamic partition management mechanism for the three-level cache architecture is constructed, including: statistics on access frequency, time distribution, and user roles of various types of data based on the data index table, analysis of data usage in different business scenarios, and calculation of data heat scores; setting up a three-level cache architecture based on the data heat scores; setting business priorities based on business types and task urgency; and establishing a data migration mechanism based on data heat scores and business priorities.

[0029] The data heat score is a quantitative indicator used to measure the activeness and importance of various types of waterway engineering data in the system. This score is calculated by comprehensively analyzing multiple factors, including data access frequency (number of calls per unit time), time distribution (the periodicity and time period of access), and user role (the proportion of access by users with different permissions and functions). The data heat score uses a weighted algorithm, assigning higher weight to recent accesses and adjusting the score based on the importance of the business scenario. The resulting numerical indicator determines the optimal storage location of data within the three-level cache architecture, enabling dynamic optimization of storage resources.

[0030] According to the data heat score, a three-level cache architecture is set, including: according to the data heat score, determining the access frequency level of the data, marking the data with an access frequency greater than the threshold P1 as high-frequency data, marking the data with an access frequency less than or equal to the threshold P1 and greater than the threshold P2 as medium-frequency data, and marking the data with an access frequency less than or equal to the threshold P2 as low-frequency data; setting the first-level cache to store high-frequency data, the first-level cache adopts a memory storage method, and the response time is less than T1; setting the second-level cache to store medium-frequency data, the second-level cache adopts a local disk method, and the response time is less than T2; setting the third-level cache to store low-frequency data, the third-level cache adopts a distributed storage method, and the response time is less than T3; wherein, T1 is less than T2, and T2 is less than T3. In this application, T1 has a value range of 50ms to 200ms, indicating millisecond-level real-time response; T2 has a value range of 250ms to 800ms, which is used for interactive query scenarios; T3 has a value range of 1000ms and above, which is used for batch analysis or archival storage.

[0031] In particular, waterway engineering data access has significant particularities: strong engineering phases, large differences in data timeliness, geographically dispersed access, and high emergency response requirements. Traditional single storage architectures are difficult to meet these complex requirements. This application proposes a three-level cache architecture based on data heat scoring, in which the data heat scoring mechanism calculates the value of data by statistically analyzing access frequency, time distribution, and user roles. For example, waterway engineering has obvious seasonality and phases, such as frequent access to monitoring data during the flood season and increased access to construction data during the dry season. The data heat scoring can dynamically reflect this cyclical change; waterway hydrological data has high timeliness, topographic survey data has medium timeliness, and geological survey data has low timeliness. The heat scoring mechanism can accurately reflect this difference; in addition, different roles such as waterway design, construction, supervision, and operation and maintenance pay attention to different data, and the heat scoring realizes data priority personalization through user role weights.

[0032] Specifically, the first-level cache (memory storage) meets the millisecond-level response requirements for real-time data such as waterway hydrological monitoring and lock operation, supports the rapid data retrieval and analysis required for waterway emergency scheduling decisions, and optimizes the access efficiency of high-frequency query data (such as current river water level and navigation status) in waterway projects.

[0033] The second-level cache (local disk) adapts to weak network or offline work scenarios in the waterway on-site construction environment. It supports data that is frequently used in waterway projects but does not require real-time response (such as construction progress records and quality inspection data), balancing waterway project data access efficiency and storage costs.

[0034] The third-level cache (distributed storage) solves the problem of long-term preservation of historical waterway data, provides unified storage capabilities for data along the entire waterway project, and covers cross-regional waterway data sharing needs; it supports waterway big data analysis and mining, such as long-term river flow trends and evolution of structure performance.

[0035] Furthermore, a data version management mechanism based on a timestamp concurrency control protocol is established, including: setting a timestamp identifier and a version number identifier for data change records; identifying the changed content by comparing the original data with the updated data based on the timestamp identifier and the version number identifier, and constructing an incremental data record containing the changed fields and corresponding metadata; selecting incremental updates or full updates based on the proportion of changed content in the incremental data record, and detecting and handling data update conflicts through a timestamp mechanism during the data update process; for updated data, recording data change records including the update time, update content, operator, and affected scope.

[0036] Furthermore, based on the timestamp identifier and version number identifier, the changed content is identified by comparing the original data with the updated data, and an incremental data record containing the changed fields and corresponding metadata is constructed, including: parsing the original data structure and the updated data structure, and establishing a field mapping relationship; comparing the values ​​of the corresponding fields in the original data and the updated data, and identifying the changed fields of all levels by traversing; for each identified changed field, obtaining the path, data type and value before and after the change of the changed field in the data structure; constructing a metadata record for each changed field, the metadata record includes the field path, field type, change information, change timestamp and operation identifier; combining the changed fields and the corresponding metadata records as an incremental data record; adding global metadata to the incremental data record, the global metadata includes the total number of changes, the change ratio and the data integrity check code.

[0037] Furthermore, incremental update or full update is selected according to the proportion of changed content in the incremental data record, and data update conflicts are detected and handled through the timestamp mechanism during the data update process, including: identifying the data to be updated according to the component code, obtaining the storage location of the original data in the three-level cache architecture through the data index table; calculating the ratio of the number of changed fields in the incremental data record to the total number of fields in the original data as the change ratio; comparing the change ratio with the preset threshold; when the change ratio is less than the preset threshold, selecting the incremental update method and adding the incremental data record to the original data; when the change ratio is greater than or equal to the preset threshold, adopting the full update method. A new method is used, and the original data is replaced with the data to be updated; before performing incremental updates or full updates, the current timestamp of the target data is retrieved from the three-layer index table structure, and compared with the timestamp of the original data recorded when the update operation starts to determine whether there is a concurrent update; when a concurrent update is detected, the data update request is processed according to the business priority; the data update result is written to the corresponding cache level, and the data distribution in the three-level cache architecture is updated according to the data heat score, and the timestamp and version number in the data index table are updated at the same time; the conflict detection and processing information of the update process is recorded in the data change log record as a historical basis for data version management.

[0038] In particular, waterway engineering data is large and distributed across different storage levels. This application uses component coding to quickly locate the target data storage location. Furthermore, by dynamically selecting incremental or full updates, data integrity is ensured while minimizing transmission overhead, making it particularly suitable for on-site waterway construction environments.

[0039] Another aspect of the present application also provides a dynamic association system for waterway models based on multi-source data indexing, including: a coding module, which constructs the component coding of the waterway project and establishes an index table of BIM model element ID, CAD drawing number, sensor equipment ID and operation and maintenance record number and component coding; an index module, which establishes a three-layer index table structure based on the component coding, which is used for standardized processing of different types of data and multi-table association retrieval; a partitioning module, which constructs a dynamic partition management mechanism based on the data access information provided by the index module, and performs multi-level caching and priority scheduling of data; a version module, which performs data updates and historical data backtracking management based on the dynamic partition management mechanism.

[0040] Compared with the existing technology, the advantages of this application are:

[0041] Waterway engineering, due to its intersectionality with multiple disciplines such as hydraulic engineering, waterway regulation, and dredging, presents a technical dilemma: diverse data sources, heterogeneous structures, and decentralized storage. Existing technologies lack a unified data indexing system and dynamic response mechanism, leading to severe information silos, inefficient data retrieval, and an inability to effectively track data version evolution.

[0042] (1) Solve the problem of data fragmentation:

[0043] The component coding system constructed in this application uses a structured expression of "project code - section number_construction type_professional code_component ID_description field," establishing a hierarchical data identification system. This unified coding system links heterogeneous data scattered across BIM models, CAD drawings, sensor networks, and operation and maintenance systems in waterway projects. This fundamentally resolves the technical dilemma of dispersed multi-disciplinary data in waterway projects.

[0044] (2) Improve access efficiency:

[0045] Through the combined application of adapter pattern design and data type recognition engine, a unified data interface standard and preprocessing pipeline were established, effectively overcoming the technical barriers in format conversion and semantic understanding of multi-source heterogeneous data (static BIM models, semi-structured CAD drawings, dynamic sensor data streams) in waterway engineering, and achieving standardization and automation of data management.

[0046] (3) Realize data evolution tracking:

[0047] The three-level cache architecture innovatively introduces a data popularity scoring mechanism. This enables dynamic data migration between memory, local disk, and distributed storage through a weighted calculation of access frequency, time distribution, and business weight. The three-tier index table structure and the three-tier cache architecture form a bidirectional linkage mechanism. The index layer provides data association mapping, while the cache layer dynamically adjusts data distribution based on access patterns. The version management system utilizes a timestamp concurrency control protocol based on incremental storage, enabling the system to maintain a complete data version evolution history with minimal storage overhead. This addresses the storage bloat issue in traditional version management systems while ensuring the efficiency and accuracy of historical data queries. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flowchart of the waterway model dynamic association method based on the multi-source data indexing mechanism of this application;

[0049] Figure 2 This is a diagram of the component coding system architecture of this application;

[0050] Figure 3 This is the structure diagram of the multi-source data index table of this application;

[0051] Figure 4 This is a schematic diagram of the dynamic partition management mechanism of this application;

[0052] Figure 5 This is a flow chart of the data version management system for this application. DETAILED DESCRIPTION

[0053] The present application is described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] Example 1

[0055] like Figure 1 As shown, the first aspect of the present invention provides a dynamic association method for a waterway model based on a multi-source data indexing mechanism, constructs a component code for a waterway project, and the component code is used to identify the same physical entity; according to the component code, a three-layer index table structure is established; a dynamic partition management mechanism of a three-level cache architecture is constructed, and the three-level cache architecture includes a first-level cache stored in memory, a second-level cache stored on a local disk, and a third-level cache stored in a distributed manner; data is migrated between caches at various levels according to data access information in the three-layer index table structure; according to the dynamic partition management mechanism, a data version management mechanism based on a timestamp concurrency control protocol is established, and the data version management mechanism generates incremental data by comparing the original data and the updated data, and selects incremental update or full update according to a threshold.

[0056] Construct a component-level coding system for waterway engineering, such as Figure 2 As shown, the specific steps include:

[0057] Design multi-level component coding rules for waterway engineering. The component coding format is as follows: component code = project code - section number_construction type_professional code_component ID_description field. Among them:

[0058] Project code (PROJECT): 3 letters used to identify the project, such as LYG for the Lianyungang project;

[0059] Section number (SECTION): 2 digits corresponding to the actual construction section, such as 01, 02;

[0060] Structure type (TYPE): 3 letters, such as NCH (channel), BRG (bridge), EMB (revetment);

[0061] DISCIPLINE: 3 letters, such as STR (Structure), MEC (Electromechanical), HYD (Hydrology);

[0062] Component ID: 6 digits, indicating the specific component serial number;

[0063] Description field (CONTENT): optional, indicating component characteristics or version information;

[0064] Establish standardized specifications for component attribute fields and define the following core attribute categories: geometric attributes: length, width, height, shape, spatial coordinates; material attributes: material type, specification, strength grade; construction attributes: construction date, construction unit, acceptance status; operation and maintenance attributes: maintenance cycle, responsible unit, health status;

[0065] Build a mapping relationship table between component codes and external data sources to achieve unified indexing: BIM model element IDs are mapped to component codes; CAD drawing numbers are associated with component codes; sensor device IDs are bound to component codes; and operation and maintenance record numbers correspond to component codes.

[0066] Develop component code automatic generation and verification tools: rule-based code automatic generation algorithm; code uniqueness check mechanism; code standardization verification function; batch code import and export interface.

[0067] Develop a multi-source data index table structure based on component coding, such as Figure 3 As shown, the specific steps include:

[0068] Design a three-tiered information organization system: The information resource layer carries pre-processed multi-source data, including static data (models, drawings), semi-structured data (documents, reports), and dynamic data (sensing, monitoring). The fusion index layer constructs data index tables through component coding, enabling structured connection between components and external information, field mapping, and update tracking. The service support layer provides data support for functional modules such as visualization analysis, information query, and risk identification.

[0069] Develop a data type recognition engine: file format recognition: supports .dwg (CAD drawings), .ifc (BIM models), .obj / .fbx (lightweight models), .las (point clouds), .csv / .json (equipment data), etc.; data structure analysis: automatically parses the internal structure of the file and extracts key fields; automatic type classification: rule-based data classification algorithm, automatically classifies into corresponding categories.

[0070] Build a data access mechanism based on the adapter model: field mapping table: defines the correspondence between different source data fields and standard fields; format template library: preset common data format conversion templates; pre-processing pipeline: automated process of data cleaning, format conversion, and field unification.

[0071] Establish a data index table based on component ID: Index table structure design: component ID, data type, data path, timestamp, version number; multi-table association mechanism: support one-to-many and many-to-many data association; fast retrieval algorithm: efficient index query based on B-tree.

[0072] Develop data association strength assessment algorithms: Access frequency statistics: record the number of accesses and time distribution of each data item; Business association analysis: evaluate the business logic association between different data; Storage optimization strategy: optimize data storage layout based on association strength.

[0073] Build a dynamic partition management mechanism based on data access frequency, such as Figure 4 As shown, the specific steps include:

[0074] Statistics on access frequency and usage scenarios of various types of data: Access frequency statistics: record the number of accesses, time distribution, and user role of each data item; Usage scenario analysis: classify and count data usage in different business scenarios; Heat evaluation model: construct a data heat scoring algorithm based on access frequency and business weight.

[0075] A three-level cache architecture was designed: Memory cache (L1) stores frequently accessed hot data, ensuring millisecond-level real-time response times; Local cache (L2) stores moderately accessed data, offering faster response times than remote storage and suitable for interactive queries; and Remote storage (L3) stores cold data and complete historical data, offering relatively longer response times and suitable for batch analysis or archiving. The response latency gradients for these three tiers increase in sequence, and the system automatically adjusts data distribution between tiers using popularity scoring and dynamic migration mechanisms.

[0076] Develop an intelligent scheduling algorithm based on task priority:

[0077] Input: data request queue Q, service priority table P;

[0078] Output: sorted request queue SQ

[0079]

[0080] Establish a data migration mechanism between partitions: Hot and cold data identification: Dynamically identify hot and cold data based on access frequency thresholds; Automatic migration strategy: Hot data is automatically migrated to high-level cache, and cold data is sunk to low-level storage; Load balancing algorithm: Dynamically adjust data distribution based on the load of each partition;

[0081] Build a partition capacity monitoring and early warning system: Capacity utilization monitoring: real-time monitoring of storage usage in each partition; dynamic capacity expansion mechanism: automatic triggering of capacity expansion when capacity reaches the threshold; strategy optimization feedback: continuous optimization of partition strategies based on operational data.

[0082] Establish a timestamp-based data version management system, such as Figure 5 As shown, the specific steps include:

[0083] S1, obtains data change request, receives data update request D2, and verifies data request and authority.

[0084] S2 locates the original data D1 through an efficient indexing mechanism and adds a timestamp and version identifier to each change data record: Timestamp format: uses the ISO8601 standard, accurate to the millisecond level (yyyy-MM-ddTHH:mm:ss.SSSZ); Version number rule: semantic version control, the format is major version number.minor version number.revision number (such as 1.2.5); Version chain construction: forms a complete version evolution chain, supports version backtracking and branch management; sets timestamps and version number identifiers to establish data lineage relationships.

[0085] S3, compare the original data D1 and the incremental data D2, construct the incremental data record ΔD, and calculate the change ratio R:

[0086]

[0087] If the change ratio R exceeds a preset threshold, step S4.1 is executed for an incremental update; otherwise, step S4.2 is executed for a full update. In this embodiment, the preset threshold ranges from 30% to 40%. Furthermore, this embodiment ensures data consistency by using transaction control to ensure update atomicity and conflict detection and resolution using a timestamp-based automatic conflict resolution mechanism.

[0088] After completing the data update, perform step S5.1 incremental merge and write back to generate a new timestamp and version package to maintain data integrity, or perform step S5.2 full merge and write back to replace the complete data and synchronously update all related metadata.

[0089] After the data is written back, step S6 is executed to write the data in layers according to the heat score mechanism. This embodiment adopts a three-level cache, namely L1, L2 and L3. Among them, L1 stores the most frequently accessed data (for example, the heat score is greater than 0.8), L2 stores the medium-frequency access data (for example, the heat score is greater than 0.3 and less than 0.8), and L3 stores the infrequently accessed data (for example, the heat score is less than 0.3).

[0090] S7, establish a data change log recording system: log structure design: timestamp, operation type, changer, change content, and scope of impact; S8, log storage strategy: use circular log files, support log archiving and compression; audit tracking function: support multi-dimensional log retrieval and analysis; change notification mechanism: key changes are automatically pushed to relevant responsible persons.

[0091] Realize the function of rapid backtracking of historical data: snapshot storage: regularly generate full data snapshots to support rapid recovery; version retrieval: fast query based on time range or version number; data restoration: support data status restoration at a specified time point; difference comparison: provide difference comparison and visual display between versions.

[0092] Example 2

[0093] In this example, a component coding system covering a 32.74-kilometer channel was constructed for the Lianyungang Port Area's secondary channel improvement project. This project includes multiple sub-projects, including channel dredging, revetment reinforcement, and the construction of new service areas. The coding system was established according to the following rules:

[0094] The project code is "LYG" (the abbreviation of Lianyungang);

[0095] The bidding section number is divided according to the actual bidding section, such as "01" and "02";

[0096] Structure types include NCH (navigation channel), BRG (bridge), EMB (revetment), BUY (navigation aid), etc.

[0097] Professional codes include STR (structure), HYD (hydrology), NAV (navigation aid), etc.

[0098] For example, the first revetment component code of the upstream approach channel of Yunshan Ship Lock is: LYG-01_EMB_STR_000001_V1.0.

[0099] A unified index table structure has been developed for the multi-source, heterogeneous data involved in waterway engineering, including BIM models, CAD drawings, sensor data, and operation and maintenance records. For example, the revetment BIM model (IFC format), related design drawings (DWG format), and structural health monitoring sensor data (JSON format) are all linked using the component code "LYG-01_EMB_STR_000001." The system automatically identifies file types and completes data linkage.

[0100] Based on actual operation statistics, a three-level cache architecture was designed:

[0101] L1 cache: stores frequently accessed channel cross-section data and real-time ship AIS data, with a response time of <100ms;

[0102] L2 cache: stores frequently accessed construction drawings and operation and maintenance records, with a response time of <500ms;

[0103] L3 storage: stores complete historical monitoring data and design archives, with a response time of <2000ms.

[0104] We implement timestamp-based version control. For example, the bank reinforcement plan underwent three revisions: V1.0 (initial version) → V1.1 (optimized reinforcement layout) → V1.2 (added anti-scour measures). The system records the time, operator, and content of each change, allowing for quick backtracking of any version.

[0105] Through the implementation of the above four steps, the project successfully built a multi-source data index management system covering the entire port channel of Lianyungang Port Area, realizing the efficient integration and dynamic management of design, construction, operation and maintenance data.

[0106] Example 3

[0107] The second aspect of the present invention also provides a waterway model dynamic association system based on a multi-source data indexing mechanism, a coding module, which constructs the component coding of the waterway project and establishes an index table of BIM model element ID, CAD drawing number, sensor equipment ID and operation and maintenance record number and component coding; an index module, which establishes a three-layer index table structure based on the component coding, which is used for standardized processing of different types of data and multi-table association retrieval; a partitioning module, which constructs a dynamic partitioning management mechanism based on the data access information provided by the indexing module, and performs multi-level caching and priority scheduling of data; a version module, which performs data updates and historical data backtracking management based on the dynamic partitioning management mechanism. The system includes: a memory and a processor, wherein the memory includes a waterway model dynamic association method program based on a multi-source data indexing mechanism, and when the waterway model dynamic association method program based on a multi-source data indexing mechanism is executed by the processor, the following steps are implemented:

[0108] Construct a component-level coding system for waterway engineering projects, formulate coding rules with a multi-level structure of division-item-type, and establish standardized specifications for component attribute fields;

[0109] Develop a multi-source data index table structure based on component coding, design data type identification and adapter patterns, and implement classified connection of static data, semi-structured data, and dynamic data;

[0110] Build a dynamic partition management mechanism based on data access frequency, design a multi-level cache system, and develop a data scheduling algorithm based on task priority;

[0111] Establish a timestamp-based data version management system, design a collaborative mechanism for incremental updates and full updates, and realize data change log records and rapid backtracking of historical data.

[0112] The invention of the present application and its implementation methods are described schematically above. This description is not restrictive. Without departing from the spirit or basic features of the present application, the present application can be implemented in other specific forms. What is shown in the accompanying drawings is only one of the implementation methods of the invention of the present application, and the actual structure is not limited to this. Therefore, if a person of ordinary skill in the art is inspired by it, without departing from the purpose of the invention, a structural method and embodiment similar to the technical solution are designed without creativity, which should all fall within the scope of protection of the present application. In addition, the word "including" does not exclude other elements or steps, and the word "one" before an element does not exclude the inclusion of "multiple" elements. Words such as first and second are used to indicate names and do not indicate any specific order.

Claims

1. A dynamic association method for waterway models based on multi-source data indexing, characterized in that: include: Construct component codes for waterway engineering, which are used to identify the same physical entity; According to the component coding, a three-layer index table structure is established; Build a dynamic partition management mechanism for a three-level cache architecture, which includes a first-level cache stored in memory, a second-level cache stored on local disks, and a third-level cache stored in distributed storage. Data is migrated between cache levels based on data access information in the three-layer index table structure; Based on the dynamic partition management mechanism, a data version management mechanism based on the timestamp concurrency control protocol is established. The data version management mechanism generates incremental data by comparing the original data and the updated data, and selects incremental update or full update based on the threshold; Component coding uses the following component coding format: Project code - Section number_Construction type_Professional code_Component ID_Description field; The project code is a three-letter letter used to identify the project to which it belongs; The section number is a two-digit number corresponding to the actual construction section; The structure type is a three-letter letter that indicates the type of structure to which the component belongs; The professional code is a three-letter code indicating the professional category to which the component belongs; Component ID is a 6-digit number, indicating the specific component serial number; The description field is optional and indicates component characteristics or version information; According to the component code, a three-layer index table structure is established, including: Set up an information resource layer to classify and store multi-source data according to the engineering code and section number of the component coding; Set up a fusion index layer and create a data index table based on the construction type, professional code and component ID in the component code; Set up a service support layer to provide data analysis and query based on the description fields in the component code; According to the multi-level component coding rules, a data type recognition engine is built. The engine matches the component identification in the data file with the component coding, and classifies and archives the data according to the construction type and professional code field; According to the identification results of the data type identification engine, the data access mechanism of the adapter mode is set up to standardize the data from different sources according to the field structure of the component code, so that the data from different sources correspond to the components of the component code one by one; A data index table based on component ID is established in the fusion index layer. The data index table uses component code as the primary key, associates data type, data path, timestamp and version number, and performs multi-level indexing and query based on the project code, section number, structure type and professional code in the component code; Data is migrated between cache levels based on data access information in the three-layer index table structure, including: Based on the data index table, statistics are collected on the access frequency, time distribution, and user roles of various types of data, data usage in different business scenarios is analyzed, and data popularity scores are calculated; Set up a three-level cache architecture based on data heat score; Set business priorities based on business type and task urgency; Establish a data migration mechanism based on data popularity scores and business priorities; According to the data heat score, a three-level cache architecture is set up, including: Based on the data heat score, the access frequency level of the data is determined. Data with an access frequency greater than the threshold P1 is marked as high-frequency data, data with an access frequency less than or equal to the threshold P1 and greater than the threshold P2 is marked as medium-frequency data, and data with an access frequency less than or equal to the threshold P2 is marked as low-frequency data. Set up the first-level cache to store high-frequency data. The first-level cache uses memory storage and the response time is less than T1; Set up a second-level cache to store IF data. The second-level cache uses a local disk, and the response time is less than T2. A third-level cache is set up to store low-frequency data. The third-level cache adopts a distributed storage method, and the response time is less than T3; among which, T1 is less than T2, and T2 is less than T3.

2. The method for dynamic association of waterway models based on multi-source data indexing according to claim 1, characterized in that: Construct component codes for waterway engineering, including: Establish a multi-level component coding rule for waterway engineering, which uses a hierarchical structure to represent engineering, section, structure and component information; According to the multi-level component coding rules, a standardized specification of component attribute fields is established, which defines the geometric attributes, material attributes, construction attributes and operation and maintenance attributes of the component; Based on the multi-level component coding rules and the standardized specifications of component attribute fields, a relationship mapping table between component codes and external data sources is established. The relationship mapping table is used to establish a unified index between BIM model element IDs, CAD drawing numbers, sensor device IDs, and operation and maintenance record numbers and component codes; Generate component coding according to multi-level component coding rules, standardization specifications and relationship mapping tables.

3. The method for dynamic association of waterway models based on multi-source data indexing according to claim 2, characterized in that: Based on the dynamic partition management mechanism, a data version management mechanism based on the timestamp concurrency control protocol is established, including: Set the timestamp and version number of data change records; Based on the timestamp and version number identifiers, the changes are identified by comparing the original data with the updated data, and an incremental data record containing the changed fields and corresponding metadata is constructed; Select incremental update or full update based on the proportion of changed content in the incremental data record, and use the timestamp mechanism to detect and handle data update conflicts during the data update process; For updated data, record the data change records including update time, update content, operator and affected scope.

4. The method for dynamic association of waterway models based on multi-source data indexing according to claim 3, characterized in that: Construct incremental data records containing changed fields and corresponding metadata, including: Parse the original data structure and update the data structure to establish field mapping relationships; Compare the values ​​of corresponding fields in the original data and the updated data, and identify the changed fields at all levels by traversing; For each identified changed field, obtain the path, data type, and values ​​before and after the change of the changed field in the data structure; Construct a metadata record for each changed field. The metadata record includes the field path, field type, change information, change timestamp, and operation identifier. Combine the changed fields and the corresponding metadata records as incremental data records; Add global metadata to incremental data records. The global metadata includes the total number of changes, the change ratio, and the data integrity check code.

5. The method for dynamic association of waterway models based on multi-source data indexing according to claim 3 is characterized in that: Select incremental update or full update based on the proportion of changes in the incremental data records, including: Identify the data to be updated based on the component code, and obtain the storage location of the original data in the three-level cache architecture through the data index table; Calculate the ratio of the number of changed fields in the incremental data record to the total number of fields in the original data as the change ratio; Compare the change ratio with the preset threshold; When the change ratio is less than the preset threshold, the incremental update method is selected and the incremental data records are added to the original data; When the change ratio is greater than or equal to the preset threshold, a full update is adopted, and the original data is replaced with the data to be updated; Before performing an incremental update or a full update, the current timestamp of the target data is retrieved from the three-layer index table structure and compared with the original data timestamp recorded at the start of the update operation to determine whether there is a concurrent update. When concurrent updates are detected, data update requests are processed according to business priorities; Write the data update results to the corresponding cache level, update the data distribution in the three-level cache architecture based on the data popularity score, and update the timestamp and version number in the data index table; The conflict detection and processing information of the update process is recorded in the data change log as a historical basis for data version management.

Images