Channel model dynamic association method based on multi-source data index

Abstract

The invention discloses a channel model dynamic association method based on a multi-source data index, and relates to channel engineering digital management. The channel model dynamic association method comprises the steps that component codes of channel engineering are constructed and used for identifying the same physical entity; establishing a three-layer index table structure according to the component codes; constructing a dynamic partition management mechanism of a three-level cache architecture, wherein the three-level cache architecture comprises a first-level cache stored in a memory, a second-level cache stored in a local disk and a third-level cache stored in a distributed manner; according to data access information in the three-layer index table structure, data migration is carried out among all levels of caches; according to the dynamic partition management mechanism, a data version management mechanism based on a timestamp concurrency control protocol is established, the data version management mechanism generates incremental data by comparing original data with updated data, and incremental updating or full updating is selected according to a threshold value. According to the method, the problem that data evolution tracking is difficult due to fragmentation of multi-source heterogeneous data and unbalanced access efficiency in navigation channel engineering is solved.

Description

Technical Field

[0001] The present application relates to the digital management of waterway engineering, and more specifically, to a dynamic association method for waterway models based on multi-source data indexing. Background Art

[0002] With the in-depth promotion of the informatization, digitization, and intelligent construction of waterway engineering, the management of waterway infrastructure has gradually evolved from the traditional static two-dimensional expression to a multi-dimensional, full-time and space data-driven mode. Modern waterway engineering involves multiple professional fields such as hydraulic structures, waterway regulation projects, dredging projects, communication and navigation facilities, etc., generating multi-source heterogeneous data sets including BIM three-dimensional models (IFC / RVT format), GIS geospatial information (Geo JSON / Shape file format), CAD engineering drawings (DWG / DXF format), Internet of Things sensor monitoring data (structured / temporal data), etc. Under the current technical framework, these data are generally stored in different business systems, using their own independent data storage structures and access interfaces, forming a decentralized and heterogeneous data island architecture.

[0003] Analyzed from the system architecture and data structure levels, the existing waterway engineering data management technologies mainly have the following technical bottlenecks:

[0004] In the existing system, data from different sources use independent coding rules. For example, BIM models use GUIDs, GIS systems use spatial indexes, and monitoring systems use device IDs, etc., resulting in different identifiers for the same physical entity in different systems. This identifier heterogeneity makes the data mapping relationship between systems need to be realized through complex intermediate tables, with a query complexity of O(n×m), and it is difficult to ensure the consistency and integrity of the mapping.

[0005] The existing technology adopts a single storage strategy and cannot be optimized according to the access characteristics of different types of data. For example, BIM model data is stored as a whole in file form, resulting in the need to load the entire model for fine-grained access; sensor data is stored in a relational database, and the performance drops significantly in high-frequency temporal query scenarios. This structural mismatch leads to unbalanced system I / O load, and the access latency of hot data is as high as several seconds.

[0006] The multi-source data formats vary significantly, lacking a unified preprocessing pipeline and data adaptation layer. The existing technology usually adopts a point-to-point conversion method. When the number of data source types is n, it is necessary to implement n²-level converters, and it cannot adaptively process undefined data formats, severely restricting the scalability of the system.

[0007] In the existing version control, the file-level snapshot method is mostly adopted, lacking efficient support for incremental updates. When the project data volume reaches the TB level, full backups lead to an exponential increase in storage overhead, and the retrieval efficiency of historical versions is low. At the same time, there is a lack of a data consistency guarantee mechanism based on timestamps, making it impossible to accurately trace the engineering status at a specific time point. Summary of the Invention

[0008] This application aims to solve the problem of difficult data evolution tracking caused by fragmentation of multi-source heterogeneous data and unbalanced access efficiency in waterway engineering. Specifically:

[0009] (1) Waterway engineering involves multiple professional fields such as hydraulic structures, waterway regulation projects, and dredging projects, generating multi-source heterogeneous data such as BIM models, GIS geographic information, CAD drawings, and sensor monitoring data. Each uses independent coding rules and data structures, forming scattered data islands.

[0010] (2) Traditional storage strategies cannot be optimized according to the access characteristics of different types of data, resulting in unbalanced system load and access latency.

[0011] (3) In the existing version control, the file-level snapshot method is mostly adopted, with insufficient support for incremental updates and difficulty in efficiently tracing the engineering status at a specific time point.

[0012] One aspect of this application provides a dynamic association method for waterway models based on multi-source data indexing, including: constructing component codes for waterway engineering, where the component codes are used to identify the same physical entity; establishing a three-layer index table structure according to the component codes; constructing a dynamic partition management mechanism for a three-level cache architecture, where the three-level cache architecture includes a first-level cache stored in memory, a second-level cache stored on a local disk, and a third-level cache stored in a distributed manner; performing data migration between caches at all levels according to the data access information in the three-layer index table structure; establishing a data version management mechanism based on a timestamp concurrency control protocol according to the dynamic partition management mechanism. The data version management mechanism generates incremental data by comparing the original data and the updated data, and selects incremental updates or full updates according to a threshold.

[0013] Among them, in waterway engineering, a physical entity refers to a specific component, structure, or device.

[0014] Further, construct the component coding for the waterway project, including: establishing a multi-level component coding rule for the waterway project, where the multi-level component coding rule uses a hierarchical structure to represent project, section, structure, and component information; according to the multi-level component coding rule, establishing a standardization specification for component attribute fields, and the standardization specification defines the geometric attributes, material attributes, construction attributes, and operation and maintenance attributes of the components; according to the multi-level component coding rule and the standardization specification of component attribute fields, establishing a relationship mapping table between component coding and external data sources, and the relationship mapping table is used to establish a unified index of BIM model element ID, CAD drawing number, sensor device ID, and operation and maintenance record number with the component coding; generating component coding according to the multi-level component coding rule, standardization specification, and relationship mapping table.

[0015] Among them, in this application, the hierarchical structure represents a top-down multi-level classification organizational structure, which specifically refers to the coding structure that gradually refines the waterway project information layer by layer according to the project level, section level, structure level, and component level in this solution. This structure can clearly express the subordinate relationship between components at all levels, facilitating systematic management and rapid positioning of specific components. The standardization specification represents the unified definition and classification standard for the attributes of waterway project components, including geometric attributes (dimensions, positions), material attributes (material, strength), construction attributes (construction methods, quality standards), and operation and maintenance attributes (maintenance cycle, usage status). This specification ensures the consistency of data descriptions in different professional fields. External data sources represent various heterogeneous data generated in the waterway project and their source systems, mainly including BIM modeling systems, CAD design systems, GIS geographic information systems, sensor monitoring systems, and operation and maintenance management systems, etc.

[0016] Further, the component coding adopts the following component coding format: project code - section number_construction type_professional code_component ID_description field; where the project code is 3 letters, used to identify the affiliated project; the section number is 2 digits, corresponding to the actual construction section; the construction type is 3 letters, indicating the type of structure to which the component belongs; the professional code is 3 letters, indicating the professional category to which the component belongs; the component ID is 6 digits, indicating the specific component serial number; the description field is an optional item, indicating component features or version information.

[0017] In particular, on the one hand, as the core component of water transportation infrastructure construction, the waterway project has significant characteristics such as long linear projects, diverse structures, complex environmental conditions, many professional intersections, and long life cycles. In addition, the waterway project extends longitudinally along the river for dozens or even hundreds of kilometers, spanning multiple administrative regions, and often adopts a sectional construction mode. The hierarchical coding structure proposed in this application (project code - section number_construction type_professional code_component ID_description field) is to adapt to the spatial distribution characteristics of the waterway project:

[0018] The project code (3 - letter) solves the identification and differentiation between different waterway projects, meeting the requirements of networked management of waterways;

[0019] The section number (2 - digit) corresponds to the common subcontracting mode of waterway projects, facilitating the division of responsible entities;

[0020] The construction type code (3 - letter) can distinguish various structures in waterway projects, such as revetments, guide dikes, spur dikes, submerged dams, navigation aids, etc.;

[0021] The professional code (3 - letter) reflects the characteristics of multi - professional intersections in waterway projects, such as hydrology, geology, structure, and electromechanics.

[0022] On the other hand, the data sources of waterway projects are highly complex. This application solves the consistency problem between 3D modeling of waterways and actual components through the mapping of BIM model element IDs and entity components; establishes the association between CAD drawing numbers and components, meeting the conversion requirements from 2D design to 3D implementation of waterway projects; associates sensor device IDs with components, satisfying the real - time data association requirements for hydrology and structural health monitoring of waterways. Integrates the operation and maintenance record numbers, solving the problem of historical data traceability in the long - term operation and maintenance management of waterway structures.

[0023] Furthermore, a three - layer index table structure is established, including: setting up the information resource layer, classifying and storing multi - source data according to the project code and section number in the component code; setting up the fusion index layer, establishing a data index table according to the construction type, professional code, and component ID in the component code; setting up the service support layer, providing data analysis and query according to the description fields in the component code; constructing a data type recognition engine according to the multi - level component coding rules. The engine matches the component identification in the data file with the component code, classifying and archiving the data according to the construction type and professional code fields; according to the recognition results of the data type recognition engine, setting up a data access mechanism in the adapter mode to standardize the conversion of data from different sources according to the field structure of the component code, so that the data from different sources corresponds one - to - one with each part of the component code; establishing a data index table based on the component ID in the fusion index layer. The data index table uses the component code as the primary key, associates the data type, data path, timestamp, and version number, and performs multi - level indexing and query through the project code, section number, construction type, and professional code in the component code.

[0024] Among them, the data access mechanism of the adapter pattern represents a heterogeneous data processing architecture based on design patterns, which is used in this application to solve the problem of inconsistent multi-source data formats in waterway engineering. This mechanism configures dedicated adapter components for each data source (such as BIM, CAD, sensor data, etc.). These adapters are responsible for converting the specific structure of the original data into the unified data structure of the system, achieving the decoupling of the data source and the processing logic. The adapter receives the original data, extracts the key identification information, and then maps it to the corresponding fields of the component code, enabling the system to process various heterogeneous data in a unified manner without knowing the internal structure of each data source.

[0025] Standardized conversion: A standardized processing process that converts the original data from different sources and in different formats in waterway engineering into a format that conforms to unified coding rules. This process includes steps such as data format unification, field mapping, structure reorganization, and metadata extraction, ensuring that various types of data can accurately correspond to fields such as the project code, section number, and structure type of the component code. Standardized conversion not only processes structured data (such as database records), but also can parse semi-structured data (such as XML files) and unstructured data (such as documents, images), extract the component information therein and standardize it, laying a foundation for subsequent data indexing and correlation analysis.

[0026] Specifically, on the one hand, the data organization of waterway engineering has the characteristics of "linear extension, point distribution, and multi-dimensional association". In this application, the information resource layer is classified and stored based on the project code and section number, adapting to the cross-regional distribution characteristics of waterway engineering and realizing the geographical spatial organization of data; solving the storage and management problems caused by the huge amount of data in long-distance waterway engineering. The fusion index layer establishes indexes based on the structure type, professional code, and component ID, solving the data management differences of different structures (such as revetments, anchorages, navigation facilities) in waterway engineering, and supporting the cross-indexing and correlation analysis of multi-professional data such as hydrology, geology, and structure. The service support layer uses descriptive fields to provide data analysis and query, adapting to the multi-objective management requirements of waterway engineering, such as flood control standards, navigation levels, ecological requirements, etc.; supporting the special analysis unique to waterway engineering, such as riverbed evolution, navigation condition assessment, and stability analysis of hydraulic structures.

[0027] On the other hand, the data types in waterway engineering are highly heterogeneous, including both traditional CAD drawings, modern BIM models, and real-time hydrological sensing data. In this application, the data type recognition engine breaks through the recognition barriers of different data formats, such as AutoCAD, Revit, Excel, database records, etc., by matching the component identification in the data file with the component code. Solving the problem of data format incompatibility generated by using different software in each stage of waterway survey, design, construction, and monitoring.

[0028] Furthermore, a dynamic partition management mechanism for a three - level cache architecture is constructed, including: according to the data index table, counting the access frequency, time distribution, and user roles of various types of data, analyzing the data usage in different business scenarios, and calculating the data heat score; setting the three - level cache architecture according to the data heat score; setting the business priority according to the business type and task urgency; establishing a data migration mechanism according to the data heat score and business priority;

[0029] Among them, the data heat score is a quantitative index used to measure the usage activity and importance of various types of data in the waterway engineering system. This score is calculated by comprehensively analyzing multi - dimensional factors such as the access frequency (the number of times called within a unit time), time distribution (the periodicity and time - period characteristics of access), and user roles (the access ratio of users with different permissions and functions). The data heat score adopts a weighted algorithm, assigns a higher weight to recent accesses, and is adjusted according to the importance of the business scenario. The finally generated numerical index determines the optimal storage location of the data in the three - level cache architecture, realizing the dynamic optimization allocation of storage resources.

[0030] According to the data heat score, set the three - level cache architecture, including: according to the data heat score, determine the access frequency level of the data, mark the data with an access frequency greater than the threshold P1 as high - frequency data, mark the data with an access frequency less than or equal to the threshold P1 and greater than the threshold P2 as medium - frequency data, and mark the data with an access frequency less than or equal to the threshold P2 as low - frequency data; set the first - level cache to store high - frequency data, the first - level cache uses the memory storage method, and the response time is less than T1; set the second - level cache to store medium - frequency data, the second - level cache uses the local disk method, and the response time is less than T2; set the third - level cache to store low - frequency data, the third - level cache uses the distributed storage method, and the response time is less than T3; where, T1 is less than T2, and T2 is less than T3. In this application, the value range of T1 is 50ms to 200ms, representing real - time response at the millisecond level; the value range of T2 is 250ms to 800ms, for interactive query scenarios; the value range of T3 is 1000ms and above, for batch analysis or archival storage.

[0031] Specifically, accessing waterway engineering data has significant particularities: strong project stages, large differences in data timeliness, scattered access regions, and high requirements for emergency response. Traditional single storage architectures are difficult to meet these complex needs. The three-level cache architecture proposed in this application is based on data heat scoring. Among them, the data heat scoring mechanism calculates the data value by statistically analyzing access frequency, time distribution, and user roles. For example, waterway engineering has obvious seasonality and stages. For instance, the access to flood season monitoring data is frequent, and the access to construction data during the dry season increases. The data heat scoring can dynamically reflect this periodic change; the timeliness of waterway hydrological data is high, the terrain survey data is medium, and the timeliness of geological exploration data is low. The heat scoring mechanism can accurately reflect this difference; in addition, different roles such as waterway design, construction, supervision, and operation and maintenance focus on different data. The heat scoring realizes personalized data prioritization through user role weights.

[0032] Specifically, the first-level cache (memory storage) meets the millisecond-level response requirements for real-time data such as waterway hydrological monitoring and lock operation, supports fast data retrieval and analysis required for waterway emergency dispatching decisions, and optimizes the access efficiency of high-frequency query data (such as the water level and navigation status of the current river section) in waterway engineering.

[0033] The second-level cache (local disk) adapts to weak network or offline working scenarios in the waterway construction site environment, supports data that is frequently used but does not require real-time response in waterway engineering (such as construction progress records and quality inspection data), and balances the access efficiency and storage cost of waterway engineering data.

[0034] The third-level cache (distributed storage) solves the problem of long-term preservation of waterway historical data, provides a unified storage capacity for waterway engineering data across the board, covers the cross-regional waterway data sharing needs; supports waterway big data analysis and mining, such as long-term river regime change trends and the performance evolution of structures.

[0035] Furthermore, a data version management mechanism based on a timestamp concurrency control protocol is established, including: setting the timestamp identifier and version number identifier of data change records; according to the timestamp identifier and version number identifier, identifying the change content by comparing the original data with the updated data, and constructing an incremental data record containing the changed fields and corresponding metadata; according to the proportion of the change content in the incremental data record, selecting incremental update or full update, and detecting and handling data update conflicts through the timestamp mechanism during the data update process; for the data after the update is completed, recording the data change record including the update time, update content, operator, and scope of influence.

[0036] Further, according to the timestamp identifier and version number identifier, identify the changed content by comparing the original data with the updated data, and construct an incremental data record containing the changed fields and corresponding metadata, including: parsing the original data structure and the updated data structure, and establishing a field mapping relationship; comparing the values of the corresponding fields in the original data and the updated data, and identifying the changed fields at all levels through traversal; for each identified changed field, obtain the path, data type, and values before and after the change of the changed field in the data structure; construct a metadata record for each changed field, where the metadata record includes the field path, field type, change information, change timestamp, and operation identifier; combine the changed fields and the corresponding metadata records as the incremental data record; add global metadata to the incremental data record, where the global metadata includes the total number of changes, change ratio, and data integrity check code.

[0037] Further, according to the ratio of the changed content in the incremental data record, select incremental update or full update, and detect and handle data update conflicts through the timestamp mechanism during the data update process, including: identifying the data to be updated according to the component code, and obtaining the storage location of the original data in the three-level cache architecture through the data index table; calculating the ratio of the number of changed fields in the incremental data record to the total number of fields in the original data as the change ratio; comparing the change ratio with a preset threshold; when the change ratio is less than the preset threshold, select the incremental update method and add the incremental data record to the original data; when the change ratio is greater than or equal to the preset threshold, adopt the full update method and replace the original data with the data to be updated; before performing incremental update or full update, retrieve the current timestamp of the target data from the three-level index table structure, and compare it with the original data timestamp recorded at the start of the update operation to determine whether there is a concurrent update; when a concurrent update is detected, process the data update request according to the business priority; write the data update result to the corresponding cache level, update the data distribution in the three-level cache architecture according to the data heat score, and update the timestamp and version number in the data index table; record the conflict detection and handling information of the update process in the data change log record as the historical basis for data version management.

[0038] In particular, the volume of waterway engineering data is large and distributed across different storage levels. This application can quickly locate the storage location of the target data through the component code. In addition, by dynamically selecting incremental / full update, it minimizes the transmission overhead while ensuring data integrity, which is particularly suitable for the on-site construction environment of waterways.

[0039] Another aspect of the present application also provides a dynamic association system for a waterway model based on multi-source data indexing, including: an encoding module, which constructs component codes for waterway projects and establishes an index table of the BIM model element IDs, CAD drawing numbers, sensor device IDs, and operation and maintenance record numbers with the component codes; an indexing module, which, according to the component codes, establishes a three-layer index table structure for the standardized processing of different types of data and the multi-table associated retrieval; a partitioning module, which, according to the data access information provided by the indexing module, constructs a dynamic partitioning management mechanism for multi-level caching and priority scheduling of data; and a version module, which performs data update and historical data backtracking management according to the dynamic partitioning management mechanism.

[0040] Compared with the prior art, the advantages of the present application are as follows:

[0041] Due to the intersection of multiple professional fields such as hydraulic construction, waterway regulation, and dredging projects in waterway projects, it has caused technical dilemmas of diverse data sources, heterogeneous structures, and scattered storage. In the prior art, there is a lack of a unified data indexing system and a dynamic response mechanism, resulting in a serious information island effect, low data retrieval efficiency, and the inability to effectively track the evolution process of data versions.

[0042] (1) Solve the problem of data fragmentation:

[0043] The component code system constructed in the present application adopts a structured expression of "project code - section number_construction type_professional code_component ID_description field", establishes a hierarchical data identification system, and associates heterogeneous data scattered in the BIM model, CAD drawings, sensor network, and operation and maintenance system in waterway projects through unified coding. Fundamentally, it solves the technical dilemma of scattered multi-professional data in waterway projects.

[0044] (2) Improve access efficiency:

[0045] Through the combined application of the adapter pattern design and the data type recognition engine, a unified data interface standard and a preprocessing pipeline are established, effectively overcoming the technical obstacles in the format conversion and semantic understanding levels of multi-source heterogeneous data (static BIM model, semi-structured CAD drawings, dynamic sensor data streams) in waterway projects, and realizing the standardization and automation of data management.

[0046] (3) Realize data evolution tracking:

[0047] The three - level cache architecture innovatively introduces a data heat score mechanism. Through the weighted calculation of access frequency, time distribution, and business weight, it realizes the dynamic migration of data between memory, local disk, and distributed storage. The three - layer index table structure and the three - level cache architecture form a two - way linkage mechanism. The index layer provides data association mapping, while the cache layer dynamically adjusts data distribution according to the access pattern. The version management system adopts a timestamp - based concurrent control protocol for incremental storage, enabling the system to maintain a complete data version evolution history with minimal storage overhead, solving the storage expansion problem in traditional version management systems while ensuring the efficiency and accuracy of historical data queries. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flowchart of the dynamic association method of the waterway model based on the multi - source data index mechanism of this application;

[0049] Figure 2 It is a structure diagram of the component coding system of this application;

[0050] Figure 3 It is a structure diagram of the multi - source data index table of this application;

[0051] Figure 4 It is a schematic diagram of the dynamic partition management mechanism of this application;

[0052] Figure 5 It is a flowchart of the data version management system of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0053] The following describes this application in detail in conjunction with the accompanying drawings of the specification and specific embodiments.

[0054] Embodiment 1

[0055] As Figure 1 shown, the first aspect of the present invention provides a dynamic association method of a waterway model based on a multi - source data index mechanism, constructs a component coding for a waterway project, and the component coding is used to identify the same physical entity; according to the component coding, a three - layer index table structure is established; a dynamic partition management mechanism of a three - level cache architecture is constructed, and the three - level cache architecture includes a first - level cache stored in memory, a second - level cache stored in a local disk, and a third - level cache stored in a distributed storage; data migration is performed between levels of caches according to the data access information in the three - layer index table structure; according to the dynamic partition management mechanism, a data version management mechanism based on a timestamp - based concurrent control protocol is established, and the data version management mechanism generates incremental data by comparing the original data and the updated data, and selects incremental update or full - volume update according to a threshold.

[0056] Construct a component - level coding system for a waterway project. As Figure 2 shown, it specifically includes the following steps:

[0057] Design a multi - level component coding rule for waterway engineering. The component coding format is as follows: Component Coding = Project Code - Section Number_Building Type_Professional Code_Component ID_Description Field. Among them:

[0058] Project Code (PROJECT): 3 letters, used to identify the affiliated project. For example, LYG represents the Lianyungang project;

[0059] Section Number (SECTION): 2 digits, corresponding to the actual construction section. For example, 01, 02;

[0060] Building Type (TYPE): 3 letters. For example, NCH (waterway), BRG (bridge), EMB (revetment);

[0061] Professional Code (DISCIPLINE): 3 letters. For example, STR (structure), MEC (mechanical and electrical), HYD (hydrology);

[0062] Component ID: 6 digits, representing the specific component serial number;

[0063] Description Field (CONTENT): Optional, representing component characteristics or version information;

[0064] Establish a standardization specification for component attribute fields, and define the following core attribute categories: Geometric attributes: length, width, height, shape, spatial coordinates; Material attributes: material type, specification, strength grade; Construction attributes: construction date, construction unit, acceptance status; Operation and maintenance attributes: maintenance cycle, responsible unit, health status;

[0065] Build a mapping relationship table between component coding and external data sources to achieve unified indexing: Map the BIM model element ID to the component coding; Associate the CAD drawing number with the component coding; Bind the sensor device ID to the component coding; Correlate the operation and maintenance record number with the component coding;

[0066] Develop a tool for automatic generation and verification of component coding: Rule - based coding automatic generation algorithm; Coding uniqueness check mechanism; Coding standardization verification function; Batch coding import and export interface.

[0067] Develop a multi - source data index table structure based on component coding, as Figure 3 shown. Specifically, it includes the following steps:

[0068] Design a three - layer information organization system: Information resource layer: It carries multi - source data that has been pre - processed, including static data (models, drawings), semi - structured data (documents, reports), and dynamic data (sensing, monitoring). Fusion index layer: Build a data index table through component coding to achieve structured connection, field mapping, and update tracking between components and external information. Service support layer: Provide data support for functional modules such as visual analysis, information query, and risk identification.

[0069] Develop a data type recognition engine: File format recognition: Support.dwg (CAD drawings),.ifc (BIM models),.obj / .fbx (lightweight models),.las (point clouds),.csv / .json (device data), etc.; Data structure analysis: Automatically analyze the internal structure of the file and extract key fields; Automatic type classification: Based on rule - based data classification algorithms, automatically classify into corresponding categories.

[0070] Build a data access mechanism with an adapter pattern: Field mapping table: Define the correspondence between different source data fields and standard fields; Format template library: Preset common data format conversion templates; Pre - processing pipeline: An automated process for data cleaning, format conversion, and field unification.

[0071] Establish a data index table based on component ID: Index table structure design: Component ID, data type, data path, timestamp, version number; Multi - table association mechanism: Support one - to - many and many - to - many data associations; Fast retrieval algorithm: An efficient index query based on B - tree.

[0072] Develop a data association strength evaluation algorithm: Access frequency statistics: Record the access times and time distribution of each data item; Business association analysis: Evaluate the business logic association degree between different data; Storage optimization strategy: Optimize the data storage layout according to the association strength.

[0073] Build a dynamic partition management mechanism based on data access frequency, as Figure 4 shown, which specifically includes the following steps:

[0074] Statistical access frequency and usage scenarios of various data: Access frequency statistics: Record the access times, time distribution, and user roles of each data item; Usage scenario analysis: Classify and count the data usage in different business scenarios; Heat evaluation model: Build a data heat score algorithm based on access frequency and business weights.

[0075] Design a three - level cache architecture: Memory cache (L1): Store hot data with high access frequency to meet millisecond - level real - time response; Local cache (L2): Store data with medium access frequency, with a response speed better than remote storage, suitable for interactive query scenarios; Remote storage (L3): Store cold data and complete historical data, with a relatively long response time, suitable for batch analysis or archiving requirements. The response latency gradients of the above three levels increase sequentially, and the system automatically adjusts the data distribution among the layers through heat scoring and dynamic migration mechanisms.

[0076] Develop an intelligent scheduling algorithm based on task priorities:

[0077] Input: Data request queue Q, business priority table P;

[0078] Output: Sorted request queue SQ

[0079] for each request r in Q do

[0080] priority = P[r.business Type] * r.urgency

[0081] r.priority = priority

[0082] end for

[0083] SQ = sort(Q, descending by priority)

[0084] return SQ

[0085] Establish an inter - zone data migration mechanism: Cold - hot data identification: Dynamically identify cold - hot data based on access frequency thresholds; Automatic migration strategy: Hot data is automatically migrated to the upper - layer cache, and cold data sinks to the lower - layer storage; Load - balancing algorithm: Dynamically adjust the data distribution according to the load conditions of each partition;

[0086] Build a partition capacity monitoring and warning system: Capacity utilization monitoring: Real - time monitor the storage usage of each partition; Dynamic expansion mechanism: Automatically trigger expansion when the capacity reaches the threshold; Policy optimization feedback: Continuously optimize the partition policy based on operation data.

[0087] Establish a data version management system based on timestamps, as Figure 5 shown, specifically including the following steps:

[0088] S1. Obtain data change requests, receive data update request D2, and verify data requests and permissions.

[0089] S2. Locate the original data D1 through an efficient indexing mechanism, and add a timestamp and version identifier to each change data record: Timestamp format: Adopt the ISO8601 standard, accurate to the millisecond level (yyyy-MM-ddTHH:mm:ss.SSSZ); Version number rule: Semantic version control, in the format of major version number.minor version number.revision number (e.g., 1.2.5); Version chain construction: Form a complete version evolution chain to support version backtracking and branch management; Set the timestamp and version number identifier to establish data lineage.

[0090] S3. Compare the original data D1 and the incremental data D2, construct the incremental data record ΔD, and calculate the change ratio R:

[0091] Identify changed fields in D2 compared to D1

[0092] For each changed field f do

[0093] ΔD[f]=D2[f]

[0094] ΔD.metadata.add(f.path, f.type, f.changeInfo)

[0095] End for

[0096] Return ΔD

[0097] When the change ratio R is greater than the preset threshold, execute step S4.1 for incremental update; otherwise, execute step S4.2 for full update; In this embodiment, the preset threshold ranges from 30% to 40%. In addition, this embodiment also ensures data consistency: Adopt transaction control to ensure update atomicity; Conflict detection and resolution: An automatic conflict resolution mechanism based on timestamps.

[0098] After completing the data update, execute step S5.1 for incremental merge and write-back to generate a new timestamp and version package to maintain data integrity, or step S5.2 for full merge and write-back to replace the complete data and synchronously update all relevant metadata.

[0099] After the data is written back, execute step S6 to write the data in layers according to the heat score mechanism. This embodiment uses three-level caches, namely L1, L2, and L3. Among them, L1 stores the highest-frequency accessed data (e.g., the heat score is greater than 0.8), L2 stores the medium-frequency accessed data (e.g., the heat score is greater than 0.3 and less than 0.8), and L3 stores the low-frequency accessed data (e.g., the heat score is less than 0.3).

[0100] S7. Establish a data change log recording system: Log structure design: Timestamp, operation type, changer, change content, scope of influence; S8. Log storage policy: Adopt circular log files, support log archiving and compression; Audit tracking function: Support multi-dimensional log retrieval and analysis; Change notification mechanism: Automatically push critical changes to relevant responsible persons.

[0101] Implement the function of quickly retrieving historical data: Snapshot storage: Regularly generate full data snapshots, support quick recovery; Version retrieval: Quick query based on time range or version number; Data restoration: Support restoring the data status at a specified time point; Difference comparison: Provide difference comparison and visual display between versions.

[0102] Embodiment 2

[0103] In this embodiment, for the secondary regulation project of the Lianyungang area's waterway to the port, a component coding system covering a 32.74-kilometer waterway range was constructed. This project includes multiple sub-projects such as waterway dredging, revetment reinforcement, and new service area construction. The coding system is established according to the following rules:

[0104] The project code uses "LYG" (the initials of Lianyungang);

[0105] The section number is divided according to the actual section, such as "01", "02";

[0106] The construction types include NCH (waterway), BRG (bridge), EMB (revetment), BUY (navigation mark), etc.;

[0107] The professional codes cover STR (structure), HYD (hydrology), NAV (navigation mark), etc.

[0108] For example, the coding of the first revetment component in the upstream approach channel of Yunshan Lock is: LYG-01_EMB_STR_000001_V1.0.

[0109] For the multi-source heterogeneous data involved in the waterway project, such as BIM models, CAD drawings, sensor data, and operation and maintenance records, a unified index table structure was developed. For example, the revetment BIM model (IFC format), relevant design drawings (DWG format), and structural health monitoring sensor data (JSON format) are all associated through the component code "LYG-01_EMB_STR_000001". The system supports automatically identifying the file type and completing data attachment.

[0110] According to the actual operation statistics, a three-level cache architecture was designed:

[0111] L1 cache: Store the frequently accessed waterway section data and real-time ship AIS data, with a response time < 100ms;

[0112] L2 Cache: Stores construction drawings and operation and maintenance records that are frequently accessed, with a response time < 500 ms;

[0113] L3 Storage: Stores complete historical monitoring data and design archives, with a response time < 2000 ms.

[0114] Version control based on timestamps is implemented. For example, the revetment structure reinforcement plan has gone through three modifications: V1.0 (initial version) → V1.1 (optimize steel bar layout) → V1.2 (add anti-scour measures). The system records the time, operator, and content of each change, and supports quick backtracking of any version.

[0115] Through the implementation of the above four steps, the project has successfully constructed a multi-source data index management system covering the entire Lianyungang area's waterway for dredging, achieving efficient integration and dynamic management of design, construction, and operation and maintenance data.

[0116] Example 3

[0117] The second aspect of the present invention also provides a dynamic association system for waterway models based on a multi-source data index mechanism, including a coding module that constructs component codes for waterway projects and establishes an index table of BIM model element IDs, CAD drawing numbers, sensor device IDs, and operation and maintenance record numbers with the component codes; an index module that, based on the component codes, establishes a three-layer index table structure for standardizing different types of data and performing multi-table association retrieval; a partitioning module that, based on the data access information provided by the index module, constructs a dynamic partitioning management mechanism for multi-level caching and priority scheduling of data; and a version module that performs data update and historical data backtracking management based on the dynamic partitioning management mechanism. The system includes: a memory and a processor. The memory includes a program for the dynamic association method of the waterway model based on the multi-source data index mechanism. When the program for the dynamic association method of the waterway model based on the multi-source data index mechanism is executed by the processor, the following steps are implemented:

[0118] Construct a component-level coding system for waterway projects, formulate coding rules including a multi-level structure of sub-projects - sub-items - types, and establish a standardization specification for component attribute fields;

[0119] Develop a multi-source data index table structure based on component codes, design a data type recognition and adapter pattern, and implement classified attachment of static data, semi-structured data, and dynamic data;

[0120] Construct a dynamic partitioning management mechanism based on data access frequency, design a multi-level caching system, and develop a data scheduling algorithm based on task priorities;

[0121] Establish a data version management system based on timestamps, design a collaborative mechanism for incremental update and full update, and implement data change log recording and quick backtracking of historical data.

[0122] The present invention and its implementation manners are schematically described above. The description is not restrictive. Without departing from the spirit or basic features of the present application, the present application can be implemented in other specific forms. What is shown in the drawings is only one of the implementation manners of the present invention, and the actual structure is not limited thereto. Therefore, if those of ordinary skill in the art are inspired by it and, without departing from the purpose of this invention, design similar structural manners and embodiments to this technical solution without creative efforts, they shall fall within the protection scope of the present application. In addition, the term "including" does not exclude other elements or steps, and the term "a" before an element does not exclude including "a plurality of" such elements. The terms such as "first" and "second" are used to represent names and do not indicate any specific order.

Claims

1. A dynamic association method for a waterway model based on multi-source data indexing, characterized in that Including: Construct component codes for waterway engineering, where the component codes are used to identify the same physical entity; Establish a three - layer index table structure according to the component codes; Construct a dynamic partition management mechanism for a three - level cache architecture, where the three - level cache architecture includes a first - level cache stored in memory, a second - level cache stored on a local disk, and a third - level cache stored in a distributed manner; Perform data migration among caches at all levels according to the data access information in the three - layer index table structure; According to the dynamic partition management mechanism, establish a data version management mechanism based on a timestamp - based concurrency control protocol. The data version management mechanism generates incremental data by comparing the original data and the updated data, and selects incremental update or full - volume update according to a threshold.

2. The method for dynamically associating a waterway model based on multi - source data indexing according to claim 1, characterized in that: Constructing component codes for waterway engineering includes: Establish a multi - level component coding rule for waterway engineering, where the multi - level component coding rule uses a hierarchical structure to represent project, section, structure, and component information; According to the multi - level component coding rule, establish a standardization specification for component attribute fields, where the standardization specification defines the geometric attributes, material attributes, construction attributes, and operation and maintenance attributes of components; According to the multi - level component coding rule and the standardization specification of component attribute fields, establish a relationship mapping table between component codes and external data sources. The relationship mapping table is used to establish a unified index between BIM model element IDs, CAD drawing numbers, sensor device IDs, and operation and maintenance record numbers and component codes; Generate component codes according to the multi - level component coding rule, standardization specification, and relationship mapping table.

3. The method for dynamically associating a waterway model based on multi - source data indexing according to claim 2, characterized in that: The component code adopts the following component coding format: Project code - section number_construction type_professional code_component ID_description field; Among them, the project code is 3 - letter, used to identify the affiliated project; The section number is 2 - digit, corresponding to the actual construction section; The construction type is 3 - letter, indicating the type of structure to which the component belongs; The professional code is 3 - letter, indicating the professional category to which the component belongs; The component ID is 6 - digit, indicating the specific component serial number; The description field is an optional item, indicating component features or version information.

4. The method for dynamically associating a waterway model based on multi - source data indexing according to claim 3, characterized in that: According to the component codes, establish a three - layer index table structure, including: Set up an information resource layer to classify and store multi - source data according to the project code and section number of the component code; Set up a fusion index layer to establish a data index table according to the construction type, professional code, and component ID in the component code; Set up a service support layer to provide data analysis and query according to the description field in the component code; According to the multi - level component coding rule, construct a data type recognition engine. The engine matches by identifying the component identifier in the data file with the component code, and classifies and archives the data according to the construction type and professional code fields. According to the recognition results of the data type recognition engine, set the data access mechanism of the adapter mode, and standardize and convert the data from different sources according to the field structure of the component code, so that the data from different sources corresponds one by one with each component of the component code; Establish a data index table based on the component ID in the fusion index layer. The data index table uses the component code as the primary key, associates the data type, data path, timestamp, and version number, and performs multi-level indexing and query through the project code, section number, structure type, and professional code in the component code.

5. The method for dynamically associating a waterway model based on multi-source data indexing according to claim 4, wherein: Perform data migration between levels of caches according to the data access information in the three-layer index table structure, including: According to the data index table, count the access frequency, time distribution, and user roles of various types of data, analyze the data usage in different business scenarios, and calculate the data heat score; Set a three-level cache architecture according to the data heat score; Set the business priority according to the business type and task urgency; Establish a data migration mechanism according to the data heat score and business priority.

6. The method for dynamically associating a waterway model based on multi-source data indexing according to claim 5, wherein: Set a three-level cache architecture according to the data heat score, including: According to the data heat score, determine the access frequency level of the data. Mark the data with an access frequency greater than the threshold P1 as high-frequency data, mark the data with an access frequency less than or equal to the threshold P1 and greater than the threshold P2 as medium-frequency data, and mark the data with an access frequency less than or equal to the threshold P2 as low-frequency data; Set the first-level cache to store high-frequency data. The first-level cache uses the memory storage method, and the response time is less than T1; Set the second-level cache to store medium-frequency data. The second-level cache uses the local disk method, and the response time is less than T2; Set the third-level cache to store low-frequency data. The third-level cache uses the distributed storage method, and the response time is less than T3; where T1 is less than T2, and T2 is less than T3.

7. The method for dynamically associating a waterway model based on multi-source data indexing according to any one of claims 2 to 6, wherein: Establish a data version management mechanism based on the timestamp concurrency control protocol according to the dynamic partition management mechanism, including: Set the timestamp identifier and version number identifier of the data change record; According to the timestamp identifier and version number identifier, identify the change content by comparing the original data with the updated data, and construct an incremental data record containing the changed fields and corresponding metadata; According to the proportion of the change content in the incremental data record, select incremental update or full update, and detect and handle data update conflicts through the timestamp mechanism during the data update process; For the data after the update is completed, record the data change record including the update time, update content, operator, and affected scope.

8. The method for dynamically associating a waterway model based on multi-source data indexing according to claim 7, wherein: Construct an incremental data record containing the changed fields and corresponding metadata, including: Parse the original data structure and update the data structure to establish field mapping relationships; Compare the values of corresponding fields in the original data and the updated data, and identify the changed fields at all levels through traversal; For each identified changed field, obtain the path, data type, and values before and after the change of the changed field in the data structure; Construct metadata records for each changed field, where the metadata records include field path, field type, change information, change timestamp, and operation identifier; Combine the changed fields and their corresponding metadata records as incremental data records; Add global metadata to the incremental data records, where the global metadata includes the total number of changes, change ratio, and data integrity verification code.

9. The method for dynamically associating a waterway model based on multi-source data indexing according to claim 7, characterized in that: Select incremental update or full update according to the ratio of the changed content in the incremental data record, including: Identify the data to be updated according to the component code, and obtain the storage location of the original data in the three-level cache architecture through the data index table; Calculate the ratio of the number of changed fields in the incremental data record to the total number of fields in the original data as the change ratio; Compare the change ratio with the preset threshold; When the change ratio is less than the preset threshold, select the incremental update method and add the incremental data record to the original data; When the change ratio is greater than or equal to the preset threshold, adopt the full update method and replace the original data with the data to be updated; Before performing incremental update or full update, retrieve the current timestamp of the target data from the three-level index table structure, and compare it with the original data timestamp recorded at the start of the update operation to determine whether there is a concurrent update; When a concurrent update is detected, process the data update request according to the business priority; Write the data update result to the corresponding cache level, update the data distribution in the three-level cache architecture according to the data heat score, and update the timestamp and version number in the data index table; Record the conflict detection and processing information of the update process in the data change log record as the historical basis for data version management.

10. A dynamic association system for a waterway model based on multi-source data indexing, characterized in that, Including: Coding module, constructing the component code of the waterway project, and establishing an index table of BIM model element ID, CAD drawing number, sensor device ID, and operation and maintenance record number with the component code; Index module, establishing a three-level index table structure according to the component code for standardized processing of different types of data and multi-table association retrieval; Partition module, constructing a dynamic partition management mechanism according to the data access information provided by the index module for multi-level caching and priority scheduling of data; Version module, performing data update and historical data backtracking management according to the dynamic partition management mechanism.

Images