Multi-source heterogeneous data collaborative synchronization and conflict processing method for integrated operation and maintenance

Abstract

The application relates to the technical field of computer network operation and maintenance, and discloses a multi-source heterogeneous data cooperative synchronization and conflict processing method for integrated operation and maintenance, which comprises the following steps: defining a unified operation and maintenance basic data metadata standard, respectively configuring field mapping and type conversion rules for different heterogeneous data sources; obtaining data from each source end based on a preset strategy and a dynamic concurrency mechanism, and converting the data into initial standardized data; extracting a globally unique identifier and performing traversal comparison with inventory data, and combining data content characteristic values to determine whether data conflict occurs; if conflict occurs, a progressive conflict resolution mechanism is sequentially executed to obtain target standardized data; and finally, the target standardized data is written into a persistent storage area, and the dynamic update of an upper panoramic monitoring view and a network link topology diagram is driven. The application effectively solves the problems of data heterogeneity, concurrency overload and multi-source data conflict, and improves the accuracy of integrated operation and maintenance data.

Description

Technical Field

[0001] This invention relates to the field of computer network operation and maintenance technology, and in particular to a method for collaborative synchronization and conflict resolution of multi-source heterogeneous data for integrated operation and maintenance. Background Technology

[0002] As enterprises deepen their digital transformation, IT operations and maintenance management is gradually evolving towards an integrated, centralized operation and monitoring model. In modern integrated operations and maintenance scenarios at the city and county levels, systems often need to simultaneously access heterogeneous data from multiple sources, such as business management systems, network device management systems, and power and environmental monitoring systems. However, existing data synchronization and integration solutions face the following prominent technical challenges in practical applications: First, data silos are severe and models are heterogeneous. Various systems use completely different communication protocols and data encapsulation formats, and lack a unified metadata standard, making it difficult for data to flow and be shared efficiently within the same resource pool.

[0003] Secondly, high-concurrency synchronization can easily lead to system overload. When collecting massive amounts of asset ledger or real-time alarm data, traditional solutions typically employ fixed concurrency strategies, which can easily put enormous concurrent access pressure on the open interfaces of the source system, and even cause the source service to become unavailable.

[0004] Finally, there is a lack of sophisticated data conflict handling mechanisms. When multiple source systems report data to the same physical entity, data conflicts such as inconsistent timestamps and mutually exclusive state logic are very likely to occur. Traditional processing methods often rely on simple and crude full coverage or manual inspection, lacking automated fine-grained conflict resolution capabilities, resulting in low-quality underlying operation and maintenance data and an inability to accurately support dynamic early warning of the upper-layer panoramic topology map. Summary of the Invention

[0005] This invention provides a method for collaborative synchronization and conflict resolution of multi-source heterogeneous data for integrated operation and maintenance. By constructing a unified metadata model and a progressive intelligent conflict resolution mechanism, it breaks down data silos between multi-source heterogeneous systems, significantly improving the accuracy of fused data and the management efficiency of integrated operation and maintenance while ensuring the stability of the source system interfaces.

[0006] This invention provides a method for collaborative synchronization and conflict resolution of multi-source heterogeneous data for integrated operation and maintenance, including: S1. Define a unified metadata standard for basic operation and maintenance data as the target data structure for the fusion of multi-source heterogeneous data. For the heterogeneous data formats of the first data source, the second data source, and the third data source, set field mapping and type conversion rules between the corresponding heterogeneous data formats and the target data structure respectively. Among them, the first data source is the business management system, the second data source is the network device management system, and the third data source is the power and environment monitoring system. S2. Using the first data source, the second data source, and the third data source as source systems, obtain multi-source heterogeneous data from the source systems based on a preset synchronization frequency and an incremental or full synchronization strategy, and use the field mapping and type conversion rules to convert the multi-source heterogeneous data into initial standardized data and store it in the cache area of ​​the unified data resource pool. S3. Extract the globally unique identifier of the initial standardized data, compare it with the existing stock standardized data in the unified data resource pool, determine whether there is a data object with the same identifier, and compare whether the data content feature values ​​of the initial standardized data and the corresponding stock standardized data are consistent, so as to determine whether the initial standardized data has a data conflict. S4. When data conflicts occur in the initial standardized data, a progressive conflict resolution mechanism is executed sequentially on the initial standardized data to obtain the target standardized data; wherein, the progressive conflict resolution mechanism includes priority adjudication triggered sequentially, field-level and data-level merging, and manual intervention flow mechanism; S5. Write the target standardized data into the persistent storage area of ​​the unified data resource pool to insert, replace or update the corresponding existing standardized data, and drive the dynamic update of the panoramic monitoring view and network link topology map of the upper-layer integrated operation management system based on the written target standardized data.

[0007] Furthermore, S1 specifically includes: S101. Construct and define a unified metadata standard for basic operation and maintenance data, and use it as the target data structure for the fusion of multi-source heterogeneous data; wherein, the standardized core fields included in the target data structure include at least a globally unique identifier, a data generation timestamp, basic device attribute information, a real-time operating status field, and a set of performance index parameters. S102. Determine the first data source, the second data source, and the third data source; parse the heterogeneous data format of the first data source and set it to use the RESTful API protocol to interact with the first data source; parse the heterogeneous data format of the second data source and set it to use the RESTful interface protocol or HTTP long connection to interact with the second data source; parse the heterogeneous data format of the third data source and set it to use the MQTT protocol to obtain the real-time data stream of the third data source through topic subscription. S103. For the heterogeneous data format, configure the field mapping and type conversion rules between the corresponding source format and the target data structure.

[0008] Furthermore, in S103, The field mapping rule is to establish a mapping dictionary between heterogeneous source data fields and standardized core fields in the target data structure, and to uniformly map the status fields or indicator fields that represent the same physical meaning but have different names in the source system to the standard fields in the target data structure. The type conversion rule is to parse and convert the different data types defined in the heterogeneous data formats into the unified data types specified by the metadata standard. This includes converting different time formats into standard timestamp formats and cleaning and converting the status description information of different systems into standard Boolean variables or system-preset standard enumeration codes.

[0009] Furthermore, S2 specifically includes: S201. Based on business requirements and data types, set preset synchronization frequencies and synchronization strategies for each of the aforementioned source systems; S202. Establish a network connection with the source system and complete authorization authentication, execute a concurrent data acquisition task to obtain the multi-source heterogeneous data, and dynamically adjust the number of concurrent threads according to the load status of the source system during the acquisition process. S203. The multi-source heterogeneous data is parsed to extract valid field information, and the valid field information is converted into initial standardized data using the field mapping and type conversion rules. S204. Perform data integrity and accuracy verification on the initial standardized data, and temporarily store the verified initial standardized data in the cache area of ​​the unified data resource pool.

[0010] Furthermore, in S201, For device status and alarm data with high real-time requirements, set to real-time synchronization frequency; for asset ledgers and configuration data with high non-real-time requirements, set to timed synchronization frequency; and for user manual requests or specific data backtracking scenarios, set to on-demand synchronization frequency. During normal system operation, an incremental synchronization strategy is set to be used, which synchronizes only newly added or changed data by comparing the timestamp or version number parameters of the data. When the system is initialized or when there is a large-scale loss or inconsistency of data, a full synchronization strategy is triggered to synchronize all data.

[0011] Furthermore, S3 specifically includes: S301. Parse the initial standardized data temporarily stored in the cache area and extract the globally unique identifier used for identity recognition; S302. The extracted globally unique identifier is compared with the existing standardized data in the persistent storage area to perform a data object consistency check. Specifically, if no existing standardized data with the same identifier is found, the initial standardized data is determined to be new data and is directly released to the database entry process. If existing standardized data with the same identifier is found, the existing standardized data is extracted and the process proceeds to S303. S303. When it is confirmed that there are data objects with the same identifier, calculate the data content feature value of the core content set of the initial standardized data and the corresponding existing standardized data respectively, and compare them; wherein, calculating the data content feature value specifically involves: using a hash algorithm to generate the hash value of the corresponding core content set of data; if the comparison finds that the data content feature values ​​of the two are completely consistent, it is determined that the data is duplicated, and only the last active timestamp of the corresponding existing standardized data is updated and the initial standardized data is discarded; S304. If the data content feature values ​​are inconsistent, the timestamps and field-level differences of the data are compared to determine whether the initial standardized data has a data conflict. If it is determined that a data conflict has occurred, a data conflict identifier is added to the initial standardized data, and the process proceeds to S4.

[0012] Furthermore, in S304, the determination of whether a data conflict has occurred by comprehensively comparing the timestamps and field-level differences of the data specifically includes: If the inconsistency in the data content feature values ​​is only reflected in the timestamp feature, and other monitoring indicators are normal changes in the sequential time, then it is determined to be a normal data update and does not constitute a data conflict. If the inconsistency in the characteristic values ​​of the data content is reflected in the core state or key attributes, and there is logical mutual exclusion or abnormal jump, then it is determined that a data conflict has occurred.

[0013] Furthermore, S4 specifically includes: S401. Perform priority decision based on the source system weight and absolute timestamp, specifically as follows: Priority weights are pre-assigned to each of the aforementioned source systems. When a data conflict occurs, data from the source system with the higher priority weight is adopted first to cover data from the source system with the lower priority weight. If the source systems that cause the conflict have the same priority weight, a timestamp priority strategy is triggered. By accurately comparing the absolute timestamps carried by the conflicting data, the updated data whose timestamp is closest to the current time is adopted. S402. If the conflict cannot be completely resolved based on the aforementioned priority decision, the intelligent merging engine is activated to perform field-level merging and data-level merging; wherein, For cases of partial field conflicts, the field-level merging method uses the preset core field of the main data source as a benchmark, extracts unique non-empty auxiliary fields from the associated data source and adds them to the benchmark, and merges conflicting individual fields according to preset business rules, including the rule that the online status of the device takes precedence over the offline status. The data-level merging addresses the situation where overall data items conflict. It parses the parameter set of conflicting data, deduplicates the configuration parameters of each data source, and merges the configuration parameters of each data source into a complete configuration parameter body without loss. S403. If, after processing by the intelligent merging engine, the core status fields still exhibit business logic conflicts, an alarm is triggered and a manual intervention process is executed; specifically: When conflicting data cannot be determined as a unique valid value even after automatic resolution strategies, the data flow process is suspended, and a data conflict alarm is automatically generated. The differences between fields of multiple conflicting data are displayed and compared intuitively on the front-end interface of the operation and maintenance management terminal. The terminal also receives manual selection instructions from the user based on the field differences to determine the final valid data. S404. Output the data formed after successfully resolving conflicts through any of the sub-steps S401-S403 and define it as the target standardized data.

[0014] Furthermore, S5 specifically includes: S501. Write the target standardized data into the persistent storage area of ​​the unified data resource pool to replace or update the corresponding existing standardized data; specifically: perform a determination operation in the persistent storage area according to the globally unique identifier of the target standardized data; if it is determined to be new data, perform an insertion operation; if it is determined to be existing changed data, perform an update operation so that the target standardized data overwrites the original existing standardized data. S502. Construct a real-time data distribution mechanism to push state change events to the upper application layer through a message middleware; wherein, the real-time data distribution mechanism is as follows: while completing the persistent writing of data, hot data is stored in a cache database, and a real-time data flow channel is constructed through a built-in message middleware or long connection mechanism; when the core state or key performance indicators of the target standardized data change, it is encapsulated as an event message and pushed to the application presentation layer in real time; S503. Based on the target standardized data, drive the dynamic update and rendering of the upper-layer panoramic monitoring view, specifically: extract the data center environment parameters, host equipment indicators and alarm status from the target standardized data, render and refresh the front-end monitoring screen in real time; if the target standardized data contains alarm attributes, trigger an alarm prompt, and highlight the data center or equipment node that has an alarm in the data center floor plan. S504. Based on the target standardized data, drive the dynamic configuration and status warning of the network link topology diagram, specifically: dynamically draw or update the network link topology diagram according to the link start point, end point, connected device and bandwidth specifications recorded in the target standardized data; when a network node or link is confirmed to have a fault, drive the corresponding network topology diagram node icon to switch to an alarm color and display a flashing effect, and at the same time highlight the relevant connection lines for warning.

[0015] The beneficial effects of this invention are as follows: This invention constructs a unified standard for basic operation and maintenance metadata and field mapping rules, compatible with multiple underlying protocols such as RESTful, HTTP, and MQTT. This breaks down data silos between business, network, and environmental systems, achieving standardized integration of heterogeneous data. During the data acquisition phase, a dynamic concurrency adjustment and fault tolerance mechanism is introduced, ensuring the real-time synchronization of massive amounts of data while effectively preventing source interfaces from crashing due to high concurrency. Simultaneously, a progressive conflict resolution mechanism is proposed. When the system accurately detects data conflicts using unique identifiers and content feature values, it sequentially resolves them through priority and timestamp adjudication, field-level and data-level intelligent merging engines, and manual intervention. This preserves valid information from multiple data sources, solves data contradictions caused by heterogeneous multi-source reporting, significantly improves the absolute accuracy of underlying data, and provides a solid data-driven foundation for the second-level dynamic updates of the upper-layer panoramic monitoring view and network topology map. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the multi-source heterogeneous data collaborative synchronization and conflict handling method for integrated operation and maintenance according to the present invention.

[0017] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0019] like Figure 1 As shown, this invention provides a method for collaborative synchronization and conflict resolution of multi-source heterogeneous data for integrated operation and maintenance, including: S1. Define a unified metadata standard for basic operation and maintenance data as the target data structure for the fusion of multi-source heterogeneous data. For the heterogeneous data formats of the first data source, the second data source, and the third data source, set field mapping and type conversion rules between the corresponding heterogeneous data formats and the target data structure respectively. Among them, the first data source is the business management system, the second data source is the network device management system, and the third data source is the power and environment monitoring system.

[0020] In one embodiment, step S1 specifically includes the following sub-steps: S101. Construct and define a unified standard for basic operation and maintenance data metadata, and use it as the target data structure for the fusion of multi-source heterogeneous data.

[0021] In integrated operation and maintenance scenarios, in order to solve the technical problems of data silos, inconsistent standards, and different formats among independent systems, a globally unified data model is built in advance at the data processing layer to form a unified data resource pool, thereby improving the efficiency of data sharing and circulation.

[0022] As a standardized data container, the target data structure contains core fields including at least globally unique identifiers (such as asset IDs or IP addresses), data generation timestamps, basic device attribute information (such as manufacturer and model), real-time operating status fields (such as alarm status and online status), and various performance indicator parameter sets (such as CPU utilization and temperature and humidity values). All heterogeneous data collected from the underlying layer is ultimately converted into the target data structure that conforms to this metadata standard.

[0023] S102. Analyze the underlying physical source end, determine the first data source, the second data source and the third data source, and analyze their heterogeneous data formats and communication protocols.

[0024] In specific embodiments of the present invention, the underlying source system includes, but is not limited to, the following three typical data sources: The primary data source is the business management system, specifically the SG-16000 2.0 system, which is deployed in the information intranet. It carries business data and outputs information such as service clusters, application systems, and task processes. This system mainly interacts with data through an open northbound interface using the RESTful API protocol.

[0025] The second data source is the network device management system, specifically the H3C intelligent network management platform, which is used for centralized management and monitoring of host servers and network devices (such as switches, routers, firewalls, etc.). The system uses a REST-style interface protocol for data interaction, or outputs real-time alarms and device performance data through HTTP long connections.

[0026] The third data source is the power and environment monitoring system, which monitors the power distribution, precision air conditioning, temperature and humidity, access control and other power and environmental equipment in the municipal headquarters computer room and district and county computer rooms in real time. The data of this system is mainly reported to the provincial unified smart IoT platform via MQTT, and the system subscribes to topics through the MQTT protocol to obtain real-time data streams.

[0027] The three data sources mentioned above use completely different interface protocols (RESTful, HTTP, MQTT) and data encapsulation formats (such as JSON, XML, etc.), forming a typical multi-source heterogeneous environment.

[0028] S103. For the above heterogeneous data formats, configure the field mapping and type conversion rules between the corresponding source format and the target data structure.

[0029] The system's data acquisition plugin invokes pre-defined cleaning and transformation rules based on the source identifier of the received data, specifically: Field mapping rules: Establish a mapping dictionary between heterogeneous source data fields and target data structure fields. For example, map the "device_status" field representing status in the first data source, the "state" field in the second data source, and the "operational status" field in the third data source to the standard "operational_state" field in the target data structure.

[0030] Type conversion rules: Force or parse the data types with different definitions in heterogeneous data to be converted into a unified data type specified by the metadata standard. For example, convert the character time format returned by the first data source to the standard Unix timestamp long integer format; or for status indicators, uniformly convert and clean the descriptions returned by different systems (such as online / offline, Up / Down, 1 / 0) into Boolean variables (True / False) or the system's preset standard enumeration codes.

[0031] S2. Using the first data source, the second data source, and the third data source as source systems, obtain multi-source heterogeneous data from the source systems based on a preset synchronization frequency and an incremental or full synchronization strategy. Then, using the field mapping and type conversion rules, convert the multi-source heterogeneous data into initial standardized data and store it in the cache area of ​​the unified data resource pool.

[0032] In one embodiment, step S2 specifically includes the following sub-steps: S201. Based on business requirements and data types, set preset synchronization frequencies and synchronization strategies for each source system.

[0033] In specific embodiments of the present invention, in order to ensure the timeliness of data acquisition while avoiding excessive pressure on the source system, differentiated synchronization frequencies and strategies are set for different types of data. Specifically, Regarding synchronization frequency, it includes real-time synchronization, scheduled synchronization, and on-demand synchronization. Real-time synchronization is used for critical data with high real-time requirements, such as equipment operating status, alarm data, and port traffic. Scheduled synchronization is used for non-real-time data such as asset ledger data and configuration item data, such as setting up scheduled polling every 24 hours or hour. On-demand synchronization is used for user manual requests (such as updating network topology views) or specific data backtracking scenarios.

[0034] Regarding synchronization strategies, there are incremental and full synchronization strategies. During normal system operation, an incremental synchronization strategy is adopted, which means that during each synchronization, parameters such as timestamps and version numbers are compared to synchronize only the data that has been added or changed since the last synchronization, thereby reducing bandwidth consumption. During system initialization, or in the event of large-scale data loss or inconsistency, a full synchronization strategy is triggered, which retrieves all asset ledgers and monitoring indicators as a whole to ensure the consistency of the underlying data base.

[0035] S202. Establish a network connection with the source system and execute concurrent data acquisition tasks to obtain multi-source heterogeneous data.

[0036] Initialize the front-end connection and complete the legitimacy authorization authentication according to the authentication methods provided by each source interface (such as token, account password, certificate, etc.). After successful authentication, the data collection plugin pulls data through the northbound interface, open interface, or IoT platform channel.

[0037] During this process, when faced with massive data extraction, a concurrent synchronization approach is adopted. This involves launching multiple data call tasks simultaneously using multi-threading to improve synchronization efficiency. To avoid excessive concurrent access pressure on the open interfaces of the primary data source (such as the SG-16000 2.0 system) or the secondary data source, the system incorporates a concurrency control mechanism and performance testing optimization strategies to dynamically adjust the request rate. Simultaneously, the acquisition plugin has a built-in fault tolerance strategy. When encountering abnormal situations such as network failures or interface call failures, it automatically triggers a retry mechanism, records error logs, and sends alarm notifications to ensure high availability during the acquisition process.

[0038] S203. Using the field mapping and type conversion rules, the acquired multi-source heterogeneous data is parsed and converted to generate initial standardized data.

[0039] Heterogeneous data, such as raw JSON, XML, or messages, obtained from various open interfaces or subscription channels, are parsed layer by layer to extract valid field information. Then, the field mapping and type conversion rules established in step S1 are called to map the extracted raw field names to standard attribute names in the target data structure, and the heterogeneous data format is converted and encapsulated into a unified structured data body for this system, thereby obtaining initial standardized data free from the constraints of the underlying source protocol.

[0040] S204. Perform integrity and accuracy verification on the initial standardized data and store it in the cache area of ​​the unified data resource pool.

[0041] First, the generated initial standardized data is validated to check its completeness (e.g., whether required fields are empty) and accuracy (e.g., whether enumerated values ​​are within the valid range). If data format errors or missing key data are found, exception handling logic is executed (e.g., discarding or re-acquiring). For the validated initial standardized data, it is not directly overwritten in the underlying database, but is temporarily pushed into the high-speed cache of the unified data resource pool (e.g., a Redis cache database). The initial standardized data remaining in the cache is the object to be processed in step S3 for conflict detection with the existing data.

[0042] S3. Extract the globally unique identifier of the initial standardized data, compare it with the existing stock standardized data in the unified data resource pool, determine whether there is a data object with the same identifier, and compare whether the data content feature values ​​of the initial standardized data and the corresponding stock standardized data are consistent, so as to determine whether the initial standardized data has a data conflict.

[0043] In one embodiment, step S3 specifically includes the following sub-steps: S301. Parse the initial normalized data in the cache and extract the globally unique identifier used for identity recognition.

[0044] During the aggregation of multi-source heterogeneous data (such as business management systems, network equipment management systems, and power and environmental monitoring systems), the first step is to identify each incoming data item. For the initial standardized data temporarily stored in the unified data resource pool cache, a parsing program extracts its globally unique identifier.

[0045] In specific operation and maintenance scenarios, this globally unique identifier can be dynamically constructed based on the device type. For example, for IT assets and network devices, "IP address + MAC address" or the asset code assigned by the system is used as the unique identifier; for environmental sensors (such as temperature and humidity meters, smoke detectors), "computer room number + device serial number + sensor port number" is used as the unique identifier. In this way, it is ensured that the same physical entity or the same monitoring indicator from different source systems has a unique digital mapping label.

[0046] S302. The extracted globally unique identifier is compared with the existing data in the unified data resource pool to perform a consistency check on the data object.

[0047] Various anomalies can occur during data synchronization, including data inconsistency, data duplication, data loss, and data conflicts. To address these anomalies, a consistency check must be performed on the data before it is officially synchronized and stored in the database. Specifically, the globally unique identifier extracted in step S301 is used as the query keyword to traverse and retrieve data in the persistent storage area (i.e., the existing database) of the unified data resource pool. If no existing standardized data with the same identifier is found, the initial standardized data is determined to be new data (such as a newly added server or a newly connected sensor), and it is directly allowed to proceed to the data entry process. If existing standardized data with the same identifier is found, it means that the data object already exists in the system. The existing standardized data will be extracted, and the process will proceed to the next step for in-depth content comparison.

[0048] S303. Calculate and compare the data content feature values ​​of the initial standardized data with the corresponding existing standardized data.

[0049] Once data objects with the same identifier are confirmed, it is necessary to further determine whether the content has undergone substantial changes or contradictions. Specifically, the core content sets of the initial standardized data and the existing standardized data (such as operating status, CPU utilization, temperature values, etc.) are extracted, and feature value extraction algorithms (such as MD5, SHA-256 hash algorithms) are used to generate corresponding data content feature values ​​(Hash values), and then the feature values ​​of the two are compared. If the feature values ​​are completely identical, it means that the two source systems pushed the exact same data (i.e., data duplication). Only the last active timestamp of the data will be updated, and redundant data packets will be discarded. If the feature values ​​are inconsistent, it means that the initial standardized data differs in content from its corresponding existing historical data, or from data with the same identifier pushed by other source systems within the same time period, thus triggering the conflict determination in step S304.

[0050] S304. Based on the combined timestamp and field-level differences, determine whether the initial standardized data has a data conflict.

[0051] By comparing information such as timestamps and unique identifiers, it is determined whether the data is consistent. If the feature values ​​are inconsistent, the specific fields that are inconsistent are further identified. If the difference is only reflected in the timestamp, and other monitoring indicators are normal time-series updates (e.g., CPU utilization changes from 40% to 45% over time), it is considered a normal data change and not a conflict. If the difference is reflected in the core status or key attributes, and there is logical mutual exclusion or abnormal jump (e.g., for servers with the same IP address, the first data source SG-16000 2.0 reports an online status at time T1, while the second data source H3C network management reports an offline status at very close time T1; or the basic information recorded for the same ledger attribute is inconsistent in the two systems), if the data is inconsistent, the conflict resolution process will begin. At this time, a data conflict identification tag is added, and the initial standardized data is sent to step S4 to trigger the progressive conflict resolution mechanism.

[0052] S4. When data conflicts occur in the initial standardized data, a progressive conflict resolution mechanism is executed sequentially on the initial standardized data to obtain the target standardized data. This progressive conflict resolution mechanism includes sequentially triggered priority adjudication, field-level and data-level merging, and manual intervention. If no data conflicts occur, the initial standardized data is directly used as the target standardized data.

[0053] In one embodiment, step S4 specifically includes the following sub-steps: S401. Perform priority decision based on source weight and timestamp.

[0054] When data inconsistency is detected and the conflict resolution process begins, a priority strategy is triggered first. Priorities for each data source can be pre-specified during automatic conflict handling. For example, in a specific operation and maintenance scenario, the business management system (such as the SG-16000 2.0 system) can be set to have the highest priority. When a conflict occurs, its data will be prioritized to overwrite the data from this system and the network device management platform (such as H3C network management system).

[0055] If the source systems causing the conflict have the same priority weight, the system automatically downgrades to a timestamp-first strategy. This strategy determines the data priority based on its timestamp, precisely compares the absolute timestamps of the conflicting data packets, and prioritizes data with the newest timestamp, adopting the most recently updated data. If, after priority adjudication, some data dimensions still cannot be uniquely identified, the process proceeds to step S402.

[0056] S402. Start the intelligent merging engine to perform field-level and data-level merging.

[0057] If a complete decision cannot be made based on the aforementioned priorities and timestamps (e.g., two pieces of data with complementary information arrive concurrently at the same timestamp), then a merging strategy is implemented, including field-level merging and data-level merging. Specifically, When merging fields at the field level, in cases of partial field conflicts, the conflicting fields can be handled separately. Using the preset core fields of the main data source as a benchmark, unique non-empty auxiliary fields from related data sources are extracted and added to the benchmark data body. Simultaneously, merging calculations are performed according to preset business rules. For example, when a conflict occurs in the device status field, online status can be prioritized over offline status based on conservative monitoring business rules.

[0058] When performing data-level operations, in cases where conflicts occur across entire data items, the data from two systems can be merged without loss. For example, when encountering conflicts in device configuration parameters, the parameter sets of each system are parsed, and the configuration parameters from both systems are deduplicated and merged into a single complete configuration parameter body. If, after deep merging, some core status fields still have irreconcilable business logic conflicts, then the process proceeds to S403.

[0059] S403, trigger an alarm and execute a manual intervention process.

[0060] When a complex data conflict is detected and the aforementioned automatic strategies fail to resolve it safely, a data conflict alarm is generated, and the data flow process is suspended until manual intervention by the user. Two (or more) conflicting data entries are displayed and compared visually on the front-end interface of the operations and maintenance management terminal, allowing the user to select the appropriate data based on actual operations and maintenance experience and offline verification.

[0061] S404, Output target standardized data.

[0062] Whether the data conflict is successfully resolved through automatic priority adjudication by the system, successfully merged through the merging engine, or ultimately confirmed by manual intervention by operations and maintenance personnel, once the data conflict is successfully resolved and a unique, accurate, and complete standardized data body is formed, it is defined as the target standardized data, thus ending the conflict resolution mechanism and entering S5 for persistent storage and view-driven processing.

[0063] S5. Write the target standardized data into the persistent storage area of ​​the unified data resource pool to insert, replace or update the corresponding existing standardized data, and drive the dynamic update of the panoramic monitoring view and network link topology map of the upper-layer integrated operation management system based on the written target standardized data.

[0064] In one embodiment, step S5 specifically includes the following sub-steps: S501. Write the target standardized data into the persistent storage area of ​​the unified data resource pool to replace or update the existing standardized data.

[0065] After processing through a progressive conflict resolution mechanism, unique, accurate, and reliable target standardized data is obtained. The system data processing layer establishes a connection with the data storage layer and writes the target standardized data into a structured database (such as a MySQL database) or a historical database. Based on the globally unique identifier of this data as a primary key or index, a judgment operation is performed in the database. Specifically, if it is newly added data, an insert operation is performed; if it is existing data that has been changed, an update operation is performed, using the target standardized data to overwrite or replace the original existing standardized data, thereby ensuring the data consistency and absolute accuracy of the underlying basic database, real-time database, and historical database.

[0066] S502. Build a real-time data distribution mechanism to push state change events to the upper application layer through message middleware.

[0067] To enable dynamic updates to the upper-layer view, while writing data, not only is hot data stored in a Redis cache database to improve query performance, but a real-time data flow channel is also built through built-in message middleware (such as RabbitMQ) or WebSocket long connection mechanisms. When the core status of the target standardized data (such as device online / offline status, alarm status transitions) or key performance indicators (such as a sudden increase in CPU utilization) changes, the unified message service encapsulates it as an event message and pushes it to the application presentation layer of the integrated operation and maintenance management system in real time.

[0068] S503: Based on target standardized data, drive the dynamic updating and rendering of the upper-level panoramic monitoring view.

[0069] After receiving the pushed target standardized data, the application presentation layer uses visualization components to construct a graphical panoramic monitoring page. It extracts data center environmental parameters, host equipment indicators, and alarm statuses from the target standardized data, rendering and refreshing the front-end monitoring dashboard in real time. For example, it updates voltage, current, and power values ​​in the mains power monitoring overview and UPS monitoring overview in real time. If the target standardized data carries alarm attributes, it drives the panoramic monitoring interface to trigger real-time alarm pop-ups and alarm prompts, and highlights the data center or equipment node experiencing the alarm in color on the 3D or 2D plan view of the data center. Through this data-driven model, it achieves second-level dynamic updates of key indicators such as the overall operating status of the data center, environmental dynamics, host equipment, and centralized alarms.

[0070] S504. Based on target standardized data, drive the dynamic configuration and status early warning of network link topology.

[0071] For network device and link data, a precise network link topology map is dynamically drawn or updated based on the integrated target standardized data (such as detailed records of the start and end points, connected devices, bandwidth specifications, etc. of each link).

[0072] Meanwhile, based on real-time updated performance metrics such as link traffic, packet loss rate, latency, and node status, the topology map provides dynamic status feedback. Specifically, normally operating nodes and links are displayed with specific colors (such as green) and smooth line animations to show the data flow. Once the target standardized data confirms that a network node or link has failed (such as link interruption or severe packet loss), not only are the relevant attributes updated immediately, but the corresponding network topology map node icons are also driven to switch to alarm colors (such as red) and flash. At the same time, the relevant connecting lines are highlighted to help maintenance personnel quickly locate the scope of the fault based on the visualized topology architecture.

[0073] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0074] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for collaborative synchronization and conflict resolution of multi-source heterogeneous data for integrated operation and maintenance, characterized in that, include: S1. Define a unified metadata standard for basic operation and maintenance data as the target data structure for the fusion of multi-source heterogeneous data. For the heterogeneous data formats of the first data source, the second data source, and the third data source, set field mapping and type conversion rules between the corresponding heterogeneous data formats and the target data structure respectively. Among them, the first data source is the business management system, the second data source is the network device management system, and the third data source is the power and environment monitoring system. S2. Using the first data source, the second data source, and the third data source as source systems, obtain multi-source heterogeneous data from the source systems based on a preset synchronization frequency and an incremental or full synchronization strategy, and use the field mapping and type conversion rules to convert the multi-source heterogeneous data into initial standardized data and store it in the cache area of ​​the unified data resource pool. S3. Extract the globally unique identifier of the initial standardized data, compare it with the existing stock standardized data in the unified data resource pool, determine whether there is a data object with the same identifier, and compare whether the data content feature values ​​of the initial standardized data and the corresponding stock standardized data are consistent, so as to determine whether the initial standardized data has a data conflict. S4. When data conflicts occur in the initial standardized data, a progressive conflict resolution mechanism is executed sequentially on the initial standardized data to obtain the target standardized data; wherein, the progressive conflict resolution mechanism includes priority adjudication triggered sequentially, field-level and data-level merging, and manual intervention flow mechanism; S5. Write the target standardized data into the persistent storage area of ​​the unified data resource pool to insert, replace or update the corresponding existing standardized data, and drive the dynamic update of the panoramic monitoring view and network link topology map of the upper-layer integrated operation management system based on the written target standardized data.

2. The method for multi-source heterogeneous data collaborative synchronization and conflict handling for integrated operation and maintenance as described in claim 1, characterized in that, S1 specifically includes: S101. Construct and define a unified metadata standard for basic operation and maintenance data, and use it as the target data structure for the fusion of multi-source heterogeneous data; wherein, the standardized core fields included in the target data structure include at least a globally unique identifier, a data generation timestamp, basic device attribute information, a real-time operating status field, and a set of performance index parameters. S102. Determine the first data source, the second data source, and the third data source; parse the heterogeneous data format of the first data source and set it to use the RESTful API protocol to interact with the first data source; parse the heterogeneous data format of the second data source and set it to use the RESTful interface protocol or HTTP long connection to interact with the second data source; parse the heterogeneous data format of the third data source and set it to use the MQTT protocol to obtain the real-time data stream of the third data source through topic subscription. S103. For the heterogeneous data format, configure the field mapping and type conversion rules between the corresponding source format and the target data structure.

3. The method for multi-source heterogeneous data collaborative synchronization and conflict handling for integrated operation and maintenance as described in claim 2, characterized in that, In S103, The field mapping rule is to establish a mapping dictionary between heterogeneous source data fields and standardized core fields in the target data structure, and to uniformly map the status fields or indicator fields that represent the same physical meaning but have different names in the source system to the standard fields in the target data structure. The type conversion rule is to parse and convert the different data types defined in the heterogeneous data formats into the unified data types specified by the metadata standard. This includes converting different time formats into standard timestamp formats and cleaning and converting the status description information of different systems into standard Boolean variables or system-preset standard enumeration codes.

4. The method for multi-source heterogeneous data collaborative synchronization and conflict handling for integrated operation and maintenance as described in claim 1, characterized in that, S2 specifically includes: S201. Based on business requirements and data types, set preset synchronization frequencies and synchronization strategies for each of the aforementioned source systems; S202. Establish a network connection with the source system and complete authorization authentication, execute a concurrent data acquisition task to obtain the multi-source heterogeneous data, and dynamically adjust the number of concurrent threads according to the load status of the source system during the acquisition process. S203. The multi-source heterogeneous data is parsed to extract valid field information, and the valid field information is converted into initial standardized data using the field mapping and type conversion rules. S204. Perform data integrity and accuracy verification on the initial standardized data, and temporarily store the verified initial standardized data in the cache area of ​​the unified data resource pool.

5. The method for multi-source heterogeneous data collaborative synchronization and conflict handling for integrated operation and maintenance as described in claim 4, characterized in that, In S201, For device status and alarm data with high real-time requirements, set to real-time synchronization frequency; for asset ledgers and configuration data with high non-real-time requirements, set to timed synchronization frequency; and for user manual requests or specific data backtracking scenarios, set to on-demand synchronization frequency. During normal system operation, an incremental synchronization strategy is set to be used, which synchronizes only newly added or changed data by comparing the timestamp or version number parameters of the data. When the system is initialized or when there is a large-scale loss or inconsistency of data, a full synchronization strategy is triggered to synchronize all data.

6. The method for multi-source heterogeneous data collaborative synchronization and conflict handling for integrated operation and maintenance as described in claim 1, characterized in that, S3 specifically includes: S301. Parse the initial standardized data temporarily stored in the cache area and extract the globally unique identifier used for identity recognition; S302. The extracted globally unique identifier is compared with the existing standardized data in the persistent storage area to perform a data object consistency check. Specifically, if no existing standardized data with the same identifier is found, the initial standardized data is determined to be new data and is directly released to the database entry process. If existing standardized data with the same identifier is found, the existing standardized data is extracted and the process proceeds to S303. S303. When it is confirmed that there are data objects with the same identifier, calculate the data content feature value of the core content set of the initial standardized data and the corresponding existing standardized data respectively, and compare them; wherein, calculating the data content feature value specifically involves: using a hash algorithm to generate the hash value of the corresponding core content set of data; if the comparison finds that the data content feature values ​​of the two are completely consistent, it is determined that the data is duplicated, and only the last active timestamp of the corresponding existing standardized data is updated and the initial standardized data is discarded; S304. If the data content feature values ​​are inconsistent, the timestamps and field-level differences of the data are compared to determine whether the initial standardized data has a data conflict. If it is determined that a data conflict has occurred, a data conflict identifier is added to the initial standardized data, and the process proceeds to S4.

7. The method for multi-source heterogeneous data collaborative synchronization and conflict handling for integrated operation and maintenance as described in claim 6, characterized in that, In step S304, the determination of whether a data conflict has occurred involves comprehensively comparing the timestamps and field-level differences of the data. Specifically, this includes: If the inconsistency in the data content feature values ​​is only reflected in the timestamp feature, and other monitoring indicators are normal changes in the sequential time, then it is determined to be a normal data update and does not constitute a data conflict. If the inconsistency in the characteristic values ​​of the data content is reflected in the core state or key attributes, and there is logical mutual exclusion or abnormal jump, then it is determined that a data conflict has occurred.

8. The method for multi-source heterogeneous data collaborative synchronization and conflict handling for integrated operation and maintenance as described in claim 1, characterized in that, S4 specifically includes: S401. Perform priority decision based on the source system weight and absolute timestamp, specifically as follows: Priority weights are pre-assigned to each of the aforementioned source systems. When a data conflict occurs, data from the source system with the higher priority weight is adopted first to cover data from the source system with the lower priority weight. If the source systems that cause the conflict have the same priority weight, a timestamp priority strategy is triggered. By accurately comparing the absolute timestamps carried by the conflicting data, the updated data whose timestamp is closest to the current time is adopted. S402. If the conflict cannot be completely resolved based on the aforementioned priority decision, the intelligent merging engine is activated to perform field-level merging and data-level merging; wherein, For cases of partial field conflicts, the field-level merging method uses the preset core field of the main data source as a benchmark, extracts unique non-empty auxiliary fields from the associated data source and adds them to the benchmark, and merges conflicting individual fields according to preset business rules, including the rule that the online status of the device takes precedence over the offline status. The data-level merging addresses the situation where overall data items conflict. It parses the parameter set of conflicting data, deduplicates the configuration parameters of each data source, and merges the configuration parameters of each data source into a complete configuration parameter body without loss. S403. If, after processing by the intelligent merging engine, the core status fields still exhibit business logic conflicts, an alarm is triggered and a manual intervention process is executed; specifically: When conflicting data cannot be determined as a unique valid value even after automatic resolution strategies, the data flow process is suspended, and a data conflict alarm is automatically generated. The differences between fields of multiple conflicting data are displayed and compared intuitively on the front-end interface of the operation and maintenance management terminal. The terminal also receives manual selection instructions from the user based on the field differences to determine the final valid data. S404. Output the data formed after successfully resolving conflicts through any of the sub-steps S401-S403 and define it as the target standardized data.

9. The method for multi-source heterogeneous data collaborative synchronization and conflict handling for integrated operation and maintenance as described in claim 1, characterized in that, S5 specifically includes: S501. Write the target standardized data into the persistent storage area of ​​the unified data resource pool to replace or update the corresponding existing standardized data; specifically: perform a determination operation in the persistent storage area according to the globally unique identifier of the target standardized data; if it is determined to be new data, perform an insertion operation; if it is determined to be existing changed data, perform an update operation so that the target standardized data overwrites the original existing standardized data. S502. Construct a real-time data distribution mechanism to push state change events to the upper application layer through a message middleware; wherein, the real-time data distribution mechanism is as follows: while completing the persistent writing of data, hot data is stored in a cache database, and a real-time data flow channel is constructed through a built-in message middleware or long connection mechanism; when the core state or key performance indicators of the target standardized data change, it is encapsulated as an event message and pushed to the application presentation layer in real time; S503. Based on the target standardized data, drive the dynamic update and rendering of the upper-layer panoramic monitoring view, specifically: extract the data center environment parameters, host equipment indicators and alarm status from the target standardized data, render and refresh the front-end monitoring screen in real time; if the target standardized data contains alarm attributes, trigger an alarm prompt, and highlight the data center or equipment node that has an alarm in the data center floor plan. S504. Based on the target standardized data, drive the dynamic configuration and status warning of the network link topology diagram, specifically: dynamically draw or update the network link topology diagram according to the link start point, end point, connected device and bandwidth specifications recorded in the target standardized data; when a network node or link is confirmed to have a fault, drive the corresponding network topology diagram node icon to switch to an alarm color and display a flashing effect, and at the same time highlight the relevant connection lines for warning.

Images