Optical transmission system management platform based on distributed storage and multi-protocol adaptation

Abstract

The invention discloses an optical transmission system management platform based on distributed storage and multi-protocol adaptation, and relates to the technical field of optical transmission network operation and maintenance. The platform comprises a data acquisition module, a distributed storage module, a data processing module, an intelligent analysis module and a user interaction module. And the data acquisition module is compatible with SNMP, NETCONF, TL1 and private protocols through a dynamic protocol analysis engine to realize unified acquisition of heterogeneous device data. The distributed storage module adopts a consistent Hash algorithm to perform fragmentation storage, and combines a multi-copy mechanism and a log structure to merge tree index optimization, thereby improving storage reliability, load balance and query efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of optical transmission network operation and maintenance management, and particularly to an optical transmission system management platform based on distributed storage and multi-protocol adaptation. Background Art

[0002] With the rapid development of optical transmission networks, the improvement of their operation and maintenance efficiency and management level has become crucial. In the face of a large amount of network management data, the existing optical transmission system management platforms are limited in storage and retrieval efficiency and are difficult to cope with the growth of data scale. Moreover, at the data collection level, the compatibility with different types of optical transmission devices is insufficient, and key data cannot be accurately and comprehensively collected, resulting in many inconveniences and limitations in operation and maintenance management. Summary of the Invention

[0003] In view of the problems existing in the prior art, an optical transmission system management platform based on distributed storage and multi-protocol adaptation of the present invention is characterized by comprising a data collection module, a distributed storage module, a data processing module, an intelligent analysis module, and a user interaction module;

[0004] The data collection module is used to collect the operating status, alarm information, and performance indicators of optical transmission devices through multi-protocol adaptation technology; the distributed storage module is used to perform distributed storage on the collected large amount of network management data; the data processing module is used to clean, transform, and integrate the collected data; the intelligent analysis module is based on big data analysis and machine learning algorithms to deeply analyze the data and realize functions such as fault warning and performance optimization; the user interaction module provides a visual interface to support the operations and decisions of operation and maintenance personnel.

[0005] Preferably, the multi-protocol adaptation includes the parsing of SNMP, NETCONF, TL1, and vendor private protocols.

[0006] Preferably, the data collection module includes a dynamic protocol parsing engine, which can automatically identify the device protocol format and perform conversion.

[0007] Preferably, the distributed storage module uses the consistent hashing algorithm to perform sharding storage on the data to balance the storage load.

[0008] Preferably, the distributed storage module further adopts a replica mechanism, and each piece of data is stored in at least three different storage nodes to improve data reliability.

[0009] Preferably, the distributed storage module uses a log-structured merge tree for index storage.

[0010] Preferably, the data processing module optimizes the query efficiency based on the index structure of the log-structured merge tree and supports batch data merging.

[0011] Preferably, the intelligent analysis module uses machine learning algorithms for anomaly detection, identifies abnormal behaviors based on the historical data of the device, and generates alarm information.

[0012] On the other hand, the present invention provides an optical transmission system management platform based on distributed storage and multi-protocol adaptation, which is characterized by including a data acquisition module, a distributed storage module, a data processing module, an intelligent analysis module, and a user interaction module; the distributed storage module is used for distributed storage of the collected massive network management data;

[0013] It is characterized in that: the distributed storage module further includes a module for blockchain certification of operation and maintenance data, and the functions of this module include:

[0014] Obtain the operation and maintenance data generated by each network device in the platform, including device identification, event type, timestamp, and event content;

[0015] Format the operation and maintenance data, and calculate its hash value through a hash algorithm;

[0016] Construct a certification structure including the device identification, event type, timestamp, and hash value;

[0017] Write the certification structure into the blockchain system to achieve tamper-proof certification of operation and maintenance data;

[0018] During data auditing or verification, recalculate the hash value according to the target operation and maintenance data extracted from the storage system, and compare it with the hash value recorded in the blockchain to determine whether the data has been tampered with.

[0019] Preferably, the distributed storage module further includes a data indexing and fast retrieval mechanism, and the functions of this mechanism include: while storing the hash value of the operation and maintenance data into the blockchain, generating a corresponding distributed storage index and storing it in the distributed storage system; adopting a query method based on blockchain smart contracts, calculating the hash value of the target data according to the query request, and verifying its integrity through the smart contract; combining the blockchain-certified data with the distributed storage index to achieve fast retrieval of historical operation and maintenance data and improve the efficiency of data auditing and backtracking.

[0020] Compared with the prior art solutions, the present invention has at least the following beneficial effects:

[0021] 1) The present invention adopts advanced distributed storage technology, disperses the massive network management data to be stored in multiple storage nodes, and realizes the efficient storage and fast retrieval of data through a distributed database management system, effectively solving the performance bottleneck faced by traditional centralized storage when the data scale grows, and improving the efficiency and response speed of optical transmission network operation and maintenance management.

[0022] 2) In the optical transmission system management platform proposed in this application of the present invention, the system involves a large amount of key operation and maintenance data such as the operating status of devices, fault alarm information, and maintenance logs. Once these data are tampered with, it will seriously affect network reliability and subsequent fault analysis. Therefore, the blockchain technology is introduced into this system to conduct evidence preservation and reinforcement for the above key data. It can significantly improve the integrity, security, and traceability of operation and maintenance data, and is especially suitable for scenarios of optical transmission systems with high requirements for multi-source heterogeneous data collaborative processing and security supervision. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the architecture of the optical transmission system management platform of the present invention.

[0024] The present invention will be further described in detail below. However, the following examples are only simple examples of the present invention and do not represent or limit the scope of the claimed rights of the present invention. The scope of protection of the present invention shall be subject to the claims. SPECIFIC EMBODIMENTS

[0025] The technical solution of the present invention will be further described below in conjunction with the drawings and through specific implementation manners.

[0026] As Figure 1 shown, the optical transmission system management platform of the present invention includes a data acquisition module, a distributed storage module, a data processing module, an intelligent analysis module, and a user interaction module. The data acquisition module communicates with various optical transmission devices to collect key data; the distributed storage module uses distributed storage technology to store a large amount of network management data; the data processing module cleans, transforms, and integrates the collected data; the intelligent analysis module conducts in-depth analysis of the data based on big data analysis and machine learning algorithms to realize functions such as fault warning and performance optimization; the user interaction module provides a visual interface to facilitate the operation and decision-making of operation and maintenance personnel.

[0027] The data acquisition module is a bridge between the management platform and optical transmission devices, and is responsible for collecting key data such as the operating status of devices, alarm information, and performance indicators. By adopting multi-protocol adaptation technology and built-in multiple communication protocol interfaces, it can communicate with optical transmission devices of different brands and models to ensure the comprehensiveness and accuracy of data acquisition. When a new device type is accessed, the data acquisition for this device can be quickly realized through the extended adaptation module.

[0028] The distributed storage module is built based on distributed storage technology, and distributes a large amount of network management data to be stored in multiple storage nodes. Each storage node is responsible for storing data within a specific range, and realizes the efficient storage and quick retrieval of data through a distributed database management system. By adopting technologies such as data sharding and replica mechanisms, it guarantees the reliability and consistency of data, and at the same time improves the scalability of the storage system, and can flexibly cope with the growth of data scale.

[0029] The data processing module cleans, transforms, and integrates the collected raw data. It removes noise and redundant information from the data, converts data in different formats into a unified format, integrates related data to form structured data, providing a high-quality data source for subsequent data analysis. Meanwhile, it conducts preliminary analysis on the data, extracts key features, and provides basic data support for the intelligent analysis module.

[0030] Based on big data analysis and machine learning algorithms, the intelligent analysis module conducts in-depth analysis on the processed data. It realizes the fault warning function. By analyzing the changing trends of device performance indicators, it can predict potential faults in advance, providing warning information for operation and maintenance personnel so that they can take timely measures to reduce the impact of faults; it provides performance optimization suggestions. According to the network operation data, it analyzes the network performance bottlenecks and proposes optimization solutions to improve the overall network performance; it supports intelligent decision-making, providing data support for operation and maintenance personnel to assist them in making reasonable operation and maintenance decisions.

[0031] The user interaction module provides an intuitive and convenient operation interface for operation and maintenance personnel. Using visualization technology, it displays data such as network topology, device status, alarm information, and performance indicators in a graphical way, facilitating operation and maintenance personnel to quickly understand the network operation status; it provides a data query function, supporting querying historical data according to various conditions such as devices, time, and indicators to meet the different data needs of operation and maintenance personnel; it supports alarm management, real-time displays alarm information, and provides an alarm handling record function, facilitating operation and maintenance personnel to track and handle alarm events; it provides a report generation function, generating various reports according to operation and maintenance requirements, providing a data basis for operation and maintenance management.

[0032] Embodiment 1

[0033] A large optical transmission network contains thousands of nodes and generates a huge amount of network management data every day. After adopting the management platform of the present invention, through the distributed storage technology, the data is dispersed and stored in multiple high-performance storage nodes, and each node is responsible for storing data within a specific range. When operation and maintenance personnel need to query the historical performance data of a specific node, the platform quickly locates and retrieves the relevant data through the distributed database management system, and the retrieval time is shortened from several minutes of the traditional platform to dozens of seconds, greatly improving the operation and maintenance efficiency.

[0034] Specifically, the distributed storage module distributes data to multiple storage nodes through a hash algorithm (such as the consistent hash algorithm Ketama Hashing), and each node stores a part of the data. For example, in the case of 10 storage nodes, assuming the total daily data volume is 864 GB; using consistent hashing for data distribution: The role of consistent hashing is to make the data as evenly distributed as possible among each storage node. Assuming there are 10 storage nodes and the data is evenly distributed, each node stores approximately: 864 GB / 10 = 86.4 GB per day.

[0035] For storage, the present invention adopts a distributed storage method to implement the operation and maintenance data blockchain evidence storage mechanism.

[0036] In the optical transmission system management platform provided by the present invention, the system needs to process a large amount of operation and maintenance data from optical network devices (such as OLT, optical modules, distributed monitoring nodes, etc.), including information such as device operation status, fault alarms, and maintenance logs. To ensure the credibility and traceability of the above data, based on the distributed storage architecture, the present invention introduces a blockchain evidence storage mechanism to perform encrypted hash processing and on-chain registration of key operation and maintenance data, thereby achieving the goals of anti-tampering and audit traceability.

[0037] Original data collection: The system receives the operation and maintenance data D from each device in real time

[0038] For example: Device ID: OLT-A1

[0039] Alarm time: April 2, 2024, 10:33:21

[0040] Alarm type: RX optical power anomaly

[0041] Measured value: -25.1dBm

[0042] Alarm level: Critical

[0043] Hash calculation: The system formats the above data and calculates its hash value H, using a secure hash algorithm (such as SHA-256):

[0044] H = SHA256(D)

[0045] Where D represents the formatted original data string.

[0046] Construct an evidence storage structure: Construct the information such as the hash value HHH, device identifier, alarm type, timestamp, etc. into a blockchain evidence storage data structure:

[0047]

[0048] On-chain registration: Write this structure as new block data into a private blockchain or consortium blockchain to complete the evidence storage of operation and maintenance data.

[0049] Data verification mechanism: When subsequent operation and maintenance personnel or the audit system need to verify whether a certain piece of data has been tampered with, the system reads this piece of data D' from the storage system, recalculates the hash value H' = SHA256(D'), and compares it with the original hash value H stored in the blockchain. If H' = H, it is considered that the data has not been tampered with.

[0050] Through the above mechanism, the present invention can significantly improve the integrity, security, and traceability of operation and maintenance data, and is particularly applicable to scenarios of optical transmission systems that require collaborative processing of multi-source heterogeneous data and have high security supervision requirements. This mechanism has little impact on the main performance of the system and has good scalability and engineering adaptability. The above-mentioned distributed storage is one of the inventive points of the present invention.

[0051] The distributed storage module is also provided with a replica mechanism, where each piece of data is stored at least 3 times and distributed across different nodes to improve disaster tolerance. If a storage node fails, the data can be read from other replica nodes to ensure high availability.

[0052] The distributed storage module also has index optimization, that is, a log-structured merge tree is used for index storage to improve data insertion and query speeds. For example, in the traditional B+ tree structure, the query time complexity is O(log n), while the log-structured merge tree can reduce the query complexity to O(1) (for some query scenarios) by batch-merging indexes.

[0053] Based on the distributed data and index optimization of the distributed storage module, the data processing module ensures the efficiency of data processing when performing data cleaning, normalization, and indexing.

[0054] The intelligent analysis module is based on the basic data support provided by the distributed storage module. Specifically: (1) The machine learning prediction model requires historical data for training; (2) The fault analysis system requires a large amount of device logs for pattern matching.

[0055] Improve data retrieval efficiency through distributed storage technology. Adopt a distributed database management system to achieve efficient storage and fast query of data. Through data sharding and replica mechanisms, ensure the reliability and consistency of data, and at the same time improve the scalability of the storage system.

[0056] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, various simple modifications can be made to the technical solutions of the present invention, and these simple modifications all fall within the protection scope of the present invention.

[0057] In addition, it should be noted that, in the case of no contradiction, the various specific technical features described in the above specific embodiments can be combined in any suitable way. To avoid unnecessary repetition, the present invention will not separately describe various possible combination methods.

[0058] In addition, any combination can be made between different embodiments of the present invention as long as it does not violate the idea of the present invention, and it should also be regarded as the content disclosed by the present invention.

Claims

1. An optical transmission system management platform based on distributed storage and multi - protocol adaptation, characterized in that, It includes a data acquisition module, a distributed storage module, a data processing module, an intelligent analysis module, and a user interaction module; The data acquisition module is used to collect the operating status, alarm information, and performance metrics of optical transmission devices through multi-protocol adaptation technology; The distributed storage module is used to perform distributed storage on the massive network management data collected; The data processing module is used to clean, transform, and integrate the collected data; The intelligent analysis module, based on big data analysis and machine learning algorithms, deeply analyzes the data to achieve functions such as fault warning and performance optimization; The user interaction module provides a visual interface to support the operations and decisions of operation and maintenance personnel.

2. The optical transmission system management platform according to claim 1, wherein The multi-protocol adaptation includes the parsing of SNMP, NETCONF, TL1, and vendor private protocols.

3. The optical transmission system management platform according to claim 2, characterized in that, The data acquisition module contains a dynamic protocol parsing engine that can automatically identify the device protocol format and perform conversion.

4. The optical transmission system management platform according to claim 1, characterized in that, The distributed storage module uses the consistent hashing algorithm to shard and store the data to balance the storage load.

5. The optical transmission system management platform according to claim 4, characterized in that, The distributed storage module further adopts a replica mechanism, and each piece of data is stored in at least three different storage nodes to improve data reliability.

6. The optical transmission system management platform according to claim 1, characterized in that, The distributed storage module uses a log-structured merge tree for index storage.

7. The optical transmission system management platform according to claim 6, wherein The data processing module optimizes the query efficiency based on the index structure of the log-structured merge tree and supports batch data merging.

8. The optical transmission system management platform according to claim 1, characterized in that, The intelligent analysis module uses machine learning algorithms for anomaly detection, identifies abnormal behaviors based on the historical data of the device, and generates alarm information.

9. An optical transmission system management platform based on distributed storage and multi-protocol adaptation, characterized in that It includes a data acquisition module, a distributed storage module, a data processing module, an intelligent analysis module, and a user interaction module; the distributed storage module is used to perform distributed storage on the massive network management data collected; It is characterized in that: the distributed storage module further includes a module for blockchain-based evidence storage of operation and maintenance data, and the functions of this module include: Obtain the operation and maintenance data generated by each network device in the platform, including device identification, event type, timestamp, and event content; Format the operation and maintenance data and calculate its hash value through a hash algorithm; Construct an evidence storage structure body containing the device identification, event type, timestamp, and hash value; Write the evidence storage structure body into the blockchain system to achieve tamper-proof evidence storage of operation and maintenance data; During data auditing or verification, recalculate the hash value based on the target operation and maintenance data extracted from the storage system, and compare it with the hash value recorded in the blockchain to determine whether the data has been tampered with.

10. The management platform according to claim 9, wherein the distributed storage module further includes a data indexing and fast retrieval mechanism, and the functions of the mechanism include: When storing the hash value of the operation and maintenance data in the blockchain, generate a corresponding distributed storage index and store it in the distributed storage system; Adopt a query method based on blockchain smart contracts, calculate the hash value of the target data according to the query request, and verify its integrity through the smart contract; Combine the blockchain evidence storage data with the distributed storage index to achieve fast retrieval of historical operation and maintenance data and improve the efficiency of data auditing and traceability.

Images