A data governance method and system for a multi-source heterogeneous communication network

Abstract

The application provides a multi-source heterogeneous communication network-oriented data management method and system, relates to the technical field of communication network data management, and comprises the following steps: an edge intelligent node is used to connect data from a multi-source heterogeneous communication network data source; the edge intelligent node adopts a main service plus micro-service component socket architecture, the edge intelligent node loads corresponding micro-service components through main service intelligent adaptation to realize the access of different data sources; the connected data is subjected to extraction processing, conversion processing and loading processing in a cloud data center, invalid data is removed, the data format is unified, and the data is loaded into corresponding source-attached libraries according to network types; based on the standardized data of the source-attached libraries, a bottom-up layered construction mode is adopted to sequentially construct a domain library and a theme library; the domain library comprises a basic support domain library and a network professional domain library, and the theme library is divided according to professional network types; a business service layer is constructed based on the theme library to provide data resource services and data asset services.

Description

Technical Field

[0001] This invention relates to the field of communication network data governance technology, and in particular to a data governance method and system for multi-source heterogeneous communication networks. Background Technology

[0002] With the rapid development of communication technology, various communication networks such as Optical Transport Network (OTN), Ethernet Transport Network, Packet Transport Network (PTN), and Synchronous Digital Hierarchy (SDH) have become core components of communication infrastructure, supporting the stable operation of diverse services. However, in the long-term network construction and operation process, these communication networks are usually composed of equipment from different periods and manufacturers, each with its own independent management system and data acquisition interface, resulting in current communication networks exhibiting significant characteristics of multi-network convergence and multi-source heterogeneity.

[0003] The existing data management methods in communication networks are relatively decentralized and have many drawbacks: First, data storage is isolated, making sharing difficult. Various communication network data are scattered and stored in local databases of different network management systems or network elements. Each data source is independent, lacking a unified data aggregation and exchange mechanism, resulting in ineffective data sharing and correlation. This data silo phenomenon severely restricts cross-network and cross-regional business linkage analysis and integrated operation and maintenance management, making it difficult to fully explore and utilize the potential value of massive amounts of network data.

[0004] Secondly, data standards are inconsistent, resulting in varying quality. The same type of data from different communication networks and equipment manufacturers can exhibit significant differences in field naming, data format, encoding rules, units of measurement, and even semantic meaning across different networks. Furthermore, data in various network systems commonly suffers from invalid, redundant, and abnormal data, lacking standardized cleaning and verification processes, directly impacting the accuracy, consistency, and usability of the data.

[0005] Third, there is a lack of a unified data governance system. Existing data management methods are mostly customized solutions for single network types, which cannot adapt to the data governance needs of multi-network convergence scenarios. This leads to practical difficulties when facing advanced application needs such as cross-network data analysis, unified resource view construction, and intelligent operation and maintenance of the entire network, including weak underlying data foundation, long data preparation cycle, and high data adaptation cost.

[0006] In conclusion, there is an urgent need for a method that can centrally manage and standardize the processing of data from various communication networks and establish an efficient data service system. This would address the existing technical problems of isolated data storage, inconsistent standards, and varying quality in communication networks, thereby improving the overall management level of communication networks and fully realizing the value of network data. Summary of the Invention

[0007] To address the aforementioned problems, this invention provides a data governance method and system for multi-source heterogeneous communication networks, which solves the technical problems of isolated data storage, inconsistent standards, and uneven quality in existing communication networks, thereby achieving unified governance and effective utilization of multi-source heterogeneous data.

[0008] This invention provides a data governance method for multi-source heterogeneous communication networks, the method comprising: Data is fed from multi-source heterogeneous communication network data sources through edge intelligent nodes; the edge intelligent nodes adopt a socket architecture of main service plus microservice components, and the edge intelligent nodes intelligently adapt and load the corresponding microservice components through the main service to realize the access of different data sources; The data is extracted, transformed, and loaded in the cloud data center. Invalid data is removed and the data format is unified. The data is then loaded into the corresponding post source library according to the network type. Based on the standardized data in the source library, a bottom-up hierarchical construction method is adopted to build a domain library and a topic library in sequence; the domain library includes a basic support domain library and a network professional domain library, and the topic library is divided according to professional network type; A business service layer is built based on the aforementioned subject library to provide data resource services and data asset services.

[0009] Furthermore, the edge intelligent node intelligently adapts and loads corresponding microservice components through the main service to enable access to different data sources, specifically including: The edge intelligent node breaks down the full communication protocol into independent microservice components; The main service actively identifies access devices, parses device feature information to determine the device type and the type of protocol to be loaded; The main service makes dynamic decisions based on the identification results and loads the corresponding microservice components from the component repository to establish a data channel with the target network; The main service manages the lifecycle of the loaded microservice components, monitors their running status, and releases resources when the connection is interrupted.

[0010] Furthermore, the conversion process includes data cleaning, data format unification, data association, and data completion; wherein, data cleaning is used to remove invalid data, including null data, outlier data, and duplicate data, using an outlier detection algorithm to identify and remove outlier data, and using a deduplication algorithm to remove duplicate data; data format unification is used to convert data of different formats into a preset standard JSON format; data association is used to establish the association relationship between data from different data sources; data completion is used to fill in missing key data, using interpolation or prediction algorithms based on historical data to fill in missing values.

[0011] Furthermore, the unified data format specifically includes: Construct a data dictionary and establish a unified business semantic mapping relationship for each communication network domain; The semantic features of the source field are extracted using the trained model, and the similarity is compared with the unified fields in the standard resource dictionary to select the matching target field and obtain the field mapping relationship. A transformation expression is generated based on the field mapping relationship to convert the source data into data that conforms to the target field specification.

[0012] Furthermore, after obtaining the field mapping relationship, it also includes: It detects differences between the source field and the target field in terms of units, precision, and value range, generates a conflict list, and automatically generates a specification conversion function based on the built-in rule library; The system automatically detects the character encoding of the source data. If it is not UTF-8, it is uniformly converted to UTF-8, and a unified code mapping table is constructed to map the business code of the source system to the unified business code, thus obtaining the encoding conversion rules. By integrating the field mapping relationships, specification conversion functions, and encoding conversion rules, a complete conversion expression is generated using a Transformer-based Seq2seq model.

[0013] Furthermore, the source library is divided into OTN transmission communication network source library, Ethernet transmission communication network source library, PTN transmission communication network source library and SDH transmission communication network source library according to network type; the source library adopts a storage method that combines relational database and non-relational database, wherein structured data is stored in relational database, and semi-structured and unstructured data is stored in non-relational database.

[0014] Furthermore, the basic support domain library is used to store basic support data for network operation, including basic equipment configuration information, basic network topology information, operation and maintenance support data, and basic system data; the network professional domain library is used to store professional operation data of professional transmission communication networks, which are classified and aggregated according to the network professional type of OTN transmission network, Ethernet transmission network, PTN transmission network, and SDH transmission network.

[0015] Furthermore, the subject library includes an OTN transmission subject library, an Ethernet transmission network subject library, a PTN transmission subject library, and an SDH transmission subject library; each subject library is partitioned and stored according to business scenarios to form a data model oriented towards business scenarios.

[0016] Furthermore, The OTN transmission topic library is based on OTN transmission professional data in the network professional domain library. The sub-topics built according to business needs include optical equipment topic, optical link topic, optical performance topic, and optical alarm topic. The Ethernet transmission network topic library takes Ethernet professional data from the network professional domain library as its core, integrates Ethernet standardized data, and constructs sub-topics according to business needs, including Ethernet device topics, Ethernet routing topics, Ethernet traffic topics, and Ethernet alarm topics. The PTN transmission topic library is based on PTN transmission professional data in the network professional domain library, and integrates PTN transmission network standardization data. The sub-topics built according to business needs include PTN device topics, PTN tag topics, PTN performance topics, and PTN alarm topics. The SDH transmission topic library is based on SDH transmission professional data in the network professional domain library, and integrates SDH transmission network standardization data. The sub-topics constructed according to business needs include SDH equipment topics, SDH link topics, SDH performance topics, and SDH alarm topics.

[0017] On the other hand, the present invention also provides a data governance system for multi-source heterogeneous communication networks, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any of the above-described methods.

[0018] In summary, this invention provides a data governance method and system for multi-source heterogeneous communication networks, which achieves the following beneficial effects compared with existing technologies: (1) This invention utilizes a socket architecture of main service plus microservice components in edge intelligent nodes to achieve unified access to multi-source heterogeneous communication network data. Combined with a hierarchical governance system of source library, domain library, topic library, and business service layer, it solves the technical problems of isolated data storage, inconsistent standards, uneven quality, and inability to achieve unified governance in existing communication networks. This invention aggregates multi-source data scattered across different networks and devices to a cloud data center. Through layer-by-layer refinement and aggregation, it forms standardized and systematic data assets, providing a high-quality data foundation for applications such as cross-network comprehensive analysis, unified resource management, and intelligent operation and maintenance, significantly improving the availability and management efficiency of communication network data.

[0019] (2) The present invention actively identifies access devices through the main service of edge intelligent nodes and dynamically loads the corresponding microservice components according to the identification results, thereby realizing adaptive adaptation to different communication network protocols. This avoids the tedious process of manually configuring protocol interfaces in traditional solutions, greatly reduces the manual cost and error risk of data access, and improves the automation level and flexible expansion capability of data collection.

[0020] (3) This invention introduces an agent-based field semantic mapping and unified exchange mechanism during the data format unification process. It utilizes a trained model to automatically identify the semantics of source fields and match them with a standard resource dictionary. Combined with specification unification, encoding conversion, and a Transformer-based Seq2seq model to automatically generate conversion expressions, it achieves intelligent and automated standardized conversion of cross-network data. Compared to the traditional method of manually writing mapping rules, this method can adaptively handle field differences between different networks and vendors, significantly improving the efficiency and accuracy of data standardization processing and effectively solving the pain point of difficulty in integrating multi-source heterogeneous communication network data.

[0021] (4) This invention forms a bottom-up, progressive data organization system by constructing a source library divided by network type, a domain library including basic support and network professional fields, and a theme library organized by business scenarios. It not only preserves the integrity and traceability of the original data, but also realizes the classification, aggregation and association of data by professional and business dimensions, so that upper-layer applications can obtain accurate and high-quality data as needed, avoiding the problems of repeated data storage and redundant processing in traditional solutions, and improving the overall operating efficiency of the data governance system. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating a data governance method and system for multi-source heterogeneous communication networks provided by the present invention. Figure 2 This is a schematic diagram illustrating the principle of a data governance method and system for multi-source heterogeneous communication networks provided by the present invention. Figure 3 This is a schematic diagram of the edge intelligent node intelligently adapting and loading corresponding microservice components for a data governance method and system for multi-source heterogeneous communication networks provided by the present invention. Figure 4 This is a schematic diagram illustrating the data format unification of a data governance method and system for multi-source heterogeneous communication networks provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0025] It should be noted that, in the description of the embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method, step, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to the method, step, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of additional identical elements in the method, step, or apparatus that includes the element.

[0026] To address the technical problems of isolated data storage, inconsistent standards, and varying quality in existing communication networks, this invention provides a data governance method and system for multi-source heterogeneous communication networks. It achieves unified access to various types of network data through a socket architecture using a main service plus microservice components at edge intelligent nodes. Data is extracted, transformed, and loaded in a cloud data center. A hierarchical governance system is constructed, comprising a source library, domain library, topic library, and business service layer. During the data format unification process, an agent-based field semantic mapping and automatic conversion mechanism is introduced, realizing fully automated governance and efficient service of multi-source heterogeneous communication network data throughout the entire process.

[0027] like Figure 1 and Figure 2 As shown, the method specifically includes: S100: Data is fed from multi-source heterogeneous communication network data sources via edge intelligent nodes. The edge intelligent nodes employ a socket architecture of a main service plus microservice components. The edge intelligent nodes intelligently adapt and load the corresponding microservice components through the main service to achieve access to different data sources. By intelligently adapting and loading the corresponding microservice components through the main service, the edge intelligent nodes achieve adaptive adaptation to different communication network protocols, avoiding the cumbersome process of manually configuring protocol interfaces required in traditional solutions. This significantly reduces the manual cost and error risk of data access, and improves the automation level and flexible scalability of data collection.

[0028] The data sources in the communication network include various types such as OTN, Ethernet, PTN, and SDH transmission networks. Multi-source data acquisition is achieved through edge intelligent nodes. These edge intelligent nodes employ a socket architecture of a main service plus microservice components, intelligently adapting to the required data interfaces and initiating data connection.

[0029] Specifically, the main service actively identifies and parses the device characteristic information of the access device, intelligently adapts and loads the corresponding microservice components, thereby establishing a data channel with the target network and initiating the data connection process. The edge intelligent node connects multi-format data from different types of data sources (such as SNMP interface protocol data, Restful interface protocol data, Telnet interface protocol data, Corba interface protocol data, XML interface protocol data, etc.) to the data lake of the cloud data center in real time or on a schedule through the intelligent data connection interface.

[0030] like Figure 3 The diagram illustrates the intelligent adaptation and loading of corresponding microservice components by an edge intelligent node according to an embodiment of the present invention. As an example, the edge intelligent node intelligently adapts and loads corresponding microservice components through the main service to achieve access to different data sources, specifically including: Step S101: The edge intelligent node decomposes all communication protocols into independent microservice components. Each communication network device protocol is encapsulated as an independent microservice component; each microservice component has independent deployment, version management, and elastic scaling capabilities. These microservice components include various protocol components such as SNMP, RESTful, CORBA, Syslog, and Telnet.

[0031] Step S102: The main service actively identifies the access device, parses the device's characteristic information to determine the device type and the protocol type to be loaded. After the communication source network is connected, the main service of the edge intelligent node actively identifies the connected device by listening to the access protocol and message characteristics. The main service parses the characteristic information of the device during registration or the first message report, including key metadata information such as device type, vendor OID, and software version, and combines it with protocol characteristics and software fingerprint rules to determine the type of the currently accessing device and the protocol type to be loaded.

[0032] Step S103: The main service dynamically decides based on the identification results and loads the corresponding microservice components from the component repository to establish a data channel with the target network. Edge intelligent nodes perform intelligent component matching and scheduling. The main service dynamically decides which microservice protocol components to invoke based on the device identification results and the parsed protocol characteristics. Specifically, the main service loads the corresponding microservice components from the component repository based on the device model and protocol version type, starts the component service, and establishes a data channel with the target network.

[0033] Step S104: The main service performs lifecycle management on the loaded microservice components, monitors their running status, and releases resources when the connection is interrupted. The main service continuously monitors the running status and health of the loaded microservice components, including periodically probing the CPU and memory usage metrics of the components, detecting the connection status with the channel network, and reconnecting when necessary. When the communication network connection is interrupted, the main service stops the component from running, releases the occupied resources, removes the component from the scheduling table, and realizes resource reclamation of the edge intelligent node.

[0034] S200: In the cloud data center, the incoming data is extracted, transformed, and loaded. Invalid data is removed, and the data format is standardized. Data is then loaded into the corresponding source library according to network type. This achieves standardized processing of multi-source heterogeneous communication network data, unifying data from different networks and formats into a standard format, laying a solid foundation for subsequent data governance and applications.

[0035] In the cloud data center, the multi-source data brought to the data lake is extracted, transformed, and loaded, and the standardized data is loaded into the corresponding library of the source library according to the network type.

[0036] It should be noted that the cloud data center performs fusion processing on the data in the data lake, specifically including three steps: extraction, transformation, and loading. This involves operations such as field mapping, data type conversion, data completion, and consistency verification. When data in the data lake is updated, a data change message is generated synchronously, notifying the service to pull the data for processing, achieving real-time updates of the source database and ensuring data consistency.

[0037] Step S201: Extraction. A multi-threaded parallel extraction method is used to extract data from various data sources in the data lake. Specifically, different extraction frequencies can be set according to the data type: real-time extraction is used for data with high real-time requirements, such as alarm data and real-time traffic data; timed extraction is used for data with lower real-time requirements, such as device configuration data and historical performance data, thereby ensuring the timeliness and integrity of the data.

[0038] Step S202: Transformation. The extracted data undergoes a standardized transformation process. As an example, this transformation includes data cleaning, data format unification, data association, and data completion. Data cleaning removes invalid data, including null values, outliers, and duplicates. An outlier detection algorithm identifies and removes outliers, and a deduplication algorithm removes duplicates. Data format unification converts data from different formats into a preset standard JSON format. Data association establishes relationships between data from different data sources. Data completion fills in missing key data using interpolation or prediction algorithms based on historical data to ensure data integrity.

[0039] To ensure data structure consistency, in this embodiment, data format uniformity specifically includes: First, a data dictionary is constructed to establish a unified business semantic mapping relationship for each communication network domain.

[0040] The semantic layer, designed to address the inconsistencies in network technology domains among OTN, PTN, SDH, and Ethernet, establishes a unified business semantic mapping relationship for each communication domain from dimensions such as resource entities, attribute fields, and performance indicators.

[0041] For example, the entity concept of "network element identifier" is represented as "managedElement.id" in OTN network, "Neid" in PTN network, "neName" in SDH network, and "sysName" in Ethernet network, and is uniformly mapped to "Element_id" through the data dictionary; while the concept of "port" is represented as "terminationPoint.toName" in OTN network, "Portid" in PTN network, "ifName" in SDH network, and "ifDescr" in Ethernet network, and is uniformly mapped to "Port_name"; the mapping relationships of other entity concepts are shown in Table 1.

[0042] Table 1. Mapping Relationships of Entity Concepts in Each Communication Network Domain

[0043] Then, the semantic features of the source field are extracted using the trained model, and the similarity is compared with the unified fields in the standard resource dictionary to select the matching target fields and obtain the field mapping relationship.

[0044] Specifically, by constructing an input text sequence, for each source component data, an input string is constructed: T_src = field_name + "[SEP]" + comment + "[SEP]" + sample_value; where field_name is the field name, such as "SDH B1 error", comment is the annotation field, such as "SDH business background error block", and sample_value is the sample value extracted from historical performance. For each source component field, T_src is used as the semantic embedding of the input BERT model, and its similarity score with the unified field in the resource dictionary is calculated. The top k keywords are selected as candidates according to the scores from high to low, and the unified field corresponding to the highest score is selected as the field mapping relationship.

[0045] Finally, a transformation expression is generated based on the field mapping relationship to convert the source data into data that conforms to the target field specification.

[0046] Furthermore, after obtaining the field mapping relationship, it also includes: It detects differences between the source field and the target field in terms of units, precision, and value range, generates a conflict list, and automatically generates a specification conversion function based on the built-in rule library; The system automatically detects the character encoding of the source data. If it is not UTF-8, it is uniformly converted to UTF-8, and a unified code mapping table is constructed to map the business code of the source system to the unified business code, thus obtaining the encoding conversion rules. By integrating the field mapping relationships, specification conversion functions, and encoding conversion rules, a complete conversion expression is generated using a Transformer-based Seq2seq model.

[0047] It should be noted that the system automatically reads the target specification definition, retrieves the specification metadata of the target field from the resource dictionary, then scans the most recent 1000 data samples, statistically analyzes the numerical distribution, unit flag, actual number of digits, and timestamp interval, analyzes the actual specification, and generates a conflict list by comparing the source specification with the target specification item by item. A built-in rule base is used, and each rule generates a specification conversion function according to the conversion logic of "conflict type + source unit = conversion function template". For example, for conversion rule R-UNIT-01 (traffic unit conversion), the template is target_value=source×factor, and the factor is predefined as: 1Mbps=1×10 6 bps, 1Gbps = 1 × 10 9 bps, 1Kbps = 1 × 10³bps. For the source field `ingressRate` of an Ethernet port, with a sample value of 120Mbps, the target_value is generated according to the rule: target_value = source × 1000000.

[0048] For string fields, the system reads the first 10 records, automatically detects the character encoding, and triggers manual detection when the detection result is below 0.9. If the detected encoding is not UTF-8, a custom transcoding function is used to convert the entire field value to UTF-8. For characters that cannot be mapped, the record is converted to an exception log. A unified code mapping table is constructed, containing vendor codes, alarm level codes, region codes, etc. The mapping table is stored in the form of a relational data table, containing three fields: source_system (source system identifier), source_code (source code value), and target_code (unified code). For example, in the alarm field mapping table: source_system is "OTN_ALARM" or "PTN_ALARM", source_code is "CRITICAL", and target_code is "1". For each code value, the mapping table is searched for source_system + source_code. If a match is found, it is directly replaced with target_code; if a match fails, a graph neural network is used for inference. Based on the source code, target code, vendor, device type, and other information of each node, the probability of the target code is inferred, and the value with the highest probability and greater than the threshold is used as the mapping result. The unified coding sets for each technical domain are shown in Table 2.

[0049] Table 2 Unified Coding Sets for Technical Domains in Various Fields

[0050] Extract triples from historical mapping data, including source sample sequence, target sample sequence, and transformation operation sequence. For example, for the sample value "2023-04-15 10:30:00", the target value is "1681536600", and the transformation operation sequence is ["TO_UNIX_TIMESTAMP", "FORMAT", "yyyy-mm-dd hh:mm:ss"], construct an encoder-decoder training pair. For a given actual sample value as the input sequence, feed it into the trained model. The model autoregressively generates transformation operations and parses the sequence into specific transformation functions. For example, mapping the OTN network source field ODU BBE to the field BBE, with historical sample values ​​ranging from 0 to 65535 and the target specification requiring unsigned values, the output of the transformation function is the corresponding processed BBE value.

[0051] Step S203: Loading Process. The transformed standardized data is loaded into the source database. As an example, the source database is categorized by network type: OTN transmission communication network source database, Ethernet transmission communication network source database, PTN transmission communication network source database, and SDH transmission communication network source database. The source database employs a storage method combining relational and non-relational databases. Structured data is stored in the relational database for easy querying and statistics; semi-structured and unstructured data are stored in the non-relational database to improve data storage flexibility. When data in the data lake is updated, the data processing flow is automatically triggered to ensure real-time updates of the source database data and maintain data consistency.

[0052] S300: Based on standardized data from the source library, it adopts a bottom-up hierarchical construction method to build domain libraries and topic libraries in sequence; the domain libraries include basic support domain libraries and network professional domain libraries, and the topic libraries are divided according to professional network types.

[0053] Based on the standardized data in the source library, a bottom-up, layered construction approach is adopted to build the domain library. That is, the scattered network element collection data and performance data in the source library are converted into standardized structured data conforming to domain model specifications. The domain library includes two categories: a basic support domain library and a network professional domain library. It integrates common data such as basic device information, topology relationships, and dictionary mappings to build a unified basic support domain library; and it constructs network professional domain libraries for different professional communication services such as OTN, Ethernet, PTN, and SDH, aggregating performance and service model data for each specialty. As the core connecting layer between the source library and the topic library, the domain library, based on the standardized data in the source library, is divided into two categories: the basic support domain library and the network professional domain library, constructing a domain-based data support system for data governance, ensuring the orderly and professional flow of data.

[0054] As an example, the basic support domain library is used to store basic support data for network operation, including basic equipment configuration information, basic network topology information, operation and maintenance support data, and basic system data; the network professional domain library is used to store professional operation data of professional transmission communication networks, which are classified and aggregated according to the network professional type of OTN transmission network, Ethernet transmission network, PTN transmission network, and SDH transmission network.

[0055] Furthermore, basic equipment configuration information includes equipment model, installation location, hardware parameters, deployment time, and responsible maintenance personnel; basic network topology information includes the connection relationship between equipment and links, network area division, and network segment allocation; maintenance support data includes maintenance personnel information, permission configuration, maintenance process specifications, and fault handling standards; and basic system data includes data encoding rules, data permission standards, and data synchronization frequency. The core function of this basic support data is to provide unified foundational support for the entire data governance system, standardize the format and association standards of data at each level, ensure the consistency of data in the source database, topic database, and business service layer, and provide a foundational guarantee for subsequent data aggregation, correlation analysis, and cross-level data flow.

[0056] It should be noted that the network professional domain database focuses on the centralized management and control of core professional data in communication networks. Based on standardized data from the source database for OTN, Ethernet, PTN, and SDH transmission networks, it is categorized and aggregated according to network professional type. The purpose of this data is to achieve centralized management of communication network professional data, clarify the inherent relationships between professional data, eliminate irrelevant data, strengthen core professional data, and provide a precise and professional data foundation for the subsequent integration and construction of thematic databases.

[0057] Based on the network professional domain library data and the basic support domain library data in the domain library, a thematic library is constructed using a bottom-up hierarchical construction approach. Specifically, core data of the corresponding professional network is extracted from the network professional domain library and combined with basic data from the basic support domain library. A bottom-up hierarchical construction approach is used to perform cross-domain data association and multi-source data fusion to construct the thematic library.

[0058] As one example, the topic library includes an OTN transport topic library, an Ethernet transport network topic library, a PTN transport topic library, and an SDH transport topic library; each topic library is partitioned and stored according to business scenarios to form a data model oriented towards business scenarios.

[0059] It should be noted that the subject database is divided according to professional network types, including OTN transmission subject database, Ethernet transmission network subject database, PTN transmission subject database, and SDH transmission subject database, corresponding to the needs of communication transmission network service scenarios, achieving precise matching of data with service scenarios. Core data of corresponding subject domains are extracted from the network professional domain database and the basic support domain database, cross-domain data association is performed, multi-source data fusion is carried out, and the subject database is partitioned and stored according to service scenarios, forming a data model oriented towards service scenarios, realizing data fusion, association, and intelligent management.

[0060] The OTN transmission topic library takes OTN transmission professional data from the network professional domain library as its core, and the sub-topics built according to business needs include optical equipment topics, optical link topics, optical performance topics, and optical alarm topics, so as to realize centralized management of optical transmission network data.

[0061] Furthermore, the optical equipment topic includes equipment configuration, operating status, fault records, maintenance records, firmware version, etc.; the optical link topic integrates fiber optic link transmission-related data, including core performance indicators such as link connection relationships, optical power, bit error rate, transmission delay, and link loss; the optical performance topic includes overall operational performance data of the OTN transmission network, including performance summary data of all OTN transmission equipment and links in the network, performance threshold alarm data, etc.; the optical alarm topic includes various alarm information of the OTN transmission network, including alarm code, alarm level, alarm duration, alarm triggering conditions, alarm cause, alarm processing records, etc.

[0062] The Ethernet Transmission Network Theme Library takes Ethernet professional data from the network professional domain library as its core, integrates Ethernet standardized data, and constructs sub-themes according to business needs, including Ethernet device themes, Ethernet routing themes, Ethernet traffic themes, and Ethernet alarm themes, to achieve centralized management of Ethernet network data.

[0063] Furthermore, the Ethernet device topic refers to the operation and management data of Ethernet network devices, including configuration information, operating status, fault records, and port configurations of routers and switches; the Ethernet routing topic refers to the operation data of Ethernet routes, including routing configuration, routing forwarding paths, routing protocol parameters, routing anomaly records, and routing convergence time; the Ethernet traffic topic refers to the traffic data of the Ethernet network, including the overall network traffic distribution, traffic change trends, peak traffic, abnormal traffic, and traffic congestion warning data; and the Ethernet alarm topic refers to various alarm data of the Ethernet network, including alarm information such as port anomalies and routing failures, as well as corresponding processing details and processing results.

[0064] The PTN transmission topic library is based on PTN transmission professional data from the network professional domain library, and integrates PTN transmission network standardization data. The sub-topics built according to business needs include PTN equipment topics, PTN tag topics, PTN performance topics, and PTN alarm topics.

[0065] Furthermore, the PTN device topic refers to PTN transmission equipment data, including equipment configuration, operating status, fault records, maintenance records, firmware version, etc.; the PTN label topic refers to MPLS-TP label-related data in the link, including core performance indicator data such as label allocation and label forwarding path (LSP); the PTN performance topic refers to the overall operating performance data of the PTN transmission network, including the physical layer performance of PTN device ports (bit errors, packet loss, etc.), service tunnel performance summary data, performance change trend data, etc.; the PTN alarm topic refers to various alarm information of the PTN transmission network, including device port alarms and link alarms, including alarm code, alarm level, alarm duration, alarm triggering conditions, alarm cause, alarm processing records, etc.

[0066] The SDH transmission topic library takes SDH transmission professional data from the network professional domain library as its core, integrates SDH transmission network standardization data, and constructs sub-topics according to business needs, including SDH equipment topics, SDH link topics, SDH performance topics, and SDH alarm topics.

[0067] Furthermore, the SDH equipment topic refers to SDH transmission equipment data, including equipment configuration, operating status, fault records, maintenance records, firmware version, etc.; the SDH link topic refers to data related to the transmission logic and physical channels of SDH links, including core performance indicators such as optical power, bit error rate, and time slot allocation; the SDH performance topic refers to the overall operational performance data of the SDH transmission network, including performance summary data of all SDH transmission equipment and links in the network, performance change trend data, performance overruns, bit errors, etc.; and the SDH alarm topic refers to various alarm information of the SDH transmission network, including optical layer alarms and multiplex section alarms, including alarm code, alarm level, alarm duration, alarm triggering conditions, alarm cause, alarm processing records, etc.

[0068] By constructing the subject database, the integration, correlation, and intelligent management of network data from various disciplines have been achieved, providing accurate and high-quality data support for upper-layer business applications.

[0069] S400: Based on the subject library, a business service layer is built to provide data resource services and data asset services.

[0070] Based on the fused data from the subject library, a business service layer is generated, which includes diversified data services such as data resource services and data asset services, and provides governance data as application services through APIs.

[0071] As an example, the business service layer provides data resource services such as data query, data export, data subscription, and data interface call via API, as well as data asset services such as data asset inventory, classification, hierarchical management, and asset ledger maintenance, thereby improving data management and sharing capabilities.

[0072] As a core support service for data flow, data resource services provide basic functions such as data query, data export, data subscription, and data interface calls. It supports precise retrieval of data from various levels of libraries (source library, domain library, and topic library) by network type, data type, and time dimension, and establishes data flow channels to ensure the orderly flow of data between various levels and services.

[0073] Data asset services focus on data security and asset management, enabling data asset inventory, classification, and hierarchical management, establishing a complete data asset ledger, clarifying the ownership, permissions, and lifecycle of data at each level, and ensuring the integrity and security of data assets.

[0074] By constructing a business service layer, we have achieved end-to-end governance of communication network data, standardized data management processes, established unified data standards, resolved data differences among various communication networks, and formed communication post source libraries, domain libraries, and topic libraries. We also provide corresponding data services through APIs, providing data support for subsequent typical applications and demonstrating broad application prospects.

[0075] On the other hand, the present invention also provides a data governance system for multi-source heterogeneous communication networks, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any of the above-described methods.

[0076] Specifically, the system includes: Edge intelligent nodes are used to draw data from multi-source heterogeneous communication network data sources. Edge intelligent nodes adopt a socket architecture of main service plus microservice components. Edge intelligent nodes can intelligently adapt and load the corresponding microservice components through the main service to realize the access of different data sources. The source library construction module is used to extract, transform, and load the imported data in the cloud data center, remove invalid data, unify the data format, and load it into the corresponding source library according to the network type. The domain library construction module is used to build the domain library based on the standardized data of the source library. It adopts a bottom-up hierarchical construction method. The domain library includes a basic support domain library and a network professional domain library. The topic library construction module is used to build the topic library based on the network professional domain library data and the basic support domain library data of the domain library. It adopts a bottom-up hierarchical construction method to build the topic library, which is divided according to professional network type. The business service layer construction module is used to build a business service layer based on the theme library, providing data resource services and data asset services.

[0077] Furthermore, the source library construction module includes an extraction processing unit, a transformation processing unit, and a loading processing unit.

[0078] It should be noted that the conversion processing unit refers to the module that performs specific exchange processing and transmission on the collected data; it further includes a data cleaning unit, a data format unification unit, a data association unit, and a data completion unit.

[0079] like Figure 4 As shown, the data format unification unit further includes a data dictionary module, a field mapping agent module, a specification unification agent module, and an encoding conversion agent module.

[0080] The data dictionary module is a semantic layer designed to address the inconsistencies in network technology domains among OTN, PTN, SDH, and Ethernet. The data dictionary establishes a unified business semantic mapping relationship for each communication network domain from dimensions such as resource entities, attribute fields, and performance indicators.

[0081] The field mapping agent module includes a semantic understanding module and a transformation logic generation module. The agent extracts semantic features from the source component fields using a trained model, compares these features with uniform fields in the resource dictionary, selects the top k candidate target fields with the highest similarity, and outputs the final matched target field.

[0082] The semantic understanding module constructs an input text sequence. For each source component data, it constructs an input string T_src = field_name + "[SEP]" + comment + "[SEP]" + sample_value, where field_name is the field name, comment is the annotation field, and sample_value is the sample value extracted from historical performance. For each source component field, T_src is used as the semantic embedding of the input BERT model. Its similarity score with the unified field in the resource dictionary is calculated. The top k keywords with the highest scores are selected as candidates, and the unified field corresponding to the highest score is selected as the mapping result.

[0083] The transformation logic generation module uses a Transformer-based Seq2seq model to generate transformation expressions. It extracts triplets (source sample sequence, target sample sequence, and transformation operation sequence) from historical mapping data to construct encoder-decoder training pairs. Given actual sample values ​​as input sequences, these are fed into the trained model, which autoregressively generates transformation operations and parses the sequences into specific transformation functions.

[0084] The specification unification intelligent agent module detects and resolves differences in units, precision, and value ranges within the communication network technology domain, ensuring data comparability. After acquiring the field mapping relationship, the agent reads sample data of the field and the specification definition of the target field, detects for conflicts, and if conflicts exist, automatically generates a specification conversion function based on the built-in normalization rule engine module to convert the data into data conforming to the specification. Furthermore, the specification unification intelligent agent module includes a specification conflict detection module and a normalization rule engine module. The specification conflict detection module automatically reads the target specification definition, then reads the target field's specification metadata from the resource dictionary, scans the most recent multiple data samples, statistically analyzes the numerical distribution, unit flags, actual number of digits, and timestamp intervals, analyzes the actual specification, and generates a conflict list by comparing the source specification with the target specification item by item. The normalization rule engine module has a built-in rule library; each rule generates conversion logic according to the conflict type + source unit = conversion function template.

[0085] The encoding conversion intelligent agent module is responsible for resolving inconsistencies between character encoding and business code encoding. It detects the encoding of source data fields; if it is not UTF-8, it converts it uniformly and extracts the encoding from the source field for comparison with the unified encoding maintained in the data dictionary. If they do not match, an alarm is reported. The encoding conversion intelligent agent module includes a character encoding detection module and a business code mapping module. For string fields, the character encoding detection module reads the first 10 records and automatically detects the character encoding. If the detection result is below 0.9, manual detection is triggered. If the detected encoding is not UTF-8, a custom conversion function is used to convert the entire field value to UTF-8. For characters that cannot be mapped, the record is converted to an exception log. The business code mapping module constructs a unified code mapping table, including vendor codes, alarm level codes, and region codes. The mapping table is stored in relational data table format and contains three fields: source_system (source system identifier), source_code (source code value), and target_code (unified code). For each code value, search the mapping table for source_system + source_code. If a match is found, replace it directly with target_code. If no match is found, use a graph neural network to infer the probability of the target code based on information such as the source code, target code, manufacturer, and device type of each node. The value with the highest probability and greater than the threshold is used as the mapping result.

[0086] By using an agent-based field semantic mapping and unified exchange mechanism, intelligent, automated, and standardized conversion of cross-network data is achieved. Compared with the traditional method of manually writing mapping rules, it can adaptively handle field differences between different networks and different manufacturers, significantly improving the efficiency and accuracy of data standardization processing.

[0087] The specific implementation methods of each module are consistent with the descriptions of the corresponding steps in the above methods, and will not be repeated here.

[0088] In summary, this invention proposes a data governance method and system for multi-source heterogeneous communication networks. First, it uses edge intelligent nodes to collect data from different types of data sources, such as OTN, Ethernet, PTN, and SDH transmission communication networks. The edge intelligent nodes adopt a socket-style architecture combining a main service and microservice components. The main service actively identifies and parses the characteristic information of the access devices, dynamically makes decisions, and loads the corresponding microservice components, thereby establishing a data channel with the target network. This enables unified collection and centralized storage of multi-source heterogeneous data, constructing a unified data lake in the cloud.

[0089] Then, in the cloud data center, the imported data is sequentially extracted, transformed, and loaded. The transformation process includes data cleaning, data format standardization, data association, and data completion: data cleaning uses outlier detection algorithms to remove invalid data and deduplication algorithms to remove duplicate data; data format standardization involves constructing a data dictionary, extracting semantic features of source fields using a trained model, comparing these features with a standard resource dictionary to generate field mapping relationships, and then automatically generating complete transformation expressions using a Transformer-based Seq2seq model, combining specification conversion functions and encoding conversion rules to standardize data of different formats into a preset JSON format; data completion uses interpolation or prediction algorithms based on historical data to fill in missing values. The processed data is then cleaned of invalid information and formatted uniformly, forming a standardized source library.

[0090] Finally, in the layered construction of data governance, a bottom-up approach is adopted: based on the source library, domain libraries are built according to network type, including basic support domain libraries and network professional domain libraries, to achieve the classification and aggregation of basic support data and professional network data; then, based on each domain library, thematic libraries for specific business scenarios are further divided and built; finally, based on the fused data of each professional thematic library, a business service layer is built to provide core APIs such as data resource services and data asset services, providing solid data support for subsequent typical applications and having broad application prospects.

[0091] It should be noted that, for the sake of simplicity, the foregoing embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0092] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0093] In the several embodiments provided in this application, it should be understood that the disclosed methods or systems can be implemented in other ways. For example, the embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0094] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0095] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0096] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0097] Those skilled in the art will understand that all or part of the circuits in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0098] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of embodiments of this disclosure upon considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

[0099] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0100] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data governance method for multi-source heterogeneous communication networks, characterized in that, The method includes: Data is fed from multi-source heterogeneous communication network data sources through edge intelligent nodes; the edge intelligent nodes adopt a socket architecture of main service plus microservice components, and the edge intelligent nodes intelligently adapt and load the corresponding microservice components through the main service to realize the access of different data sources; The data is extracted, transformed, and loaded in the cloud data center. Invalid data is removed and the data format is unified. The data is then loaded into the corresponding post source library according to the network type. Based on the standardized data in the source library, a bottom-up hierarchical construction method is adopted to build a domain library and a topic library in sequence; the domain library includes a basic support domain library and a network professional domain library, and the topic library is divided according to professional network type; A business service layer is built based on the aforementioned subject library to provide data resource services and data asset services.

2. The data governance method for multi-source heterogeneous communication networks according to claim 1, characterized in that, The edge intelligent node loads corresponding microservice components through intelligent adaptation of the main service to enable access to different data sources, specifically including: The edge intelligent node breaks down the full communication protocol into independent microservice components; The main service actively identifies access devices, parses device feature information to determine the device type and the type of protocol to be loaded; The main service makes dynamic decisions based on the identification results and loads the corresponding microservice components from the component repository to establish a data channel with the target network; The main service manages the lifecycle of the loaded microservice components, monitors their running status, and releases resources when the connection is interrupted.

3. The data governance method for multi-source heterogeneous communication networks according to claim 1, characterized in that, The transformation process includes data cleaning, data format unification, data association, and data completion. Data cleaning removes invalid data, including null values, outliers, and duplicates. An outlier detection algorithm is used to identify and remove outliers, and a deduplication algorithm is used to remove duplicates. Data format unification converts data from different formats into a preset standard JSON format. Data association establishes relationships between data from different data sources. Data completion fills in missing key data using interpolation or prediction algorithms based on historical data.

4. The data governance method for multi-source heterogeneous communication networks according to claim 3, characterized in that, The unified data format specifically includes: Construct a data dictionary and establish a unified business semantic mapping relationship for each communication network domain; The semantic features of the source field are extracted using the trained model, and the similarity is compared with the unified fields in the standard resource dictionary to select the matching target field and obtain the field mapping relationship. A transformation expression is generated based on the field mapping relationship to convert the source data into data that conforms to the target field specification.

5. A data governance method for multi-source heterogeneous communication networks according to claim 4, characterized in that, After obtaining the field mapping relationship, the following is also included: It detects differences between the source field and the target field in terms of units, precision, and value range, generates a conflict list, and automatically generates a specification conversion function based on the built-in rule library; The system automatically detects the character encoding of the source data. If it is not UTF-8, it is uniformly converted to UTF-8, and a unified code mapping table is constructed to map the business code of the source system to the unified business code, thus obtaining the encoding conversion rules. By integrating the field mapping relationships, specification conversion functions, and encoding conversion rules, a complete conversion expression is generated using a Transformer-based Seq2seq model.

6. A data governance method for multi-source heterogeneous communication networks according to any one of claims 1 to 5, characterized in that, The source library is divided into OTN transmission communication network source library, Ethernet transmission communication network source library, PTN transmission communication network source library and SDH transmission communication network source library according to network type. The source library adopts a storage method that combines relational database and non-relational database, wherein structured data is stored in relational database, and semi-structured and unstructured data is stored in non-relational database.

7. A data governance method for multi-source heterogeneous communication networks according to claim 6, characterized in that, The basic support domain library is used to store basic support data for network operation, including basic equipment configuration information, basic network topology information, operation and maintenance support data, and basic system data; the network professional domain library is used to store professional operation data of professional transmission communication networks, which are classified and aggregated according to the network professional type of OTN transmission network, Ethernet transmission network, PTN transmission network, and SDH transmission network.

8. A data governance method for multi-source heterogeneous communication networks according to claim 7, characterized in that, The subject libraries include OTN transmission subject libraries, Ethernet transmission network subject libraries, PTN transmission subject libraries, and SDH transmission subject libraries; each subject library is partitioned and stored according to business scenarios to form a data model oriented towards business scenarios.

9. A data governance method for multi-source heterogeneous communication networks according to claim 8, characterized in that, The OTN transmission topic library is based on OTN transmission professional data in the network professional domain library. The sub-topics built according to business needs include optical equipment topic, optical link topic, optical performance topic, and optical alarm topic. The Ethernet transmission network topic library takes Ethernet professional data from the network professional domain library as its core, integrates Ethernet standardized data, and constructs sub-topics according to business needs, including Ethernet device topics, Ethernet routing topics, Ethernet traffic topics, and Ethernet alarm topics. The PTN transmission topic library is based on PTN transmission professional data in the network professional domain library, and integrates PTN transmission network standardization data. The sub-topics built according to business needs include PTN device topics, PTN tag topics, PTN performance topics, and PTN alarm topics. The SDH transmission topic library is based on SDH transmission professional data in the network professional domain library, and integrates SDH transmission network standardization data. The sub-topics constructed according to business needs include SDH equipment topics, SDH link topics, SDH performance topics, and SDH alarm topics.

10. A data governance system for multi-source heterogeneous communication networks, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 9.

Images