A multi-tenant hierarchical control-based unified data collection method for a telecommunications network

Abstract

The application discloses a kind of unified data acquisition methods of telecommunication network based on multi-tenant hierarchical control, belong to telecommunication network management technical field, static acquisition template is generated with initial acquisition task, wash and standardize original data, store standard data set, dynamic tenant request processing, combined with hierarchical flow limiting, monitor whole link state including solve the technical problem of the quick response of data acquisition, priority control and whole process monitoring, the application improves data acquisition efficiency and quality, guarantees the timeliness, accuracy of data acquisition, to adapt to the rapid update of network equipment, realize the "zero code" adaptation of new equipment access, establish dynamic resource allocation mechanism based on tenant level, construct end-to-end acquisition quality monitoring system.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of telecommunication network management, and particularly relates to a unified data collection method for a telecommunication network based on multi-tenant hierarchical control. BACKGROUND

[0002] In the existing telecommunication network management system, with the popularization of 5G, IPv6 and other technologies, the equipment renewal brought by the upgrading and reconstruction of mobile and fixed networks makes the data collection problem of the existing network element adaptation layer more prominent:

[0003] New access equipment has a long cycle, and when new equipment is connected to the existing network, the original professional network management needs to be developed again, which cannot support quickly, and the data collection and application are tightly coupled, resulting in repeated development and a long cycle.

[0004] The collection task lacks priority control, and there is repeated collection, resource preemption, data cannot be synchronized in time, and key tasks are difficult to guarantee.

[0005] The collection process lacks effective monitoring and early warning, there is no audit monitoring log, the data integrity and accuracy cannot be guaranteed, and problem troubleshooting is difficult. SUMMARY

[0006] The purpose of the application is to provide a unified data collection method for a telecommunication network based on multi-tenant hierarchical control, which solves the technical problems of fast response, priority control and full-process monitoring of data collection.

[0007] To achieve the above purpose, the application adopts the following technical solutions:

[0008] A unified data collection method for a telecommunication network based on multi-tenant hierarchical control, comprising the following steps:

[0009] Step 1: The dynamic hierarchical collection engine provides a collection instruction template, which is configured by the operation and maintenance personnel; the dynamic hierarchical collection engine generates collection instructions for various target equipment according to the configured collection instruction template, generates data collection tasks automatically according to the data collection tasks, and collects the original data of the network element type, link type and tenant awareness type generated by the target equipment;

[0010] Step 2: The data cleaning engine calls the original data, pre-processes the original data, verifies the data integrity using an LSTM-based time series prediction model, unifies the format of the pre-processed data, obtains standard data, and generates processing operation logs;

[0011] Step 3: The fast data storage module calls the standard data, selects the corresponding storage mode according to the data type and query performance, performs data extraction, format standardization and storage management, and outputs an accessible data set;

[0012] Step 4: When a tenant issues a data collection request to the dynamic hierarchical collection engine, the execution module verifies the tenant's identity and determines whether the current system thread count has reached the flow control threshold: if yes, the tenant flow control is performed; if no, the two-level flow control engine is used to control the tenant flow control, which specifically includes device-level flow control and network-level flow control according to the tenant level, the device-level flow control uses the token bucket algorithm to control the flow of ordinary tenants, and the network-level flow control uses the weighted round-robin algorithm and the preemptive resource allocation method to control the flow of important tenants and special tenants; a new data collection task is generated according to the flow control result, and a new collection task list is constructed, and the original data of the target device is collected according to the new collection task list;

[0013] In network-level flow control, the PID controller is used to shape the network traffic of important tenants and special tenants, dynamically adjust the network device forwarding rules, and optimize the execution of key collection instructions;

[0014] Before executing any new data collection task, the corresponding collection instruction is matched and compared with the historical instructions in the cache database: if matched, the accessible data set is accessed to find the matching standard data and output; if not matched, the collection instruction is issued to the target device through the protocol adapter, and the real-time original data returned by the target device is received and saved, and output;

[0015] Step 5: The monitoring module monitors the collection state, preprocessing progress, storage state and flow control state of the original data, generates a monitoring result report and alarm information, and displays them through a visual display interface.

[0016] Preferably, in step 1, the dynamic hierarchical collection engine performs the following steps:

[0017] Step 1-1: Generate an atomized collection instruction template and an instruction cache pool through a protocol adapter, and configure the collection instruction in the instruction cache pool to the collection instruction template by an operation and maintenance personnel;

[0018] The collection instruction template includes device type, protocol and data type, and the data type includes network element type, link type and tenant awareness type;

[0019] Step 1-2: Generate a data collection task for the target device according to the device type, protocol and data type, and generate an original data collection task list according to the preset static collection rule;

[0020] Step 1-3: According to the original data collection task list, the collection instruction is issued to the target device through the protocol adapter, the original data returned by the target device is received and cached;

[0021] The type of original data includes network element type, link type and tenant awareness type.

[0022] Preferably, when performing step 2, the data cleaning engine specifically performs the following steps:

[0023] Step 2-1: Call the original data, perform integrity detection, accuracy detection, consistency detection and repeatability detection on the original data, and generate a detection report;

[0024] Step 2-2: According to the detection report, the missing value processing, error value correction, abnormal value processing and repeated value deletion processing are carried out on the original data, and the cleaned data is obtained;

[0025] Step 2-3: The preprocessed data is obtained by performing format conversion, data normalization, encoding conversion and data aggregation on the cleaned data;

[0026] Step 2-4: Construct a data integrity verification model based on LSTM time series prediction to verify the data integrity of the preprocessed data;

[0027] At the same time, an abnormality judgment model is constructed to detect and judge the abnormality of the preprocessed data, and the specific formula is as follows:

[0028] ;

[0029] Wherein, is the standard deviation of the sliding window; represents the original data value actually collected from the data source, represents the predicted value predicted by the LSTM-based time series prediction model; Alert represents the abnormality detection judgment value, which takes the value of 1 or 0;

[0030] Step 2-5: According to the result of step 2-4, the verified standard data is obtained;

[0031] Step 2-6: Establish a rule management module to configure and manage various rules in the preprocessing process of steps 2-1 to 2-3, including data detection rules, cleaning rules and conversion rules; record processing log and running state;

[0032] Step 2-7: Establish a metadata management module to manage the metadata involved in the process of steps 2-1 to 2-5, including data source metadata, cleaning rule metadata and conversion rule metadata; data source metadata includes database table structure and field type.

[0033] Preferably, when performing step 3, the storage scheme selected by the fast data storage module includes memory storage scheme, database storage scheme and file system storage scheme.

[0034] Preferably, when performing step 4, the dynamic hierarchical collection engine specifically performs the following steps:

[0035] Step 4-1: The tenant initiates a data collection request to the dynamic hierarchical collection engine, which verifies the legitimacy of the tenant's identity, including the tenant's name, token, and permission level;

[0036] When the verification is passed, step 4-2 is executed; when the verification is not passed, an error message is returned that the tenant is invalid;

[0037] Step 4-2: Real-time monitoring of the current number of threads, if the preset flow limiting threshold is reached, the tenant's collection data request is terminated immediately and returned; otherwise, step 4-3 is executed;

[0038] Step 4-3: Determine the tenant's permissions and implement different flow limiting methods according to the tenant's permissions:

[0039] For ordinary tenants: device-level flow limiting is used, specifically, a fixed-capacity token bucket is initialized for each ordinary tenant, and tokens are continuously added to the bucket at a preset rate. The tenant consumes the corresponding number of tokens when requesting resources, and if the tokens are insufficient, the request is rejected. Real-time monitoring of the token bucket state, dynamic adjustment of token generation rate and bucket capacity according to overall load and tenant behavior;

[0040] For important tenants or special tenants: network-level flow limiting is used, specifically, a PID controller is used for network traffic shaping, and different tenant level traffic shaping strategies are set. The traffic shaping strategy is sent to the network device; the PID controller monitors network traffic in real time, and dynamically adjusts the forwarding rules of the network device according to the monitoring data and the strategy; for special tenants, when they execute key commands, the PID controller interrupts the non-key traffic of other level tenants, ensuring that the special tenant's traffic is prioritized;

[0041] Step 4-5: Generate new data collection tasks according to the flow limiting results, and build a new collection task list;

[0042] Step 4-6: Before executing any new data collection task, match the collection instructions corresponding to the new data collection task with the historical instructions in the cache database: if they match, access the accessible data set, find the matching standard data, and output; if they do not match, send the collection instructions to the target device through the protocol adapter, receive and save the real-time raw data returned by the target device, and output and cache;

[0043] Step 4-7: Return the results of step 4-6 to the tenant requestor.

[0044] The present invention discloses a unified data acquisition method for telecommunications networks based on multi-tenant hierarchical control, which solves the technical problems of rapid response, priority control and full-process monitoring of data acquisition. The present invention improves the efficiency and quality of data acquisition, ensures the timeliness and accuracy of data acquisition, adapts to the rapid upgrading of network equipment, realizes "zero-code" adaptation for new equipment access, establishes a dynamic resource allocation mechanism based on tenant level, and constructs an end-to-end acquisition quality monitoring system. Attached Figure Description

[0045] Figure 1 This is the main flowchart of the present invention;

[0046] Figure 2 This is a system architecture diagram of the present invention;

[0047] Figure 3 This is a flowchart of step 4 of the present invention;

[0048] Figure 4 This is a logic flowchart of the two-level rate limiting engine of the present invention. Detailed Implementation

[0049] Depend on Figures 1-4 The method for unified data collection in a telecommunications network based on multi-tenant hierarchical control, as shown, includes the following steps:

[0050] Step 1: The dynamic hierarchical acquisition engine provides acquisition instruction templates, which are configured by operation and maintenance personnel. Based on the configured acquisition instruction templates, the dynamic hierarchical acquisition engine generates acquisition instructions for various target devices according to device type, protocol and data type, automatically generates data acquisition tasks, and collects the original data of network element type, link type and tenant awareness type generated by the target devices according to the data acquisition tasks.

[0051] When executing step 1, the dynamic hierarchical acquisition engine performs the following steps:

[0052] Step 1-1: Generate atomic acquisition instruction templates and instruction cache pools through the protocol adapter. The operation and maintenance personnel then retrieve the acquisition instructions from the instruction cache pool and configure the acquisition instruction templates.

[0053] The data collection instruction template includes device type, protocol, and data type. Data types include network element type, link type, and tenant awareness type.

[0054] Steps 1-2: Generate data acquisition tasks for the target device according to device type, protocol, and data type; generate a list of raw data acquisition tasks based on preset static acquisition rules.

[0055] Steps 1-3: Based on the original data acquisition task list, send acquisition commands to the target device through the protocol adapter, receive the original data returned by the target device, and cache it;

[0056] The types of original data include network element type, link type and tenant awareness type.

[0057] In this embodiment, a protocol adapter (SNMP / Netconf / FTP, etc.) + instruction cache pool technology is adopted, and the professional, network, collection protocol and device type are used as dimensions, the atomized collection instruction is used, and the template configuration service is provided.

[0058] The data collection types include network element type (configuration, resource, performance and alarm), link type (signaling and user plane) and user awareness type (complaint and dial test). Since the network data collection involves many professionals, network elements and long duration, in order to avoid repetition, facilitate backtracking and data processing, the file name and file format are usually specified.

[0059] The collection instruction configuration provides the self-defined configuration device collection instruction.

[0060] The collection task configuration collects the configured collection instructions, and provides the self-defined collection task configuration function.

[0061] Step 2: The data cleaning engine calls the original data, pre-processes the original data, verifies the data integrity by using the LSTM-based time series prediction model, unifies the format of the pre-processed data, obtains the standard data, and generates a processing operation log;

[0062] When step 2 is performed, the data cleaning engine specifically performs the following steps:

[0063] Step 2-1: Call the original data, and perform integrity detection, accuracy detection, consistency detection and repeatability detection on the original data to generate a detection report.

[0064] Integrity detection: Check whether the data record has missing values, such as empty columns or incomplete records, and at the same time, count the proportion of missing data to evaluate the integrity degree of the data.

[0065] Accuracy detection: According to the preset business rules and data type specifications, the correctness of the data is verified. For example, check whether the date field conforms to the date format, whether the numerical value field is within a reasonable value range, and perform format verification on specific fields such as ID card number and mobile phone number.

[0066] Consistency detection: Ensure that the same data is expressed consistently in different tables or different records. For example, check whether the names of the same customer in different tables are uniform, and whether there are contradictions in the address information.

[0067] Repeatability detection: By setting a unique identification field or calculating a data fingerprint, repeated records are identified to avoid data redundancy interference on subsequent analysis.

[0068] Step 2-2: According to the detection report, missing value processing, error correction, outlier processing and repeated value deletion processing are performed on the original data to obtain cleaned data.

[0069] Missing value processing: Multiple processing strategies are supported, such as deleting records containing missing values, filling missing values of numerical and categorical data with mean, median or mode, and predicting missing values through machine learning algorithms.

[0070] Error correction: According to the preset rules and reference data, the error data is corrected. For example, the address library is used to correct the wrong address information, and the dictionary table is used to replace the misspelled words.

[0071] Repeated value deletion: According to the detected repeated records, one of them is retained and the rest is deleted to ensure the uniqueness of the data.

[0072] Outlier processing: Statistical methods (such as 3σ principle) or machine learning algorithms are used to identify abnormal data, and according to business needs, the abnormal values are deleted as noise data or corrected.

[0073] Step 2-3: Format conversion, data normalization, encoding conversion and data aggregation are performed on the cleaned data to obtain preprocessed data.

[0074] Format conversion: The storage format of data is converted, such as converting the date format from “YYYY-MM-DD” to “DD / MM / YYYY”, and converting numerical data to a specified precision.

[0075] Data normalization: Numerical data of different dimensions are converted to the same value range, common methods include min-max normalization, Z-score standardization, etc., which facilitates data comparison and analysis.

[0076] Encoding conversion: Encoding processing is performed on categorical data, such as converting “male” and “female” in the gender field to numerical codes “0” and “1”, or using one-hot encoding to meet the requirements of machine learning algorithms and other applications.

[0077] Data aggregation: Data is aggregated according to specific dimensions, such as summing and averaging sales data by month and region to generate aggregated data tables.

[0078] Step 2-4: Construct a data integrity verification model based on LSTM time series prediction to verify the data integrity of the preprocessed data.

[0079] The data integrity verification model based on LSTM time series prediction is as follows:

[0080] The input layer receives a historical 5-minute data volume sequence , a prediction window = 30s:

[0081] ;

[0082] wherein X represents time series data, t represents a time index, n is a historical window size, τ is a prediction step, and θ is a set of parameters (weights and biases) of the LSTM neural network model.

[0083] Network structure: 2-layer LSTM (hidden units 128) + 1-layer Dense

[0084] Loss function: Huber Loss ( ).

[0085] At the same time, an anomaly determination model is constructed to perform anomaly detection and determination on the preprocessed data, and the specific formula is as follows:

[0086] ;

[0087] wherein is a sliding window standard deviation, and the window size = 60 sampling points. represents an original data value actually collected from a data source, represents a predicted value predicted by the LSTM-based time series prediction model; Alert represents an anomaly detection and determination value, and takes a value of 1 or 0;

[0088] Step 2-5: According to the result of step 2-4, the verified standard data is obtained.

[0089] Step 2-6: A rule management module is established to configure and manage various rules in the preprocessing process of steps 2-1 to 2-3, including data detection rules, cleaning rules and conversion rules; record processing logs and running states;

[0090] In this embodiment, the rule management module is used to configure and manage various rules in the data cleaning process, including data detection rules, cleaning rules and conversion rules. Users can create, modify and delete rules through a visual interface, support version management and backup recovery of rules. In addition, the module also has the function of setting the priority of rules, which can be executed according to the priority order when there is a conflict between multiple rules, to ensure the accuracy and consistency of data cleaning work.

[0091] Step 2-7: Establish a metadata management module to manage the metadata involved in the process of steps 2-1 to 2-5, including data source metadata, cleaning rule metadata and conversion rule metadata; the data source metadata includes database table structure and field type.

[0092] In this embodiment, the metadata management module is responsible for managing the metadata involved in the data cleaning process, including data source metadata (such as database table structure, field type), cleaning rule metadata, conversion rule metadata, etc. It provides metadata browsing, querying, editing functions, supports automatic discovery and collection of metadata, helps users better understand the data cleaning process and data blood relationship by establishing the association between metadata, and provides support for data governance.

[0093] Step 3: The fast data storage module retrieves standard data, selects the corresponding storage mode according to the data type and query performance, performs data extraction, format standardization and storage management, and outputs the accessible data set;

[0094] When performing step 3, the storage scheme selected by the fast data storage module includes memory storage scheme, database storage scheme and file system storage scheme.

[0095] Step 4: When a tenant issues a data collection request to the dynamic hierarchical collection engine, the execution module verifies the tenant's identity, determines whether the current system thread number reaches the flow limiting threshold: yes, then perform tenant flow control; no, then use two-level flow limiting engine to control the flow of tenants, which specifically includes implementing device-level flow control and network-level flow control according to the tenant level, the device-level flow control uses the token bucket algorithm to control the flow of ordinary tenants, and the network-level flow control uses the weighted round robin algorithm and the pre-emptive resource allocation method to control the flow of important tenants and special tenants; generate a new data collection task according to the flow limiting result, and construct a new collection task list, and collect the original data of the target device according to the new collection task list;

[0096] When network-level flow control is performed, the PID controller is used to shape the network traffic of important tenants and special tenants, dynamically adjust the network device forwarding rules, and optimize the execution of key collection instructions;

[0097] Before executing any new data collection task, match and compare its corresponding collection instruction with the historical instructions in the cache database: if they match, access the accessible data set to find the matching standard data and output; if they do not match, issue the collection instruction to the target device through the protocol adapter, receive and save the real-time original data returned by the target device, and output;

[0098] When performing step 4, the dynamic hierarchical collection engine specifically performs the following steps:

[0099] Step 4-1: The tenant initiates a data collection request to the dynamic hierarchical collection engine, and the dynamic hierarchical collection engine verifies the legality of the tenant's identity, including verifying the tenant name, token and permission level;

[0100] When the verification passes, step 4-2 is executed; when the verification fails, an error message that the tenant is invalid is returned;

[0101] Step 4-2: Real-time monitoring of the current number of threads is performed, and if the preset flow limiting threshold is reached, the data collection request issued by the tenant is immediately terminated and returned; otherwise, step 4-3 is executed.

[0102] Step 4-3: The tenant permission is judged, and different flow limiting methods are implemented according to the tenant permission:

[0103] For ordinary tenants: device-level flow limiting is adopted, specifically, a fixed-capacity token bucket is initialized for each ordinary tenant, and tokens are continuously added to the bucket at a preset rate, and the tenant consumes a corresponding number of tokens when requesting resources, and if the tokens are insufficient, the request is rejected; the token bucket state is monitored in real time, and the token generation rate and bucket capacity are dynamically adjusted according to the overall load and tenant behavior;

[0104] For important tenants or special-level tenants: network-level flow limiting is adopted, specifically, a PID controller is used for network traffic shaping, and traffic shaping strategies for different tenant levels are set and issued to network devices; the PID controller monitors network traffic in real time, and dynamically adjusts the forwarding rules of network devices according to the monitoring data and strategies; for special-level tenants, when they execute key commands, the PID controller interrupts the non-key traffic of tenants of other levels to ensure that the traffic of special-level tenants is prioritized;

[0105] Step 4-5: New data collection tasks are generated according to the flow limiting results, and a new collection task list is constructed;

[0106] Step 4-6: Before executing any new data collection task, the corresponding collection instruction is matched and compared with the historical instructions in the cache database: if they match, the accessible data set is accessed, the matching standard data is found, and outputted; if they do not match, the collection instruction is issued to the target device through the protocol adapter, and the real-time raw data returned by the target device is received and saved, and outputted and cached;

[0107] Step 4-7: The results of step 4-6 are returned to the tenant requestor.

[0108] In this embodiment, the efficiency and security of data collection are guaranteed through multi-level authentication and dynamic flow limiting mechanism:

[0109] Tenant authentication: First, the identity legitimacy of the requesting tenant is authenticated to ensure that only authorized users can trigger collection tasks.

[0110] Tenant flow limiting: Real-time monitoring of the current number of threads is performed, and if the preset flow limiting threshold is reached, the request is immediately terminated and returned, avoiding system overload.

[0111] Network element flow control and hierarchical strategy:

[0112] Implement differentiated flow control according to tenant level.

[0113] Ordinary user: Hard concurrent limit (token bucket algorithm), strictly limit the number of concurrent connections, exceed the maximum value directly reject new connection, guarantee system stability;

[0114] Important user: Queue priority scheduling (weighted round robin algorithm), exceed the maximum number of concurrent connections, request automatically into the queue, according to the priority order processing, balance resource allocation;

[0115] Special user: Can interrupt other tasks (preemptive resource allocation), enjoy the highest priority, execute network element command can force interrupt other tenants of the task being executed, ensure real-time response of critical business.

[0116] Two-level flow control engine:

[0117] This embodiment proposes a two-level flow control engine, including device-level flow control and network-level flow control. Device-level flow control uses token bucket algorithm, suitable for ordinary tenants to control their use of computing resources. Network-level flow control is based on PID traffic shaping technology, suitable for important and special tenants to control their network traffic.

[0118] Device level: suitable for ordinary tenants. Reasonable flow control of collection tasks at device level solves the problem of resource preemption and insufficient collection performance caused by frequent collection tasks in the past.

[0119] Add tokens to the bucket at a fixed rate, the capacity of the bucket is limited. When a tenant requests resources, it must take out the corresponding number of tokens from the bucket, if the token in the bucket is insufficient, the request is rejected.

[0120] Initialize token bucket: initialize a token bucket for each ordinary tenant, set the size (maximum number of tokens) and token addition rate of the bucket.

[0121] Token addition: the system adds tokens to the token bucket at a preset rate.

[0122] Resource request processing: when a tenant requests resources, check the number of tokens in its token bucket. If enough, take out the corresponding number of tokens from the bucket and handle the request; if not enough, reject the request.

[0123] Monitoring and adjustment: real-time monitoring of the state of the token bucket, dynamically adjust the token addition rate and the size of the bucket according to the system load and tenant behavior.

[0124] Network level: Suitable for important / special level tenants. PID-based traffic shaping controls network traffic by centrally controlling network traffic, achieving precise control of network traffic for important and special level tenants. PID controller dynamically adjusts the forwarding rules of network devices according to network policy and tenant level to control the entry and output of traffic.

[0125] PID controller configuration: Configure the PID controller to define traffic shaping policies for different tenant levels.

[0126] Policy delivery: Deliver traffic shaping policies to network devices (such as switches, routers).

[0127] Traffic monitoring and adjustment: PID controller monitors network traffic in real time and dynamically adjusts the forwarding rules of network devices according to monitoring data and policies.

[0128] Priority processing: For special level tenants, when executing critical commands, the PID controller can interrupt non-critical traffic of other tenants to ensure that special level tenant traffic is prioritized.

[0129] Mathematical modeling of dynamic flow limiting algorithm:

[0130] Tenant priority weight calculation:

[0131] Define the weight coefficient of tenant ui at time t:

[0132] ;

[0133] : Preset priority (special level = 3, important = 2, ordinary = 1);

[0134] : Current system load rate (0-100%);

[0135] : Last successful execution time of this tenant;

[0136] Coefficient = 0.6, = 0.3, = 0.1 (optimized through network search);

[0137] Thread allocation control law:

[0138] Use PID controller to dynamically adjust thread pool size:

[0139] ;

[0140] : Actual load deviation from target load in the first period. Parameter setting value: = 1.2, = 0.5, = 0.3.

[0141] Step 5: The monitoring module monitors the collection state, preprocessing progress, storage state, and flow limiting state of the original data, generates monitoring result reports and alarm information, and displays them through a visual display interface.

[0142] In this embodiment, the monitoring module monitors and records the running state of the data cleaning engine in real time:

[0143] Running monitoring: Real-time display of data cleaning task execution progress, resource usage (such as CPU, memory occupation), data processing volume, etc. The running state of the task is presented through a visual interface, and alarm information is sent in time when an exception occurs.

[0144] Log recording: Detailed recording of each operation step, data processing result, and error information in the data cleaning process, facilitating user problem tracing and analysis. Log data supports query and retrieval by time, task, module, etc. dimensions, providing basis for data cleaning engine optimization and maintenance.

[0145] Through real-time monitoring of the collection interface state, monitoring of the completeness and timeliness of the collected data, a full-link monitoring node from the device interface to the application layer is realized: heartbeat detection → integrity algorithm → alarm push, and the problem is alarmed at the first time.

[0146] Compared with the prior art, the present application has significant improvement in key performance: the access efficiency is shortened from 3-6 months per device to less than 7 days, thanks to the templated configuration; the task success rate is improved to 99.5%, realizing QoS guarantee; resource utilization is optimized, with single-device peak load stably controlled below 70% from 150%; fault location is shortened from 2 hours of manual troubleshooting to less than 1 minute of automatic alarm response, realizing fast and accurate positioning.

[0147] The present application realizes comprehensive innovation in development efficiency, resource scheduling, quality control, and architecture scalability: through componentized abstraction of collection capabilities, the new device access period is compressed from months to hours, and the repeated development cost is reduced by more than 50%; a tenant-level driven dynamic flow limiting model is established to ensure that the response time of special tasks is ≤100ms, and the resource preemption rate is reduced by 80%; a collection full-link monitoring closed loop is constructed to realize real-time early warning of data completeness and timeliness, and the fault location time is shortened to within 10 minutes; at the same time, it supports hybrid access of soft collection / hard collection, is compatible with new network protocols such as 5G and IPv6, and has the ability to adapt to network evolution in the next 10 years.

[0148] The application discloses a unified data acquisition method based on multi-tenant hierarchical control of a telecommunication network, solves the technical problems of quick response, priority control and whole-process monitoring of data acquisition, improves data acquisition efficiency and quality, guarantees timeliness and accuracy of data acquisition, adapts to quick update and replacement of network equipment, realizes "zero code" adaptation of new equipment access, establishes a dynamic resource allocation mechanism based on tenant levels, and constructs an end-to-end acquisition quality monitoring system.

Claims

1. A method for unified data collection of a multi-tenant hierarchical control based telecommunications network, characterized in that: Comprise the following steps: Step 1: the dynamic hierarchical collection engine provides a collection instruction template, and is configured by an operation and maintenance personnel; the dynamic hierarchical collection engine generates collection instructions for various target devices according to the configured collection instruction template, generates data collection tasks automatically according to the device type, protocol and data type, and collects original data of the network element type, link type and tenant awareness type generated by the target device according to the data collection task; Step 2: the data cleaning engine calls the original data, pre-processes the original data, verifies the data integrity by using an LSTM-based time series prediction model, unifies the format of the pre-processed data, obtains standard data, and generates a processing operation log; Step 3: the fast data storage module calls the standard data, selects a corresponding storage mode according to the data type and query performance, performs data extraction, format standardization and storage management, and outputs an accessible data set; Step 4: when a tenant issues a data collection request to the dynamic hierarchical collection engine, the execution module verifies the tenant identity, judges whether the current system thread number reaches the flow limiting threshold: yes, then performs tenant flow limiting control; no, then uses a two-level flow limiting engine to perform tenant flow limiting control, which specifically includes implementing device-level flow limiting and network-level flow limiting according to the tenant level, the device-level flow limiting uses a token bucket algorithm to limit the flow of ordinary tenants, and the network-level flow limiting uses a weighted round robin algorithm and a preemptive resource allocation method to limit the flow of important tenants and special-level tenants; a new data collection task is generated according to the flow limiting result, and a new collection task list is constructed, and the original data of the target device is collected according to the new collection task list; When network-level flow limiting is performed, a PID controller is used to perform traffic shaping on the network traffic of important tenants and special-level tenants, dynamically adjust the network device forwarding rules, and execute key collection instructions; Before executing any new data collection task, the corresponding collection instruction is matched and compared with the historical instructions in the cache database: if matched, the standard data matched is found out from the accessible data set and output; if not matched, the collection instruction is issued to the target device through the protocol adapter, and the real-time original data returned by the target device is received and saved, and output; Step 5: the monitoring module monitors the collection state, pre-processing progress, storage state and flow limiting state of the original data, generates a monitoring result report and alarm information, and displays them through a visual display interface.

2. The method of claim 1, wherein the method further comprises: When step 1 is executed, the dynamic hierarchical collection engine performs the following steps: Step 1-1: generate an atomized collection instruction template and an instruction cache pool through a protocol adapter, and the operation and maintenance personnel call the collection instructions in the instruction cache pool to configure the collection instruction template; The collection instruction template includes the device type, protocol and data type, and the data type includes the network element type, link type and tenant awareness type; Step 1-2: generate a data collection task for the target device according to the device type, protocol and data type, and generate an original data collection task list according to the preset static collection rule; Step 1-3: issue the collection instruction to the target device through the protocol adapter according to the original data collection task list, receive the original data returned by the target device, and cache them; The types of the original data include network element type, link type and tenant awareness type.

3. The method of claim 1, wherein the method further comprises: determining a tenant of the network device; and determining a tenant-specific configuration file associated with the tenant of the network device. In step 2, the data cleaning engine specifically performs the following steps: Step 2-1: Call the original data, and perform integrity detection, accuracy detection, consistency detection and repeatability detection on the original data to generate a detection report; Step 2-2: According to the detection report, the original data is processed to obtain cleaned data, including missing value processing, error value correction, abnormal value processing and repeated value deletion processing; Step 2-3: The cleaned data is converted into preprocessed data by format conversion, data normalization, encoding conversion and data aggregation; Step 2-4: A data integrity verification model based on LSTM time series prediction is constructed to verify the data integrity of the preprocessed data; At the same time, an abnormality judgment model is constructed to detect and judge the abnormality of the preprocessed data, and the specific formula is as follows: ； wherein, is the sliding window standard deviation; represents the original data value actually collected from the data source, represents the predicted value predicted by the LSTM-based time series prediction model; Alert represents an abnormality detection judgment value, which takes the value of 1 or 0; Step 2-5: According to the result of step 2-4, the standard data after verification is obtained; Step 2-6: A rule management module is established to configure and manage various rules in the preprocessing process of steps 2-1 to 2-3, including data detection rules, cleaning rules and conversion rules; record processing logs and running status; Step 2-7: A metadata management module is established to manage the metadata involved in the process of steps 2-1 to 2-5, including data source metadata, cleaning rule metadata and conversion rule metadata; the data source metadata includes database table structure and field type.

4. The method of claim 1, wherein the method further comprises: determining a tenant of the network device; and determining a tenant-specific configuration file associated with the tenant of the network device. In step 3, the storage scheme selected by the fast data storage module includes memory storage scheme, database storage scheme and file system storage scheme.

5. The method of claim 1, wherein the method further comprises: determining a tenant of the network device; and determining a tenant-specific configuration file associated with the tenant of the network device. In step 4, the dynamic hierarchical collection engine specifically performs the following steps: Step 4-1: The tenant initiates a data collection request to the dynamic hierarchical collection engine, and the dynamic hierarchical collection engine verifies the legality of the tenant identity, including verifying the tenant name, token and permission level; When the verification is passed, step 4-2 is executed; when the verification is not passed, an error message that the tenant is invalid is returned; Step 4-2: Real-time monitoring of the current thread number, if the preset flow limiting threshold is reached, the collection data request issued by the tenant is terminated immediately and returned; otherwise, step 4-3 is executed; Step 4-3: Judge the tenant's permission, and implement different flow limiting methods according to the tenant's permission: For ordinary tenants: device-level flow limiting is adopted, specifically, a fixed capacity token bucket is initialized for each ordinary tenant, and tokens are continuously added to the bucket at a preset rate, the tenant consumes the corresponding number of tokens when requesting resources, if the tokens are insufficient, the request is refused; real-time monitoring of the token bucket state, dynamically adjusting the token generation rate and bucket capacity according to the overall load and tenant behavior; For important tenants or special tenants: network-level flow limiting is adopted, specifically, a PID controller is used for network traffic shaping, and different tenant level traffic shaping strategies are set, and the traffic shaping strategies are sent to the network device; The PID controller monitors network traffic in real time, dynamically adjusts the forwarding rules of network devices according to the monitoring data and strategies; for a special level tenant, when it executes a key command, the PID controller interrupts the non-key traffic of other level tenants, ensuring that the traffic of the special level tenant is prioritized to pass through; Step 4-5: generate new data collection tasks according to the result of flow limiting, and build a new collection task list; Step 4-6: Before executing any new data collection task, match the corresponding collection instruction with the historical instruction in the cache database: if matched, access the accessible data set, find the matched standard data, and output; if not matched, issue the collection instruction to the target device through the protocol adapter, receive and save the real-time raw data returned by the target device, and output and cache; Step 4-7: return the result of step 4-6 to the tenant requester.

Images