Telecommunication network unified data acquisition method based on multi-tenant hierarchical control

Abstract

The invention discloses a telecommunication network unified data acquisition method based on multi-tenant hierarchical control, which belongs to the technical field of telecommunication network management, and comprises the following steps: generating a static acquisition template and an initial acquisition task, cleaning and standardizing original data, storing a standard data set, processing dynamic tenant requests, and combining hierarchical current limiting. Monitoring of the full-link state comprises solving the technical problems of quick response, priority control and whole-process monitoring of data acquisition, the data acquisition efficiency and quality are improved, and the timeliness and accuracy of data acquisition are guaranteed so as to adapt to quick upgrading of network equipment, realize'zero code 'adaptation of new equipment access, and improve the data acquisition efficiency and quality. A dynamic resource allocation mechanism based on tenant levels is established, and an end-to-end acquisition quality monitoring system is constructed.

Description

Technical Field

[0001] The present invention belongs to the technical field of telecommunication network management, and in particular relates to a telecommunication network unified data collection method based on multi-tenant hierarchical control. Background Art

[0002] In the existing telecommunications network management system, with the promotion of technologies such as 5G and IPv6, and the upgrade and transformation of mobile and fixed networks leading to equipment updates, data collection issues at the existing network element adaptation layer are becoming increasingly prominent: The cycle of newly accessed equipment is long. When new equipment is connected to the existing network, the original professional network management needs to be redeveloped and cannot be quickly supported. In addition, data collection and application are tightly coupled, resulting in repeated development and long cycles.

[0003] The collection tasks lack priority control, resulting in repeated collection, resource preemption, and data failure to synchronize in a timely manner, making it difficult to ensure key tasks.

[0004] The collection process lacks effective monitoring and early warning, there are no audit monitoring logs, the integrity and accuracy of the data cannot be guaranteed, and problem troubleshooting is difficult. Summary of the Invention

[0005] The purpose of the present invention is to provide a unified data collection method for telecommunication networks based on multi-tenant hierarchical control, which solves the technical problems of rapid response, priority control and full-process monitoring of data collection.

[0006] To achieve the above object, the present invention adopts the following technical solutions: A method for unified data collection in a telecommunications network based on multi-tenant hierarchical control, comprising the following steps: Step 1: The dynamic hierarchical collection engine provides a collection instruction template, which is configured by the operation and maintenance personnel. Based on the configured collection instruction template, the dynamic hierarchical collection engine generates collection instructions for various target devices according to device type, protocol, and data type. It automatically generates data collection tasks and collects the raw network element, link, and tenant perception data generated by the target devices according to the data collection tasks. Step 2: The data cleaning engine retrieves and preprocesses the raw data. It uses an LSTM-based time series prediction model to verify data integrity, unifies the format of the preprocessed data, obtains standard data, and generates a processing operation log. Step 3: The fast data storage module retrieves standard data, selects the appropriate storage method based on the data type and query performance, performs data extraction, format standardization, and storage management, and outputs an accessible data set; Step 4: When a tenant sends a data collection request to the dynamic hierarchical collection engine, the execution module verifies the tenant's identity and determines whether the current number of system threads reaches the current limiting threshold. If yes, tenant current limiting control is performed; if not, a two-level current limiting engine is used to perform tenant current limiting control, specifically implementing device-level current limiting and network-level current limiting according to the tenant level. Device-level current limiting uses a token bucket algorithm to limit the flow of ordinary tenants, and network-level current limiting uses a weighted polling algorithm and a preemptive resource allocation method to limit the flow of important tenants and special tenants respectively. A new data collection task is generated based on the current limiting result, and a new collection task list is constructed. The original data of the target device is collected according to the new collection task list. When limiting traffic at the network level, a PID controller is used to shape the network traffic of important tenants and special tenants, dynamically adjust the forwarding rules of network devices, and prioritize the execution of key collection instructions; Before executing any new data collection task, the corresponding collection instruction is matched and compared with the historical instructions in the cache database: if there is a match, the accessible data set is accessed to find the matching standard data and output it; if there is no match, the collection instruction is sent to the target device through the protocol adapter, and the real-time raw data returned by the target device is received, saved, and output; Step 5: The monitoring module monitors the acquisition status, preprocessing progress, storage status and current limiting status of the raw data, generates monitoring result reports and alarm information, and displays them through a visual display interface.

[0007] Preferably, when executing step 1, the dynamic hierarchical collection engine executes the following steps: Step 1-1: Generate an atomic collection instruction template and instruction cache pool through the protocol adapter. The operation and maintenance personnel call the collection instruction in the instruction cache pool and configure the collection instruction template; The collection instruction template includes the device type, protocol, and data type. The data types include network element type, link type, and tenant perception type. Step 1-2: Generate data collection tasks for target devices based on device type, protocol, and data type, and generate a raw data collection task list based on preset static collection rules; Steps 1-3: According to the raw data collection task list, send collection instructions to the target device through the protocol adapter, receive the raw data returned by the target device, and cache it; The types of raw data include network element type, link type, and tenant perception type.

[0008] Preferably, when executing step 2, the data cleaning engine specifically performs the following steps: Step 2-1: Retrieve the original data, perform integrity testing, accuracy testing, consistency testing, and repeatability testing on the original data, and generate a test report; Step 2-2: According to the test report, the original data is processed for missing values, error value correction, outlier processing and duplicate value deletion to obtain the cleaned data; Step 2-3: Perform format conversion, data normalization, encoding conversion, and data aggregation on the cleaned data to obtain preprocessed data; Step 2-4: Build a data integrity verification model based on LSTM time series prediction to verify the data integrity of the preprocessed data; At the same time, an anomaly determination model is constructed to perform anomaly detection and determination on the preprocessed data. The specific formula is as follows: ; in, is the sliding window standard deviation; Indicates the original data value actually collected from the data source. Indicates the predicted value predicted by the LSTM-based time series prediction model; Alert indicates the abnormality detection judgment value, which is 1 or 0; Step 2-5: Based on the results of step 2-4, obtain the verified standard data; Step 2-6: Establish a rule management module to configure and manage various rules used in the preprocessing process from step 2-1 to step 2-3, including data detection rules, cleaning rules, and conversion rules; and record processing logs and operation status; Step 2-7: Establish a metadata management module to manage the metadata involved in steps 2-1 to 2-5, including data source metadata, cleaning rule metadata, and conversion rule metadata; data source metadata includes database table structure and field types.

[0009] Preferably, when executing step 3, the storage scheme selected by the fast data storage module includes a memory storage scheme, a database storage scheme and a file system storage scheme.

[0010] Preferably, when executing step 4, the dynamic hierarchical collection engine specifically performs the following steps: Step 4-1: The tenant initiates a data collection request to the dynamic hierarchical collection engine. The dynamic hierarchical collection engine verifies the legitimacy of the tenant's identity, including verifying the tenant name, token, and permission level; If the verification is successful, execute step 4-2; if the verification is unsuccessful, return an error message indicating that the tenant is invalid. Step 4-2: Monitor the number of currently executing threads in real time. If the preset current limit threshold is reached, immediately terminate the tenant's data collection request and return; otherwise, execute step 4-3. Step 4-3: Determine tenant permissions and implement different flow control methods based on tenant permissions: For ordinary tenants: Device-level current limiting is adopted. Specifically, a fixed-capacity token bucket is initialized for each ordinary tenant, and tokens are continuously added to the bucket at a preset rate. Tenants consume a corresponding number of tokens when requesting resources. If there are insufficient tokens, the request is rejected. The token bucket status is monitored in real time, and the token generation rate and bucket capacity are dynamically adjusted according to the overall load and tenant behavior. For important or premium tenants: network-level traffic limiting is implemented. Specifically, a PID controller is used to shape network traffic, set traffic shaping policies for different tenant levels, and distribute the traffic shaping policies to network devices. The PID controller monitors network traffic in real time and dynamically adjusts the forwarding rules of network devices based on monitoring data and policies. For premium tenants, when they execute critical commands, the PID controller interrupts non-critical traffic from other tenants, ensuring that premium tenants' traffic passes first. Step 4-5: Generate new data collection tasks based on the current limiting results and build a new collection task list; Steps 4-6: Before executing any new data collection task, the corresponding collection instruction is compared with the historical instructions in the cache database. If a match is found, the accessible data set is accessed to find the matching standard data and output it. If a match is found, the collection instruction is sent to the target device through the protocol adapter, and the real-time raw data returned by the target device is received, saved, output, and cached. Step 4-7: Return the result of step 4-6 to the tenant requester.

[0011] The present invention discloses a unified data collection method for telecommunications networks based on multi-tenant hierarchical control, which solves the technical problems of rapid response, priority control, and full-process monitoring of data collection. The present invention improves data collection efficiency and quality, ensures the timeliness and accuracy of data collection, adapts to the rapid upgrading of network equipment, realizes "zero-code" adaptation of new equipment access, establishes a dynamic resource allocation mechanism based on tenant level, and builds an end-to-end collection quality monitoring system. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is the main flow chart of the present invention; Figure 2 It is a system architecture diagram of the present invention; Figure 3 is a flow chart of step 4 of the present invention; Figure 4 It is a logic flow chart of the two-stage current limiting engine of the present invention. DETAILED DESCRIPTION

[0013] Depend on Figure 1-Figure 4 A method for unified data collection in a telecommunications network based on multi-tenant hierarchical control is shown, comprising the following steps: Step 1: The dynamic hierarchical collection engine provides a collection instruction template, which is configured by the operation and maintenance personnel. Based on the configured collection instruction template, the dynamic hierarchical collection engine generates collection instructions for various target devices according to device type, protocol, and data type. It automatically generates data collection tasks and collects the raw network element, link, and tenant perception data generated by the target devices according to the data collection tasks. When executing step 1, the dynamic hierarchical collection engine performs the following steps: Step 1-1: Generate an atomic collection instruction template and instruction cache pool through the protocol adapter. The operation and maintenance personnel call the collection instruction in the instruction cache pool and configure the collection instruction template; The collection instruction template includes the device type, protocol, and data type. The data types include network element type, link type, and tenant perception type. Step 1-2: Generate data collection tasks for target devices based on device type, protocol, and data type, and generate a raw data collection task list based on preset static collection rules; Steps 1-3: According to the raw data collection task list, send collection instructions to the target device through the protocol adapter, receive the raw data returned by the target device, and cache it; The types of raw data include network element type, link type, and tenant perception type.

[0014] In this embodiment, the protocol adapter (SNMP / Netconf / FTP, etc.) + instruction cache pool technology is adopted, with professional, network, collection protocol, and device type as dimensions, using atomic collection instructions and providing template configuration services.

[0015] Data collection types include network element types (configuration, resources, performance, alarms), link types (signaling, user plane), and user perception types (complaints, dialing tests). Given that network data collection involves many professions and network elements and lasts for a long time, in order to avoid duplication and facilitate backtracking and data processing, it is usually necessary to specify file names and file formats.

[0016] Collection instruction configuration provides custom configuration device collection instructions.

[0017] The collection task configuration will be summarized in the configured collection instructions, providing the function of customizing the collection task configuration.

[0018] Step 2: The data cleaning engine retrieves and preprocesses the raw data. It uses an LSTM-based time series prediction model to verify data integrity, unifies the format of the preprocessed data, obtains standard data, and generates a processing operation log. When executing step 2, the data cleaning engine specifically performs the following steps: Step 2-1: Retrieve the original data, perform integrity testing, accuracy testing, consistency testing, and repeatability testing on the original data, and generate a test report; Completeness check: Check whether there are missing values ​​in the data records, such as if a column field is empty or some records are incomplete. At the same time, count the proportion of missing data and evaluate the degree of data completeness.

[0019] Accuracy testing: Verify data correctness based on pre-set business rules and data type specifications. For example, check whether date fields conform to the date format, whether numeric fields are within a reasonable value range, and perform format verification on specific fields such as ID numbers and mobile phone numbers.

[0020] Consistency testing: Ensures that the same data is represented consistently across different tables or records. For example, checking whether the name of a customer is consistent across different tables or whether there are any inconsistencies in address information.

[0021] Duplicate detection: Identify duplicate records by setting unique identification fields or calculating data fingerprints to prevent data redundancy from interfering with subsequent analysis.

[0022] Step 2-2: According to the test report, the original data is processed for missing values, error value correction, outlier processing and duplicate value deletion to obtain the cleaned data; Missing value handling: Supports multiple handling strategies, such as deleting records containing missing values, filling missing values ​​for numerical and categorical data using the mean, median, or mode, and predicting missing values ​​using machine learning algorithms.

[0023] Error correction: Corrects incorrect data based on pre-set rules and reference data. For example, it can correct incorrect address information using the address library and replace misspelled words using the dictionary.

[0024] Deduplication: Based on the detected duplicate records, retain one and delete the rest to ensure data uniqueness.

[0025] Outlier processing: Use statistical methods (such as the 3σ principle) or machine learning algorithms to identify abnormal data and process it according to business needs, such as deleting outliers as noise data or correcting them.

[0026] Step 2-3: Perform format conversion, data normalization, encoding conversion, and data aggregation on the cleaned data to obtain preprocessed data; Format conversion: Convert the storage format of data, such as converting the date format from "YYYY-MM-DD" to "DD / MM / YYYY", and converting numeric data to a specified precision.

[0027] Data normalization: converting numerical data of different dimensions into the same value range. Common methods include minimum-maximum normalization and Z-score normalization to facilitate data comparison and analysis.

[0028] Encoding conversion: Encoding categorized data, such as converting "male" and "female" in the gender field into digital codes "0" and "1", or using one-hot encoding to meet the requirements of applications such as machine learning algorithms.

[0029] Data aggregation: Summarize and calculate data according to specific dimensions, such as summing and averaging sales data by month and region to generate an aggregated data table.

[0030] Step 2-4: Build a data integrity verification model based on LSTM time series prediction to verify the data integrity of the preprocessed data; The data integrity verification model for LSTM time series prediction is as follows: The input layer receives the historical 5-minute data sequence , prediction window =30s: ; Where X represents time series data, t represents the time index, n is the history window size, τ is the prediction step size, and θ is the set of parameters (weights and biases) of the LSTM neural network model.

[0031] Network structure: 2 layers of LSTM (128 hidden units) + 1 layer of Dense; Loss function: Huber Loss( ).

[0032] At the same time, an anomaly determination model is constructed to perform anomaly detection and determination on the preprocessed data. The specific formula is as follows: ; in, is the sliding window standard deviation; window size = 60 sampling points. Indicates the original data value actually collected from the data source. Indicates the predicted value predicted by the LSTM-based time series prediction model; Alert indicates the abnormality detection judgment value, which is 1 or 0; Step 2-5: Based on the results of step 2-4, obtain the verified standard data; Step 2-6: Establish a rule management module to configure and manage various rules used in the preprocessing process from step 2-1 to step 2-3, including data detection rules, cleaning rules, and conversion rules; and record processing logs and operation status; In this embodiment, the rule management module is used to configure and manage various rules used in the data cleaning process, including data detection rules, cleaning rules, and transformation rules. Users can create, modify, and delete rules through a visual interface, and the module supports rule version management and backup and recovery. Furthermore, the module features rule priority settings. When multiple rules conflict, they are executed in order of priority, ensuring the accuracy and consistency of data cleaning.

[0033] Step 2-7: Establish a metadata management module to manage the metadata involved in steps 2-1 to 2-5, including data source metadata, cleaning rule metadata, and conversion rule metadata; data source metadata includes database table structure and field types.

[0034] In this embodiment, the metadata management module is responsible for managing metadata involved in the data cleansing process, including data source metadata (such as database table structure and field types), cleansing rule metadata, and transformation rule metadata. It provides metadata browsing, querying, and editing functions, supports automatic metadata discovery and collection, and establishes relationships between metadata to help users better understand the data cleansing process and data relationships, providing support for data governance.

[0035] Step 3: The fast data storage module retrieves standard data, selects the appropriate storage method based on the data type and query performance, performs data extraction, format standardization, and storage management, and outputs an accessible data set; When executing step 3, the storage solutions selected by the fast data storage module include a memory storage solution, a database storage solution, and a file system storage solution.

[0036] Step 4: When a tenant sends a data collection request to the dynamic hierarchical collection engine, the execution module verifies the tenant's identity and determines whether the current number of system threads reaches the current limiting threshold. If yes, tenant current limiting control is performed; if not, a two-level current limiting engine is used to perform tenant current limiting control, specifically implementing device-level current limiting and network-level current limiting according to the tenant level. Device-level current limiting uses a token bucket algorithm to limit the flow of ordinary tenants, and network-level current limiting uses a weighted polling algorithm and a preemptive resource allocation method to limit the flow of important tenants and special tenants respectively. A new data collection task is generated based on the current limiting result, and a new collection task list is constructed. The original data of the target device is collected according to the new collection task list. When limiting traffic at the network level, a PID controller is used to shape the network traffic of important tenants and special tenants, dynamically adjust the forwarding rules of network devices, and prioritize the execution of key collection instructions; Before executing any new data collection task, the corresponding collection instruction is matched and compared with the historical instructions in the cache database: if there is a match, the accessible data set is accessed to find the matching standard data and output it; if there is no match, the collection instruction is sent to the target device through the protocol adapter, and the real-time raw data returned by the target device is received, saved, and output; When executing step 4, the dynamic hierarchical collection engine specifically performs the following steps: Step 4-1: The tenant initiates a data collection request to the dynamic hierarchical collection engine. The dynamic hierarchical collection engine verifies the legitimacy of the tenant's identity, including verifying the tenant name, token, and permission level; If the verification is successful, execute step 4-2; if the verification is unsuccessful, return an error message indicating that the tenant is invalid. Step 4-2: Monitor the number of currently executing threads in real time. If the preset current limit threshold is reached, immediately terminate the tenant's data collection request and return; otherwise, execute step 4-3. Step 4-3: Determine tenant permissions and implement different flow control methods based on tenant permissions: For ordinary tenants: Device-level current limiting is adopted. Specifically, a fixed-capacity token bucket is initialized for each ordinary tenant, and tokens are continuously added to the bucket at a preset rate. Tenants consume a corresponding number of tokens when requesting resources. If there are insufficient tokens, the request is rejected. The token bucket status is monitored in real time, and the token generation rate and bucket capacity are dynamically adjusted according to the overall load and tenant behavior. For important or premium tenants: network-level traffic limiting is implemented. Specifically, a PID controller is used to shape network traffic, set traffic shaping policies for different tenant levels, and distribute the traffic shaping policies to network devices. The PID controller monitors network traffic in real time and dynamically adjusts the forwarding rules of network devices based on monitoring data and policies. For premium tenants, when they execute critical commands, the PID controller interrupts non-critical traffic from other tenants, ensuring that premium tenants' traffic passes first. Step 4-5: Generate new data collection tasks based on the current limiting results and build a new collection task list; Steps 4-6: Before executing any new data collection task, the corresponding collection instruction is compared with the historical instructions in the cache database. If a match is found, the accessible data set is accessed to find the matching standard data and output it. If a match is found, the collection instruction is sent to the target device through the protocol adapter, and the real-time raw data returned by the target device is received, saved, output, and cached. Step 4-7: Return the result of step 4-6 to the tenant requester.

[0037] In this embodiment, the efficiency and security of data collection are guaranteed through multi-level authentication and dynamic current limiting mechanisms: Tenant authentication: First, the requesting tenant's identity is authenticated to ensure that only authorized users can trigger the collection task.

[0038] Tenant current limiting: Real-time monitoring of the number of currently executing threads. If the preset current limiting threshold is reached, the request is immediately terminated and returned to avoid system overload.

[0039] Network element current limiting and classification strategy: Implement differentiated flow control based on tenant level.

[0040] Ordinary users: Hard concurrency limit (token bucket algorithm) strictly limits the number of concurrent connections. When the maximum value is exceeded, new connections will be directly rejected to ensure system stability. Important users: Queue priority scheduling (weighted round-robin algorithm). After the maximum number of concurrent connections is exceeded, requests are automatically queued and processed in order of priority to balance resource allocation. Special user: can interrupt other tasks (preemptive resource allocation), enjoy the highest priority, and can forcibly interrupt other tenants' ongoing tasks when executing network element commands to ensure real-time response to critical business.

[0041] Two-stage current limiting engine: This embodiment proposes a two-level throttling engine, including device-level throttling and network-level throttling. Device-level throttling uses a token bucket algorithm and is applicable to ordinary tenants to control their use of computing resources. Network-level throttling uses PID-based traffic shaping technology and is applicable to key and premium tenants to control their network traffic.

[0042] Device level: Applicable to ordinary tenants. This feature allows for reasonable flow control of data collection tasks at the device level, resolving issues such as resource preemption and insufficient data collection performance caused by frequent data collection tasks.

[0043] Tokens are added to the bucket at a fixed rate, and the bucket has a limited capacity. When a tenant requests a resource, they must take the corresponding number of tokens from the bucket. If there are insufficient tokens in the bucket, the request is rejected.

[0044] Initialize token bucket: Initialize a token bucket for each common tenant and set the bucket size (maximum number of tokens) and token addition rate.

[0045] Token Addition: The system adds tokens to the token bucket at a preset rate.

[0046] Resource request processing: When a tenant requests a resource, the system checks the number of tokens in its token bucket. If sufficient, it takes the corresponding number of tokens from the bucket and processes the request; if insufficient, it rejects the request.

[0047] Monitoring and adjustment: Monitor the status of the token bucket in real time and dynamically adjust the token addition rate and bucket size based on system load and tenant behavior.

[0048] Network-level: Applicable to key / premium tenants. PID-based traffic shaping centrally controls network traffic, enabling precise control of network traffic for key and premium tenants. The PID controller dynamically adjusts network device forwarding rules based on network policies and tenant tiers to control incoming and outgoing traffic.

[0049] PID controller configuration: Configure the PID controller and define traffic shaping policies for different tenant levels.

[0050] Policy delivery: Deliver traffic shaping policies to network devices (such as switches and routers).

[0051] Traffic monitoring and adjustment: The PID controller monitors network traffic in real time and dynamically adjusts the forwarding rules of network devices based on monitoring data and policies.

[0052] Priority processing: For premium tenants, when they execute critical commands, the PID controller can interrupt the non-critical traffic of other tenants to ensure that the premium tenant's traffic passes first.

[0053] Mathematical modeling of dynamic current limiting algorithm: Tenant priority weight calculation: Define the weight coefficient of tenant ui at time t: ; : Default priority (Special=3, Important=2, Normal=1); : Current system load rate (0-100%); : The last successful execution time of the tenant; coefficient =0.6, =0.3, =0.1 (optimized by web search); Thread allocation control law: Use PID controller to dynamically adjust the thread pool size: ; : No. The deviation between the actual load and the target load during the cycle. Parameter setting value: =1.2, =0.5, =0.3.

[0054] Step 5: The monitoring module monitors the acquisition status, preprocessing progress, storage status and current limiting status of the raw data, generates monitoring result reports and alarm information, and displays them through a visual display interface.

[0055] In this embodiment, the monitoring module monitors and records the running status of the data cleaning engine in real time: Operation monitoring: Real-time display of data cleaning task execution progress, resource usage (such as CPU and memory usage), data processing volume and other indicators, presenting the task's running status through a visual interface, and issuing timely alarm information when anomalies occur.

[0056] Logging: Detailed records of every step, data processing results, and error messages during the data cleaning process facilitate problem tracing and analysis. Log data can be queried and retrieved by time, task, module, and other dimensions, providing a basis for optimizing and maintaining the data cleaning engine.

[0057] By real-time monitoring of the acquisition interface status and monitoring of the integrity and timeliness of the collected data, a full-link monitoring node from the device interface to the application layer is implemented: heartbeat detection → integrity algorithm → alarm push, and an alarm is issued immediately when a problem occurs.

[0058] Compared with the existing technology, this invention has significant improvements in key performance: access efficiency is shortened from 3-6 months per device to less than 7 days, thanks to templated configuration; the task success rate is increased to 99.5%, achieving QoS guarantee; resource utilization is optimized, and the peak load of a single device is stably controlled from 150% to below 70%; fault location is shortened from 2 hours of manual investigation to less than 1 minute for automatic alarm response, achieving fast and accurate positioning.

[0059] This invention has achieved comprehensive innovation in development efficiency, resource scheduling, quality control and architectural scalability: through the componentized abstraction of acquisition capabilities, the new equipment access cycle is compressed from months to hours, and the cost of repeated development is reduced by more than 50%; a tenant-level driven dynamic current limiting model is established to ensure that the response time of special tasks is ≤100ms and the resource preemption rate is reduced by 80%; a full-link acquisition monitoring closed loop is constructed to achieve real-time early warning of data integrity and timeliness, and the fault location time is shortened to within 10 minutes; at the same time, it supports soft / hard acquisition hybrid access, is compatible with new network protocols such as 5G and IPv6, and has the ability to adapt to network evolution in the next 10 years.

[0060] The present invention discloses a unified data collection method for telecommunications networks based on multi-tenant hierarchical control, which solves the technical problems of rapid response, priority control, and full-process monitoring of data collection. The present invention improves data collection efficiency and quality, ensures the timeliness and accuracy of data collection, adapts to the rapid upgrading of network equipment, realizes "zero-code" adaptation of new equipment access, establishes a dynamic resource allocation mechanism based on tenant level, and builds an end-to-end collection quality monitoring system.

Claims

1. A method for unified data collection in a telecommunications network based on multi-tenant hierarchical control, characterized by: The steps include: Step 1: The dynamic hierarchical collection engine provides a collection instruction template, which is configured by the operation and maintenance personnel. Based on the configured collection instruction template, the dynamic hierarchical collection engine generates collection instructions for various target devices according to device type, protocol, and data type. It automatically generates data collection tasks and collects the raw network element, link, and tenant perception data generated by the target devices according to the data collection tasks. Step 2: The data cleaning engine retrieves and preprocesses the raw data. It uses an LSTM-based time series prediction model to verify data integrity, unifies the format of the preprocessed data, obtains standard data, and generates a processing operation log. Step 3: The fast data storage module retrieves standard data, selects the appropriate storage method based on the data type and query performance, performs data extraction, format standardization, and storage management, and outputs an accessible data set; Step 4: When a tenant sends a data collection request to the dynamic hierarchical collection engine, the execution module verifies the tenant's identity and determines whether the current number of system threads reaches the current limiting threshold. If yes, tenant current limiting control is performed; if not, a two-level current limiting engine is used to perform tenant current limiting control, specifically implementing device-level current limiting and network-level current limiting according to the tenant level. Device-level current limiting uses a token bucket algorithm to limit the flow of ordinary tenants, and network-level current limiting uses a weighted polling algorithm and a preemptive resource allocation method to limit the flow of important tenants and special tenants respectively. A new data collection task is generated based on the current limiting result, and a new collection task list is constructed. The original data of the target device is collected according to the new collection task list. When limiting traffic at the network level, a PID controller is used to shape the network traffic of important tenants and special tenants, dynamically adjust the forwarding rules of network devices, and prioritize the execution of key collection instructions; Before executing any new data collection task, the corresponding collection instruction is matched and compared with the historical instructions in the cache database: if there is a match, the accessible data set is accessed to find the matching standard data and output it; if there is no match, the collection instruction is sent to the target device through the protocol adapter, and the real-time raw data returned by the target device is received, saved, and output; Step 5: The monitoring module monitors the acquisition status, preprocessing progress, storage status and current limiting status of the raw data, generates monitoring result reports and alarm information, and displays them through a visual display interface.

2. The method for unified data collection in a telecommunications network based on multi-tenant hierarchical control according to claim 1, characterized in that: When executing step 1, the dynamic hierarchical collection engine performs the following steps: Step 1-1: Generate an atomic collection instruction template and instruction cache pool through the protocol adapter. The operation and maintenance personnel call the collection instruction in the instruction cache pool and configure the collection instruction template; The collection instruction template includes the device type, protocol, and data type. The data types include network element type, link type, and tenant perception type. Step 1-2: Generate data collection tasks for target devices based on device type, protocol, and data type, and generate a raw data collection task list based on preset static collection rules; Steps 1-3: According to the raw data collection task list, send collection instructions to the target device through the protocol adapter, receive the raw data returned by the target device, and cache it; The types of raw data include network element type, link type, and tenant perception type.

3. The method for unified data collection in a telecommunications network based on multi-tenant hierarchical control according to claim 1, characterized in that: When executing step 2, the data cleaning engine specifically performs the following steps: Step 2-1: Retrieve the original data, perform integrity testing, accuracy testing, consistency testing, and repeatability testing on the original data, and generate a test report; Step 2-2: According to the test report, the original data is processed for missing values, error value correction, outlier processing and duplicate value deletion to obtain the cleaned data; Step 2-3: Perform format conversion, data normalization, encoding conversion, and data aggregation on the cleaned data to obtain preprocessed data; Step 2-4: Build a data integrity verification model based on LSTM time series prediction to verify the data integrity of the preprocessed data; At the same time, an anomaly determination model is constructed to perform anomaly detection and determination on the preprocessed data. The specific formula is as follows: ； in, is the sliding window standard deviation; Indicates the original data value actually collected from the data source. Indicates the predicted value predicted by the LSTM-based time series prediction model; Alert indicates the abnormality detection judgment value, which is 1 or 0; Step 2-5: Based on the results of step 2-4, obtain the verified standard data; Step 2-6: Establish a rule management module to configure and manage various rules used in the preprocessing process from step 2-1 to step 2-3, including data detection rules, cleaning rules, and conversion rules; and record processing logs and operation status; Step 2-7: Establish a metadata management module to manage the metadata involved in steps 2-1 to 2-5, including data source metadata, cleaning rule metadata, and conversion rule metadata; data source metadata includes database table structure and field types.

4. The method for unified data collection in a telecommunications network based on multi-tenant hierarchical control according to claim 1, characterized in that: When executing step 3, the storage solutions selected by the fast data storage module include a memory storage solution, a database storage solution, and a file system storage solution.

5. The method for unified data collection in a telecommunication network based on multi-tenant hierarchical control according to claim 1, characterized in that: When executing step 4, the dynamic hierarchical collection engine specifically performs the following steps: Step 4-1: The tenant initiates a data collection request to the dynamic hierarchical collection engine. The dynamic hierarchical collection engine verifies the legitimacy of the tenant's identity, including verifying the tenant name, token, and permission level; If the verification is successful, execute step 4-2; if the verification is unsuccessful, return an error message indicating that the tenant is invalid. Step 4-2: Monitor the number of currently executing threads in real time. If the preset current limit threshold is reached, immediately terminate the tenant's data collection request and return; otherwise, execute step 4-3. Step 4-3: Determine tenant permissions and implement different flow control methods based on tenant permissions: For ordinary tenants: Device-level current limiting is adopted. Specifically, a fixed-capacity token bucket is initialized for each ordinary tenant, and tokens are continuously added to the bucket at a preset rate. Tenants consume a corresponding number of tokens when requesting resources. If there are insufficient tokens, the request is rejected. The token bucket status is monitored in real time, and the token generation rate and bucket capacity are dynamically adjusted according to the overall load and tenant behavior. For important tenants or premium tenants: network-level traffic limiting is implemented. Specifically, a PID controller is used to shape network traffic. Traffic shaping policies are set for different tenant levels and distributed to network devices. The PID controller monitors network traffic in real time and dynamically adjusts the forwarding rules of network devices based on monitoring data and policies. When a premium tenant executes a critical command, the PID controller interrupts the non-critical traffic of other tenants, ensuring that the premium tenant's traffic passes first. Step 4-5: Generate new data collection tasks based on the current limiting results and build a new collection task list; Steps 4-6: Before executing any new data collection task, the corresponding collection instruction is compared with the historical instructions in the cache database. If a match is found, the accessible data set is accessed to find the matching standard data and output it. If a match is found, the collection instruction is sent to the target device through the protocol adapter, and the real-time raw data returned by the target device is received, saved, output, and cached. Step 4-7: Return the result of step 4-6 to the tenant requester.

Images