Intelligent network technology service management system and method based on big data

Abstract

The invention discloses an intelligent network technology service management system and method based on big data, and particularly relates to the technical field of intelligent network technology services, and the system comprises a data collection module which is used for configuring a distributed crawler, an API interface and a sensor to obtain multi-source heterogeneous data from a user terminal, network equipment and a third-party platform, comprising user behavior data, network flow data, equipment state data and service quality logs; the data storage module is used for storing data by adopting a fusion framework of a column database and a time sequence database; a column database and a time sequence database are fused, a partition storage strategy is matched, unified storage of structured / unstructured data is achieved, cross-database ETL batch processing bottlenecks are eliminated, a data processing module integrates a streaming computing engine and a batch processing framework, high-concurrency real-time analysis is supported, query delay is reduced from the hour level to the second level, and the query efficiency is improved. And automatically triggering computing node / bandwidth elastic expansion when responding to burst traffic.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent network technology services, and particularly relates to a management system and method. Specifically, it discloses an intelligent network technology service management system and method based on big data. Background Art

[0002] With the rapid development of Internet technology, the global network service scale has shown exponential growth.

[0003] The prior art uses a hierarchical architecture and a rule-driven mechanism to implement network management: collecting traffic metadata, splitting and storing it in a relational database, a data warehouse, and a time series database according to data types, and using an ETL tool to achieve cross-database queries; anomaly detection triggers alarms based on preset thresholds, and supplements them with a model trained offline to generate a behavior baseline; resource scheduling adopts a historical peak over-allocation strategy and manual expansion, and load balancing depends on the weighted round-robin algorithm; at the security level, it is through full-flow encryption and centralized auditing; engineering verification builds a 1:1 mirror environment to test strategy changes and updates during downtime. The prior art realizes network management through rule-driven plus manual intervention, but there are still four core defects:

[0004] Data fragmentation: Multi-source heterogeneous data depends on a hierarchical storage architecture. Cross-database queries require ETL batch processing and stream-batch separation calculations, resulting in data islands and processing delays, and it is difficult to support high-concurrency real-time analysis requirements.

[0005] Analysis lag: Resource scheduling is based on a historical peak over-allocation strategy and manual decision-making for expansion, lacking a dynamic elastic framework. Sudden traffic is likely to cause service degradation or resource misallocation.

[0006] Security loss: Full-flow IPSec encryption causes transmission performance loss. Centralized log auditing depends on offline aggregation and manual verification, and it is difficult to balance security intensity and business continuity.

[0007] Inefficient verification: Strategy changes require building a 1:1 physical mirror environment for simulation testing, relying on downtime maintenance windows to update configurations. The verification cycle is long and it is impossible to quickly iterate complex scenarios.

[0008] Therefore, it is necessary to provide an intelligent network technology service management system and method with data fusion, real-time decision-making, fine-grained security, and engineering agility. Summary of the Invention

[0009] To overcome the above-mentioned defects of the prior art, embodiments of the present invention provide an intelligent network technology service management system and method based on big data. By constructing a unified data lake to integrate multi-source heterogeneous data, sub-second anomaly detection is realized based on a stream-batch integrated engine; dynamic baselines and online models are used to predict attacks and resource requirements in real time; a resource elasticity framework is designed to scale resources on demand in seconds, combined with intent-driven networks to automatically optimize traffic paths; microservice-based encryption and blockchain audit chains are deployed on the security side; digital twin platforms are introduced in engineering verification to support gray release, and downtime is compressed to solve the problems of difficult processing of massive data, slow response to sudden demands, high cost of security protection, and low efficiency of engineering verification.

[0010] To achieve the above object, the present invention provides the following technical solutions: An intelligent network technology service management system based on big data, comprising:

[0011] Data collection module: Configure distributed crawlers, API interfaces, and sensors to obtain multi-source heterogeneous data from user terminals, network devices, and third-party platforms, including user behavior data, network traffic data, device status data, and service quality logs;

[0012] Data storage module: Adopt a fusion architecture of columnar databases and time-series databases to store data, and implement data partitioning strategies and data lifecycle management strategies based on data sources and access frequencies to store massive structured and unstructured data;

[0013] Data processing and analysis module: Include a distributed streaming computing engine, a network performance processing sub-module, and an intelligent analysis sub-module;

[0014] Early warning signal generation module: Set thresholds according to the anomaly index of intelligent analysis to generate hierarchical early warning signals;

[0015] Intelligent decision-making module: Take different measures according to the hierarchical early warning signals, and optimize resource allocation based on user priorities and service level agreements;

[0016] Service management module: Execute resource dynamic scheduling instructions, dynamically configure network resources, and adjust network bandwidth, computing nodes, and cache resources in real time according to policies;

[0017] Security and privacy protection module: Include access control, encrypted transmission, and audit log deposition;

[0018] Visualization and interaction module: Provide a real-time monitoring dashboard, a policy configuration interface, and an API interaction interface, and display analysis results and system operation status in the form of a visualization dashboard and an API interface.

[0019] An intelligent network technology service management method based on big data, the method is as follows:

[0020] Step 1: Collect network service data in real time and build a multi-dimensional data warehouse;

[0021] Step 2: Clean and standardize multi-source data to generate a standardized data stream;

[0022] Step 3: Extract real-time features based on stream computing, and perform anomaly detection using network performance processing and intelligent analysis;

[0023] Step 4: Use the results of intelligent analysis to output service demand prediction results and take measures;

[0024] Step 5: Generate a dynamic resource allocation strategy according to the prediction results and send it to the execution layer through the API;

[0025] Step 6: Execute dynamic scheduling and collect feedback data, and iteratively optimize model parameters based on the online learning mechanism and A / B testing.

[0026] Technical effects and advantages of the present invention:

[0027] 1. The present invention adopts the integration of columnar database and time series database, combined with the partition storage strategy, to achieve the unified storage of structured / unstructured data and eliminate the cross-database ETL batch processing bottleneck;

[0028] 2. The data processing module of the present invention integrates a stream computing engine and a batch processing framework, supports high-concurrency real-time analysis, reduces the query latency from hours to seconds, and automatically triggers the elastic expansion of computing nodes / bandwidth when responding to burst traffic;

[0029] 3. The security module of the present invention uses the lightweight QUIC protocol to replace IPSec to achieve transport layer encryption, combined with the data classification strategy, encrypts sensitive data / plainsensitive data, and improves the transmission performance;

[0030] 4. The visualization and interaction module of the present invention builds a virtualization simulation test platform, supports online gray release and A / B testing of policy changes, and can complete multi-scenario verification without downtime maintenance. Description of the Drawings

[0031] Appendix Figure 1 is the method step diagram of the present invention.

[0032] Appendix Figure 2 is the system block diagram of the present invention. Detailed Embodiments

[0033] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0034] As shown in the attached Figure 1 drawing, the present invention proposes an intelligent network technology service management system based on big data, which is characterized by including: a data collection module, a data storage module, a data processing and analysis module, an early warning signal generation module, an intelligent decision-making module, a service management module, a security and privacy protection module, and a visualization and interaction module.

[0035] Specifically, the collection module of the data collection module includes a user behavior data collection sub-module, a network traffic collection sub-module, and a device status monitoring sub-module.

[0036] Among them, the user behavior data collection sub-module captures user click, browsing, and transaction behavior data on the terminal device through the embedding technology;

[0037] Among them, the network traffic collection sub-module analyzes the traffic type and source based on the NetFlow protocol and the deep packet inspection DPI technology;

[0038] Among them, the device status monitoring sub-module obtains the load data of the server CPU, memory, disk, and network interface in real time through the SNMP protocol.

[0039] Specifically, in the data storage module, the columnar database uses Apache Parquet or ClickHouse to store user behavior logs and network traffic detail data, and supports column pruning and predicate pushdown to optimize query performance.

[0040] Among them, the time series database uses InfluxDB or TimescaleDB to store device status metrics and service quality monitoring data, slices by time and automatically compresses expired data, and the device status metrics include CPU, memory, and network latency.

[0041] Among them, the data partitioning strategy divides the storage cluster based on the data source and data type, and realizes load balancing through the consistent hashing algorithm. The data comes from user terminals, network devices, and third-party platforms, and the data types include structured and unstructured data.

[0042] Among them, the data life cycle management strategy: data is divided into hot data, warm data, and cold data according to access frequency, and is automatically migrated through the LRU algorithm. Among them, hot data is stored in SSD, warm data is stored in HDD, and cold data is archived to object storage.

[0043] Specifically, the data processing and analysis module includes a distributed streaming computing engine, a network performance processing sub-module, and an intelligent analysis sub-module.

[0044] Among them, the distributed streaming computing engine uses Apache Flink or Spark Streaming to perform window aggregation and feature extraction on real-time data streams.

[0045] Among them, the network performance processing module includes a network transmission rate average model, a network connection persistence model, and a network fluctuation amplitude model;

[0046] The specific network transmission rate model is:

[0047]

[0048] Measure the average transmission capacity of the network within the observation period, where μ represents the average transmission rate, n is the total number of samples, and R i represents the instantaneous rate of the i-th sampling point, with the unit of bps or Mbps.

[0049] The specific network connection persistence model is:

[0050]

[0051] Among them, S is the effective service duration, such as the time when data transmission is successful, and D is the interruption duration, such as the time when the connection fails.

[0052] The ratio represents that the higher the value of C, the higher the network stability. When D = 0, C → ∞, indicating no interruption.

[0053] The specific network fluctuation amplitude model is:

[0054]

[0055] Calculate the standard deviation of the rate value R i and the mean μ to quantify the rate fluctuation degree and evaluate the stability of the network transmission rate.

[0056] Specifically, the intelligent analysis module includes calculating the network fluency index, stability index, and security index.

[0057] The network fluency index is denoted as NFI, and the specific formula is:

[0058]

[0059] where R max is the maximum transmission rate. The higher the average rate and the smaller the fluctuation, the higher the smoothness. The standard deviation reflects the impact of instantaneous fluctuations on the user experience.

[0060] The network stability index is denoted as NSI, specifically:

[0061]

[0062] where represents the connection persistence ratio, S is the effective service duration, D is the interruption duration, and ΔR = R max - R min represents the rate range, reflecting the sudden fluctuation, and R min is the minimum transmission rate.

[0063] This index represents that the higher the proportion of the effective service duration and the smaller the rate fluctuation, the stronger the stability. Combining the interruption duration and the rate fluctuation, the network reliability is comprehensively evaluated.

[0064] The security index is denoted as NSeI, specifically:

[0065]

[0066] where E secure is the coverage rate of security protection means such as access control and data encryption, E total is the total number of security requirement items such as the number of security measures required by the standard, N slice is the network slice isolation degree such as the number of different user groups or service types supported by 5G network slices, and N max is the maximum number of supported slices such as the terminal can access up to 8 slices at the same time, and α, β are weight coefficients.

[0067] This index indicates that the higher the coverage rate of security measures and the stronger the slice isolation ability, the higher the network security.

[0068] Specifically, the process of setting thresholds according to the anomaly index to generate hierarchical warning signals is as follows: first, set the warning thresholds for the network smoothness index, the network stability index, and the network security index respectively, and then generate multi-level warning signals, including yellow warning, orange warning, and red warning.

[0069] The warning threshold for the network smoothness index is set as follows:

[0070] Normal (green): NFI ≥ F1·NFI 基线 ;

[0071] Minor warning (yellow): F2·NFI 基线≤NFI < F1·NFI 基线 ;

[0072] Medium warning (orange): F3·NFI 基线 ≤NFI < F2·NFI 基线 ;

[0073] Severe warning (red): NFI < F3·NFI 基线 ;

[0074] Among them, F1 is between 80% - 90%, F2 is between 60% - 70%, and F3 is between 40% - 50%. The specific values are determined according to the actual situation.

[0075] Among them, the warning threshold of the network stability index is set as follows:

[0076] Normal (green): NSI ≥ S1;

[0077] Mild warning (yellow): S2 ≤ NSI < S1;

[0078] Medium warning (orange): S3 ≤ NSI < S2;

[0079] Severe warning (red): NSI < S3;

[0080] Among them, S1 is between 8 - 9, S2 is between 5 - 6, and S3 is between 3 - 4. The specific values are determined according to the actual situation.

[0081] Among them, the warning threshold of the network security index is set as follows:

[0082] Normal (green): NSeI ≥ Se1;

[0083] Mild warning (yellow): Se2 ≤ NSeI < Se1;

[0084] Medium warning (orange): Se3 ≤ NSeI < Se2;

[0085] Severe warning (red): NSeI < Se3;

[0086] Among them, Se1 is between 0.85 - 0.9, Se2 is between 0.75 - 0.8, and Se3 is between 0.6 - 0.65. The specific values are determined according to the actual situation.

[0087] Among them, the multi - level warning signal generation rules include independent trigger and combination rules:

[0088] Among them, for independent trigger, when any index triggers a warning, a global signal is generated according to the highest level.

[0089] Example: NFI = red, NSI = yellow → Global red.

[0090] The combination rules are as follows:

[0091] In the case of double warnings, that is, when two indices simultaneously trigger medium-level warnings, the global warning level is upgraded to severe;

[0092] In the case of continuous triggering, that is, when the same warning signal has not been restored for 3 consecutive cycles, it is automatically upgraded by one level.

[0093] Specifically, the intelligent decision-making module includes a measure-taking unit based on hierarchical warning signals, a dynamic programming unit, and a priority scheduling unit.

[0094] The measures taken according to the hierarchical warning signals are as follows:

[0095] Yellow warning: Record logs, initiate preliminary diagnosis, and check the device load;

[0096] Orange warning: Assign a dedicated person to conduct troubleshooting, adjust network configuration, and expand bandwidth;

[0097] Red warning: Emergency shutdown, initiate emergency response plan, and switch to backup link.

[0098] The dynamic programming unit uses a dynamic programming algorithm to calculate the optimal path for virtual machine migration to minimize service interruption time;

[0099] The priority scheduling unit reserves bandwidth and computing resources for high-priority users according to the service level agreement (SLA).

[0100] Specifically, the service management module includes a fault self-healing unit and an abnormal traffic isolation function.

[0101] Among them, the fault self-healing unit triggers Kubernetes container migration and load balancing strategies when a server outage is detected;

[0102] Among them, the abnormal traffic isolation function redirects DDoS attack traffic to the cleaning center by issuing flow table rules through the SDN controller.

[0103] Specifically, the security and privacy protection module includes a homomorphic encryption data desensitization sub-module, a zero-trust access control sub-module, and a blockchain audit and evidence storage sub-module.

[0104] Among them, the homomorphic encryption data desensitization sub-module: ensures the irreversibility of user privacy data during the analysis process, and is configured to perform fully homomorphic encryption operations on user privacy fields during data processing, so that the encrypted data maintains plaintext calculation equivalence during machine learning model inference and the original data cannot be deduced from the ciphertext.

[0105] Among them, the trusted access control sub-module: Based on the RBAC (Role-Based Access Control) model and the dynamic token verification mechanism, it restricts the access rights of unauthorized users to network resource scheduling instructions, specifically including:

[0106] The modification operation of the resource allocation policy requires approval by at least two levels of administrators;

[0107] The query request for sensitive data needs to be attached with a one-time dynamic verification code.

[0108] Among them, the blockchain audit and evidence storage sub-module: Adopts the consortium chain architecture to record data access logs and resource scheduling operations. Each block contains the operation type, timestamp, and the hash value of the executor's identity, and realizes log tampering detection through smart contracts. When detecting inconsistent block data, an alarm is triggered.

[0109] It should be specifically noted that the visualization and interaction module includes a real-time data monitoring dashboard, a prediction result visualization sub-module, a policy configuration interface, a policy simulator, and an interactive API interface.

[0110] Among them, the real-time data monitoring dashboard: Dynamically renders the following visualization elements:

[0111] Network traffic topology diagram: Displays the real-time traffic between nodes in a force-directed graph, and the edge width is positively correlated with the traffic value;

[0112] Server load heat map: Renders a three-dimensional rack model based on OpenGL, and maps the CPU / memory usage rate with color gradients;

[0113] Service response time curve: Displays the deviation band between the predicted value and the actual value superimposed on the time axis.

[0114] Among them, the prediction result visualization sub-module: Displays the predicted values of service demand for the next 24 hours through an interactive line chart, and supports clicking on a time point to view the detailed prediction confidence interval;

[0115] Marks abnormal events in the scatter plot, where DDoS attack events are marked with red pulse icons, and hardware failure events are marked with yellow exclamation mark icons.

[0116] Among them, the policy configuration interface: Provides the following functions:

[0117] Drag-and-drop policy generator: Drags virtual machine migration rules and bandwidth threshold policy components to the canvas and automatically generates a policy description file in JSON format.

[0118] Among them, the policy simulator: Loads the historical data set and runs the policy simulation, and outputs the resource utilization rate change curve and the potential conflict detection report.

[0119] Among them, the interactive API interface: Based on the RESTful protocol, it opens resource scheduling, data query, and model training interfaces, and supports JSON and Protobuf data formats.

[0120] Specifically, a method for managing intelligent network technology services based on big data is as Figure 2 shown, and specifically includes the following steps:

[0121] Step 1: Collect network service data in real time and build a multi-dimensional data warehouse;

[0122] Step 2: Clean and standardize multi-source data to generate a standardized data stream;

[0123] Step 3: Extract real-time features based on stream computing, and perform anomaly detection using network performance processing and intelligent analysis;

[0124] Step 4: Use historical data to train a prediction model and output the prediction result of service demand;

[0125] Step 5: Generate a dynamic resource allocation strategy according to the prediction result and send it to the execution layer through the API;

[0126] Step 6: Execute dynamic scheduling and collect feedback data, and iteratively optimize the model parameters based on the online learning mechanism and A / B testing.

[0127] The generation of the resource allocation strategy in Step 5 includes:

[0128] Dynamically adjust the number of Pod replicas in the Kubernetes cluster to adapt to the predicted load;

[0129] Use the genetic algorithm to solve the optimal task distribution plan for edge computing nodes.

[0130] The iterative optimization of the model parameters in Step 6 includes:

[0131] Update the weight parameters of the LSTM model based on the online learning mechanism;

[0132] Compare the actual effects of different scheduling strategies through A / B testing and select the optimal strategy.

Claims

1. An intelligent network technology service management system based on big data, characterized in that, Including: Data acquisition module: Configure distributed crawlers, API interfaces, and sensors to obtain multi-source heterogeneous data from user terminals, network devices, and third-party platforms, including user behavior data, network traffic data, device status data, and service quality logs; Data storage module: Adopt a fusion architecture of columnar databases and time-series databases to store data, and implement data partitioning strategies and data lifecycle management strategies based on data sources and access frequencies to store massive structured and unstructured data; Data processing and analysis module: Include a distributed streaming computing engine, a network performance processing sub-module, and an intelligent analysis sub-module; Early warning signal generation module: Set thresholds according to the anomaly index of intelligent analysis to generate hierarchical early warning signals; Intelligent decision-making module: Take different measures according to the hierarchical early warning signals, and optimize resource allocation based on user priorities and service level agreements; Service management module: Execute resource dynamic scheduling instructions, dynamically configure network resources, and adjust network bandwidth, computing nodes, and cache resources in real time according to policies; Security and privacy protection module: Include access control, encrypted transmission, and audit log archiving; Visualization and interaction module: Provide a real-time monitoring dashboard, a policy configuration interface, and an API interaction interface, and display analysis results and system operating status in the form of a visualization dashboard and an API interface.

2. The intelligent network technology service management system based on big data according to claim 1, characterized in that: The acquisition module of the data acquisition module includes a user behavior data acquisition sub-module, a network traffic acquisition sub-module, and a device status monitoring sub-module: The user behavior data acquisition sub-module captures user click, browsing, and transaction behavior data on terminal devices through the embedding technology; The network traffic acquisition sub-module analyzes traffic types and sources based on the NetFlow protocol and deep packet inspection DPI technology; The device status monitoring sub-module obtains load data in real time through the SNMP protocol.

3. A big data-based intelligent network technology service management system according to claim 1, characterized in that: The columnar database uses Apache Parquet or ClickHouse to store user behavior logs and network traffic detail data, and supports column pruning and predicate pushdown to optimize query performance; The time-series database uses InfluxDB or TimescaleDB to store device status metrics and service quality monitoring data, slices by time, and automatically compresses expired data; The data partitioning strategy divides the storage cluster based on data sources and data types, and realizes load balancing through the consistent hashing algorithm; The data lifecycle management strategy divides data into hot data, warm data, and cold data according to access frequencies, and automatically migrates through the LRU algorithm.

4. A big data-based intelligent network technology service management system according to claim 1, characterized in that: The distributed streaming computing engine specifically is: Use Apache Flink or Spark Streaming to perform window aggregation and feature extraction on real-time data streams; The network performance processing sub-module includes a network transmission rate model, a network connection persistence model, and a network fluctuation amplitude model; The network transmission rate model specifically is: where n is the total number of samplings, and R i represents the instantaneous rate of any sampling point, with the unit of bps or Mbps; The specific network connection persistence model is as follows: Where S is the effective service duration and D is the interruption duration; The specific network fluctuation amplitude model is as follows: The intelligent analysis module includes: network fluency index, stability index, and security index; The network fluency index is denoted as NFI, and specifically: where R max is the maximum transmission rate; The network stability index is denoted as NSI, and specifically: where ΔR = R max - R min represents the rate range, and R min is the minimum transmission rate; The security index is denoted as NSeI, and specifically: Among which E secure is the coverage rate of security protection means, E total is the total number of security requirements, N slice is the network slice isolation degree, N max is the maximum number of supportable slices, and α, β are weight coefficients.

5. An intelligent network technology service management system based on big data according to claim 1, characterized in that: The process of generating the hierarchical warning signal is specifically as follows: first, set the warning thresholds for the network fluency index, network stability index, and network security index respectively, and then generate multi-level warning signals, including yellow warning, orange warning, and red warning.

6. An intelligent network technology service management system based on big data according to claim 1, characterized in that: The intelligent decision-making module includes taking measures according to the hierarchical warning signal, a dynamic programming unit, and a priority scheduling unit: The measures taken according to the hierarchical warning signal are as follows: Yellow warning: record the log and start the preliminary diagnosis; Orange warning: assign a dedicated person to conduct a check and adjust the network configuration; Red warning: perform an emergency shutdown and start the emergency plan; The dynamic programming unit includes calculating the optimal path for virtual machine migration using the dynamic programming algorithm to minimize the service interruption time; The priority scheduling unit includes reserving bandwidth and computing resources for high-priority users according to the service level agreement SLA.

7. An intelligent network technology service management method based on big data, characterized in that: The method steps are as follows: Step 1: Collect network service data in real time and construct a multi-dimensional data warehouse; Step 2: Clean and standardize the multi-source data to generate a standardized data stream; Step 3: Extract real-time features based on stream computing, and perform anomaly detection using network performance processing and intelligent analysis; Step 4: Use the intelligent analysis results to output the service demand prediction results and take measures; Step 5: Generate a dynamic resource allocation strategy according to the prediction results and send it to the execution layer through the API; Step 6: Execute dynamic scheduling and collect feedback data, and iteratively optimize the model parameters based on the online learning mechanism and A / B testing.

8. A method for managing intelligent network technology services based on big data according to claim 7, characterized in that, The generation of the resource allocation strategy in step 5 includes: Dynamically adjust the number of Pod replicas in the Kubernetes cluster to adapt to the predicted load; Use the genetic algorithm to solve the optimal task distribution scheme for edge computing nodes.

9. An intelligent network technology service management method based on big data according to claim 7, characterized in that, The iterative optimization of the model parameters in step 6 includes: Update the weight parameters of the LSTM model based on the online learning mechanism; Compare the actual effects of different scheduling strategies through A / B testing and select the optimal strategy.

Images