Method for realizing exception control and real-time monitoring based on probe technology

Abstract

The invention discloses a method for realizing exception control and real-time monitoring based on a probe technology, relates to the technical field of computer network and system monitoring, and aims to overcome the defects of a traditional system in the aspects of real-time performance, accuracy, resource consumption, deployment and maintenance complexity, expansion flexibility, data security privacy and the like. Firstly, a probe is deployed in a key system component, performance and service data are collected, a machine learning algorithm is applied to automatically identify and early warn potential abnormity, and an instant response mechanism is triggered. And data sources such as video monitoring, network traffic and system logs are accessed, and deep potential safety hazards are revealed by using a multi-modal data fusion algorithm. An edge computing technology is introduced, local data preprocessing is realized, transmission delay and load of a central server are reduced, finally, differential privacy is applied to protect information security, a federal learning technology is combined, data privacy is guaranteed, meanwhile, cross-data-source collaborative learning is promoted, and model training efficiency and accuracy are enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of computer network and system monitoring, and in particular to a method for realizing abnormality control and real-time monitoring based on probe technology. Background Art

[0002] With the rapid development of information technology, the scale of networks and systems is expanding, and the complexity is also increasing. Traditional monitoring methods often have problems such as insufficient real-time performance, great impact on system performance, and difficulty in deployment and maintenance, which can hardly meet the current needs of safe and stable operation of networks and systems. Probe technology is a non-intrusive or low-intrusive monitoring method. By dynamically inserting monitoring code (probe) when the software or system is running, it can collect and analyze the status data of the system or network in real time. It has the advantages of high real-time performance, little impact on system performance, and easy deployment and maintenance.

[0003] To ensure a clear understanding of the key terms mentioned in the background technology section of the present invention, the following are definitions of some core terms: (1) Probe technology: a monitoring method that allows monitoring code to be dynamically inserted while the software or system is running to collect and analyze system or network status data in real time; (2) Multi-sensor fusion technology: Integrate data from multiple different types of sensors and use algorithms to process, correlate, and comprehensively analyze these data to obtain more accurate, reliable, and comprehensive information than a single sensor. (3) Near-source processing: technology that processes data in processing units deployed near the data source or at the edge of the network (network edge, device terminal, etc.); (4) Differential privacy: a mathematical framework that protects individual data from being leaked during statistical analysis or machine learning model training; (5) Federated learning: a distributed machine learning approach that allows multiple parties (devices, organizations, or data centers) to jointly train machine learning models without sharing the original data; (6) Multimodal data fusion algorithm: refers to integrating data from different types of sensors (video, audio, temperature, humidity, etc.) to obtain more comprehensive and accurate information.

[0004] However, the current monitoring methods based on probe technology still have some limitations in application: 1. Lack of real-time performance: Traditional monitoring systems often rely on regular data sampling and batch processing, which leads to delayed responses to network attacks and system failures, and cannot respond to new or unknown attack patterns in a timely manner. For example, according to a Gartner report, traditional monitoring systems take an average of several minutes to hours to detect network attacks. This delay makes rapid response and defense difficult, especially when facing rapidly changing network threats; 2. Insufficient accuracy: Detection systems based on static rules are prone to false positives or missed positives. A study conducted by Fireeye showed that enterprises receive more than 17,000 security alerts on average every day, but more than 99% of them are false positives. This not only increases the workload of security analysts, but also reduces the efficiency of responding to real threats; 3. High resource consumption: Traditional monitoring systems require a large amount of computing resources to process and analyze data. For example, according to IDC analysis, enterprise spending on data storage and processing grows by more than 20% each year. This resource consumption not only increases the operating costs of enterprises, but also limits the scalability of monitoring systems. 4. Difficult to deploy and maintain: Traditional monitoring systems require complex configuration and maintenance. A survey conducted by Forrester shows that more than 60% of IT decision makers believe that the deployment and maintenance of monitoring systems is one of the main challenges they face. This is not only because the configuration of monitoring systems is complex, but also because as the network environment changes, monitoring systems need to be constantly updated and adjusted; 5. Insufficient scalability and flexibility: With the development of technologies such as cloud computing and the Internet of Things, the network environment has become more dynamic and changeable. Traditional monitoring systems often find it difficult to adapt to such changes and lack the necessary scalability and flexibility. According to a Cisco report, global IP traffic is expected to triple by 2023, which requires monitoring systems to be able to quickly adapt to the growth of network scale; 6. Data privacy and security issues: With the increase in data breaches, data privacy and security have become a key issue in monitoring technology. According to a research report by the Ponemon Institute, the average cost of a data breach reached $3.86 million in 2018, which emphasizes the importance of protecting data privacy and security in monitoring technology.

[0005] Therefore, a more advanced and flexible monitoring technology is urgently needed to make up for the above shortcomings and ensure the safe and stable operation of the network and system. As a non-intrusive or low-intrusive monitoring method, probe technology, combined with emerging technologies such as intelligent control, multi-sensor fusion, near-source processing, differential privacy and federated learning, is expected to overcome the limitations of existing technologies and provide a more comprehensive and effective abnormal control and real-time monitoring solution. Summary of the invention

[0006] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a method for realizing abnormal control and real-time monitoring based on probe technology. The technical solution adopted is: combining probe technology with intelligent control, multi-sensor fusion technology, near-source processing accelerated response, differential privacy and federated learning technology to realize real-time monitoring and abnormal control of the system. Specifically, by deploying probes in the system, collecting system data in real time, and using multi-sensor fusion technology to process the data, the accuracy and reliability of the data are improved. At the same time, near-source processing accelerated response technology is used to achieve rapid response and processing of abnormal situations. In addition, differential privacy and federated learning technology are used to protect the privacy and security of user data.

[0007] To achieve the above objectives, a method for abnormal control and real-time monitoring based on probe technology is implemented through the following technical solutions: S1: Probe technology integrates intelligent control. The abnormal control and real-time monitoring methods based on probe technology also incorporate elements of intelligence and automation. Combining big data analysis, artificial intelligence algorithms and other technical means, intelligent abnormal detection and early warning are realized to improve defense efficiency. At the same time, automation technology also enables probe technology to automatically respond to and handle abnormal behaviors, reducing the burden on operation and maintenance personnel and improving work efficiency; S2: Multi-sensor fusion technology, combined with high-speed scanning capabilities, intelligent data processing and wireless communication technology, further enhances the function and performance of the probe. By integrating multiple data sources such as video surveillance, network traffic monitoring, system logs, and using multimodal data fusion algorithms for comprehensive analysis, the accuracy and comprehensiveness of anomaly detection are improved. Cross-modal correlation analysis can reveal deeper security risks and judge potential threats by combining personnel behavior patterns with network traffic anomalies; S3: Near-source processing accelerates response, integrating edge computing technology to migrate part of the monitoring processing logic to a location close to the data source, reducing data transmission delays and improving response speed; S4: Data processing and analysis, using differential privacy technology to protect personal privacy information and ensure data security and privacy. Using federated learning technology, collaborative learning and analysis between multiple data sources can be achieved while protecting data privacy.

[0008] Preferably, in said S1, further comprising: The probe collects various performance indicators of the system in real time, such as CPU usage, memory usage, network bandwidth, etc., as well as business-related data, such as request response time and error rate. Through machine learning algorithms, it can automatically identify and predict potential abnormal behaviors, and issue early warnings so that timely measures can be taken. Once abnormal behavior is detected, the preset response measures are automatically triggered to isolate the abnormal source, adjust resource allocation, restart services, etc. This automated response mechanism reduces dependence on manual intervention and improves the self-healing ability of the system. It can also perform intelligent optimization based on the real-time status and historical data of the system, adjust system parameters, optimize resource allocation, etc., to improve the overall performance and stability of the system.

[0009] Preferably, in said S2, further comprising: Select appropriate sensor types according to monitoring requirements, such as temperature sensors, humidity sensors, pressure sensors, etc. These sensors should be able to cover all key parameters of the target system. Clean, denoise and standardize the raw data to improve the quality and availability of the data. Use specific algorithms and rules to fuse data from different sensors. These algorithms can comprehensively consider the data characteristics and relevance of different sensors to obtain more accurate and comprehensive information.

[0010] Preferably, in said S3, further comprising: Deploy near-source processing nodes near the data source (network edge, device terminal, etc.) to pre-process the data collected by the probe, including data cleaning, format conversion, etc. By setting thresholds or rules, perform preliminary filtering of abnormal data to reduce unnecessary data transmission. Migrating part of the processing logic to edge devices can reduce the load on the central server and improve the overall processing capacity and scalability of the system. Use machine learning algorithms or preset rules to detect abnormal behavior in real time. Once an anomaly is detected, an alarm is immediately triggered and an abnormal report is generated.

[0011] Preferably, in said S4, further comprising: During data collection and transmission, random noise is added to perturb the data so that attackers cannot accurately infer specific information about individuals from the data. By setting a privacy budget, the number and frequency of data queries are limited to ensure that the privacy of the data can be maintained during multiple queries and that personal privacy information will not be leaked during data transmission and storage. Local model training is performed on each device or system instead of directly transmitting raw data. The central server is responsible for collecting and aggregating the model parameters of each device or system. During the federated learning process, differential privacy technology is used to perturb the model parameters to further enhance privacy protection.

[0012] Compared with the prior art, the present invention discloses a method for realizing abnormal control and real-time monitoring based on probe technology. The present invention has the following beneficial effects: 1. Probe technology integrated with intelligent control: By collecting system performance indicators and business data in real time, using machine learning algorithms to automatically identify and predict potential anomalies, early warning and automatic triggering of response measures, reducing the burden of operation and maintenance and improving system security and stability; 2. Multi-sensor fusion technology: Combining multiple sensor data and using multi-modal data fusion algorithms to improve the accuracy and comprehensiveness of anomaly detection and reveal deep-seated safety risks; 3. Near-source processing accelerates response: Through edge computing technology, the monitoring processing logic is migrated to the vicinity of the data source to reduce latency, improve response speed and system processing capabilities; 4. Differential privacy and federated learning: Add random noise to data collection and transmission, limit the number and frequency of queries, and protect personal privacy. Use federated learning technology to conduct local model training and global model updates to enhance privacy protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those skilled in the art, other relevant drawings can be obtained based on these drawings without creative work. Figure 1 It is a flow chart of the probe fusion intelligent control and multi-sensor data integration of the present invention; Figure 2 It is a flow chart of the near-source processing and federated learning integration strategy of the present invention. DETAILED DESCRIPTION

[0014] Step 1: Probe technology integrated with intelligent control S11 Probe deployment and configuration: deploy probes on key system components (servers, network devices, databases, etc.), configure probes to collect key performance indicators, such as CPU usage, memory usage, network traffic, etc., and set the trigger conditions for the probes to automatically alarm when the CPU usage exceeds the preset threshold; S12 uses intelligent processing and machine learning algorithms to analyze the data collected by the probe, automatically identify abnormal patterns, set automated response strategies, and automatically restrict access to related IPs when abnormal traffic is detected. Use the Random Forest algorithm to set the number of decision trees n_estimators, which defaults to 100; it can be increased to a higher value based on computing resources. Set the maximum depth max_depth of each tree to prevent overfitting. Set the minimum number of samples required to split internal nodes min_samples_split, which defaults to 'auto'. This algorithm is used to distinguish normal behavior from abnormal behavior; During the real-time monitoring and early warning stage, S13 uses a real-time monitoring interface to display the system status and data collected by the probe. When a potential anomaly is detected, the system automatically issues an early warning to prompt the operation and maintenance personnel to take action.

[0015] Step 2: Multi-sensor fusion technology S21 selects appropriate sensor types according to monitoring requirements: temperature sensor, humidity sensor, pressure sensor, video monitoring sensor, network traffic monitoring sensor, system log monitoring sensor, etc., to ensure that the sensor can cover all key parameters of the target system; deploy sensors at key locations of the system to ensure that the sensors can collect the required data in real time; S22 cleans the raw data, removes noise and outliers, and improves the quality and availability of the data; further processes the data to eliminate possible interference information and ensure the accuracy of the data; standardizes the data collected by different sensors so that they have a unified format and unit to facilitate subsequent data fusion and analysis; S23 selects the appropriate fusion algorithm based on monitoring requirements and data characteristics. Using the selected fusion algorithm, data from different sensors are fused to obtain more comprehensive and accurate information: (1) Kalman filter fusion algorithm: Based on the system dynamic model and measurement model, the Kalman filter algorithm is used to iteratively update the state estimation and error covariance; Q: Process noise covariance matrix, describing the uncertainty of the system. The initial value can be set based on experience and can be adjusted later based on the actual error; R: Observation noise covariance matrix, reflecting the degree of sensor measurement error. Similarly, the initial value can be set based on the error range given in the sensor specification; P: estimated error covariance matrix, initialized to a reasonable small value, and gradually converged with iteration; (2) Weighted Average fusion algorithm, based on the weight of each sensor, calculates the weighted average as the fusion result: weights: weight vector of each sensor data, usually determined based on factors such as sensor accuracy and historical performance; (3) Neural Network (NN) fusion algorithm, training the neural network model, taking multiple sensor data as input, and outputting the fused state estimate: hidden_layers: the number of hidden layers and the number of neurons; activation_function: activation function, ReLU, Sigmoid, etc. learning_rate: learning rate; S24 combines information from multiple data sources to conduct cross-modal correlation analysis to reveal deeper security risks. It determines potential threats by combining personnel behavior patterns with network traffic anomalies, and uses the fused data to perform anomaly detection to promptly discover abnormal behaviors in the system. The fused data and analysis results are output to the monitoring platform or management interface for operation and maintenance personnel to view and analyze.

[0016] Step 3: Near-source processing speeds up response S31 deploys near-source processing nodes near data sources (network edge, device terminals, etc.). These nodes are responsible for preprocessing the data collected by the probes, including data cleaning, format conversion, etc. S32 sets thresholds by using percentiles of data. Values ​​exceeding these percentiles are considered abnormal or train an anomaly detection model (isolation forest, autoencoder, etc.) to identify abnormal patterns. The output of the model can be used as a rule for abnormality. Data points exceeding the threshold are marked as abnormal for further analysis. Data points that do not exceed the threshold are considered normal and can be temporarily ignored to reduce unnecessary data transmission. This step helps to reduce the burden of subsequent processing and improve the efficiency of the overall system; S33 uses machine learning algorithms or preset rules to detect abnormal behavior in real time on near-source processing nodes. Once an anomaly is detected, an alarm is triggered immediately and an anomaly report is generated. This helps to quickly respond to and handle potential security threats; S34 defines specific automatic response rules based on different types of anomalies and business impact analysis. For network attacks, the rules may include cutting off suspicious connections; for service failures, the rules may include restarting services, etc. This automated response mechanism improves the system's self-healing ability, can respond to abnormal events quickly and effectively, reduces reliance on manual intervention and potential damage, and improves system stability and security; Based on the collected real-time and historical data, S35 can intelligently optimize and adjust the system to improve the overall performance and stability of the system. It can adjust resource allocation, optimize system parameters, etc.

[0017] Step 4: Differential Privacy and Federated Learning During the data collection phase, S41 disturbs the data by adding random noise, and selects the most appropriate noise adding method (Laplace mechanism, Gaussian mechanism, etc.) according to the data type and application scenario to ensure the privacy and security of the data. This step is to prevent attackers from accurately inferring specific information about individuals from the data: (1) Laplace noise: For a query function ff, its output is a real value. Add Laplace distributed noise to the query result: NoisyResult=f(D)+Laplace(0,Δf / ε) Where Δf is the sensitivity of the query function (i.e., the maximum impact of changing a data point on the query result), and ε is the privacy parameter (privacy budget) that controls the noise level; (2) Gaussian Mechanism: Similar to the Laplace mechanism, but uses Gaussian distribution on the noise distribution: NoisyResult=f(D)+N(0,(2⋅(Δf)² / ε²)) Where N(μ,σ²) represents a normal distribution with mean μ and variance σ²; Set a privacy budget ϵ. Based on business needs and privacy protection goals, determine the scope of the privacy budget and limit the number and frequency of data queries to ensure that data privacy is maintained during multiple queries. S42 selects the appropriate machine learning model architecture (logistic regression, neural network, etc.) according to the application scenario, and defines the loss function and optimization algorithm. Extract useful features, which may include standardization, normalization and other operations. Use each participant's own private data for training in the local environment, and use mini-batch gradient descent or other optimization algorithms to iteratively update the model parameters. According to the determined privacy budget ε, differential privacy technology is used to add noise to the model parameters or gradient information to prevent attackers from inferring the original data of a single user through this information. Encryption technology (TLS / SSL) can be added to ensure the security of data during network transmission. The noisy model parameters or gradient information are then encrypted and uploaded to the central server. The central server receives and aggregates the data of each participant and updates the global model; S43 uses algorithms such as federated averaging, and the central server updates the global model, regularly evaluates system performance, and adjusts privacy budgets and other parameters when necessary to ensure long-term effectiveness and security. At the same time, it realizes collaborative learning and analysis among multiple data sources, improving the accuracy of the anomaly detection model; S44 continuously verifies the effectiveness of privacy protection mechanisms throughout the process of differential privacy and federated learning. By simulating attacks and data analysis, it ensures that personal privacy information will not be leaked during data transmission and storage. S45 uses the updated global model to detect anomalies on new data. Once abnormal behavior is detected, an alarm is triggered immediately. It can automatically respond according to the preset strategy to isolate the affected system, limit network traffic or notify the security team, and generate an abnormal report.

Claims

1. A method for realizing abnormal control and real-time monitoring based on probe technology, characterized in that: The method comprises the following steps: S1: Deploy probes on key system components (servers, network devices, databases, etc.), configure probes to collect key performance indicators (CPU usage, memory usage, network traffic, etc.), and set trigger conditions (automatic alarm when CPU usage exceeds the preset threshold). Use machine learning algorithms (random forest algorithm, set the number of decision trees n_estimators to 100, and the maximum depth max_depth to prevent overfitting) to analyze probe data, automatically identify abnormal patterns, and set automated response strategies; S2: Select appropriate sensor types (temperature sensors, humidity sensors, pressure sensors, video surveillance sensors, network traffic monitoring sensors, etc.) according to monitoring requirements to ensure that all key parameters of the target system are covered; clean, denoise and standardize the raw data to improve data quality and availability; use specific multimodal data fusion algorithms (Kalman Filter, initialize the process noise covariance matrix Q, observation noise covariance matrix R and estimation error covariance matrix P; or weighted average method, determine the weight vector weights of each sensor data; or neural network, define the number of hidden layers hidden_layers, activation function activation_function and learning rate learning_rate) to fuse data from different sensors to obtain more accurate and comprehensive information; S3: Deploy near-source processing nodes near data sources (network edge, device terminals, etc.), which are responsible for preprocessing the data collected by the probes, including but not limited to data cleaning, format conversion, and other operations; set thresholds by using percentiles of the data, and values ​​exceeding these percentiles are considered abnormal, or train an anomaly detection model (isolation forest, autoencoder, etc.) to identify abnormal patterns; mark data points exceeding the threshold as abnormal and conduct further analysis, and temporarily ignore data points that do not exceed the threshold to reduce unnecessary data transmission; use machine learning algorithms or preset rules to detect abnormal behavior in real time on near-source processing nodes, and once an abnormality is detected, trigger an alarm and generate an abnormality report immediately; S4: Add random noise (Laplace noise or Gaussian noise) to perturb the data during data collection and transmission, select the most appropriate noise adding method according to the data type and application scenario, set the privacy budget ε to limit the number and frequency of data queries, and ensure the security and privacy of the data; use private data in the local environment of each participant to train the model (logistic regression, neural network, etc.), and use mini-batch gradient descent or other optimization algorithms to iteratively update the model parameters; according to the determined privacy budget ε, use differential privacy technology to add noise to the model parameters or gradient information to prevent attackers from inferring the original data of a single user through this information; encrypt and upload the noisy model parameters to the central server, the central server receives and aggregates the data of each participant, and updates the global model.

2. The method according to claim 1, characterized in that The S1 further comprises: The probe collects system performance indicators and business-related data in real time, and automatically identifies and predicts potential abnormal behaviors through the random forest algorithm.

3. The method according to claim 1, characterized in that The S2 further includes: Select the appropriate sensor type according to monitoring requirements, and clean, denoise and standardize the raw data to improve data quality and availability.

4. The method according to claim 1, characterized in that The S3 further includes: Deploy near-source processing nodes near the data source (network edge, device terminal, etc.) to pre-process the data collected by the probe, including data cleaning and format conversion; set thresholds or rules to perform preliminary filtering of abnormal data to reduce unnecessary data transmission.

5. The method according to claim 1, characterized in that The S4 further comprises: Random noise is added to perturb the data during data collection and transmission, and a privacy budget is set to limit the number and frequency of data queries to ensure data security and privacy.

Images