Data anomaly analysis method for equipment providing edge computing service

Abstract

The invention provides a data anomaly analysis method for equipment providing edge computing service, which comprises the following steps: S1, data acquisition: edge node equipment reports data to a data center through an equipment agent, and the data is transmitted in a message mode through Rabbit MQ message middleware; s2, a rule engine is formulated, and the data center identifies and filters abnormal data through the rule engine; furthermore, the rule engine solves the conventional rule definition through Spring-spel, and carries out business rule customization on the data through drools; s3, data transmission: using kettle as ETL, extracting equipment alarm data filtered by a rule engine from Rabbit MQ, performing rule matching and distribution on the data, falling the alarm data into an HDFS distributed file system, and keeping the alarm data in a model preprocessing folder as a data set of model training; and S4, creating an AI model, and training the equipment alarm data retained every day to generate an alarm classification model, an alarm score model and an alarm rating model.

Description

Technical Field

[0001] The present invention relates to a data analysis method, and in particular to a method for analyzing abnormal data of a device providing edge computing services. Background Art

[0002] In the edge computing industry, different devices may have a negative impact on their ability to provide edge computing services due to their respective hardware configurations, network configurations, and changes in the surrounding environment. Especially with the advent of the Internet of Everything and the high-bandwidth, low-latency era of 5G, the edge computing industry has higher requirements for the efficient use and high-quality distribution of edge nodes. Data-driven automated analysis has become an urgent need to address the challenges of device management complexity, scalability, and flexibility.

[0003] Prior art 1: In an edge computing environment, devices may often have multi-dimensional alarms. For example, the system agent version is too low, causing the device to fail to correctly receive the service delivery, thereby affecting the computing capacity and even causing it to fail to operate normally. At this time, users usually need to provide feedback, and the operator manually restarts the agent or re-delivers the service. Operators need to check hardware and network anomalies based on device data and make adjustments after troubleshooting. However, the disadvantage of prior art 1 is that although some problems can be solved based on operational experience, the troubleshooting and processing process is time-consuming and labor-intensive due to the large number of devices. Some faults may require the intervention of developers, resulting in excessively high troubleshooting costs, while some devices may actually have no problems, resulting in useless workload.

[0004] Prior Art 2: Alarm levels and penalty items are usually manually configured based on their own work experience. However, the disadvantage of prior art 2 is that the manually configured alarm levels and penalty items are subjective, may cause errors due to user feedback and experience analysis, and lack traceability. Due to the wide variety of alarms, the requirements for manual operation are high, and the setting of different alarm thresholds may also lead to various problems. Summary of the invention

[0005] The present invention provides a method for analyzing abnormal data of a device providing edge computing services, which is used to enable the user of the device to discover the problem of insufficient ability of the device to provide edge computing services as early as possible. The technical solution is as follows: A method for analyzing abnormal data of a device providing edge computing services, comprising the following steps: S1: Data collection: The edge node device reports data to the data center through the device agent, and the data is transmitted in the form of messages through the Rabbit MQ message middleware; S2: Formulate a rule engine, through which the data center identifies and filters abnormal data; further, the rule engine uses Spring-spel to solve conventional rule definitions and uses drools to customize business rules for data; S3: Data transmission: Use kettle as ETL to extract device alarm data filtered by the rule engine from Rabbit MQ, match the rules and distribute the data, and store the alarm data in the HDFS distributed file system and save it in the model preprocessing folder as the data set for model training. S4: Create an AI model and train it on the daily retained device alarm data to generate models for alarm classification, alarm scoring, and alarm rating.

[0006] Furthermore, in step S1, the data includes various dimensional information of the device itself, as well as alarm data, hardware configuration information, and network information of the device.

[0007] In step S1, the architecture of the data center adopts the Pigsty cluster for data archiving, querying and analysis.

[0008] Furthermore, in step S2, the Spring-spel is used to solve conventional rule definitions and filter configured one-dimensional rule definitions; the drools is used to customize business rules for data and filter complex rules or rules with customized processes.

[0009] Furthermore, in step S3, the kettle copies a full data stream back to the business database for real-time data query and display, ensuring that users can obtain device status and alarm information in a timely manner.

[0010] Furthermore, in step S4, creating an AI model includes the following steps: S11: Use the dictionary to create a data frame, construct the alarm indicators of the equipment, and realize data preparation; S12: Convert raw data into a format suitable for model input and create feature engineering; S13: Divide the original data into data sets according to a certain ratio; S14: Use XGBoost's XGBRegressor regression model for model training; S15: Using XGBoost model feature_importances = model.feature_importances_ attribute to extract the importance of each alarm classification feature and convert it into a percentage format. In addition, the impact of different alarm classifications on settlement is calculated, and the relationship between alarm classification and settlement impact weight is matched to perform impact analysis.

[0011] Furthermore, in step S11, the dictionary uses the python pandas data analysis library, and the alarm indicators include disk packet loss, network type, network protocol, uplink bandwidth and settlement bandwidth.

[0012] Furthermore, in step S12, the classification features need to be converted into numerical form and encoded.

[0013] Furthermore, in step S13, the square error is used as the loss function and the number of trained trees is specified as 76.

[0014] Furthermore, the method further comprises step S16: manually or automatically updating the business configuration of the alarm classification according to the importance of the alarm feature, and generating an update log and a model processing result as a basis for configuration optimization.

[0015] The device data anomaly analysis method for providing edge computing services has built a set of data based on the device report, which can independently determine whether the alarm threshold is met through the rule engine, and realize the function of dynamically updating the alarm level in combination with the AI ​​algorithm module. It replaces manual analysis through automated analysis of data-driven and intelligent classification AI algorithms, and displays device problems in a real-time and concise manner, so that users can discover device anomalies as early as possible, understand how to handle and repair them so that the device can stably provide edge computing services, thereby improving user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a structural diagram of the method for analyzing abnormal data of devices providing edge computing services; Figure 2 It is a page schematic diagram of the rule making engine of the present invention; Figure 3 It is a flowchart of creating an AI model. DETAILED DESCRIPTION

[0017] like Figure 1 As shown, the device data anomaly analysis method for providing edge computing services includes the following steps: S1: Data collection: The edge node device reports data to the data center through the device agent, and the data is transmitted in the form of messages through the Rabbit MQ message middleware; The edge node device reports data through the device agent in the form of messages, and reports various dimensional information of its own device (such as CPU usage, memory usage, disk status, as well as device alarm data, hardware configuration information, network information, etc.) to the back-end data center through the Rabbit MQ message middleware. In the edge computing environment, edge nodes are widely distributed and large in number, and real-time data collection and transmission are key. Rabbit MQ message middleware can efficiently process a large number of concurrent messages, ensuring the reliability and low latency of data during transmission.

[0018] The main architecture of the data center adopts Pigsty cluster for data archiving, querying and analysis. Pigsty cluster provides high availability and high scalability data storage and processing capabilities, supports real-time and offline analysis of large-scale data, and realizes data persistence.

[0019] S2: Formulate a rule engine, through which the data center identifies and filters abnormal data; further, the rule engine uses Spring-spel to solve conventional rule definitions and uses drools to customize business rules for data; The data in the data center covers regular data and abnormal data. Since model training and model analysis are required for abnormal data, the definition of abnormal data becomes particularly important. By cleaning the data (such as whether the network protocol meets TCP / UDP, whether the network type meets, whether the NAT type meets the full cone, whether the peak packet loss rate is greater than 30, etc.), the problem data can be accurately located and the size of the data set can be reduced. After preliminary screening, more accurate and detailed data can be filtered out, which can improve the execution efficiency of the model algorithm.

[0020] like Figure 2 As shown, the present invention uses Spring-spel to solve conventional rule definitions and uses drools to customize business rules for data. The Spring-spel can solve conventional rule definitions and filter configured one-dimensional rule definitions. The drools can customize business rules for data (such as filtering devices with a CPU utilization rate of less than 80% when the number of CPU threads is greater than 10) and filter complex or customized rules.

[0021] In edge computing environments, rule engines can help quickly identify and filter abnormal data, improve data quality and model training efficiency. For example, the rule engine can filter out node data with network delays exceeding a certain threshold, or filter out node data with frequent alarms within a specific time period, so as to accurately locate the problem node and optimize it.

[0022] S3: Data transmission: Use kettle as ETL to extract device alarm data filtered by the rule engine from Rabbit MQ, match the rules and distribute the data, and store the alarm data in the HDFS distributed file system and save it in the model preprocessing folder as the data set for model training. In the edge computing environment, Kettle can efficiently process and distribute large-scale data, and HDFS can provide reliable distributed storage, supporting offline analysis and model training of large-scale data. Since the data of the present invention is structured data and does not require deep processing, kettle is only used as an ETL (Extract, Transform, Load) tool to extract the device alarm data filtered by the rule engine from RabbitMQ, match the rules and distribute the data; the alarm data that meets the container module, system log module, disk module, etc. will fall into the HDFS distributed file system and need to be retained in the model preprocessing folder as a data set for model training.

[0023] In addition, Kettle copies a full data stream back to the business database for real-time data query and display, ensuring that users can obtain device status and alarm information in a timely manner.

[0024] S4: Create an AI model and train it on the daily retained device alarm data to generate models for alarm classification, alarm scoring, and alarm rating.

[0025] The AI ​​model provided by the present invention is crucial for the alarm profiling of equipment in an edge computing environment. Through a multi-dimensional analysis method, it can comprehensively, accurately and quickly understand the specific circumstances and impacts of the alarm, improve fault handling efficiency and system reliability, and provide strong support for equipment management and optimization.

[0026] Create AI models such as Figure 3 As shown, the steps included are as follows: S11: Data preparation: Use a dictionary (python pandas data analysis library) to create a data frame and construct the alarm indicators of the device, which include disk packet loss, network type, network protocol, uplink bandwidth, settlement bandwidth, etc.

[0027] S12: Feature Engineering: The process of converting raw data into data suitable for model input. For categorical features (such as "network type" and "network protocol"), they need to be converted into numerical form to encode the categorical features.

[0028] The original data is standardized and normalized by using StandardScaler or MinMaxScaler in the scikit-learn library. Packet loss and retransmission may cause points or dirty data (data points that deviate from the normal range), so the data specifications need to be unified.

[0029] S13: Dataset partitioning: The most basic dataset partitioning method is used, which is usually divided according to a certain ratio (such as 80% training set, 20% test set) (suitable for situations where the data volume is large and the model complexity is low).

[0030] S14: Model training: Use XGBoost's XGBRegressor regression model for training, such as model =xgb.XGBRegressor(objective='reg:squarederror', n_estimators=76, learning_rate=0.1, max_depth=5); The squared error (objective='reg:squarederror':) is used as the loss function, the number of trained trees is specified as 76 (this number is to evaluate the impact of different numbers of trees on model performance through cross-validation, so as to select the best number of trees), and the default configurations are used for the tree depth (max_depth = 5) and learning rate (learning_rate=0.1).

[0031] S15: Impact analysis: Use the feature_importances = model.feature_importances_ attribute of the XGBoost model to extract the importance of each alarm classification feature, convert it into a percentage format, calculate the impact of different alarm classifications on settlement, and match the relationship between alarm classification and settlement impact weight.

[0032] S16: Configuration update: According to the importance of the alarm characteristics, the business configuration of the alarm classification is manually or automatically updated, and an update log and model processing results are generated as the basis for configuration optimization.

[0033] In this way, the present invention can dynamically adjust the weight and configuration of alarm classification, improve the accuracy and efficiency of equipment management, and thus enhance the overall resource supply capacity.

[0034] The following terms are used in this article: RabbitMQ: is an open source message broker software (also known as message-oriented middleware) that implements the Advanced Message Queuing Protocol (AMQP). The RabbitMQ server is written in Erlang, while clustering and failover are built on the Open Telecom Platform framework. All major programming languages ​​have client libraries for communicating with the broker interface.

[0035] Pigsty: is an open source PostgreSQL cluster management tool that provides a series of functions for managing PostgreSQL clusters, such as cluster deployment, monitoring, scaling, backup and recovery, etc. Since Pigsty is a tool specifically for managing PostgreSQL clusters, it has good compatibility with PostgreSQL.

[0036] Spring-spel: Spring Expression Language (SpEL for short) is a powerful expression language that supports querying and manipulating object graphs at runtime. The language syntax is similar to Unified EL, but provides additional features, most notably method invocation and basic string template capabilities.

[0037] Drools: Drools Rule Engine is an open source rule engine that is mainly used to implement rule management in Java applications. The Drools rule engine separates rule definition and management from application code, allowing rules to run independently of applications. This improves the reliability and maintainability of rules, and also makes rule updates and management more convenient.

[0038] Kettle: It is an open source ETL tool written in pure Java. It is green and does not require installation. It can extract data efficiently and stably (data migration tool). There are two script files in Kettle, transformation and job. Transformation completes the basic conversion of data, and job completes the control of the entire workflow.

[0039] HDFS: Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware. It has a lot in common with existing distributed file systems. But at the same time, it is also very different from other distributed file systems. HDFS is a highly fault-tolerant system suitable for deployment on cheap machines. HDFS can provide high-throughput data access and is very suitable for applications on large-scale data sets.

[0040] XGBoost is an optimized distributed gradient boosting library designed to be efficient, flexible, and portable. It implements machine learning algorithms under the Gradient Boosting framework. XGBoost provides parallel tree boosting (also known as GBDT, GBM) that can solve many data science problems quickly and accurately.

[0041] ETL is the abbreviation of Extract-Transform-Load, which is used to describe the process of extracting, transforming and loading data from the source to the destination. The term ETL is more commonly used in data warehouses, but its application is not limited to data warehouses.

[0042] Scikit-learn (formerly scikits.learn and also known as sklearn) is a free software machine learning library for the Python programming language. It features a variety of classification, regression, and clustering algorithms, including support vector machines, random forests, gradient boosting, k-means, and DBSCAN, and is designed to be used in conjunction with the Python numerical science libraries NumPy and SciPy.

[0043] The present invention optimizes the process through AI alarm classification automation, which has the following advantages: (1) The rule determination mechanism uses spring-spel as the first-level data screening. If parallel conditions or chain conditions are configured, the drools rules are referenced to solve the data screening problem.

[0044] (2) Update and iterate the alarm classification, level, and score of different types of equipment through AI models.

[0045] (3) The APP interaction mode is changed to be device-based, and the business status of the device and the system agent status can be displayed through the alarm levels, deduction items, and solutions of various dimensions formulated by the present invention, and updates or re-issuances are allowed.

Claims

1. A method for analyzing abnormal data of a device providing edge computing services, characterized in that: The following steps are involved: S1: Data collection: The edge node device reports data to the data center through the device agent, and the data is transmitted in the form of messages through the Rabbit MQ message middleware; S2: Formulate a rule engine, through which the data center identifies and filters abnormal data; further, the rule engine uses Spring-spel to solve conventional rule definitions and uses drools to customize business rules for data; S3: Data transmission: Use kettle as ETL to extract device alarm data filtered by the rule engine from Rabbit MQ, match the rules and distribute the data, and store the alarm data in the HDFS distributed file system and save it in the model preprocessing folder as the data set for model training. S4: Create an AI model and train it on the daily retained device alarm data to generate models for alarm classification, alarm scoring, and alarm rating.

2. The device data anomaly analysis method for providing edge computing services according to claim 1, characterized in that: In step S1, the data includes various dimensional information of the device itself, as well as the device's alarm data, hardware configuration information, and network information.

3. The device data anomaly analysis method for providing edge computing services according to claim 1, characterized in that: In step S1, the architecture of the data center adopts the Pigsty cluster for data archiving, querying and analysis.

4. The device data anomaly analysis method for providing edge computing services according to claim 1, characterized in that: In step S2, the Spring-spel is used to solve conventional rule definitions and filter configured one-dimensional rule definitions; the drools is used to customize business rules for data and filter complex rules or rules with customized processes.

5. The device data anomaly analysis method for providing edge computing services according to claim 1, characterized in that: In step S3, the kettle copies a full data stream back to the business database for real-time data query and display, ensuring that users can obtain device status and alarm information in a timely manner.

6. The device data anomaly analysis method for providing edge computing services according to claim 1, characterized in that: In step S4, creating an AI model includes the following steps: S11: Use the dictionary to create a data frame, construct the alarm indicators of the equipment, and realize data preparation; S12: Convert raw data into a format suitable for model input and create feature engineering; S13: Divide the original data into data sets according to a certain ratio; S14: Use XGBoost's XGBRegressor regression model for model training; S15: Using XGBoost model feature_importances = model.feature_importances_ attribute to extract the importance of each alarm classification feature and convert it into a percentage format. In addition, the impact of different alarm classifications on settlement is calculated, and the relationship between alarm classification and settlement impact weight is matched to perform impact analysis.

7. The device data anomaly analysis method for providing edge computing services according to claim 6, characterized in that: In step S11, the dictionary uses the python pandas data analysis library, and the alarm indicators include disk packet loss, network type, network protocol, uplink bandwidth and settlement bandwidth.

8. The device data anomaly analysis method for providing edge computing services according to claim 6, characterized in that: In step S12, the classification features need to be converted into numerical form and encoded.

9. The device data anomaly analysis method for providing edge computing services according to claim 6, characterized in that: In step S13, the square error is used as the loss function and the number of trees to be trained is specified as 76.

10. The device data anomaly analysis method for providing edge computing services according to claim 6, characterized in that: The method further includes step S16: manually or automatically updating the business configuration of the alarm classification according to the importance of the alarm feature, and generating an update log and a model processing result as a basis for configuration optimization.

Images