An edge gateway device log analysis method based on real-time stream processing
By using real-time stream processing and distributed servers to parse edge gateway logs, the problems of large log data volume and high real-time requirements were solved, achieving efficient and timely log parsing and ensuring the normal operation and information acquisition of edge gateway devices.
Patent Information
- Application Number
- CN202310599727.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-23
- Publication Date
- 2025-11-28
- Estimated Expiration
- 2043-05-23
AI Technical Summary
Edge gateway devices generate large volumes of log data with high real-time requirements. Existing batch processing methods lead to data loss and high latency, affecting normal device operation and information acquisition.
A real-time stream processing approach is adopted, which transmits log data to a distributed server for parsing via the MQTT protocol. By combining an adaptive sampling interval algorithm and an incremental long short-term memory network, server resources are dynamically adjusted to achieve efficient parsing.
It enables real-time processing of edge gateway logs, reduces the risk of data loss, improves system availability and flexibility, and ensures the normal operation of critical applications.
Smart Images
Figure CN116582423B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to an implementation method of edge gateway device log analysis, which realizes edge gateway device log analysis by real-time collection of logs, real-time transmission of logs and real-time analysis of logs. BACKGROUND
[0002] Edge gateway plays an important role in the fields of edge computing and Internet of Things. In the field of edge computing, edge gateway provides efficient data processing and distribution tasks for Internet of Things applications; in the field of Internet of Things, edge gateway can not only transmit data to the cloud, but also connect various sensors and actuators and other devices to realize centralized monitoring and management of the devices. In the era of big data and Internet of Everything, the amount of log data in edge gateway devices is growing rapidly.
[0003] As one of the important components in the fields of edge computing and Internet of Things, the normal operation of edge gateway is crucial, and the log data generated during its operation hides important information related to device status, external interference, device abnormalities, etc. These logs are usually unstructured data in text form. Therefore, it is particularly important to analyze the edge gateway device logs, which not only helps to analyze device faults and quickly locate the fault position, but also further analyzes various indicators of the devices and behavior information of the applications, which is related to the normal operation of the entire cloud-edge system and Internet of Things system.
[0004] Edge gateway device logs usually contain timestamp, log level, log content, log source field and other additional messages. Log analysis refers to extracting the corresponding timestamp, log level, log source, log event template and parameter information from a log message, and finally converting the unstructured data in text form into structured data; among them, extracting the log event template and parameter information is the key to log analysis. The constant part in the log content is retained, and the parameter part is replaced with special symbols to obtain the log event template, which together with the extracted timestamp, log level and log source field constitutes the log template.
[0005] Existing log analysis methods usually use clustering, frequent item, heuristic and other methods to extract log templates, and process the entire log data set in a batch processing manner. However, the importance and rich functionality of edge gateway make it generate a large amount of log data every day, and these logs are usually considered as unbounded data stream. The device itself usually has a function of deleting logs at regular intervals. If the device log data is not analyzed in time, data loss may occur, resulting in lack of important information sources. Therefore, from the perspective of the characteristics of edge gateway and log generation, the present application considers using a real-time stream processing method to analyze the logs, avoiding the problems of high delay and data backlog caused by batch data processing.
[0006] The development of Internet of Things technology makes more and more data generated and transmitted in the form of stream, so it is essential to realize stream processing of data. Stream processing can process data in real time, quickly respond to changes in data, and support processing of multiple data types. More importantly, stream processing can adapt to changing data traffic and processing requirements through horizontal expansion and automatic scaling, improving the availability and flexibility of the system.
[0007] Real-time transmission of logs can ensure that logs are processed immediately. MQTT is a lightweight communication protocol based on the publish-subscribe mode, which can provide real-time and reliable message service for remote devices and is crucial for data transmission in the field of Internet of Things. MQTT is based on TCP / IP protocol, which can avoid data loss and ensure service quality during data transmission. Therefore, the MQTT protocol is used in the present application to transmit edge gateway device logs in real time to a distributed server. SUMMARY
[0008] Since the edge gateway device logs are generated at a high speed, in real time and in large quantities, using the data processing function of the gateway itself to analyze the logs will make the gateway heavily loaded, which may affect other critical applications. The present application decouples the log parsing task from other applications on the edge gateway by transmitting log data to a distributed server for processing, so as to quickly and efficiently perform the log parsing task.
[0009] The general process of the present application includes log collection, log transmission, log parsing, and result output, which can be summarized into three parts: log data stream collection and caching, real-time log data stream parsing, and log template post-processing and fusion. For log data stream collection and caching, the present application uses a custom collection program to listen to the edge gateway and collect device logs in real time, and transmits them from the edge gateway to a distributed cache queue in real time through the MQTT protocol. For real-time log data stream parsing, the distributed server reads data from the cache queue in parallel to perform the parsing task, and the preliminary results of the parsing are output to a management table. For log template post-processing and fusion, after the log templates are de-duplicated or re-parsed, the log templates are corrected using a similarity measurement method, and the final results are output to an external table.
[0010] The specific steps of the method of the present application are as follows:
[0011] Step 1. Use a log collection program to collect logs in real time at different sampling intervals based on a sampling interval adaptive adjustment algorithm based on log generation rate, and perform data cleaning.
[0012] Step 2. Configure the MQTT client in the edge gateway and the distributed cache queue, and collect the logs in real time through the MQTT protocol from the edge gateway to the distributed cache queue, and set a rolling window in the cache queue.
[0013] Step 3. The log parsing program in the distributed server listens to the cache queue and loads the pre-set template set and rule library into the memory, and when data is detected, reads and parses the data in parallel.
[0014] Step 4. Based on the incremental long short-term memory network algorithm, the required number of servers is predicted, and when the server processing capacity is insufficient, the server is increased to reduce the log processing delay.
[0015] Step 5. Post-process the templates obtained in step 3 periodically, mainly to remove duplicate templates, filter templates with missing information, and re-interpret inaccurate templates.
[0016] Step 6. Use the similarity measurement method to fuse the templates obtained in step 5, and the result after fusion is the final log template set.
[0017] The present application is aimed at edge gateway device logs in the field of edge computing and Internet of Things, based on a sampling interval adaptive adjustment algorithm for predicting log generation rate, using a custom log collection program to collect log data in real time and perform data cleaning according to the predicted value at different sampling intervals, to reduce the transmission pressure and processing pressure of the logs; using a data-driven log parsing application program to listen to the cache queue in real time, and using a distributed server to parse the logs in parallel to improve the parsing efficiency and data processing throughput; in addition, based on the incremental long short-term memory network algorithm, the required number of servers is predicted in real time, and when the server processing capacity is insufficient, the server is increased to process the logs waiting in time and reduce the log processing delay. BRIEF DESCRIPTION OF DRAWINGS
[0018] Figure 1 is an edge gateway device log parsing architecture based on real-time stream processing.
[0019] Figure 2 is a process diagram of log parsing.
[0020] Figure 3 is an incremental long short-term memory network server number prediction diagram. DETAILED DESCRIPTION
[0021] The present application will be further described below in conjunction with the accompanying drawings, please refer to Figure 1 . Figure 1The log analysis architecture of the edge gateway device based on real-time stream processing is shown, which is composed of log data stream collection and caching, log data stream real-time analysis, and log template post-processing and fusion.
[0022] (1) Log data stream collection and caching
[0023] The log data stream collection uses a custom collection program to collect device logs, predicts whether it is a log generation peak period in the future period of time based on a weighted moving average method-long short-term memory network combination algorithm, and sets different sampling intervals according to the prediction value, so that more log data is collected during the log peak period and less log data is collected during the non-peak period.
[0024] The log data is transmitted to the distributed cache queue through the MQTT communication protocol, and the use of the MQTT protocol can ensure the real-time and reliability of the log transmission. The distributed cache queue caches the real-time log stream, and the design of the cache queue is to avoid the situation that the server processing is not timely when data surge occurs, resulting in high delay.
[0025] (2) Log data stream real-time analysis
[0026] The data-driven log analysis program listens to the distributed cache queue, and immediately reads the data and executes the analysis task when it listens to the cache queue and data arrives. The analysis program is distributed in multiple servers, and all servers read logs in parallel for analysis to improve log processing throughput and reduce delay.
[0027] The analysis mainly includes extracting log fields and matching parameters in log content to splice the log template. In addition, during the log analysis process, the number of servers is predicted according to the server processing capacity and resource utilization, and the waiting log data is processed in time by dynamically increasing the number of servers, thereby reducing the delay of log data stream real-time analysis as a whole.
[0028] (3) Log template post-processing and fusion
[0029] The log template post-processing mainly deletes duplicate templates and invalid templates, and reanalyzes inaccurate templates. The templates are grouped according to their length, and the similarity between the templates in the group is measured and adjusted using a similarity measurement method, and the templates with higher quality, higher accuracy and higher coverage are retained.
[0030] Please refer to Figure 2 , Figure 2is one of the specific processes in each server to perform log parsing tasks. The log stream comes from the distributed cache queue, the preprocessing extracts the fields of the log, the parameter matching replaces the parameters in the log content with "*", the splicing is the splicing of the log fields and the log content to obtain the structured log template, and the post-processing deletes the duplicate templates and invalid templates and adjusts the templates.
[0031] Please refer to Figure 3 , Figure 3 is a process of predicting the number of servers using an incremental long short-term memory network. The basic data set includes server processing capacity, resource usage, running status, server number, log arrival rate, and log waiting time, which is used to train the LSTM model. Then predict the current number of servers needed, and finally decide whether to add the number of servers to the distributed system to share the log parsing task and reduce the log processing delay. The algorithm uses a batch of new data every certain period of time to train and update the model in an incremental way and make predictions for the next period of time. This process is repeated until the log parsing task is completed.
[0032] The specific steps and detailed description of the edge gateway device log parsing method based on real-time stream processing proposed in this embodiment are as follows:
[0033] (1) Step 1: Custom log collection program, based on log generation rate sampling interval adaptive adjustment algorithm to collect logs in real time at different sampling intervals, and perform data cleaning
[0034] Edge gateway will generate a large number of logs in the case of heavy load or abnormal running, and the number of logs generated under the condition of small load or stable running is relatively small. Configure a custom log collection program in the edge gateway to collect edge gateway device logs at different sampling intervals. In the log generation peak period, the sampling interval is short, and each collection is the log in the interval time period, with large data volume. In the off-peak period, the sampling interval is relatively long, and each collection is only the log at the sampling time, with small data volume.
[0035] This embodiment proposes a sampling interval adaptive adjustment algorithm based on log generation rate to predict the log generation rate in the future period of time. The algorithm is composed of weighted moving average method and long short-term memory network. The algorithm first uses the weighted moving average method to smooth the historical data set, then uses the processed data to train the long short-term memory network model to predict the log generation rate in the future period of time, and the log collection program sets different sampling intervals according to the predicted value.
[0036] The historical data set is a time series data reflecting the resource utilization, load size and running status of the edge gateway, which is composed of CPU utilization, memory utilization, network bandwidth, task number, running status and log generation rate of the edge gateway at multiple historical time points. Over time, these observations have varying degrees of impact on the log generation rate, and the weighted moving average method can give higher weights to recent observations and lower weights to earlier observations, thereby smoothing the historical data and reducing random fluctuations and noise.
[0037] The long short-term memory network can better handle complex time series data by retaining more important information and ignoring secondary information, and can learn complex patterns and features in the data through continuous training, with higher accuracy in prediction tasks. The algorithm combining the weighted moving average method and the long short-term memory network can better utilize the smoothing of the former and the memory capacity of the latter model, improving the accuracy and stability of the model.
[0038] The long short-term memory network model is trained using the smoothed data, the log generation rate is the label of the data set, and the other observations are the input features. The output is the log generation rate in the future period. According to the prediction result, if the log generation rate is large, set a smaller sampling interval, otherwise set a larger sampling interval; if the log generation rate in the future period changes greatly, adjust the sampling interval at the appropriate time point. The pseudo code for predicting the log generation rate and setting the sampling interval according to the proposed algorithm is as follows:
[0039]
[0040] Periodically update the model asynchronously, use new relevant data and log generation rate to train the model, and predict the log generation rate in the next stage. By continuously training the model with new data, the model can be gradually optimized and the accuracy can be increased. At the same time, it can also well adapt to the uncertainty of the log generation rate of the edge gateway device. In addition, asynchronously training the model will not affect the normal operation of the task.
[0041] Compared with the collection program that always maintains the same sampling interval, this custom collection program is more flexible and targeted, and can reduce the collection of irrelevant log data, reduce the transmission pressure and processing pressure of data.
[0042] Clean the collected log data, delete the data with missing log fields and incomplete logs, and delete the log data with unreasonable length according to the maximum and minimum length of historical logs. The purpose of data cleaning is to reduce unnecessary network traffic and server resource consumption, and to transmit the cleaned data.
[0043] (2) Step 2: Configure MQTT clients in the edge gateway and distributed cache queue. The collected logs are transmitted in real time from the edge gateway to the distributed cache queue through the MQTT protocol. A rolling window is set in the cache queue.
[0044] MQTT is a lightweight communication protocol based on the publish-subscribe model, running on the TCP / IP protocol stack. The two parties of communication communicate information through MQTT topics. MQTT is simple, easy to implement, and provides real-time and reliable message services for remote device connections. Using MQTT-based communication protocols to transmit log data can reduce network transmission costs and pressure, and ensure the real-time nature of log stream transmission.
[0045] In the edge gateway and distributed cache system, configure MQTT clients. The MQTT server creates topics, and log data is managed by the MQTT server using topics. The MQTT client in the distributed cache system subscribes to the MQTT server topic, and log data is published to the topic by the MQTT client in the edge gateway. After the server finds that the MQTT client in the cache system subscribes to the topic information, it pushes the received log data to the cache queue. Once the log collection program collects logs, it immediately transmits them. Logs are continuously generated, and transmission is continuously performed.
[0046] The distributed cache queue is a distributed system composed of multiple servers, with high throughput. After transmission, log data is cached on multiple servers. A rolling window is set in the cache queue, with a window size of the amount of logs parsed at a time, which is set based on experience or domain knowledge.
[0047] (3) Step 3: The log parsing program in the distributed server listens to the cache queue and loads the pre-set template set and rule library into memory. When data is detected, it reads and parses the data in parallel.
[0048] The log parsing program is deployed in the distributed server and is a data-driven application, i.e., the program depends on data, and data is the input of the program. The log parsing application is composed of a listening module, a preprocessing module, a parameter matching module, and an output module.
[0049] In the process of log analysis, the same module in the analysis program is distributed in multiple servers, the function realized by the listening module is completed by the main thread, the function of other modules is completed by the sub-thread, when a sub-thread completes the preprocessing operation, the next sub-thread executes the parameter matching operation, and then the next sub-thread arranges the result output, and other threads continue to execute the analysis operation of the next log. That is, inside the server, data passes through each thread in the form of a stream, and the same stream processing operation is performed in each server. Therefore, in the entire log analysis program, the distributed servers read data in parallel, perform analysis in parallel, and output results in parallel, thereby realizing parallel analysis of log streams, which not only can fully utilize server resources and reduce the response time of log processing, but also can improve the analysis efficiency, and ensure the reliability and real-time performance of the analysis.
[0050] An edge gateway device log is composed of a message header and a message content, the message header is composed of a log framework, mainly including a timestamp, a log level, a log source and the like; the message content is a specific log event, composed of a constant string and a variable, the variable is usually a runtime parameter value or device attribute information. The variable is replaced by a special symbol, and together with the constant part, a log event template is formed, and the core of log analysis is to extract the log event template.
[0051] The pre-set template set describes the format of the log, including various fields of the log, field types, field orders, field formats and constant values, and the template set is obtained by analyzing historical samples of the edge gateway device log. The rule library records the timestamp conversion rule, the log data splitting rule, the invalid character and its processing method table, the log field list, the case conversion rule, the regular expression, the keyword, the frequent item list, the key-value pair splitting and the like.
[0052] After the analysis program is started, the state of the listening buffer queue is maintained, and the pre-set template set and the rule library are loaded into the memory, and the function is realized by the listening module. When the log data is transmitted to the buffer queue, the program can listen to the change, and immediately read the data in parallel to execute the specific analysis task. The specific analysis task is realized by the preprocessing and parameter matching modules, the preprocessing operation performs format conversion and unification on the log data, and extracts the timestamp, log level and log source and the like; the parameter matching operation matches the parameter field in the message content, and further extracts the log event template.
[0053] For the read edge gateway device log data, the preprocessing module processes the possible invalid characters according to the invalid characters defined in the rule library and the processing method, and converts and processes the time stamp and data expression form; finally, according to the definition of fields, keywords and data formats in the regular expression and template set, the time stamp, log level and log source fields of the log are extracted. If the rule library cannot be applied to convert the data format or extract the log field, the preprocessing rule is defined according to the domain knowledge and added to the rule library. The preprocessed log data has a more unified form, and the correctness of the content is increased.
[0054] After the log data is preprocessed, the parameters in the log content need to be extracted, which is completed by the parameter matching module. For the message content of a log, the parameter matching module uses "*" to replace the matched parameters according to the regular expression and keyword recorded in the rule library, and retains the constant part, so as to obtain the log event template. For example, the message content of a log is: Forwarding data to Cloud: {"sensor_id":"123","temperature":25.5,"humidity":40.2}, the numbers in it are the parameters appearing in the device running process, which are replaced by "*", and the obtained log event template is: Forwarding data to Cloud: {"sensor_id":"*","temperature":*,"humidity":*}. If the rule library cannot be applied for parameter matching, the expression is defined according to the domain knowledge, such as device ID and name, IP address, username, status value, device attribute value, etc., and the custom rule is added to the rule library. The output module splices the log event template obtained after the processing of the parameter matching module with the log fields extracted in the preprocessing to obtain the structured log template data, and outputs it to the management table for temporary storage.
[0055] (4) Step 4: Predict the required number of servers based on the incremental long short-term memory network algorithm, increase the server when the server processing capacity is not enough, and reduce the log processing delay
[0056] When the processing capacity of the existing distributed server is insufficient to process the data in the cache queue or the data arrival rate is much greater than the server processing capacity, the phenomenon of data waiting will occur, thereby violating the purpose of real-time processing. Log parsing is a computationally intensive task, and in order to reduce the log data processing delay as much as possible, an adaptive method needs to be set to automatically control and adjust the computing resources to meet the resource requirements of log parsing. The present application uses the method of increasing servers to reduce data processing delay and maintain the real-time performance of log parsing. A monitor is set to monitor and record the processing capacity of the distributed server, the server resource usage, the server running status, the server quantity, the data arrival rate and the waiting time of the logs in each cache queue in real time. The waiting time of the logs is the difference between the processing time and the arrival time.
[0057] In view of the high requirement of log parsing on real-time performance, the present application proposes an incremental long short-term memory network (hereinafter abbreviated as "incremental LSTM") algorithm, which trains a model to predict the required server quantity in a period of time in real time, and dynamically increases or decreases the server according to the prediction result. The process is as follows:
[0058] 4-1: Collect a batch of historical data, which consists of the processing capacity of the distributed server, the server resource usage, the server running status, the server quantity, the log arrival rate and the waiting time of the logs in each cache queue, as the basic data used for training the model;
[0059] 4-2: Preprocess the collected data to remove outliers, fill in missing values, normalize, etc. to improve the effect of model training;
[0060] 4-3: Train the model using the incremental LSTM algorithm on the preprocessed data, and use the trained model to predict the server quantity in the future period of time
[0061] 4-4: Increase or decrease the server quantity according to the prediction result to meet the computing resources required for parsing tasks;
[0062] 4-5: Use the data of the next time period to train the model again and update the parameters to predict the server quantity required in the next time period.
[0063] 4-6: Use the new data to repeat steps 4-2 to 4-6.
[0064] The monitor can collect the data mentioned in the new 4-1 after the server finishes processing the log data of each window. The core of the incremental LSTM algorithm is to always use the latest data to train the model and update the parameters to predict the number of servers in the next time period. This process continues as the log parsing task is performed. As the training data increases, the model is gradually optimized, and the accuracy increases. The time period of each prediction is the difference between the time when the first log arrives and the time when the last log arrives.
[0065] By continuously training the model using real-time data streams and predicting the number of servers needed in the next time period in real time, the number of servers can be increased or decreased as needed based on the log parsing situation, making the resources required for parsing tasks scalable. Increasing the number of servers when the processing capacity is insufficient can timely process the logs in the cache queue, reduce the pressure of log data processing, effectively reduce the log processing delay, and meet the real-time requirements.
[0066] (5) Step 5: Regularly post-process the templates parsed in step 3, mainly to remove duplicate templates, filter templates with missing information, and reparse inaccurate templates
[0067] The number of templates is increasing, and logically forms an unbounded template dataset. There may be duplicate or incorrect templates in these templates that need to be post-processed.
[0068] Set a time interval to periodically read a batch of template data, first remove duplicate templates, and then calculate the information entropy of the fields to measure the amount of information in the template and further judge the degree of information loss. The formula for calculating the field information entropy is as follows:
[0069] H = -p*log2 p
[0070] Where p is the probability of the occurrence of a field, which is obtained by calculating the ratio of the number of occurrences to the total number of fields. The information entropy of all fields is calculated and summed to obtain the information entropy of the template. The information entropy of the template reflects the degree of information loss of the template. A lower information entropy may be due to a higher degree of information loss or an invalid template.
[0071] In order to process templates with different information entropy ranges, the present application proposes an information entropy-based post-processing method selection algorithm. For the information entropy of all templates, first, the data distribution is counted, and two thresholds (one large and one small) are set according to the data distribution. Compare the information entropy with the threshold value. If the information entropy is lower than the smallest threshold value, it means that the template has a lot of information loss, and the template is directly deleted. If the information entropy is between the two thresholds, reparse the corresponding log. If it is greater than the maximum threshold, it is retained. This algorithm uses three different post-processing methods to more accurately post-process the templates. The pseudo code of this process is as follows:
[0072]
[0073]
[0074] (6) Step 6: The templates obtained in step 5 are fused using a similarity measure method, and the fused result is the final log template set.
[0075] Template fusion mainly improves the quality and accuracy of templates by merging or adjusting two or more templates. The template set obtained in step 5 is further grouped by template length, and the similarity between two templates is measured by calculating the edit distance between them. The core idea is to calculate the minimum number of fields required to change one template into another. Let the lengths of templates A and B be m and n, respectively. The edit distance between A and B is defined as follows:
[0076]
[0077] where d(i, j) represents the edit distance between the first i fields of template A and the first j fields of template B, and cost(i, j) represents the replacement cost of the ith field of A and the jth field of B.
[0078] Based on the obtained set of edit distances, an empirical value method or unsupervised learning method is used to set a threshold. If the edit distance between two templates is greater than 0 but less than the set threshold, the templates are adjusted by merging, modifying, deleting certain fields or certain templates, etc. to make the templates more covering, and the templates with greater coverage are retained, otherwise both templates are retained. If the edit distance between two templates is greater than the set threshold, both templates are retained.
[0079] The templates after edit distance fusion are the final template set, which are saved to an external table for permanent storage. The parsed log templates can be used for device anomaly detection, fault diagnosis, device performance analysis, and device security monitoring and threat detection, etc.
Claims
1. An edge gateway device log parsing method based on real-time stream processing, characterized in that The method comprises the following steps: Step 1. Collect logs in real time with different sampling intervals using a log collection program, and perform data cleaning; Step 2. Configure an MQTT client in the edge gateway and the distributed cache queue; The collected logs are transmitted in real time to the distributed cache queue through the MQTT protocol by the edge gateway, and a rolling window is set in the cache queue; Step 3. The log analysis program in the distributed server listens to the cache queue, loads a pre-set template set and rule library into the memory, and reads the data in parallel for analysis when data is detected; Step 4. Predict the required number of servers based on the incremental long short-term memory network algorithm; Step 5. Periodically post-process the templates obtained in step 3, including removing duplicate templates, filtering templates with too much missing information, and reanalyzing inaccurate templates; Step 6. Use a similarity measurement method to fuse the templates obtained in step 5, and the fused result is the final log template set; Step 1 collects logs in real time with different sampling intervals, specifically: First, smooth the historical time series data; Second, use the processed data to train a long short-term memory network model to predict the log generation rate in the future; Finally, the log collection program collects logs at different sampling intervals according to the prediction results; during the peak period, the sampling interval is short, and the data volume is large; during the off-peak period, the sampling interval is relatively long, and only the log at the sampling time is collected, with small data volume; Step 4 specifically is: 4-1: Collect a batch of historical data, including distributed server processing capacity, server resource usage, server running status, server quantity, log arrival rate, and log waiting time in each cache queue, as basic data for training the model; 4-2: Preprocess the collected data to remove outliers, fill in missing values, and normalize; 4-3: Train the model using the incremental LSTM algorithm on the preprocessed data, and use the trained model to predict the server quantity in the future; 4-4: Increase or decrease the number of servers according to the prediction results to meet the computing resources required for analysis tasks; 4-5: Use the data in the next time period to train the model again and update the parameters to predict the number of servers required in the next time period; 4-6: Use the new data to repeat steps 4-2 to 4-6.
2. The edge gateway device log parsing method based on real-time stream processing according to claim 1, characterized in that: The smoothing process uses the weighted moving average method.
3. The edge gateway device log parsing method based on real-time stream processing according to claim 1, characterized in that: In step 1, the collected log data is cleaned by deleting log field missing and incomplete log data, and deleting log data with unreasonable length according to the maximum and minimum length of historical logs.
4. The edge gateway device log parsing method based on real-time stream processing according to claim 1, characterized in that: The log analysis program in step 3 is a data-driven application program composed of a listening module, a preprocessing module, a parameter matching module, and an output module, and the same modules are distributed in multiple servers; When log streams arrive at the cache queue, the distributed server reads logs in parallel, performs analysis operations in parallel, and outputs results in parallel to improve throughput and reduce latency.
5. The edge gateway device log parsing method based on real-time stream processing according to claim 1, characterized in that: The pre-set template set describes the format of the log, including various fields of the log, field types, field orders, field formats and constant values; the template set is obtained by analyzing log historical samples of the edge gateway device; the rule library records timestamp conversion rules, log data splitting rules, invalid character and processing method tables, log field lists, case conversion rules, regular expressions, keywords, frequent item lists, key-value pair splitting rules.
6. The edge gateway device log parsing method based on real-time stream processing according to claim 4, characterized in that: The listening module listens to the state of the cache queue and loads the pre-set template set and the rule library into the memory; The preprocessing module processes possible invalid characters according to invalid characters and processing methods defined in the rule library, and converts and processes timestamps and data expression forms; The timestamp, log level and log source of the log are extracted according to the definition of fields, keywords and data formats in the regular expression and the template set; The parameter matching module replaces the matched parameters with symbols and retains the constant part according to the regular expression and the keyword recorded in the rule library, thereby obtaining a log event template; The output module splices the log event template obtained after processing by the parameter matching module and the log fields extracted in the preprocessing, obtains structured log template data, and outputs the data to the management table for temporary storage.
7. The edge gateway device log parsing method based on real-time stream processing according to claim 1, characterized in that: In step 5, a post-processing method selection algorithm based on information entropy is used to post-process log templates in different information entropy ranges; Specifically, double thresholds are set according to the distribution of information entropy data, and by comparing the information entropy with the threshold size, the template is selected to be deleted, retained or reanalyzed.