Intelligent real-time monitoring and early warning method and system for log information

Abstract

The invention discloses an intelligent real-time monitoring and early warning method and system for log information, and relates to the technical field of intelligent real-time monitoring and early warning, and the method comprises the steps: collecting log data from a business system through an integrated log collector; when the Kafka cluster works normally, transmitting the log data to an Elasticsearch cluster through a Logstash pipeline, and carrying out persistent storage on the log data; real-time monitoring and anomaly detection are performed on log data through a deep learning algorithm based on an LSTM model, and request path reconstruction is performed in combination with a log link tracking tool. According to the method, real-time collection and rapid transmission of log data are ensured, the real-time performance of system monitoring and the data processing efficiency are improved, the processing capacity of large-scale log data is enhanced, the overall operation and maintenance analysis efficiency is improved, a request path can be rebuilt through a log link tracking tool, and the operation and maintenance efficiency is improved. And the specific position and service of the abnormality can be quickly positioned, so that the troubleshooting time is shortened, and the system stability is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent real-time monitoring and early warning, and in particular to an intelligent real-time monitoring and early warning method and system for log information. Background Art

[0002] In today's information age, the stability and security of business systems are crucial to the operation of enterprises. Log information, as a record of system operation status, plays an irreplaceable role in fault diagnosis, performance optimization and security management. With the widespread application of big data and cloud computing technologies, log data analysis technology has also made great progress. At present, the mainstream technical solutions include log collection, transmission, storage and subsequent data mining and analysis. These technologies have played a positive role in improving the efficiency of system operation and maintenance.

[0003] However, with the increasing complexity of business systems, log data has shown an explosive growth. Existing log monitoring technologies face many challenges. Traditional log collection and transmission methods often have low efficiency when processing large-scale, high-concurrency log data. Especially in scenarios with huge data volumes and high real-time requirements, the processing capabilities of existing technologies are stretched. Existing log data analysis algorithms have obvious deficiencies in real-time monitoring and anomaly detection. Due to the lack of deep learning and complex pattern recognition technology, these algorithms cannot meet actual needs in terms of accuracy and recall when identifying unknown attacks and abnormal behaviors. Existing log analysis systems often ignore the importance of log link tracking, resulting in the inability to quickly locate the root cause of the problem when an anomaly occurs, thereby affecting the speed of fault recovery. Existing log analysis technologies have problems including rigid data processing processes, which are difficult to adapt to changes in the needs of different business scenarios, lack of effective early warning mechanisms, and are often discovered only after the problem occurs, making it impossible to achieve pre-emptive prevention. The system has poor scalability and is difficult to cope with the growing data processing needs. These problems seriously limit the application of existing technologies in the field of intelligent real-time monitoring and early warning, making it difficult for enterprises to effectively ensure the stability and security of the system when facing a complex and changing business environment. Summary of the invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by the present invention is: the existing intelligent real-time monitoring and early warning technology has the problems of low efficiency in processing large-scale log data, poor accuracy of real-time monitoring and anomaly detection, lack of effective log link tracking means, and how to achieve efficient and intelligent real-time monitoring and early warning.

[0006] To solve the above technical problems, the present invention provides the following technical solutions: an intelligent real-time monitoring and early warning method for log information, comprising collecting log data from a business system through an integrated log collector; when the Kafka cluster works normally, the log data is transmitted to the Elasticsearch cluster through a Logstash pipeline for persistent storage; real-time monitoring and anomaly detection of log data are performed through a deep learning algorithm based on an LSTM model, and request path reconstruction is performed in combination with a log link tracking tool.

[0007] As a preferred solution of the intelligent real-time monitoring and early warning method of log information described in the present invention, the integrated log collector includes recording log data in an asynchronous manner and sending the log to the Kafka cluster. When the Kafka service cannot receive the log data, the log data is temporarily stored locally, and the Kafka cluster status is monitored based on the heartbeat mechanism.

[0008] As a preferred solution of the intelligent real-time monitoring and early warning method for log information described in the present invention, the Kafka cluster includes a distributed design, and manages the consistency of node information through Zookeeper. When an agent in the Kafka cluster goes offline, the remaining agents automatically take over the data transmission without interruption.

[0009] As a preferred solution of the intelligent real-time monitoring and early warning method for log information described in the present invention, the Logstash pipeline includes transmitting log data to an Elasticsearch cluster, and the Elasticsearch cluster stores the log data through a distributed index.

[0010] As a preferred solution of the intelligent real-time monitoring and early warning method for log information described in the present invention, the deep learning algorithm based on the LSTM model includes the input of the LSTM model as a log template sequence and a component sequence, and judges whether the current log is abnormal by predicting the next log key and component.

[0011] As a preferred solution of the intelligent real-time monitoring and early warning method for log information described in the present invention, the deep learning algorithm based on the LSTM model includes classification of abnormal logs based on the softmax activation function.

[0012] As a preferred solution of the intelligent real-time monitoring and early warning method of log information described in the present invention, the log link tracking tool includes SkyWalking or Zipkin, which tracks the log request path, marks the hierarchical relationship of the log by trace_id and span_id, and reconstructs the complete request link. The trace_id remains unchanged in the same request, and the trace_id of different requests is also different. Each service call will generate a unique span_id. If there is an upstream service call, the span_id will also be recorded as the parent_id to form a hierarchical relationship.

[0013] Another object of the present invention is to provide an intelligent real-time monitoring and early warning system for log information, which can transmit log data to the Elasticsearch cluster through the Logstash pipeline for persistent storage when the Kafka cluster is operating normally, thereby solving the problems of single point failure in the data transmission process and inconvenience of large-scale query and analysis of log data storage in the current intelligent real-time monitoring and early warning technology.

[0014] As a preferred solution of the intelligent real-time monitoring and early warning system for log information described in the present invention, it includes: a data acquisition module, a transmission and storage module, and a log link tracking module; the data acquisition module is used to collect log data from the business system through an integrated log collector; the transmission and storage module is used to transmit the log data to the Elasticsearch cluster through the Logstash pipeline for persistent storage when the Kafka cluster is working normally; the log link tracking module is used to perform real-time monitoring and anomaly detection on the log data through a deep learning algorithm based on the LSTM model, and to reconstruct the request path in combination with the log link tracking tool.

[0015] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a step of an intelligent real-time monitoring and early warning method for log information.

[0016] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of an intelligent real-time monitoring and early warning method for log information.

[0017] Beneficial effects of the present invention: the intelligent real-time monitoring and early warning method for log information provided by the present invention collects log data from the business system through an integrated log collector, thereby ensuring the real-time and comprehensiveness of the log data, providing a high-quality data source for subsequent analysis, improving the real-time performance of system monitoring, and enabling the operation and maintenance team to quickly respond to changes in the system state. When the Kafka cluster works normally, the log data is transmitted to the Elasticsearch cluster through the Logstash pipeline for persistent storage, thereby ensuring high reliability and low latency of the log data during transmission. The log data is monitored in real time and anomaly detection is performed through a deep learning algorithm based on the LSTM model, and the request path is reconstructed in combination with a log link tracking tool, thereby improving the accuracy and timeliness of anomaly detection, reducing false positives and missed positives, and being able to quickly locate anomalies through request path reconstruction, thereby shortening the fault handling time. The present invention achieves better results in improving the real-time performance of system monitoring, improving the overall operation and maintenance analysis efficiency, and improving the accuracy of anomaly detection and the speed of fault location. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0019] Figure 1 This is an overall flow chart of an intelligent real-time monitoring and early warning method for log information provided by the first embodiment of the present invention.

[0020] Figure 2 This is a system overall flow chart of an intelligent real-time monitoring and early warning method for log information provided by the third embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0022] Example 1, reference Figure 1 , is an embodiment of the present invention, and provides an intelligent real-time monitoring and early warning method for log information, comprising:

[0023] S1: Collect log data from business systems through the integrated log collector.

[0024] Furthermore, the integrated log collector includes recording log data in an asynchronous manner, sending logs to the Kafka cluster, and temporarily storing log data locally when the Kafka service cannot receive it, and monitoring the Kafka cluster status based on the heartbeat mechanism.

[0025] It should be noted that the architecture of the log collector FC-log is divided into three components: data collection, data transmission, and persistent storage. The data collection component is integrated into the business system in the form of a jar package, which consists of a log module and a Kafka module. It supports data collection for both physical machine services and cloud services. The log module is mainly responsible for assembling log data, adding custom fields such as the current service name, log level, and thread number to the log information, and performing desensitization, formatting, and other pre-processing on the assembled logs to facilitate subsequent queries and analysis. The Kafka module receives data from the log module and decides whether to send the logs directly to the Kafka cluster or temporarily store them in local files based on the current system situation. If the isActive flag is true, it means that the Kafka cluster The group can receive information normally, and the log data is sent directly to the Kafka cluster. Otherwise, the log data is temporarily saved locally until the Kafka cluster resumes the receiving function, and then the data is synchronized. The Kafka cluster is both a data forwarding intermediary that supports multiple consumers to subscribe to log data, and a temporary data storage center that saves messages that cannot be consumed in time to avoid data loss. The Elasticsearch cluster is a data persistence storage carrier. Compared with traditional relational data, Elasticsearch has more efficient retrieval performance and more flexible expansion capabilities, and is more in line with the storage requirements of distributed system logs. Logstash pipelines are used for data transmission between Kafka and Elasticsearch.

[0026] It should also be noted that Elasticsearch, referred to as ES, is an open source full-text retrieval system based on the Apache Lucene toolkit that supports distributed storage. It can be used to collect, analyze, and store data. It has flexible scalability, real-time storage speed, and efficient retrieval performance that effectively solves the problem of rapid retrieval of large-scale data. The biggest competitive advantage of Elasticsearch is its ultra-high search performance. Different from most relational databases that organize data in the form of relational tables and non-relational databases that organize data using key-value pairs, ES uses indexes as storage units and inverted indexes as storage structures. The inverted index is stored in the form of "keyword-document" and consists of a "word dictionary" and an "inverted file". The word dictionary is a collection of strings that record all word segments that appear in a document. The inverted list records the list of documents corresponding to the word and the position of the word in the document. The inverted file is the physical file where these lists are stored. When querying, the inverted index can return all eligible documents at once, and the query efficiency is much higher than that of the forward index.

[0027] It should also be noted that the heartbeat mechanism and local cache mechanism are used to ensure the stability of data transmission. The Kafka module will regularly check the health status of the Kafka cluster. When the Kafka cluster fails to receive messages due to an abnormality, the log will be temporarily stored in a local file. After the Kafka cluster resumes normal operation, the data will be written back to the Kafka cluster through the disk reading thread to avoid data loss due to Kafka abnormalities. The system itself will not affect the normal execution of business processes due to log problems. When the log writing speed is greater than the reading speed, the memory will be quickly filled up. At this time, a new write request is initiated, either overwriting the original data or discarding the new data, and data loss will inevitably occur. The double buffering mechanism can be applied to solve this problem. Unlike physical machines, services are isolated from each other in a cloud environment. Each service instance runs in a container group (Pod), which is equivalent to an independent virtual machine with separate storage space. If an accident causes the service to restart, Po d will be deleted and recreated, and the logs that have not been processed in time will disappear along with the service. In this case, special processing of cloud services is required to meet the local persistent storage requirements of logs. In the data storage part, data integrity is ensured by backing up different storage centers. Each major futures trading market will set up a storage center, and the log data will be sent to the nearest storage center for storage. In the three trading markets of Asia, South America and Europe, an Elasticsearch sub-cluster is set up to receive log data, and the data distributed in various places are aggregated to the central cluster for analysis. The advantage of this is that it shares the data pressure of reading and writing. The sub-cluster can provide nearly real-time log information, and the staff of the local trading market can find problems in time. The main cluster gathers complete data and can perform more comprehensive analysis. When one of the data centers is unavailable, other centers can temporarily receive data on its behalf and synchronize the data back after the center resumes work.

[0028] It should also be noted that by integrating the log collector to collect log data from the business system, real-time capture and preliminary processing of log data are achieved, the response speed of system operation and maintenance is improved, the accuracy and reliability of log information are ensured, and a solid foundation is provided for subsequent anomaly detection and early warning.

[0029] S2: When the Kafka cluster is working normally, the log data is transmitted to the Elasticsearch cluster through the Logstash pipeline for persistent storage.

[0030] Furthermore, the Kafka cluster includes a distributed design that manages node information consistency through Zookeeper. When an agent in the Kafka cluster goes offline, the remaining agents automatically take over the data transmission without interruption.

[0031] It should be noted that the Logstash pipeline includes transferring log data to an Elasticsearch cluster, which stores log data through distributed indexes.

[0032] It should also be noted that, compared with traditional messaging services, Kafka uses the concept of partitions to ensure that load balancing and orderliness can be improved in the case of multiple concurrent consumers. The number of partitions is the maximum number of concurrent consumers. As long as the number of consumer groups does not exceed the number of partitions, each consumer group can be assigned to an exclusive partition to ensure load balancing. When consuming messages, each partition is assigned to a consumer group, so that each partition will only be consumed by one consumer in the consumer group, thereby ensuring that consumers can consume messages in that partition in order. Kafka has a certain degree of fault tolerance. When a proxy in the Kafka cluster goes offline abnormally, Zookeeper will automatically update the proxy information, and messages that failed to be submitted in this process will also try to be resubmitted. If some components of the system fail, When the system is ineffective, it will not affect the whole system. The Kafka cluster uses Zookeeper to ensure the consistency of node information. Taking the case of adding a new topic in Kafka, when the Kafka node sends a write request for the new topic to the follower node of Zookeeper, the follower node immediately sends the request to the leader node. After receiving the request, the leader node initiates a vote to all follower nodes through atomic broadcast, that is, whether the follower nodes agree to add the topic. The follower nodes return the voting results to the leader node. The leader node determines whether to add the topic based on the voting results. If more than half of the follower nodes agree to the operation, the topic is created successfully and data writing is allowed. At the same time, the result is notified to the follower nodes, and the follower nodes return it to the client.

[0033] It should also be noted that Logstash is an excellent pluggable data collection and processing tool engine that supports the collection of multiple data sources. The pipeline is an independent running unit in Logstash. Each pipeline contains two required elements, input and output, and an optional element filter. The development community provides more than 200 plug-ins to coordinate different inputs, filters, and outputs. Functions include input plug-ins (Inputplugin) for collecting data from multiple data sources. The most common data source types are text data, databases, and message queues; filter plug-ins (Filterplugin) are used to process data, parse unstructured log data into structured and queryable data through regular expressions, and rename, delete, replace, and modify fields; output plug-ins (Outputplugin) are used to output data to a specified target, which can be a specific file or network interface, or a persistent database, ES, etc.

[0034] It should also be noted that the distributed characteristics of the Kafka cluster improve the throughput and fault tolerance of data transmission, and the distributed index of the Elasticsearch cluster enables efficient query and long-term preservation of log data, making historical data analysis possible.

[0035] S3: Use a deep learning algorithm based on the LSTM model to monitor log data in real time and detect anomalies, and use the log link tracing tool to reconstruct the request path.

[0036] Furthermore, the deep learning algorithm based on the LSTM model includes the input of the LSTM model as a log template sequence and a component sequence, and determines whether the current log is abnormal by predicting the next log key and component.

[0037] It should be noted that the deep learning algorithm based on the LSTM model includes classification of abnormal logs based on the softmax activation function.

[0038] It should also be noted that log link tracking tools include SkyWalking or Zipkin, which track the log request path, mark the hierarchical relationship of the log by trace_id and span_id, and reconstruct the complete request link; the trace_id remains unchanged in the same request, and the trace_id of different requests is also different; each service call will generate a unique span_id, and if there is an upstream service call, the span_id will also be recorded as the parent_id to form a hierarchical relationship.

[0039] It should also be noted that a log item is composed of a fixed constant part and a dynamic variable part. The constant part describes the behavior or function of the current program, and the variable part reflects the dynamic context information during the program running process. The constant part is also called a log template or log key. The log template parsing process is to obtain the corresponding template from the unstructured log data and filter the dynamic parameters for subsequent analysis.

[0040] It should also be noted that the model detects log anomalies from the event dimension. In addition to the log message, each log item also contains component information, indicating which service the log comes from. In the model, K is used to represent the log template set, K = {k1, k2, ..., k m}, S represents the component set, S={s1,s2,…,s n}, l1 is used to represent the template value at position i in the log template sequence, K has m possible values, and the probability of each value is affected by the adjacent templates, / is used to represent the value at position i in the component sequence, the same log template may appear in different components, j iThe value of is affected by the log template. In business requests, anomalies may cause changes in business flows. Such failures can be located by changes in component sequences. For this situation, the MC-LSTM anomaly detection model is suitable to process log key sequences and component sequences at the same time. m represents the log item at time t. The combined sequence is obtained through templating and feature extraction, which is expressed as:

[0041] W={(k1, s1), (k2, s2),..., (k t ,s t )}

[0042] The input to the MC-LSTM model is a window W of h log keys and components, expressed as:

[0043] W={(k t-h ,s t-h ), (k t-h+1 ,s t-h+1 ),...,(k t-1 ,s t-1 )}

[0044] Each pair of sequences comes from the log term m t , combined sequence M e ={(k1, s1), (k1, s2), (k3, s3), (k4, s5)}, window size h = 2, then the training data is {(k1, s1), (k1, s2)→(k3, s3)}, {(k1, s2), (k3, s3)→(k4, s5)}, the input of the MC-LSTM model is a time window W of size h; the MC-LSTM model is a multi-class classifier, using cross entropy as the loss function in the training model, and using cross entropy as the loss function of the log key model and the component model, expressed as:

[0045]

[0046]

[0047] loss1 is the loss used to calculate the prediction of the logkey, loss2 is the loss used to calculate the prediction of the component, M is the total number of training samples, n is the number of logkey or component types, k is the number of j is the actual markup of the log template, s j is the actual markup of the component, p j is the probability value of the log key or component, and the Adam algorithm is used to minimize the loss function. In model detection, a log sequence for detection is input, which contains the most recent h log keys w, expressed as:

[0048] w={mt-h , ..., m t-2 , m t-1}

[0049] The probability distributions P1 and P2 of log key detection and component detection are obtained, expressed as:

[0050]

[0051] The goal of the model is to make the next sequence reflected by the detection data sequence be (k i |k i ∈K) or (s i |s i ∈S) has the largest probability, and the possible log keys k and components s are sorted according to the output probability of the softmax activation function. If any probability does not belong to the first g sequences, the result is considered abnormal. The specific calculation process is: define r1 as k t The prediction result of r2 is s t The prediction results are expressed as follows:

[0052] r=r1|r2

[0053] As long as one of r1 and r2 is 1, the final result r is 1, and the log is marked as an abnormal log. The accuracy of the combined LSTM model is affected by the window length h and the normal log range g. Increasing h can improve the accuracy, but the larger the better. Excessive increase in the value of h will affect the performance of the model. When training the model, set h according to the number of services called by the request. After testing, setting h to the minimum number of logs for a single request has the best performance. The setting of the normal log range g will affect the log detection results. If g is set too small, normal logs are easily misjudged as abnormal logs; if g is set too large, abnormal logs are easily missed. It is more appropriate to set g to the maximum number of branches of the request, that is, the maximum number of downstream interface calls that may appear in a request.

[0054] It should also be noted that there is no unified standard for the log information generated by each business system. The lack of certain key fields will affect subsequent use. For example, in distributed services, the lack of thread information will cause interference when troubleshooting irrelevant logs. The lack of request numbers is not conducive to forming log context relationships and it is difficult to build request links. In the log collection stage, some custom fields need to be added to improve the analyzability of the logs. These fields will eventually be stored in Elasticsearch in the form of indexes, as shown in Table 1.

[0055] Table 1 Key fields table

[0056]

[0057] In the case of distributed deployment of microservices, a request will pass through multiple services and generate multiple log information. When querying related log information, how to accurately obtain all related logs is a problem that needs to be solved urgently. In distributed link tracking, ready-made components such as SkyWalking and Zipkin can solve this problem, but such components themselves are highly invasive, and applying them to all logs will consume more resources. Adjustment, upgrade and maintenance require additional maintenance costs. Drawing on the idea of ​​link tracking, logs are grouped by span_id through the marker symbol trace_id to associate the log hierarchy relationship; trace_id remains unchanged in the same request, and trace_id is different for different requests; each service call will generate a unique span_id, and if there is an upstream service call, its span_id will be recorded as parent_id to form a hierarchical relationship.

[0058] It should also be noted that the LSTM model can learn the time series features in log data, effectively identify complex patterns and potential abnormal behaviors, and through the log link tracking tool, it can reconstruct the request path and quickly locate the specific location and service where the anomaly occurs, thereby shortening the troubleshooting time and improving system stability.

[0059] Example 2 is an embodiment of the present invention, which provides an intelligent real-time monitoring and early warning method for log information. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0060] First, the experimental setting includes three main stages: log collection, data transmission and persistent storage, and real-time monitoring and anomaly detection of log data. In the log collection stage, the integrated log collector FC-log is deployed in multiple business systems. The collector uses asynchronous recording to send log data to the Kafka cluster. When the Kafka service is unavailable, the data is temporarily stored locally and the Kafka cluster status is monitored through the heartbeat mechanism. The focus of this stage is to verify the log collection rate and data accuracy; in the data transmission and persistent storage stage, the log data in the Kafka cluster is transferred to the Elasticsearch cluster through the Logstash pipeline; the experiment focuses on data transmission speed and data retention time, especially the transmission efficiency and long-term storage capacity under high concurrency; in the real-time monitoring and anomaly detection stage, a deep learning algorithm based on the LSTM model is used to process log data. The model input is a log template sequence and a component sequence. The next log key and component are predicted to identify anomalies. behavior; this stage focuses on the improvement of error detection rate and system response time. According to experimental results, the log collection rate is increased to 1000logs / s, which is twice that of the traditional method; the data transmission speed reaches 10MB / s, which is an increase of 900%, enhancing the data processing capability under high concurrency, and the data retention time is extended to 30 days, which is an increase of 1250%, which is conducive to long-term data analysis and troubleshooting; in terms of error detection rate, the present invention reaches 95%, which is an improvement over the 70% of the traditional method; the system response time is shortened from 5 minutes to 10 seconds, which improves the fault response speed and reduces the risk of business interruption; the accuracy of log data is improved to 99.9%, which is an improvement over the 90% of the traditional method, ensuring the reliability of log information and the effectiveness of analysis; in summary, the present invention shows innovation and advantages in efficiency, reliability and functionality, and effectively solves the problem of log management in distributed systems; the above report combines experimental data, elaborates on the role and beneficial effects of the invention content, and compares it with the prior art.

[0061] Example 3, reference Figure 2 , is an embodiment of the present invention, which provides an intelligent real-time monitoring and early warning system based on log information, including a data acquisition module, a transmission and storage module, and a log link tracking module.

[0062] The data collection module is used to collect log data from the business system through the integrated log collector; the transmission and storage module is used to transmit the log data to the Elasticsearch cluster through the Logstash pipeline for persistent storage when the Kafka cluster is working normally; the log link tracking module is used to monitor and detect anomalies of log data in real time through a deep learning algorithm based on the LSTM model, and to reconstruct the request path in combination with the log link tracking tool.

[0063] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0064] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0065] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0066] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc. It should be noted that the above embodiments are only used to illustrate the technical solution of the present invention and are not limited. Although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

[0067] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An intelligent real-time monitoring and early warning method for log information, characterized in that: include: Collect log data from business systems through an integrated log collector; When the Kafka cluster is working normally, the log data is transferred to the Elasticsearch cluster through the Logstash pipeline for persistent storage; The deep learning algorithm based on the LSTM model is used to monitor log data in real time and detect anomalies, and the log link tracking tool is used to reconstruct the request path.

2. The intelligent real-time monitoring and early warning method for log information according to claim 1, characterized in that: The integrated log collector includes recording log data in an asynchronous manner, sending logs to the Kafka cluster, and temporarily storing log data locally when the Kafka service cannot receive the data, and monitoring the Kafka cluster status based on the heartbeat mechanism.

3. The intelligent real-time monitoring and early warning method for log information according to claim 2, characterized in that: The Kafka cluster adopts a distributed design and manages node information consistency through Zookeeper. When an agent in the Kafka cluster goes offline, the remaining agents automatically take over the data transmission without interruption.

4. The intelligent real-time monitoring and early warning method for log information according to claim 3, characterized in that: The Logstash pipeline includes transferring log data to an Elasticsearch cluster, which stores the log data via a distributed index.

5. The intelligent real-time monitoring and early warning method for log information according to claim 4, characterized in that: The deep learning algorithm based on the LSTM model includes the input of the LSTM model as a log template sequence and a component sequence, and determines whether the current log is abnormal by predicting the next log key and component.

6. The intelligent real-time monitoring and early warning method for log information according to claim 5, characterized in that: The deep learning algorithm based on the LSTM model includes classification of abnormal logs based on the softmax activation function.

7. The intelligent real-time monitoring and early warning method for log information according to claim 6, characterized in that: The log link tracking tools include SkyWalking or Zipkin, which track the log request path, mark the hierarchical relationship of the log by trace_id and span_id, and reconstruct the complete request link. The trace_id remains unchanged in the same request, and the trace_id of different requests is also different. Each service call will generate a unique span_id. If there is an upstream service call, the span_id will also be recorded as the parent_id to form a hierarchical relationship.

8. A system using the intelligent real-time monitoring and early warning method of log information as claimed in any one of claims 1 to 7, characterized in that: Including data collection module, transmission and storage module, log link tracking module; The data collection module is used to collect log data from the business system through an integrated log collector; The transmission and storage module is used to transmit log data to the Elasticsearch cluster through the Logstash pipeline for persistent storage when the Kafka cluster is working normally; The log link tracking module is used to perform real-time monitoring and anomaly detection on log data through a deep learning algorithm based on the LSTM model, and to reconstruct the request path in combination with the log link tracking tool.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the intelligent real-time monitoring and early warning method for log information described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the intelligent real-time monitoring and early warning method for log information described in any one of claims 1 to 7 are implemented.

Images