Log processing method and system, log management platform, and electronic device

Abstract

The present invention relates to the technical field of computers, and provides a log processing method and system, a log management platform, and an electronic device. The method comprises: acquiring real-time log data of each service system, and storing the log data in a message queue; according to a preset time window, writing the log data in the message queue into a target file of a specified storage directory; and in response to completion of data writing of a current time window, on the basis of the storage directory of the target file, calling a log analysis interface of a log analysis service to perform log analysis on the target file. The log processing method in the present invention reduces the risk incident rate of service systems, and improves the localization accuracy of an anomaly condition of a service log.

Description

Log processing method and system, log management platform and electronic equipment

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority from application number: 202311507477.4, application date: November 13, 2023, and invention title: Log processing method and system, log management platform and electronic device. The entire contents of this Chinese patent application are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of computer technology, and in particular to a log processing method and system, a log management platform, and an electronic device. Background Art

[0004] With the continuous development of more refined services, numerous business systems have emerged on the business side to provide users with various refined services. Managing and utilizing the log data from these business systems is key to improving the service experience. Related technologies typically conduct risk checks on log data periodically or perform anomaly checks after a risk occurs. This fails to promptly identify potential risks in logs, resulting in a high risk incident rate. Furthermore, each risk check is labor-intensive, making it difficult to ensure the accuracy of the results.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field.

[0006] Summary of the Invention

[0007] The purpose of the present disclosure is to provide a log processing method and system, a log management platform and an electronic device.

[0008] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0009] According to a first aspect of the present disclosure, a log processing method is provided, which includes: obtaining real-time log data of each business system and storing the log data in a message queue; writing the log data in the message queue into a target file in a specified storage directory according to a preset time window; in response to completing the data writing in the current time window, calling a log analysis interface of a log analysis service based on the storage directory of the target file to perform log analysis on the target file.

[0010] Optionally, before calling the log analysis interface to perform log analysis on the target file, the method further includes: mapping the storage directory of the target file to a container corresponding to the log analysis service, so that the log analysis service shares the storage directory of the target file.

[0011] Optionally, before calling the log analysis interface to perform log analysis on the target file, the method also includes: in response to completing data writing in the current time window, sending a message containing the storage directory of the target file to the designated topic of the message queue, so that the log analysis service obtains the storage directory of the target file by consuming the designated topic.

[0012] Optionally, the storage directory of the target file is determined based on the storage path of the target file and the current time.

[0013] Optionally, before writing the log data in the message queue to the target file in the designated storage directory, the method further includes: preprocessing the log data, wherein the preprocessing includes at least one of data format verification, target data filtering, and key data analysis.

[0014] Optionally, writing the log data in the message queue to a target file in a specified storage directory according to a preset time window includes: aggregating the log data in the preset time window by an aggregation function to obtain aggregated data in a target format, where the target format is a data format agreed upon with the log analysis service; and writing the aggregated data in the target format to a target file in the specified storage directory;

[0015] Alternatively, the log data includes various types of log data corresponding to each business system, and each type of log data within a preset time window is aggregated separately through an aggregation function to obtain aggregated data in a target format corresponding to each type of log data, where the target format is the data format agreed upon with the log analysis service; the aggregated data corresponding to each type of log data is written into a target file in a specified storage directory.

[0016] Optionally, the method further includes: if a task failure occurs during the process of writing the log data into the target file, performing state recovery from a most recent checkpoint in a preset manner.

[0017] Optionally, based on the storage directory of the target file, the log analysis interface of the log analysis service is called to perform log analysis on the target file, including: sending a log analysis request containing the storage directory of the target file to the log analysis service, so that the log analysis service obtains the corresponding log data to be analyzed based on the storage directory of the target file, and performs log analysis on the log data to be analyzed.

[0018] Optionally, the method further includes: receiving a response message containing analysis results returned by the log analysis service; parsing the response message to obtain abnormal information in the analysis results; generating alarm information based on the abnormal information, and sending the alarm information to the target object.

[0019] Optionally, the method also includes: receiving a response message containing analysis results returned by the log analysis service; parsing the response message to determine whether there is abnormal information in the analysis results; if there is abnormal information in the analysis results, requesting an access token from the alarm interface; within the authorization time limit of the access token, calling the alarm interface through the authorized access token to trigger the generation of an alarm event for the abnormal information.

[0020] According to a second aspect of the present disclosure, a log processing system is provided, comprising: a real-time processing module configured to obtain real-time log data of each business system and store the log data in a message queue; write the log data in the message queue to a target file in a specified storage directory according to a preset time window; and, in response to completion of writing data in the current time window, send a log analysis request containing the storage directory of the target file to a log analysis module;

[0021] The log analysis module is configured to obtain corresponding log data to be analyzed based on the storage directory of the target file included in the received log analysis request, and perform log analysis on the log data to be analyzed.

[0022] Optionally, the system further comprises: an alarm service module;

[0023] The log analysis module is configured to return analysis results including abnormal information to the real-time processing module;

[0024] The real-time processing module is configured to send an access token request for the alarm interface to the alarm service module;

[0025] The alarm service module is configured to return a first authorized access token to the real-time processing module based on receiving the access token request;

[0026] The real-time processing module is configured to send an alarm request including the abnormality information to the alarm service calling module through the first authorized access token within the authorization time limit of the first authorized access token;

[0027] The alarm service module is configured to trigger generation of an alarm event for the abnormal information based on the received alarm request.

[0028] Optionally, the log analysis module is configured to return analysis results including exception information to the real-time processing module;

[0029] The real-time processing module is configured to send an alarm request including the abnormal information and its corresponding original log access address to the alarm service module;

[0030] The alarm service module is configured to generate a corresponding alarm event based on the abnormal information, so that the alarm processing object obtains the original log data through the target IP address based on the original log access address and the corresponding second authorized access token, and processes the alarm event based on the original log data; the second authorized access token is an access token for the original log data.

[0031] According to a third aspect of the present disclosure, a log management platform is provided, including the log processing system in the above embodiment.

[0032] According to a fourth aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the log processing method in the above embodiment is implemented.

[0033] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the log processing method as in the above-mentioned embodiment.

[0034] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0036] FIG1 schematically shows one of the flow charts of a log processing method according to an embodiment of the present disclosure.

[0037] FIG2 schematically shows a time window diagram according to an embodiment of the present disclosure.

[0038] FIG3 schematically shows a flow chart of writing message queue data into a target file according to an embodiment of the present disclosure.

[0039] FIG4 schematically shows a second flowchart of a log processing method according to an embodiment of the present disclosure.

[0040] FIG5 schematically shows one of the schematic diagrams of a log processing system according to an embodiment of the present disclosure.

[0041] FIG6 schematically shows a second schematic diagram of a log processing system according to an embodiment of the present disclosure.

[0042] FIG7 schematically shows a block diagram of a log management platform according to an embodiment of the present disclosure.

[0043] FIG8 schematically shows a module diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.

[0045] In addition, the described features, structures or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, well-known methods, devices, implementations or operations are not shown or described in detail to avoid blurring various aspects of the present disclosure.

[0046] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0047] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0048] In one embodiment of the present disclosure, a log processing method is proposed. FIG1 shows a flow chart of the log processing method. As shown in FIG1 , the log processing method can be applied to a real-time processing service, and includes at least the following steps:

[0049] Step S110: acquiring real-time log data of each business system and storing the log data in a message queue.

[0050] Step S120 , writing the log data in the message queue into a target file in a designated storage directory according to a preset time window.

[0051] Step S130 , in response to the completion of data writing in the current time window, based on the storage directory of the target file, the log analysis interface of the log analysis service is called to perform log analysis on the target file.

[0052] The log processing method of the embodiment of the present disclosure can obtain the real-time log data of each business system and store the log data in the message queue; write the log data in the message queue to the target file of the specified storage directory according to the preset time window; in response to the completion of the data writing of the current time window, based on the storage directory of the target file, call the log analysis interface of the log analysis service to perform log analysis on the target file. On the one hand, it can realize the real-time analysis and processing of the log data of each business system, quickly and timely discover the potential risks existing in the log, reduce the risk accident rate, and avoid the greater accident risk caused by the accumulation of errors, while reducing the workload of operation and maintenance personnel. On the other hand, by storing the log data of the message queue in the target file of the specified storage directory, the log analysis service can accurately locate the data to be analyzed based on the storage directory, which is convenient for operation and maintenance personnel to accurately locate anomalies based on the analysis results, thereby improving the accuracy of anomaly detection.

[0053] In order to make the technical solution of the present disclosure clearer, the steps of the log processing method are described below.

[0054] In step S110 , real-time log data of each business system is acquired and stored in a message queue.

[0055] In this exemplary embodiment, each business system can be a business system in which a service provider provides various services to its service objects. During the operation of each business system, a series of log records, i.e., log data, will be generated. The real-time processing service can collect log data from each business system through various log collection tools, for example, log collection components such as Logstash, Filebeat, and rsyslog; or it can query and obtain log data from the log database, which is not limited in this example. Since the data format of the log data of each business system may be different, a log collection component such as Filebeat can be selected for log data collection, so that the log data of each business system can be unified in format through Filebeat. The collected log data is forwarded to a stream processing platform (such as Kafka).

[0056] In step S120 , the log data in the message queue is written into a target file in a designated storage directory according to a preset time window.

[0057] In this exemplary embodiment, the file storage directory can be configured in advance, and the message queue data can be written to the file according to the preset time window. For example, a streaming computing engine (such as Flink) can be used to use the message queue as the input source and the file system as the output sink to write the log data to the target file in the specified directory. The infinite real-time data stream can be split into data buckets of finite size by a preset time window (such as 5 minutes, 10 minutes, 15 minutes, etc.). As shown in Figure 2, the log data corresponding to each user is divided into different data buckets for storage according to the time window, and the log data within a time window can be written to a file.

[0058] For example, the following steps can be used to write log data in a message queue to a target file in a specified storage directory:

[0059] Aggregate the log data within the preset time window through the aggregation function to obtain the aggregated data in the target format, and write the aggregated data in the target format into the target file in the specified storage directory.

[0060] In this exemplary embodiment, the target format is the data format agreed upon with the log analysis service. For example, for log data in json format, the target format for writing to the file can be agreed upon as jsonArray. In this example, the input format of the log data can be agreed upon with the log analysis service in advance. The log data can be aggregated through an accumulator. The aggregation function is a function that performs incremental calculations based on the state of the intermediate calculation results. It is an iterative calculation method. During the window processing process, the data of the entire window does not need to be cached, so the execution efficiency is high. For example, the aggregation function is an aggregate operator, which applies the aggregation function to each window, aggregates the values ​​in an incremental manner, and keeps the state of each window in an accumulator.

[0061] Alternatively, in some embodiments, log data can be categorized by business system, and the expected corresponding log data can be written to a file based on the business system. The log data can include various types of log data corresponding to each business system. Each type of log data within a preset time window is aggregated using an aggregation function to obtain aggregated data in a target format corresponding to each type of log data. The aggregated data corresponding to each type of log data is then written to a target file in a designated storage directory.

[0062] In this exemplary embodiment, log data can be written to files separately according to business systems. That is, log data from different business systems can be aggregated and written to corresponding target files. In this example, business system identifiers can be added to storage directories or file names to facilitate subsequent anomaly detection, especially when troubleshooting anomalies from business systems with similar log data, greatly improving anomaly detection efficiency.

[0063] In some embodiments, before the log data in the message queue is written to the target file in the designated storage directory, the log data may be preprocessed, where the preprocessing includes at least one of data format verification, target data filtering, and key data analysis.

[0064] In this exemplary embodiment, data format verification refers to checking whether the log data conforms to the specified format, for example, checking whether the read log data is in the unified json format collected by the Filebeat component, and discarding data that does not conform to the format. Target data filtering refers to identifying and filtering out log data that does not need to be paid attention to or is unknown or obviously erroneous in the log. For example, target data filtering can be achieved by pre-configuring filtering conditions, which can shield data that the user does not pay attention to or is useless, thereby reducing the amount of data processing. Key data parsing refers to parsing out key information in the log data, such as the business system name, event time, etc., to facilitate subsequent classification statistics or classification feedback of the analysis results, etc., in order to achieve rapid response. The log data can be pre-processed in the order of data format verification, target data filtering, and key data parsing, or at least one of the above pre-processing can be performed according to actual needs. This example does not limit this.

[0065] In some embodiments, the storage directory of the target file can be named according to specified rules. For example, the storage directory of the target file can be determined based on the storage path of the target file and the current time, and the target file can be named based on the current time. For example, the data storage directory can be configured, with the current date as the secondary directory and the file name as the tertiary directory. The current time point can be spliced ​​into the file name in the file name, so that data can be quickly located based on the file storage directory, and fast data reverse query can be achieved during the abnormality troubleshooting process. For example, the data storage directory can be configured as outputPath, the secondary directory corresponding to the current date is yyyyMMdd, and the file name spliced ​​at the current time point is: part_yyyyMMddHHmm, then the file storage directory is outputPath / yyyyMMdd / part_yyyyMMddHHmm.

[0066] In some embodiments, if a task fails in the process of writing log data to the target file, the state is restored from the most recent checkpoint in a preset manner.

[0067] In this exemplary embodiment, the preset method can be a configured fault tolerance mechanism. When a task fails, the fault tolerance mechanism is used to restore the most recent data state. For example, Flink's Checkpoint mechanism, when enabled, causes the state to be persisted with the Checkpoint, preventing data loss and ensuring consistency during recovery.

[0068] For example, as shown in Figure 3, the process of writing input data from a message queue to the file system involves configuring the connection information for the input message queue. For example, a Kafka message queue (Kafka cluster) can consist of one or more servers, with each server node acting as a broker. Brokers store topic data. Each message published to the Kafka cluster has a category, called a topic. A topic can have multiple partitions, and data within a topic is stored on each broker (server node) by partition. The Kafka cluster uses a publish-subscribe model for message queues. Data publishers are called producers, and data consumers are called consumers. Consumers can consume data from multiple topics, and each consumer belongs to a specific consumer group. First, configure the connection information for the Kafka message queue. This information may include the Kafka service address, subscribed topics, consumer group ID, and the starting consumer location (the starting point for reading data from the message queue, such as the starting broker). Based on the configured connection information, log data is read. After the log data is preprocessed by the format validator, filter, and parser, it enters the accumulator for aggregation according to the preset time window, and the aggregation results are written to the target file according to the specified storage directory.

[0069] In step S130 , in response to the completion of data writing in the current time window, a log analysis interface of the log analysis service is called based on the storage directory of the target file to perform log analysis on the target file.

[0070] In this exemplary embodiment, when data writing within a time window is complete, log analysis can be performed on the target file by calling the log analysis service's log analysis interface, thereby enabling real-time processing of log data. The log analysis service is an application that provides log data analysis for various business systems. It can be executed on various computing devices, such as physical servers, cloud servers, server clusters, or distributed systems corresponding to log analysis. It can also be deployed on the same or partially identical computing devices as the current real-time processing service, but this example does not limit this.

[0071] Before performing log analysis, you need to obtain the corresponding log data based on the storage directory of the target file. You can obtain the log data of the target file through various methods such as file sharing and data transmission.

[0072] Exemplarily, before calling the log analysis interface to perform log analysis on the target file, the storage directory of the target file may be mapped to a container corresponding to the log analysis service so that the log analysis service shares the storage directory of the target file.

[0073] In this exemplary embodiment, the log analysis service and the real-time processing service can run in different containers respectively. When the containers are deployed, the storage directory of the target file is mapped to the corresponding containers of the log analysis service and the real-time processing service respectively, so that the log analysis service and the real-time processing service share the storage directory of the target file, thereby ensuring that the log analysis service can accurately obtain the data to be analyzed without data link communication, simplifying the data processing process and improving the real-time performance of data processing.

[0074] Alternatively, before calling the log analysis interface to perform log analysis on the target file, in response to completing data writing in the current time window, a message containing the storage directory of the target file is sent to the specified topic of the message queue, so that the log analysis service obtains the storage directory of the target file by consuming the specified topic.

[0075] In this exemplary embodiment, the real-time processing service can send a specified message to a specified topic of a message queue (such as a Kafka message queue) to indicate the storage directory of the target file corresponding to the current time window when the data writing of the current time window is completed. The specified message can be customized by pre-configuration. The completion of writing the data of a time window to the target file will trigger the sending of the specified message to the specified topic, so that the log analysis service obtains the storage directory when consuming the specified topic (i.e., obtaining the data of the specified topic). This example actively sends a message to the message queue topic so that the log analysis service obtains the storage directory information when consuming the topic. It can use the existing data processing process to achieve the acquisition of key information (storage directory), reduce the information interaction process between the real-time processing service and the log analysis service, simplify the data processing process, and improve the real-time performance of data processing.

[0076] Exemplarily, the following steps can be used to implement log analysis of the target file based on the storage directory of the target file and call the log analysis interface of the log analysis service: send a log analysis request containing the storage directory of the target file to the log analysis service, so that the log analysis service obtains the corresponding log data to be analyzed based on the storage directory of the target file, and performs log analysis on the log data to be analyzed.

[0077] In this exemplary embodiment, the log analysis service can be an existing log analysis system, such as a log analysis system built based on natural language and deep learning, or a log analysis system built according to actual needs, which is not limited in this example. The real-time processing service triggers the log analysis service to perform corresponding log analysis by sending a log analysis request to the log analysis service. The log analysis request may include the storage directory of the target file or the storage address of the target file, and may also include information such as the analysis type and the analysis parameters that need to be obtained. This example does not limit this. The log analysis of the log analysis service can be an exception analysis, such as an exception matching based on a preset template library, and determining the matched data as abnormal data; it can also be a statistical indicator analysis, such as clustering statistics on log data to obtain various statistical indicators; it can also be other types of log analysis such as application health analysis, which is not limited in this example. This example can perform customized analysis of logs by sending requests to improve system flexibility.

[0078] In some embodiments, an abnormality alarm can be issued based on the analysis results. For example, the real-time processing service receives a response message containing the analysis results returned by the log analysis service; parses the response message to obtain abnormality information in the analysis results; generates alarm information based on the abnormality information, and sends the alarm information to the target object.

[0079] In this exemplary embodiment, the response message may include the name or identifier of the abnormal system (such as an application name list), the log data address (such as a file storage directory), the IP address, the abnormal type template and other information, and may also include other information such as the response time, which is not limited in this example. The response message is parsed to obtain the content of each field of the message (such as the data field), and an alarm message is generated based on the content of the field including the abnormal information. The alarm information may include the cause of the alarm, the source of the alarm, the alarm level and other information. The alarm information is sent to the target object (such as the corresponding business person in charge or operation and maintenance personnel, etc.) via email, internal platform or text message, so that the target object can handle the abnormality in the alarm in a timely manner, realize the timely discovery and timely processing of the abnormality, and improve the efficiency of the abnormal response.

[0080] In other embodiments, the real-time processing service receives a response message containing the analysis results returned by the log analysis service; parses the response message to determine whether there is abnormal information in the analysis results; if there is abnormal information in the analysis results, requests an access token from the alarm interface; within the authorization time limit of the access token, calls the alarm interface through the authorized access token to trigger the generation of an alarm event for the abnormal information.

[0081] In this exemplary embodiment, it is possible to determine whether there is abnormal information through the specific field information of the analysis result. For example, it is possible to determine whether there is abnormal information through the data field of the analysis result. If the data field is empty, it is determined that there is no abnormal information. In the case of abnormal information, the alarm interface can be called to generate a corresponding alarm event, that is, the alarm service is called to perform alarm processing. In order to ensure data security, an access token for the alarm service is obtained before calling the alarm service. Exemplarily, an authorization token for the alarm service can be obtained by requesting a login interface through account information such as a username and password. The authorization token can have a specified validity period (such as 24 hours). If the validity period is exceeded, the authorization token becomes invalid and a new authorization token needs to be obtained. This example realizes the alarm function by calling the existing alarm service interface to generate an alarm event, which can perform real-time alarms for abnormal situations and ensure the immediacy of the alarm; at the same time, the information security of the alarm service is guaranteed by the access token.

[0082] 4 , a log processing method in another embodiment provided by the present disclosure may be performed between a stream processing platform, a real-time processing service, a log analysis service, and an alarm service, and specifically includes the following steps:

[0083] Step S401: Collect corresponding log data from each business system through a log collector and store it in a message queue of a stream processing platform.

[0084] Step S402: Process the log data in the service consumption message queue in real time.

[0085] Step S403: The real-time processing service writes the log data in the message queue into a target file in a designated storage directory according to a preset time window.

[0086] Step S404 : In response to completing the data writing in the current time window, the real-time processing service calls the log analysis interface of the log analysis service to perform log analysis on the target file.

[0087] Step S405: The log analysis service obtains the data to be analyzed based on the storage directory of the target file.

[0088] Step S406: The log analysis service performs an exception analysis on the data to be analyzed.

[0089] Step S407: The log analysis service returns the analysis result to the real-time processing service.

[0090] Step S408: The real-time processing service verifies the analysis results and determines whether an anomaly exists.

[0091] Step S409: The real-time processing service requests an authorization token from the alarm service.

[0092] Step S410: The alarm service verifies the authority and returns authorization token information to the real-time processing service.

[0093] Step S411: The real-time processing service calls the alarm interface of the alarm service based on the authorization token to generate an alarm event.

[0094] In the above embodiments, the log analysis service and the alarm service can be existing log services, and the real-time processing service can be deployed in various servers, such as physical servers, cloud servers, server clusters or distributed systems, etc. Existing services can be accessed through interface calls.

[0095] The log processing method provided by the present invention utilizes stream processing tools to realize real-time processing of log data. It can quickly and accurately locate abnormal information in a large amount of log data and promptly issue an alarm for abnormal situations. This facilitates operation and maintenance personnel to promptly discover and handle abnormal situations in a large amount of log data, thereby avoiding greater accidents and losses as much as possible, reducing the workload of manual log review, and realizing unified log management of complex business systems.

[0096] The following describes a system embodiment of the present disclosure, which can be used to execute the log processing method described above. For details not disclosed in the system embodiment of the present disclosure, please refer to the embodiment of the log processing method described above.

[0097] 5 , the present disclosure further provides a log processing system 500 , which may include a real-time processing module 510 and a log analysis module 520 , wherein:

[0098] The real-time processing module 510 is configured to obtain real-time log data from each business system and store the log data in a message queue; write the log data in the message queue to a target file in a specified storage directory according to a preset time window; and in response to completing the data writing in the current time window, send a log analysis request containing the storage directory of the target file to the log analysis module;

[0099] The log analysis module 520 is configured to obtain corresponding log data to be analyzed based on the storage directory of the target file included in the received log analysis request, and perform log analysis on the log data to be analyzed.

[0100] In this example implementation, the real-time processing module 510 may be used to process log data in real time, and the log analysis module 520 may be used to perform various analyses on the real-time log data. The two modules may run in different containers, respectively.

[0101] In some embodiments of the present disclosure, based on the aforementioned solution, the system 500 may further include: an alarm service module 530;

[0102] The log analysis module 420 is configured to return analysis results including exception information to the real-time processing module;

[0103] The real-time processing module 510 is configured to send an access token request for the alarm interface to the alarm service module;

[0104] The alarm service module 530 is configured to return a first authorized access token to the real-time processing module based on receiving the access token request;

[0105] The real-time processing module 510 is configured to send an alarm request including abnormality information to the alarm service calling module through the first authorized access token within the authorization time limit of the first authorized access token;

[0106] The alarm service module 530 is configured to trigger the generation of an alarm event for abnormal information based on the received alarm request.

[0107] In this example embodiment, the real-time processing module can issue an exception alarm by sending an alarm request to the alarm service module. Prior to this, the real-time processing module can first obtain access rights to the alarm service interface. Specifically, the authorization token (first authorization access token) can be obtained by sending an authorization request based on a username and password.

[0108] In some embodiments of the present disclosure, based on the aforementioned solution, the log analysis module 520 is configured to return analysis results including exception information to the real-time processing module;

[0109] The real-time processing module 510 is configured to send an alarm request including abnormal information and its corresponding original log access address to the alarm service module;

[0110] The alarm service module 530 is configured to generate corresponding alarm events based on the abnormal information, so that the alarm processing object obtains the original log data through the target IP address based on the original log access address and the corresponding second authorized access token, and processes the alarm event based on the original log data; the second authorized access token is an access token for the original log data.

[0111] In this example implementation, the original log access address refers to the access address of the log data in the business system, the alarm request can include information such as affiliated monitoring platform identification, alarm sending platform IP address, alarm event details, the alarm event details can include alarm exception type and original log access address (such as original log link), and the original log file can be obtained by accessing the original log link. Even if the alarm service module and the real-time processing module and the log analysis module are deployed in different devices like this, the original log data can still be obtained easily, and the alarm service module is convenient for checking the cause of the alarm. This example takes into account the security of log data, and similarly, the access token (the second authorized access token) of the original log data can be obtained by logging in with a user name and password. The access IP restriction of the original log data can also be increased, for example, only open access rights to designated IP, so as to protect the safety of the original log data, improve data security.

[0112] In some embodiments of the present disclosure, based on the aforementioned scheme, the real-time processing module 510 is configured to map the storage directory of the target file to the corresponding container of the log analysis service before calling the log analysis interface to perform log analysis on the target file, so that the log analysis service shares the storage directory of the target file.

[0113] In some embodiments of the present disclosure, based on the aforementioned scheme, the real-time processing module 510 is configured to, before calling the log analysis interface to perform log analysis on the target file, send a message containing the storage directory of the target file to the designated topic of the message queue in response to completing data writing in the current time window, so that the log analysis service obtains the storage directory of the target file by consuming the designated topic.

[0114] In some embodiments of the present disclosure, based on the aforementioned solution, the storage directory of the target file is determined based on the storage path of the target file and the current time.

[0115] In some embodiments of the present disclosure, based on the aforementioned scheme, the real-time processing module 510 is configured to preprocess the log data in the message queue before writing the log data to the target file in the specified storage directory, and the preprocessing includes at least one of data format verification, target data filtering and key data analysis.

[0116] In some embodiments of the present disclosure, based on the aforementioned solution, the real-time processing module 510 is configured to write the log data in the message queue to a target file in a designated storage directory according to a preset time window through the following steps:

[0117] Aggregate the log data within the preset time window using an aggregation function to obtain aggregated data in the target format, which is the data format agreed upon with the log analysis service.

[0118] Writes the aggregated data in the target format to a target file in the specified storage directory.

[0119] Alternatively, the log data includes various types of log data corresponding to various business systems. The real-time processing module 510 is configured to write the log data in the message queue into a target file in a designated storage directory according to a preset time window through the following steps:

[0120] Aggregate each type of log data within a preset time window using an aggregation function to obtain aggregated data in the target format corresponding to each type of log data. The target format is the data format agreed upon with the log analysis service.

[0121] Write the aggregated data corresponding to each type of log data into the target file in the specified storage directory.

[0122] In some embodiments of the present disclosure, based on the aforementioned solution, the real-time processing module 510 is configured to: if a task failure occurs during the process of writing log data to the target file, restore the state from the most recent checkpoint in a preset manner.

[0123] In some embodiments of the present disclosure, based on the aforementioned solution, the real-time processing module 510 is configured to implement log analysis on the target file by calling the log analysis interface of the log analysis service based on the storage directory of the target file through the following steps:

[0124] A log analysis request including the storage directory of the target file is sent to the log analysis service, so that the log analysis service obtains the corresponding log data to be analyzed based on the storage directory of the target file and performs log analysis on the log data to be analyzed.

[0125] In some embodiments of the present disclosure, based on the aforementioned solution, the real-time processing module 510 is configured to:

[0126] Receive a response message containing analysis results returned by the log analysis module 520;

[0127] Parse the response message to obtain abnormal information in the analysis results;

[0128] Generates warning information based on abnormal information and sends the warning information to the target object.

[0129] In some embodiments of the present disclosure, based on the aforementioned solution, the real-time processing module 510 is configured to:

[0130] Receive a response message containing analysis results returned by the log analysis module 520;

[0131] Analyze the response message to determine whether there is any abnormal information in the analysis result;

[0132] If there is abnormal information in the analysis result, request an access token from the alarm service module 530;

[0133] Within the authorization time limit of the access token, the alarm service module 530 is called by the authorized access token to trigger the generation of an alarm event for abnormal information.

[0134] For example, as shown in Figure 6, the log processing system of the present invention may include a stream storage unit 610, a real-time processing unit 620, a log analysis unit 630 and an alarm unit 640. The stream storage unit 610 is used to store the log data collected by the log collector from each business system into a message queue; the real-time processing unit 620 is used to consume the data in the message queue and write the log data in the message queue into a target file in a specified storage directory according to a preset time window; when all the data in a time window is written to the target file, the real-time processing unit 620 calls the log analysis unit 630 through an interface to perform log analysis on the target data (log data in the target file). Natural language technology and deep learning technology can be used for log analysis. The real-time processing unit 620 calls the alarm unit 640 through an interface based on the analysis results to issue an exception alarm. The real-time processing unit 620 can also trigger the alarm unit 640 to send exception information or alarm information to the operation and maintenance personnel to handle the exception in time.

[0135] The specific details of each of the above log processing systems have been described in detail in the corresponding log processing methods, so they will not be repeated here.

[0136] Embodiments of the present disclosure also provide a log management platform, including the log processing system of any of the aforementioned embodiments. Figure 7 illustrates a block diagram of a log management platform according to some embodiments of the present disclosure. Referring to Figure 7, the present disclosure provides a log management platform 700, including the log processing system 710 of any of the aforementioned embodiments. Log management can be performed using a system consisting of a terminal and a server.

[0137] It should be noted that although several modules or units of the device for execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0138] In this exemplary embodiment, an electronic device capable of implementing the above method is also provided.

[0139] Those skilled in the art will appreciate that various aspects of the present invention may be implemented as systems, methods, or program products. Therefore, various aspects of the present invention may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0140] The electronic device 800 according to this embodiment of the present invention is described below with reference to Figure 8. The electronic device 800 shown in Figure 8 is only an example and should not limit the functions and scope of use of the embodiment of the present invention.

[0141] As shown in FIG8 , electronic device 800 is implemented as a general-purpose computing device. Components of electronic device 800 may include, but are not limited to, the aforementioned at least one processing unit 810, the aforementioned at least one storage unit 820, a bus 830 connecting various system components (including storage unit 820 and processing unit 810), and a display unit 840.

[0142] The storage unit stores program code, which can be executed by the processing unit 810, so that the processing unit 810 performs the steps of various exemplary embodiments of the present invention described in the log processing method described above in this specification. For example, the processing unit 810 can execute step S110 as shown in Figure 1: obtain real-time log data of each business system and store the log data in a message queue; step S120: write the log data in the message queue to a target file in a specified storage directory according to a preset time window; step S130: in response to completing the data writing in the current time window, call the log analysis interface of the log analysis service based on the storage directory of the target file to perform log analysis on the target file.

[0143] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 8201 and / or a cache memory unit 8202 , and may further include a read-only memory unit (ROM) 8203 .

[0144] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0145] Bus 830 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0146] The electronic device 800 can also communicate with one or more external devices (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a viewer to interact with the electronic device 800, and / or any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 850. Furthermore, the electronic device 800 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 860. As shown, the network adapter 860 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown, other hardware and / or software modules can be used in conjunction with the electronic device 800, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0147] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0148] In this exemplary embodiment, a computer-readable storage medium is also provided, storing a program product capable of implementing the methods described above. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product comprising program code. When the program product is executed on a terminal device, the program code is configured to cause the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the "Exemplary Methods" section above.

[0149] According to an embodiment of the present invention, a program product for implementing the above-mentioned method can be a portable compact disc read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, and the program can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0150] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0151] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0152] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0153] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0154] Furthermore, the figures above are merely illustrative of the processes included in the methods according to exemplary embodiments of the present disclosure and are not intended to be limiting. It is readily understood that the processes illustrated in the figures above do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0155] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow from the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

[0156] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A log processing method, characterized in that, It includes: Obtain the real-time log data of each business system and store the log data in a message queue; Write the log data in the message queue into a target file in a specified storage directory according to a preset time window; In response to completing the data writing of the current time window, based on the storage directory of the target file, call the log analysis interface of the log analysis service to perform log analysis on the target file.

2. The method according to claim 1, wherein Before calling the log analysis interface to perform log analysis on the target file, the method further includes: Map the storage directory of the target file to the corresponding container of the log analysis service so that the log analysis service can share the storage directory of the target file.

3. The method according to claim 1, wherein Before calling the log analysis interface to perform log analysis on the target file, the method further includes: In response to completing the data writing of the current time window, send a message containing the storage directory of the target file to a specified topic of the message queue, so that the log analysis service can obtain the storage directory of the target file by consuming this specified topic.

4. The method according to any one of claims 1 to 3, characterized in that The storage directory of the target file is determined based on the storage path of the target file and the current time.

5. The method according to claim 1, characterized in that, Before writing the log data in the message queue into a target file in a specified storage directory, the method further includes: Preprocess the log data, and the preprocessing includes at least one of data format verification, target data filtering, and key data parsing.

6. The method according to claim 1 or 5, characterized in that, Writing the log data in the message queue into a target file in a specified storage directory according to a preset time window includes: Perform aggregation processing on the log data within a preset time window through an aggregation function to obtain aggregated data in a target format, and the target format is a data format agreed with the log analysis service; Write the aggregated data in the target format into a target file in a specified storage directory; Alternatively, the log data includes various types of log data corresponding to each business system, Perform aggregation processing on each type of log data within a preset time window through an aggregation function respectively to obtain aggregated data in a target format corresponding to each type of log data, and the target format is a data format agreed with the log analysis service; Write the aggregated data corresponding to each type of log data into a target file in a specified storage directory.

7. The method according to claim 1, wherein The method further includes: If a task running failure occurs during the process of writing the log data into the target file, perform status recovery from the nearest checkpoint through a preset method.

8. The method according to claim 1, wherein Based on the storage directory of the target file, calling the log analysis interface of the log analysis service to perform log analysis on the target file includes: Send a log analysis request containing the storage directory of the target file to the log analysis service, so that the log analysis service can obtain the corresponding log data to be analyzed based on the storage directory of the target file and perform log analysis on the log data to be analyzed.

9. The method according to claim 8, characterized in that, The method further includes: Receive a response message containing an analysis result returned by the log analysis service; Parse the response message to obtain the exception information in the analysis result; Generate an alarm message based on the exception information and send the alarm message to a target object.

10. The method according to claim 8, wherein The method further includes: Receive a response message containing analysis results returned by the log analysis service; Parse the response message to determine whether there is any abnormal information in the analysis results; If there is abnormal information in the analysis results, request an access token from the alarm interface; Within the authorization time limit of the access token, call the alarm interface through the authorized access token to trigger the generation of an alarm event for the abnormal information.

11. A log processing system, characterized in that, Including: A real-time processing module configured to obtain real-time log data of each business system and store the log data in a message queue; Write the log data in the message queue to a target file in a specified storage directory according to a preset time window; In response to completing the data writing of the current time window, send a log analysis request including the storage directory of the target file to the log analysis module; A log analysis module configured to obtain corresponding log data to be analyzed based on the storage directory of the target file included in the received log analysis request and perform log analysis on the log data to be analyzed.

12. The system according to claim 11, wherein The system further includes: an alarm service module; The log analysis module is configured to return an analysis result including abnormal information to the real-time processing module; The real-time processing module is configured to send an access token request for the alarm interface to the alarm service module; The alarm service module is configured to return a first authorized access token to the real-time processing module based on the received access token request; The real-time processing module is configured to, within the authorization time limit of the first authorized access token, send an alarm request including the abnormal information to the alarm service module through the first authorized access token; The alarm service module is configured to trigger the generation of an alarm event for the abnormal information based on the received alarm request.

13. The system according to claim 11, wherein The log analysis module is configured to return an analysis result including abnormal information to the real-time processing module; The real-time processing module is configured to send an alarm request including the abnormal information and its corresponding original log access address to the alarm service module; The alarm service module is configured to generate a corresponding alarm event based on the abnormal information, so that the alarm processing object can obtain the original log data through the target IP address based on the original log access address and the corresponding second authorized access token, and process the alarm event based on the original log data; The second authorized access token is an access token for the original log data.

14. An electronic device, including: One or more processors; A storage device for storing one or more programs, which when executed by the one or more processors, cause the one or more processors to implement the method according to any one of claims 1 to 10.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program, when executed by the processor, implements the method according to any one of claims 1 to 10.

Images