Anomaly Detection Method, Device, Equipment and Medium for Log Data

Abstract

An embodiment of the present invention discloses a method, apparatus, device, and medium for detecting anomalies in log data. The method includes: obtaining, according to a target key field, log data to be converted that matches the target key field from the log data associated with each server to be detected; processing the log data to be converted based on the data format mapped in an index template to obtain corresponding log data to be used, and establishing a target log data index corresponding to the log data to be used based on the log data grouping method of the index template and the log data to be used; determining, based on the target log data index and a preset alarm rule, target anomaly log data of the server to be detected and a target anomaly event corresponding to the server to be detected from the log data to be used; and determining the anomaly cause of the server to be detected according to the target anomaly event and an anomaly cause analysis model. The technical solution of the present invention improves the efficiency of anomaly detection and anomaly handling of log data.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular, to an abnormal detection method, device, equipment and medium for log data. Background Art

[0002] Currently, the booming development of the mobile Internet has made people inseparable from the services provided by computers and intelligent devices in their daily lives. However, these services face various threats to network security during actual operation, and service interruptions often occur due to vulnerability attacks and changes in the operating state. Therefore, it is necessary to perform abnormal detection on the log data of the server.

[0003] In the existing log anomaly detection methods, operation and maintenance personnel search for log information using keywords, use Linux script tools for inspection, and perform positioning and cause analysis based on existing experience to find the actual fault points. Or log collection is pre-performed through Flume, then the data is sent to Kafka, and finally stored in Hdfs for offline analysis to ensure data consistency and regular troubleshooting. The disadvantages of the first implementation method are as follows: the scanning granularity of the script tool is not fine enough, the source of the problem cause overly relies on the existing experience of operation and maintenance personnel, and the manual workload is relatively large, resulting in slow troubleshooting speed and difficulty in expansion. The disadvantages of the second implementation method are as follows: problem troubleshooting is not timely, searching for a large amount of information is time-consuming, and it is impossible to alarm and discover in the first time. A large amount of log data is difficult to be fully utilized, and the accuracy of mining the root cause is relatively low.

[0004] Therefore, the current log alarm method and detection system involve a large amount of operation and maintenance work, and the operation and maintenance cost is relatively high. By means of a real-time log detection system, abnormal information is collected and matched, providing a basis for operation and maintenance personnel when analyzing abnormal problems, and improving the accuracy and efficiency of troubleshooting. Summary of the Invention

[0005] The present invention provides an abnormal detection method, device, equipment and medium for log data to automatically obtain and analyze the log data of the server, and determine the abnormal log data and the corresponding abnormal causes.

[0006] According to one aspect of the present invention, an abnormal detection method for log data is provided, and the method includes:

[0007] Obtaining the log data to be converted that matches the target keyword field from the log data associated with each server to be detected according to the target keyword field;

[0008] Wherein, the target keyword field includes: request time field, user address field, service address field, service port field, request parameter field, thread field, and device MAC address field;

[0009] Process the to-be-converted log data based on the data format mapped in the index template to obtain the to-be-used log data corresponding to the to-be-converted log data, and establish a target log data index corresponding to the to-be-used log data based on the log data grouping method of the index template and the to-be-used log data;

[0010] Based on the target log data index and the preset alarm rules, determine the target abnormal log data of the to-be-detected server and the target abnormal event corresponding to the to-be-detected server from the to-be-used log data;

[0011] Determine the abnormal cause of the to-be-detected server according to the target abnormal event and the abnormal cause analysis model.

[0012] According to another aspect of the present invention, there is provided an abnormal detection device for log data, and the device includes:

[0013] A to-be-converted log data acquisition module, configured to acquire the to-be-converted log data matching the target key fields from the log data associated with each to-be-detected server according to the target key fields;

[0014] Wherein, the target key fields include: a request time field, a user address field, a service address field, a service port field, a request parameter field, a thread field, and a device MAC address field;

[0015] A target log data index establishment module, configured to process the to-be-converted log data based on the data format mapped in the index template to obtain the to-be-used log data corresponding to the to-be-converted log data, and establish a target log data index corresponding to the to-be-used log data based on the log data grouping method of the index template and the to-be-used log data;

[0016] A target abnormal log data determination module, configured to determine the target abnormal log data of the to-be-detected server and the target abnormal event corresponding to the to-be-detected server from the to-be-used log data based on the target log data index and the preset alarm rules;

[0017] An abnormal cause determination module, configured to determine the abnormal cause of the to-be-detected server according to the target abnormal event and the abnormal cause analysis model.

[0018] According to another aspect of the present invention, there is provided an electronic device, and the electronic device includes:

[0019] At least one processor;

[0020] And a memory communicatively connected to the at least one processor;

[0021] Wherein, the memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor is enabled to execute the abnormal detection method of log data according to any embodiment of the present invention.

[0022] According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement the abnormal detection method of log data according to any embodiment of the present invention when executed.

[0023] The technical solution of the embodiment of the present invention obtains the log data to be converted that matches the target key field from the log data associated with each server to be detected according to the target key field; processes the log data to be converted based on the data format mapped in the index template to obtain the corresponding log data to be used, and establishes a target log data index corresponding to the log data to be used based on the log data grouping method of the index template and the log data to be used; determines the target abnormal log data of the server to be detected and the corresponding target abnormal event of the server to be detected from the log data to be used based on the target log data index and the preset alarm rule; determines the abnormal cause of the server to be detected according to the target abnormal event and the abnormal cause analysis model. The technical solution of the present invention improves the efficiency of abnormal detection and abnormal processing of log data, solves the problem of low efficiency of abnormal detection of log data in the prior art, realizes real-time abnormal detection of log data, and determines the abnormal cause.

[0024] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. Description of the Drawings

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0026] Figure 1 It is a flowchart of an abnormal detection method for log data provided in Embodiment 1 of the present invention;

[0027] Figure 2 It is a flowchart of an abnormal detection method for log data provided in Embodiment 2 of the present invention;

[0028] Figure 3A schematic diagram of Bayesian reason classification provided in the second embodiment of the present invention;

[0029] Figure 4 A schematic structural diagram of an abnormal detection device for log data provided in the third embodiment of the present invention;

[0030] Figure 5 A schematic structural diagram of an electronic device provided in the fourth embodiment of the present invention. Detailed implementation manners

[0031] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0033] Embodiment 1

[0034] Figure 1 It is a flowchart of an abnormal detection method for log data provided in the first embodiment of the present invention. This embodiment is applicable to the situation where abnormal logs generated by a server need to be detected and resolved. This method can be executed by an abnormal detection device for log data, and this device can be implemented in the form of hardware and / or software. This device can be configured in an abnormal detection server for log data. As Figure 1 shown, this method includes:

[0035] S110. Obtain the log data to be converted that matches the target key field from the log data associated with each server to be detected according to the target key field.

[0036] Among them, the target keyword fields include: request time field, user address field, service address field, service port field, request parameter field, thread field, and device MAC address field. The target keyword fields are important fields selected or set in advance, used to represent key information in the log data. The server to be detected can be any server that needs to perform log anomaly detection. For example, the server to be detected is an application server, and the application server can provide services for the corresponding client. When providing services, the client can send requests to the application server. The request time field is used to represent the time information related to the request. The user address field can represent the address information of the user. The service address field is used to represent the address of the application server. The service port field is used to represent the port information of the server. The request parameter field is used to represent the specific parameters of the request. The thread field refers to the thread information called to process the request. The device MAC field is used to represent the MAC address of the terminal used by the user. The log data refers to the log data generated by the server to be detected during operation. The log data to be converted refers to the log data that matches the target keyword fields, such as the log data containing the target keyword fields.

[0037] Specifically, the server to be detected can be configured so that the server to be detected sends the log data containing the target keyword fields in the generated log data to the log data anomaly detection device, and uses the log data containing the target keyword fields as the log data to be converted. Corresponding interfaces and protocols can be configured between the server to be detected and the log data anomaly detection device for transmitting the log data to be converted. The acquisition of the log data to be converted can be the active sending of the server to be detected, or the active acquisition of the log data anomaly detection device.

[0038] Based on the above technical solution, the obtaining of the log data to be converted that matches the target keyword fields from the log data associated with each server to be detected includes: collecting the log data containing the target keyword fields from the log data associated with each server to be detected based on a parallel collection method, and determining the log data as the log data to be converted.

[0039] Among them, the parallel collection method means that each server to be detected has a corresponding log transmission channel. The number of servers to be detected is one or more. The servers to be detected can be, but are not limited to, application servers, system servers, and network servers.

[0040] In practical applications, there can be multiple servers that require anomaly detection of log data. Therefore, a parallel collection method needs to be adopted for the collection and transmission of log data. The log data containing the target key fields in each server can be extracted, which can be in the form of filtering. The extracted log data containing the target key fields is used as the log data to be converted.

[0041] S120. Process the log data to be converted based on the data format mapped in the index template to obtain the log data to be used corresponding to the log data to be converted, and establish a target log data index corresponding to the log data to be used based on the log data grouping method of the index template and the log data to be used.

[0042] Among them, the index template refers to the template used to establish log indexes, which can be an ES index template. The data format can be the data format mapped by the index template. For example, the data format is the Json data format mapped in the ES index template. The log data to be used refers to the log data obtained after format conversion of the log data to be converted. The log data grouping method refers to the grouping method of the log data to be used in the index. For example, the log data containing the address field is divided into one group. The target log data index can be the index established according to the index template corresponding to the log data to be used.

[0043] Specifically, the log data to be converted can be processed based on the data format mapped in the index template to obtain the log data to be used corresponding to the log data to be converted, and a target log data index corresponding to the log data to be used can be established based on the log data grouping method of the index template and the log data to be used. The advantage of this is that the log data to be used can be quickly searched, improving the efficiency of log anomaly detection.

[0044] Based on the above technical solution, the processing of the log data to be converted based on the data format mapped in the index template to obtain the log data to be used corresponding to the log data to be converted, and establishing a target log data index corresponding to the log data to be used based on the log data grouping method of the index template and the log data to be used includes: converting the format of the log data to be converted according to the data format mapped in the index template to obtain the log data to be used corresponding to the log data to be converted; grouping the log data to be used based on the log data grouping method of the index template to establish a target log data index corresponding to the log data to be used.

[0045] Specifically, the data format of the log data to be converted can be converted according to the data format mapped in the index template. For example, the data to be converted containing key information can be processed, matched with the predefined template of the log, and parsed into the Json data mapped in the ES index template. For example: "project" -> "zjgg", "level" -> "error", "time" -> "2022-03-10 15-30-30", "ip" -> "10.0.0.1", etc. The obtained Json format data is the log data to be used. Further, the log data to be used is grouped according to the log data grouping method, and the log data of the same type or the log data containing the same fields are divided into the same group, and the target log data index corresponding to the log data to be used is established.

[0046] On the basis of the above technical solution, after processing the log data to be converted according to the data format mapped in the index template to obtain the log data to be used corresponding to the log data to be converted, and establishing the target log data index corresponding to the log data to be used based on the log data grouping method of the index template and the log data to be used, it further includes: sending the log data to be used to the log storage server for storage, so that when performing anomaly detection on the log data of the server to be detected, the log data to be used is obtained from the log storage server based on the address of the log storage server and the target transmission protocol.

[0047] Among them, the log storage server can be a server for storing log data, and the target transmission protocol refers to from the log storage server

[0048] Specifically, after establishing the target log data index, the log data to be used can be sent to the log storage server in batches in a fixed quantity first, so as to reduce the pressure on the collection end or the storage pressure of the server to be detected. Further, when it is necessary to perform anomaly detection on the log data of the server to be detected, the log data to be used can be directly obtained from the log data storage server, the address of the log server is obtained by using the secure hash algorithm, and the syslog protocol is used for transmission, and sent to the device or service cluster for anomaly detection of log data, and sent to the backup cluster as a historical record.

[0049] S130. Determine the target anomaly log data of the server to be detected and the target anomaly event corresponding to the server to be detected from the log data to be used based on the target log data index and the preset alarm rule.

[0050] Among them, the preset alarm rule can be an alarm rule pre-set by the operation and maintenance personnel, which is used to determine the abnormal log data to be detected, that is, the target abnormal log data. Correspondingly, the event corresponding to the target abnormal log data is the target abnormal event, and the target abnormal event is used to represent the abnormality occurring in the server to be detected. For example, the abnormal event can be request timeout or request failure.

[0051] Specifically, the log data to be used by the server to be detected can be found through the target log data index, and it can be determined according to the preset alarm rule which of the log data to be used is abnormal. If it is abnormal, it can be determined as the target abnormal log data, and the event corresponding to the target abnormal log data is used as the target abnormal event corresponding to the server to be detected. The advantage of such a setting is that the service log can be processed in real time, key information can be accurately collected, the complexity of full-text retrieval is reduced, abnormal information can be fed back in time, and the impact of faults on the enterprise is reduced. The task volume of troubleshooting by operation and maintenance personnel is reduced and the efficiency of cause finding is improved, which is beneficial to maintaining the rapid growth of the system business and the high availability of the service.

[0052] Based on the above technical solution, determining the target abnormal log data of the server to be detected and the target abnormal event corresponding to the server to be detected from the log data to be used based on the target log data index and the preset alarm rule includes: determining the log data to be analyzed based on the target log data index, and performing matching analysis on the log data to be analyzed based on the preset alarm rule, so as to determine the current log data to be analyzed as the target abnormal log data when the current log data to be analyzed exceeds the set range of the preset alarm rule;

[0053] Determining the preset abnormal event corresponding to the preset alarm rule as the target abnormal event corresponding to the server to be detected.

[0054] Among them, the log data to be analyzed can be understood as the log data that needs to be analyzed for abnormality. Any log data to be used found through the target log data index can be used as the log data to be analyzed, and the preset event can be the event name pre-set corresponding to the preset alarm rule.

[0055] Specifically, one or more can be selected from the log data to be used as the log data to be analyzed through the target log data index, and it is analyzed whether the log data to be analyzed exceeds the set range of the preset alarm rule. For example, it can be determined the request time field in the log data to be analyzed, and the time value of the field content can be used to know the time when the request is processed. Then, the processing time is compared with the request time threshold in the pre-set alarm rule. If it exceeds the request time threshold, it means that the request processing time is too long, and it can be considered that the corresponding target abnormal event is network delay.

[0056] S140. Determine the abnormal cause of the server to be detected based on the target abnormal event and the abnormal cause analysis model.

[0057] Among them, the abnormal cause analysis model can be a Bayesian network model pre-trained through a large number of abnormal events and abnormal causes, used to analyze the target abnormal event and output the corresponding abnormal cause.

[0058] Specifically, the target abnormal event can be used as the input of the abnormal cause analysis model. Correspondingly, the abnormal cause analysis model can output the corresponding abnormal cause as the abnormal cause of the server to be detected.

[0059] Based on the above technical solution, the step of determining the abnormal cause of the server to be detected according to the target abnormal event and the abnormal cause analysis model includes: determining the corresponding abnormal information based on the target abnormal event, and analyzing the abnormal features in the abnormal information based on the abnormal cause analysis model to obtain the abnormal cause output by the server to be detected.

[0060] Among them, the abnormal information refers to the information of the server to be detected related to the abnormal event and the information of the client corresponding to the server to be detected. The abnormal features include: request parameters, user permissions, load data, audit compliance, service status, and network traffic. The request parameters refer to the parameter information related to the request. The user permissions can be the permissions of the user corresponding to the client that issues the request. The load data refers to the load information of the server to be detected. The audit compliance refers to the audit result of the log data.

[0061] Specifically, the information associated with the abnormal event can be determined according to the abnormal event, and the abnormal features can be analyzed from the information. Then, the conditional probability of the abnormal features can be calculated through the Bayesian network model, and the cause corresponding to the maximum likelihood function can be used as the root cause of the abnormality, and the abnormal cause can be output.

[0062] Based on the above technical solution, the step of determining the abnormal cause of the server to be detected according to the target abnormal event and the abnormal cause analysis model further includes: determining the target abnormal handling solution corresponding to the abnormal cause from the pre-set abnormal handling solution library based on the abnormal cause; performing abnormal handling on the server to be detected based on the target abnormal handling solution.

[0063] Among them, the pre-set abnormal handling solution library is a pre-established database, in which the handling solutions corresponding to different abnormal causes are stored.

[0064] Specifically, according to the exception cause, an exception handling solution corresponding to the current exception cause can be found from a preset exception handling library, and this exception handling solution can be used as the target exception handling solution for the server to be detected, so that the operation and maintenance personnel can handle the exception of the server to be detected based on this target exception handling solution.

[0065] The technical solution of the embodiment of the present invention is as follows: obtaining, according to a target keyword field, log data to be converted that matches the target keyword field from log data associated with each server to be detected; processing the log data to be converted based on the data format mapped in an index template to obtain corresponding log data to be used, and establishing a target log data index corresponding to the log data to be used based on the log data grouping method of the index template and the log data to be used; determining, based on the target log data index and a preset alarm rule, target abnormal log data of the server to be detected and a target abnormal event corresponding to the server to be detected; and determining the exception cause of the server to be detected according to the target abnormal event and an exception cause analysis model. The technical solution of the present invention improves the efficiency of abnormal detection and exception handling of log data, solves the problem of low efficiency of abnormal detection of log data in the prior art, realizes real-time abnormal detection of log data, and determines the exception cause.

[0066] Embodiment 2

[0067] Figure 2 FIG. is a flowchart of an abnormal detection method for log data provided by Embodiment 2 of the present invention. This embodiment is a preferred embodiment of the above-mentioned embodiment, and the specific implementation manner can refer to the technical solution of this embodiment. Among them, the same or corresponding technical terms as those in the above-mentioned embodiment will not be described in detail here.

[0068] As Figure 2 shown, the method includes:

[0069] Collecting log data generated by the server in real time according to the keyword field

[0070] Cleaning, converting and creating an index for the log according to the index template

[0071] Storing and backing up the log data in slices according to the service level

[0072] Matching the index according to the alarm rule and processing in combination with the alarm strategy

[0073] Using a Bayesian network to discover the exception cause and storing it in the database

[0074] Converting the collected log into the data format of the index template, storing the data in slices according to the service level, processing in combination with the alarm strategy, and using a Bayesian network to find the root cause. As Figure 3As shown in the figure, it is a schematic diagram of Bayesian cause classification. A Bayesian network is a probabilistic graphical model that calculates the probability of the occurrence of cause event A under the condition that abnormal event B has occurred. The result of global causal relationship can be obtained through the product of multiple local functions, and the cause corresponding to the maximum likelihood function is used as the root cause of the abnormality.

[0075] (1) Description of log data collection

[0076] Collect in parallel from multiple servers, and the content includes: request time, client IP address, service address and port number, request parameters, executed thread, device mac address, etc.

[0077] (2) Description of log format conversion

[0078] Process the data containing key information, match the predefined template of the log, and parse it into Json data mapped in the ES index template. For example: "project" -> "zjgg", "level" -> "error", "time" -> "2022-03-10 15-30-30", "ip" -> "10.0.0.1", etc.

[0079] (3) Description of log distribution and storage

[0080] Send in batches with a fixed amount of data to reduce the pressure on the collection end. Use the secure hash algorithm to obtain the corresponding log server address, and use the syslog protocol for transmission, and send it to the cluster for real-time alarm calculation and the backup cluster for historical records.

[0081] (4) Description of log alarm discovery

[0082] Distinguish the types of alarms, such as system operation anomalies, application service anomalies, network traffic anomalies, etc. Include memory overflow, traversal of request paths, unauthorized access, problems with uploaded parameters, etc.

[0083] (5) Description of log alarm records

[0084] After matching the corresponding alarm rules, find the causal chain of abnormal problems through the Bayesian network, collect abnormal information and abnormal reasons separately and count the number of times. Make a judgment according to the threshold, send text messages and emails to the system administrator, and provide a preset abnormal handling plan according to the service id number.

[0085] The technical solution of the embodiment of the present invention is as follows: obtain the to-be-converted log data matching the target key field from the log data associated with each server to be detected according to the target key field; process the to-be-converted log data based on the data format mapped in the index template to obtain the corresponding to-be-used log data, and establish a target log data index corresponding to the to-be-used log data based on the log data grouping method of the index template and the to-be-used log data; determine the target abnormal log data of the server to be detected and the corresponding target abnormal event of the server to be detected from the to-be-used log data based on the target log data index and the preset alarm rule; determine the abnormal cause of the server to be detected according to the target abnormal event and the abnormal cause analysis model. The technical solution of the present invention improves the efficiency of abnormal detection and processing of log data, solves the problem of low efficiency of abnormal detection of log data in the prior art, realizes real-time abnormal detection of log data, determines the abnormal cause, performs real-time processing on service logs, accurately collects key information, reduces the complexity of full-text retrieval, can timely feedback abnormal information, reduces the impact of faults on enterprises. It reduces the task volume of operation and maintenance personnel to troubleshoot and improves the efficiency of finding the cause, which is beneficial to maintaining the rapid growth of system services and the high availability of services.

[0086] Embodiment III

[0087] Figure 4 It is a schematic structural diagram of an abnormal detection device for log data provided by Embodiment III of the present invention. As Figure 4 shown, the device includes:

[0088] The to-be-converted log data acquisition module 310 is configured to obtain the to-be-converted log data matching the target key field from the log data associated with each server to be detected according to the target key field;

[0089] Wherein, the target key field includes: request time field, user address field, service address field, service port field, request parameter field, thread field, and device MAC address field;

[0090] The target log data index establishment module 320 is configured to process the to-be-converted log data based on the data format mapped in the index template to obtain the to-be-used log data corresponding to the to-be-converted log data, and establish a target log data index corresponding to the to-be-used log data based on the log data grouping method of the index template and the to-be-used log data;

[0091] The target abnormal log data determination module 330 is configured to determine the target abnormal log data of the server to be detected and the corresponding target abnormal event of the server to be detected from the to-be-used log data based on the target log data index and the preset alarm rule;

[0092] Anomaly cause determination module 340, configured to determine the anomaly cause of the server to be detected according to the target anomaly event and the anomaly cause analysis model.

[0093] Based on the above device, the log data to be converted acquisition module 310 includes:

[0094] Parallel acquisition module, configured to acquire log data including the target keyword field from the log data associated with each server to be detected based on a parallel acquisition method, and determine the log data as log data to be converted; wherein, the number of servers to be detected is one or more.

[0095] Based on the above device, the target log data index establishment module 320 includes:

[0096] Format conversion module, configured to perform format conversion processing on the log data to be converted according to the data format mapped in the index template, so as to obtain the log data to be used corresponding to the log data to be converted;

[0097] Index establishment module, configured to group the log data to be used based on the log data grouping method of the index template, so as to establish a target log data index corresponding to the data to be used.

[0098] Based on the above device, it further includes:

[0099] Log data storage module, configured to send the log data to be used to a log storage server for storage, so as to obtain the log data to be used from the log storage server based on the address of the log storage server and the target transmission protocol when performing anomaly detection on the log data of the server to be detected.

[0100] Based on the above device, the target anomaly log data determination module 330 includes:

[0101] Log data analysis and matching module, configured to determine the log data to be analyzed based on the target log data index, and perform matching analysis on the log data to be analyzed based on the preset alarm rule, so as to determine the current log data to be analyzed as the target anomaly log data when the current log data to be analyzed exceeds the set range of the preset alarm rule;

[0102] Anomaly event determination module, configured to determine the preset anomaly event corresponding to the preset alarm rule as the target anomaly event corresponding to the server to be detected.

[0103] Based on the above device, the anomaly cause determination module 340 includes:

[0104] Anomaly cause analysis module, configured to determine corresponding anomaly information based on the target anomaly event, and analyze anomaly features in the anomaly information based on the anomaly cause analysis model to obtain the anomaly cause output by the server to be detected;

[0105] Wherein, the anomaly features include: request parameters, user permissions, load data, audit compliance, service status, and network traffic.

[0106] Based on the above device, it further includes:

[0107] Target anomaly solution determination module, configured to determine a target anomaly solution corresponding to the anomaly cause from a preset anomaly handling solution library based on the anomaly cause;

[0108] Anomaly handling module, configured to perform anomaly handling on the server to be detected based on the target anomaly handling solution.

[0109] The technical solution of the embodiment of the present invention obtains the log data to be converted that matches the target key field from the log data associated with each server to be detected according to the target key field; processes the log data to be converted based on the data format mapped in the index template to obtain the corresponding log data to be used, and establishes a target log data index corresponding to the log data to be used based on the log data grouping method of the index template and the log data to be used; determines the target anomaly log data of the server to be detected and the target anomaly event corresponding to the server to be detected from the log data to be used based on the target log data index and the preset alarm rule; determines the anomaly cause of the server to be detected according to the target anomaly event and the anomaly cause analysis model. The technical solution of the present invention improves the efficiency of anomaly detection and anomaly handling of log data, solves the problem of low efficiency of anomaly detection of log data in the prior art, realizes real-time anomaly detection of log data, determines the anomaly cause, performs real-time processing on service logs, accurately collects key information, reduces the complexity of full-text retrieval, can feedback anomaly information in time, reduces the impact of faults on enterprises. Reduces the task volume of operation and maintenance personnel to troubleshoot and improves the efficiency of finding the cause, which is beneficial to maintaining the rapid growth of system services and the high availability of services.

[0110] The anomaly detection device for log data provided by the embodiment of the present invention can execute the anomaly detection method for log data provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

[0111] Embodiment 4

[0112] Figure 5Schematic diagram of a structure of an electronic device provided in Embodiment 4 of the present invention. The electronic device is intended to represent various forms of digital computers, such as, for example, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as, for example, personal digital assistants, cellular telephones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0113] As Figure 5 shown, the electronic device 40 includes at least one processor 41 and a memory communicatively connected to the at least one processor 41, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc. The memory stores a computer program executable by the at least one processor. The processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 into the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected to each other via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0114] Multiple components in the electronic device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a magnetic disk, an optical disc, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0115] The processor 41 may be various general-purpose and / or special-purpose processing components having processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 41 executes the various methods and processes described above, such as the method for anomaly detection of log data.

[0116] In some embodiments, the method for anomaly detection of log data may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed onto the electronic device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the above-described method for anomaly detection of log data may be performed. Alternatively, in other embodiments, the processor 41 may be configured to perform the method for anomaly detection of log data by any other suitable means (e.g., by means of firmware).

[0117] The various embodiments of the systems and techniques described above in this document may be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), application specific standard products (ASSP), systems on a chip (SOC), complex programmable logic devices (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a special-purpose or general-purpose programmable processor that receives data and instructions from a storage system, at least one input device, and at least one output device, and transmits the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0118] The computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The computer programs may be executed entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine, or entirely on the remote machine or server.

[0119] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0120] To provide for interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can also be used to provide for interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0121] The systems and techniques described herein can be implemented in a computing system that includes backend components (such as, for example, a data server), or a computing system that includes middleware components (such as, for example, an application server), or a computing system that includes frontend components (such as, for example, a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (such as, for example, a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0122] A computing system may include a client and a server. The client and the server are generally far from each other and usually interact via a communication network. The relationship between the client and the server is created by computer programs that run on respective computers and have a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0123] It should be understood that various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitations are imposed herein.

[0124] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. An abnormal detection method for log data, characterized in that, Including: Obtaining the log data to be converted that matches the target keyword field from the log data associated with each server to be detected according to the target keyword field; Wherein, the target keyword field includes: request time field, user address field, service address field, service port field, request parameter field, thread field, and device MAC address field; Processing the log data to be converted based on the data format mapped in the index template to obtain the log data to be used corresponding to the log data to be converted, and establishing a target log data index corresponding to the log data to be used based on the log data grouping method of the index template and the log data to be used; Determining the target abnormal log data of the server to be detected and the target abnormal event corresponding to the server to be detected from the log data to be used based on the target log data index and the preset alarm rule; Determining the abnormal cause of the server to be detected according to the target abnormal event and the abnormal cause analysis model; Wherein, the abnormal cause analysis model is a Bayesian network model trained in advance through abnormal events and abnormal causes; The determining the abnormal cause of the server to be detected according to the target abnormal event and the abnormal cause analysis model includes: Determining the corresponding abnormal information based on the target abnormal event, and analyzing the abnormal characteristics from the abnormal information; Calculating the conditional probability of the abnormal characteristics through the Bayesian network model, and taking the cause corresponding to the maximum likelihood function as the abnormal cause of the server to be detected.

2. The method according to claim 1, characterized in that, The obtaining the log data to be converted that matches the target keyword field from the log data associated with each server to be detected according to the target keyword field includes: Collecting the log data containing the target keyword field from the log data associated with each server to be detected based on a parallel collection method, and determining the log data as the log data to be converted; wherein, the number of servers to be detected is multiple.

3. The method according to claim 1, wherein The processing the log data to be converted based on the data format mapped in the index template to obtain the log data to be used corresponding to the log data to be converted, and establishing a target log data index corresponding to the log data to be used based on the log data grouping method of the index template and the log data to be used includes: Converting the format of the log data to be converted according to the data format mapped in the index template to obtain the log data to be used corresponding to the log data to be converted; Grouping the log data to be used based on the log data grouping method of the index template to establish a target log data index corresponding to the log data to be used.

4. The method according to claim 1, wherein After the processing the log data to be converted based on the data format mapped in the index template to obtain the log data to be used corresponding to the log data to be converted, and establishing a target log data index corresponding to the log data to be used based on the log data grouping method of the index template and the log data to be used, it further includes: Send the to-be-used log data to a log storage server for storage, so that when performing anomaly detection on the log data of the to-be-detected server, obtain the to-be-used log data from the log storage server based on the address of the log storage server and the target transfer protocol.

5. The method according to claim 1, characterized in that, Determining the target anomaly log data of the to-be-detected server and the target anomaly event corresponding to the to-be-detected server from the to-be-used log data based on the target log data index and the preset alarm rules includes: Determine the to-be-analyzed log data based on the target log data index, and perform matching analysis on the to-be-analyzed log data based on the preset alarm rules. When the current to-be-analyzed log data exceeds the set range of the preset alarm rules, determine the current to-be-analyzed log data as the target anomaly log data; Determine the preset anomaly event corresponding to the preset alarm rule as the target anomaly event corresponding to the to-be-detected server.

6. The method according to claim 1, characterized in that The anomaly features include: request parameters, user permissions, load data, audit compliance, service status, and network traffic.

7. The method according to claim 1, characterized in that After determining the anomaly cause of the to-be-detected server according to the target anomaly event and the anomaly cause analysis model, it further includes: Determine the target anomaly handling solution corresponding to the anomaly cause from the preset anomaly handling solution library based on the anomaly cause; Perform anomaly handling on the to-be-detected server based on the target anomaly handling solution.

8. An abnormal detection device for log data, characterized in that, It includes: A to-be-converted log data acquisition module, configured to acquire to-be-converted log data that matches the target keyword field from the log data associated with each to-be-detected server according to the target keyword field; Among them, the target keyword fields include: request time field, user address field, service address field, service port field, request parameter field, thread field, and device MAC address field; A target log data index establishment module, configured to process the to-be-converted log data based on the data format mapped in the index template to obtain to-be-used log data corresponding to the to-be-converted log data, and establish a target log data index corresponding to the to-be-used log data based on the log data grouping method of the index template and the to-be-used log data; A target anomaly log data determination module, configured to determine the target anomaly log data of the to-be-detected server and the target anomaly event corresponding to the to-be-detected server from the to-be-used log data based on the target log data index and the preset alarm rules; An anomaly cause determination module, configured to determine the anomaly cause of the to-be-detected server according to the target anomaly event and the anomaly cause analysis model; Among them, the anomaly cause analysis model is a Bayesian network model that has been trained in advance through anomaly events and anomaly causes; The anomaly cause determination module includes an anomaly cause analysis module, configured to: Determine the corresponding anomaly information based on the target anomaly event, and analyze the anomaly features from the anomaly information; Calculate the conditional probability of the anomaly features through the Bayesian network model, and use the cause corresponding to the maximum likelihood function as the anomaly cause of the to-be-detected server.

9. An electronic device, characterized in that, The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the abnormal detection method of the log data according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for implementing the abnormal detection method of the log data according to any one of claims 1-7 when executed by a processor.

Images