Log data processing method and device, electronic equipment and readable storage medium

The log data processing method addresses the lack of comprehensive business monitoring by analyzing log data to determine field types and channels, ensuring real-time detection and resolution of issues, thus stabilizing business systems.

CN119065914APending Publication Date: 2024-12-03SF TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310645392.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-06-01
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Current alarm monitoring systems in business systems lack comprehensive business monitoring, failing to achieve real-time detection and handling of issues, particularly in database tables and third-party services, leading to instability.

Method used

A log data processing method that performs field analysis on business system logs to determine field types and data channels, applies detection rules, and acquires detection data through specific channels to ensure comprehensive and real-time monitoring.

Benefits of technology

Enables all-round business monitoring, allowing timely detection and resolution of issues, thereby enhancing the stability of business systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119065914A_ABST
    Figure CN119065914A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a log data processing method and device, electronic equipment and a readable storage medium, and relates to the technical field of data processing. According to the embodiment of the invention, field analysis is carried out on the log data of the to-be-monitored service system to obtain the log field in the log data, the field type of the log field and the second field data, the data channel and the detection rule of the detection data corresponding to the log field are determined according to the field type of the log field, and the detection result of the to-be-monitored service system is obtained based on the data channel. And obtaining detection data of the log field, and detecting second field data of the log field according to the detection data and the detection rule. Thus, by determining the data channel of the detection data corresponding to the log field, the detection data in all aspects of the log data can be obtained, then comprehensive detection of the service can be achieved based on the detection data and the detection rule, problems can be found and processed in time, and the stability of a service system is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a log data processing method, device and readable storage medium. Background Art

[0002] With the rapid popularization of the Internet, more and more companies and users use online transactions, which are characterized by a large number of users and high frequency. The requirements for the stability of online business are getting higher and higher, and there are also higher standards for the perception of problems. A complete monitoring solution has become a solid backing to ensure the stability of the system.

[0003] The current business system alarms are mainly non-business alarms such as system CPU load, memory usage, disk usage, network status, etc. As for business alarms, they are less involved and cannot achieve all-round business monitoring. Summary of the invention

[0004] The embodiments of the present application provide a log data processing method, device, electronic device and readable storage medium, which can realize all-round monitoring of a business system and improve the stability of the business system.

[0005] The embodiments of the present invention can be implemented in the following ways:

[0006] In a first aspect, an embodiment of the present invention provides a log data processing method, including:

[0007] Perform field parsing on the log data of the business system to be monitored to obtain the log field in the log data and the field type and second field data of the log field;

[0008] Determine, according to the field type of the log field, a data channel and a detection rule for the detection data corresponding to the log field; wherein each field type is correspondingly configured with a data channel and a detection rule;

[0009] Based on the data channel, acquiring detection data of the log field;

[0010] The second field data of the log field is detected according to the detection data and the detection rule.

[0011] In an optional implementation, the detecting the second field data of the log field according to the detection data and the detection rule includes:

[0012] Extracting field data from the detection data to obtain first field data of a detection field in the detection data;

[0013] Based on the detection rule, the first field data of the detection field in the detection data is compared with the second field data of the log field to obtain a detection result.

[0014] In an optional implementation, comparing the first field data of the detection field in the detection data with the second field data of the log field based on the detection rule includes:

[0015] If the field type of the log field is a data table type, comparing the field state of the detection field in the detection data with the field state of the log field;

[0016] If the field type of the log field is an interface type, comparing the interface call mode data and the interface return result data of the detection field in the detection data with the second field data of the log field;

[0017] If the field type of the log field is a delay type, the expected data of the detection field in the detection data is compared with the second field data of the log field.

[0018] In an optional implementation manner, after detecting the second field data of the log field according to the detection data and the detection rule, the method further includes:

[0019] Determining target detection results that do not meet the standards and target detection rules used to obtain the target detection results;

[0020] Determining the alarm level of the business system to be monitored according to the target detection rule;

[0021] A security alert policy corresponding to the alert level is initiated.

[0022] In an optional implementation manner, determining the alarm level of the service system to be monitored according to the target detection rule includes:

[0023] Determine the alarm score corresponding to the target detection rule according to a preset rule score data table; the rule score data table is used to indicate the alarm score corresponding to each detection rule;

[0024] According to the alarm score corresponding to the target detection rule, the alarm level of the service system to be monitored is determined; each alarm level corresponds to an alarm score range.

[0025] In an optional implementation manner, after detecting the second field data of the log field according to the detection data and the detection rule, the method further includes:

[0026] Determine the number of times each IP address in the log data accesses the domain name, and the frequency of each IP address calling the interface of the business system to be monitored;

[0027] The target IP whose access times exceed the preset access threshold and / or whose calling frequency exceeds the preset frequency threshold is marked.

[0028] In an optional implementation manner, obtaining the detection data of the log field based on the data channel includes:

[0029] If the field type of the log field is a data table type, the field data of the log field is obtained from the database through a preconfigured database interface, and / or the field data of the log field is obtained from a third-party data source based on a preset key, and the detection data is obtained according to the obtained field data;

[0030] If the field type of the log field is an interface type, a third-party interface is called to obtain interface return result data, and interface call method data is obtained according to an interface identifier of the third-party interface, and detection data is obtained according to the interface return result data and the interface call method data;

[0031] If the field type of the log field is a delayed type, the expected data of the log field after a preset time period is determined, and the detection data is obtained according to the expected data of the log field.

[0032] In a second aspect, an embodiment of the present invention provides a log data processing device, the log data processing device comprising a field processing module, a mode determination module, a data acquisition module and a data detection module;

[0033] A field processing module, used for performing field analysis on the log data of the monitored business system to obtain the log field in the log data and the field type and second field data of the log field;

[0034] A method determination module, used to determine the data channel and detection rule of the detection data corresponding to the log field according to the field type of the log field; wherein each field type is correspondingly configured with a data channel and a detection rule;

[0035] A data acquisition module, used for acquiring the detection data of the log field based on the data channel;

[0036] The data detection module is used to detect the second field data of the log field according to the detection data and the detection rule.

[0037] In a third aspect, an embodiment of the present invention provides an electronic device, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the log data processing method described in any of the above-mentioned embodiments when executing the computer program.

[0038] In a fourth aspect, an embodiment of the present invention provides a readable storage medium, wherein the readable storage medium includes a computer program, and when the computer program is executed, the electronic device where the readable storage medium is located is controlled to execute the log data processing method described in any of the above embodiments.

[0039] The log data processing method, device, electronic device and readable storage medium provided by the embodiment of the present invention perform field analysis on the log data of the monitored business system to obtain the log field in the log data and the field type and second field data of the log field, determine the data channel and detection rule of the detection data corresponding to the log field according to the field type of the log field, obtain the detection data of the log field based on the data channel, and detect the second field data of the log field according to the detection data and the detection rule. In this way, by determining the data channel of the detection data corresponding to the log field, all-round detection data of the log data can be obtained, and then comprehensive detection of the business can be achieved based on the detection data and the detection rules, problems can be discovered and handled in a timely manner, and the stability of the business system is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The technical solution and other beneficial effects of the present application will be made apparent by describing in detail the specific implementation methods of the present application in conjunction with the accompanying drawings.

[0041] Figure 1 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention.

[0042] Figure 2 A flow chart of a log data processing method provided by an embodiment of the present invention.

[0043] Figure 3 A further flow chart of a log data processing method provided by an embodiment of the present invention is provided.

[0044] Figure 4 A schematic diagram of a framework of a monitoring platform provided in an embodiment of the present invention.

[0045] Figure 5 A block diagram of a log data processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0047] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0048] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or reference letters in different examples, and this repetition is for the purpose of simplification and clarity, which itself does not indicate the relationship between the various embodiments and / or settings discussed. In addition, the various specific processes and material examples provided by the present application, but those of ordinary skill in the art can appreciate the application of other processes and / or the use of other materials.

[0049] At present, the existing alarm monitoring can be roughly divided into two categories: non-business alarms and business alarms. Non-business alarms are represented by Zabbix, which mainly monitors the CPU load, memory usage, disk usage, and network status of the business system. Business alarms mainly customize some scheduled tasks to perform data comparison and verification, which may have a certain delay and cannot achieve real-time monitoring, or make some basic verifications in the business logic, but generally based on the verification of the upstream service or system processing results that are considered reasonable. This generally cannot achieve comprehensive verification. For example, if a field is set to a state, it is impossible to achieve state verification of all database tables (including the current business system and external business systems) and third-party services, and it is not reusable. There will be a lot of verification logic in the business implementation.

[0050] Based on the above research, an embodiment of the present invention provides a log data processing method, device, electronic device and readable storage medium, which performs field analysis on the log data of the monitored business system to obtain the log field in the log data and the field type and second field data of the log field, and determines the data channel and detection rule of the detection data corresponding to the log field according to the field type of the log field, obtains the detection data of the log field based on the data channel, and detects the second field data of the log field according to the detection data and the detection rule. In this way, by determining the data channel of the detection data corresponding to the log field, based on the data channel, the detection data of each log field can be obtained, and then the detection data of the log field can be detected through the detection data and the detection rule, so that the business can be fully detected in real time, problems can be found and handled in time, and the stability of the business system is improved.

[0051] See also Figure 1 , Figure 1 A schematic diagram of the structure of an electronic device provided in this embodiment. Figure 1 As shown, the electronic device 100 includes a log data processing device 10, a memory 20, a processor 30, and a communication unit 40. The memory 20, the processor 30, and the communication unit 40 are electrically connected to each other directly or indirectly to achieve signal transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0052] In this embodiment, the log data processing device 10 includes at least one software function module that can be stored in the memory 20 in the form of software or firmware. The processor 30 is used to execute the software function module stored in the memory 20. When the electronic device 100 is running, the processor 30 communicates with the memory 20 through a bus, and the processor 30 executes the software function module to implement the log data processing method described in this embodiment.

[0053] Among them, the memory 20 can be, but is not limited to, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable read-only memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.

[0054] The processor 30 is used to perform one or more functions described in this embodiment. In some embodiments, the processor 30 may include one or more processing cores (e.g., a single-core processor (S) or a multi-core processor (S)). By way of example only, the processor 30 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), an application specific instruction set processor (ASIP), a graphics processing unit (GPU), a physical processing unit (PPU), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a controller, a microcontroller unit, a reduced instruction set computer (RISC) or a microprocessor, or any combination thereof.

[0055] For ease of description, only one processor is described in the electronic device 100. However, it should be noted that the electronic device 100 in this embodiment may also include multiple processors, so the steps performed by one processor described in this embodiment may also be performed jointly or individually by multiple processors. For example, if the processor of the electronic device performs step A and step B, it should be understood that step A and step B may also be performed jointly by two different processors or individually in one processor. For example, the processor performs step A and the second processor performs step B, or the processor and the second processor perform steps A and B together.

[0056] In this embodiment, the process definition method disclosed in any implementation manner may be applied to the processor 30 or implemented by the processor 30 .

[0057] The communication unit 40 is used to establish a communication connection between the electronic device 100 and other devices through a network, and to send and receive data through the network.

[0058] In some embodiments, the network may be any type of wired or wireless network, or a combination thereof. By way of example only, the network may include a wired network, a wireless network, a fiber optic network, a telecommunications network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a wide area network (WAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, or a near field communication (NFC) network, or any combination thereof.

[0059] In this embodiment, the electronic device may be a notebook, an ultra-mobile personal computer (UMPC), a personal digital assistant (PDA), a physical server, or other devices, and may also be a server cluster built based on multiple physical servers. This embodiment does not impose any restrictions on the specific type of electronic device.

[0060] Understandably, Figure 1 The structure shown is for illustration only. The electronic device may also have Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown. Figure 1 The components shown may be implemented in hardware, software or a combination thereof.

[0061] based on Figure 1 This embodiment provides a log data processing method. Figure 1 The electronic device shown in the figure executes the steps of the log data processing method provided by this embodiment. Figure 2 The log data processing method provided in this embodiment includes steps S101 to S104.

[0062] Step S101: performing field parsing on the log data of the business system to be monitored to obtain the log fields in the log data and the field types and second field data of the log fields.

[0063] In this embodiment, the business system refers to a system that executes a specific business. The type of business is not specifically limited in this embodiment. For example, it can be a system that executes a logistics business or a system that executes an e-commerce business.

[0064] In an optional implementation, there may be multiple business systems, each of which is in communication connection with the electronic device. The business system to be monitored may be any one of the business systems.

[0065] In an optional implementation, a unique system code may be configured for each business system, and the system code of each business system may be stored in an electronic device, and the electronic device may obtain the log data of the corresponding business system through the system code. Specifically, when it is necessary to obtain the log data of the business system to be monitored, the system code of the business system to be monitored may be matched with the system codes of each business system to obtain a target business system that matches the system code of the business system to be monitored, and then the log data may be read from the target business system.

[0066] In an optional implementation, a log collector Filebeat can also be configured to collect log data of each business system through the log collector, and the log data of each business system is associated with the system code of the business system. Afterwards, for the business system to be monitored, the system code of the business system to be monitored can be matched with each system code in the log collector to obtain a target system code that matches the system code of the business system to be monitored, and the log data associated with the target system code is determined as the log data of the business system to be monitored.

[0067] In this embodiment, when collecting the log data of the business system, the log fields to be monitored can be formulated according to the business needs, and then when the business system is running, the operation data of the business system can be collected based on the formulated log fields, and the log data of the business system can be obtained. Based on this, the obtained log data includes the formulated log fields and the field data corresponding to the log fields. It can be understood that the field data of the log fields represent the operation data of the business system at runtime, reflecting the operation status or business status of the business system at runtime.

[0068] In this embodiment, each log field in the log data corresponds to a field type, and the log fields are distinguished by the field type. It can be understood that there may be multiple log fields or only one log field under the same field type.

[0069] In order to improve the accuracy of log detection, in this embodiment, after the log data of the business system to be monitored is obtained, the log data may be preprocessed.

[0070] Considering that the collected log data may contain duplicate data or invalid data, based on this, the preprocessing of the log data in this embodiment may include a filtering operation of the log data.

[0071] By filtering the log data, you can remove duplicate data and data irrelevant to the detection requirements in the log data, and reduce the interference of invalid data. Among them, you can use Logstash to filter the log data and perform other operations. Logstash is an open source data collection engine with real-time pipeline capabilities. Logstash can dynamically unify data from different sources and standardize the data to the selected target output. It provides a large number of plug-ins that can parse, enrich, convert and buffer any type of data.

[0072] Considering that if the formats of the fields in the collected log data are inconsistent, inaccurate detection will be caused. For example, if the formats of the time field, IP field, user agent field, etc. are not unified, the field meaning to be expressed by the field data of each field cannot be accurately obtained, thereby causing inaccurate detection. Based on this, in this embodiment, the preprocessing of the log data can also include unifying the field formats of the log data.

[0073] The log4j component can be used to unify the format of log data. Log4j is an open source project of Apache. By using Log4j, the output format of each log data can be controlled.

[0074] After the log data is preprocessed, the preprocessed log data can be subjected to business detection. When the preprocessed log data is subjected to business detection, the log data can be firstly subjected to field analysis to determine the log field and the second field data of the log field in the log data, and then the second field data of the log field can be detected to detect whether there is a problem in the business.

[0075] Optionally, in this embodiment, the field parsing of the log data can be performed by first dividing the log data into fields, and then extracting the field data corresponding to each log field, so as to obtain the second field data of the log field. For example, for the log data {"ip":"113.89.238.214","Time":"2023.4.11"}, after field division, log fields such as IP, userAgent, and Time can be obtained. After data extraction, the second field data corresponding to the IP log field can be obtained as 113.89.238.214, and the second field data corresponding to the Time log field can be obtained as 2023.4.11.

[0076] Optionally, in this embodiment, the log data may be divided into fields using a regular expression, and the second field data of each log field may be extracted.

[0077] In this embodiment, the log data includes business log data and request log data of the business system to be monitored, wherein the business log data mainly refers to business data such as transaction data, inventory data, order data, etc., and the business log data is mainly stored based on data tables. The request log data mainly refers to the interface request data of the business system to be monitored, including data of calling a third-party interface through the business system to be monitored. Therefore, in this embodiment, the field types of the log fields in the log data can be mainly divided into two types, namely, data table type and interface type.

[0078] Considering that there are delayed businesses in the actual business operation process, for example, for a certain order in the business, the payment time is set to 30 minutes. After 30 minutes, the field status of the payment field corresponding to the order must change, that is, it becomes payment completed or payment failed. Based on this, the log data can also contain delayed type fields.

[0079] It can be understood that for the log field extracted from the log data, if the log field is extracted from the data table in the business log data, the field type of the log field is the data table type; if the log field is extracted from the request log data, the field type of the log field is the interface type; if the log field requires a delayed detection status, the field type of the log field is a delay type.

[0080] It can be understood that the field type division of the log field described above is only one of the implementation methods listed in this embodiment. The field type division of the log field can be set according to the actual application scenario and is not specifically limited.

[0081] Step S102: Determine the data channel and detection rule of the detection data corresponding to the log field according to the field type of the log field.

[0082] In this embodiment, the detection data refers to the verification data of the log field, that is, the correct field data of the log field. When performing business detection in this embodiment, the verification data is compared with the log data of the log field in the log data to detect whether the log data of the log field is in a normal state. Therefore, how to obtain comprehensive and correct verification data is the key to ensure the accuracy of detection.

[0083] Considering that the data channels for verification data of log fields of different log types are different, for example, for verification data of data table type, it is necessary to obtain field data in the data table from the database or third-party data source as detection data; for verification data of interface type, it is necessary to call a third-party interface to obtain detection data; and for verification data of delay type, it is necessary to use expected data as detection data.

[0084] Based on this, in order to obtain comprehensive and correct verification data, in this embodiment, for each field type, the data channel of the verification data corresponding to the field type is first determined, and then each field type is associated with its corresponding data channel to obtain channel type mapping data. The channel type mapping data is used to indicate the mapping relationship between each field type and the data channel.

[0085] Based on this, after obtaining the log fields and field types of the log fields in the log data, for each log field in the log data, the data channel of the detection data corresponding to the log field can be obtained from the channel type mapping data according to the field type of the log field.

[0086] Optionally, for each log field, the field type of the log field can be matched with the field type in the type data channel to obtain a target field type that matches the field type of the log field, and then the data channel corresponding to the target field type is determined as the field type of the log field.

[0087] Taking into account the different field types of log fields, the detection focus is different. For example, for log fields with a data table type, the main focus is to detect whether the data value and data status of the log field are consistent with the data value and data status in other data tables (in the database or third-party data source); for log fields with an interface type, the main focus is to detect whether the interface call method corresponding to the log field is consistent with the set call method; for log fields with a delay type, the main focus is to detect whether the state of the log field after the set time is consistent with the expected state.

[0088] In order to realize the detection of various types of log fields, in this embodiment, detection rules are configured corresponding to log fields of different field types, and each field type is associated with its corresponding detection rule to obtain rule type mapping data, which is used to indicate the mapping relationship between each field type and the detection rule.

[0089] Based on this, after obtaining the log fields and field types of the log fields in the log data, for each log field in the log data, the detection rule corresponding to the log field can be obtained from the rule type mapping data according to the field type of the log field.

[0090] In an optional implementation, a system rule package may be set for different business systems, and the system rule package stores the detection rules corresponding to each field type of the business system, that is, the system rule package of each business system stores the rule type mapping data of the business system. The system rule package of each business system is associated with the system code of the business system. When detecting the business of the monitored business system, the system rule package of the monitored business system may be first obtained according to the system code of the monitored business system, and then, for the log field of the monitored business system, according to the field type of the log field, the detection rule corresponding to the log field is obtained from the system rule package, and then, according to the detection rule corresponding to the log field and the detection data, the second field data of the log field is detected.

[0091] This embodiment configures corresponding system rule packages for different business systems. When the verification rules of a business system need to be updated, the verification rules in the system rule package of the business system can be directly updated, thereby improving the flexibility of detection.

[0092] Step S103: Based on the data channel, the detection data of the log field is obtained.

[0093] After obtaining the data channel of the detection data corresponding to the log field, the detection data can be obtained based on the corresponding data channel.

[0094] In order to obtain detection data from various data channels and realize comprehensive detection of services, in this embodiment, corresponding data acquisition methods are set for different data channels, and detection data is obtained from the data channels based on the data acquisition methods. For example, for a data channel of a database type, a database call interface can be set, and detection data is obtained from the database based on the acquisition method of the interface call. For another example, for a data channel of a third-party interface, the third-party interface can be called and the data generated by calling the third-party interface can be used as detection data.

[0095] Step S104: Detect the second field data of the log field according to the detection data and the detection rule.

[0096] In this embodiment, for each log field, after obtaining the detection data and detection rules corresponding to the log field, the detection rules corresponding to the log field can be run, and the second field data of the log field can be detected according to the detection data corresponding to the log field.

[0097] In order to detect the second field data of the log field, in this embodiment, according to the detection data and the detection rule, the step of detecting the second field data of the log field may include:

[0098] Step a1: extracting field data from the detection data to obtain first field data of the detection field in the detection data;

[0099] Step a2: Based on the detection rule, compare the first field data of the detection field in the detection data with the second field data of the log field to obtain a detection result.

[0100] The detection field in the detection data refers to a field in the detection data that is the same as the log field.

[0101] In order to better illustrate this embodiment, the field data of the detection field in the detection data is defined as the first field data, and the field data of the log field in the log data is defined as the second field data for illustration.

[0102] For each log field, when detecting the second field data of the log field according to the detection data and detection rules corresponding to the log field, the field data of the detection data corresponding to the log field can be first extracted to extract the field data of the corresponding field from the detection data, that is, the first field data of the detection field that is the same as the log field is extracted from the detection data, and then the detection rules of the log field are run to compare the first field data of the detection field with the second field data of the log field extracted from the log data to compare whether the first field data of the detection field extracted from the detection data and the second field data of the log field extracted from the log data match. If they do not match, a detection result of failing to meet the detection standard of the log field is obtained. If they match, a detection result of meeting the detection standard of the log field is obtained.

[0103] The log data processing method provided in this embodiment determines the data channel and detection rule of the detection data corresponding to the log field according to the field type of the log field, and then after obtaining the detection data of the log field based on the data channel, the second field data of the log field can be detected according to the detection data and the detection rule. In this way, for each type of log field, the corresponding detection data and detection rules can be obtained, and then when the detection is performed according to the detection data and detection rules of each log field, a comprehensive detection of the business can be achieved, problems can be discovered and handled in a timely manner, and the stability of the business system can be improved.

[0104] In order to fully obtain the detection data of each field type and implement the detection of log fields of each field type, in this embodiment, based on the data channel, the step of obtaining the detection data of the log field includes:

[0105] Step b1: If the field type of the log field is a data table type, the field data of the log field is obtained from the database through a pre-configured database interface, and / or the field data of the log field is obtained from a third-party data source based on a preset key, and the detection data is obtained according to the obtained field data.

[0106] Step b2: If the field type of the log field is an interface type, call the third-party interface to obtain the interface return result data, and obtain the interface call method data according to the interface identifier of the third-party interface, and obtain the detection data according to the interface return result data and the interface call data.

[0107] Step b3: If the field type of the log field is a delay type, then the expected data of the log field after a preset time period is determined, and the detection data is obtained according to the expected data of the log field.

[0108] Among them, for the log field whose field type is a data table type, its detection data needs to be obtained from the data table in the database of the business system to be monitored and other business systems or the data table of a third-party data source.

[0109] In this embodiment, for the database of the business system, a database interface can be configured, and a connection can be established with the database of the business system through the database interface, and then the detection data can be obtained from the database. For a third-party data source, in order to ensure data security and avoid data leakage, the detection data can be obtained from the third-party data source of the third-party service through a key agreed with the third-party service.

[0110] Optionally, for a log field whose field type is a data table type, after establishing a connection with the database through the configured database interface, the target data table including the log field can be searched from the database, and then the field data of the log field can be extracted from the target data table, and the extracted field data can be used as the detection data of the log field.

[0111] Optionally, for a log field whose field type is a data table type, when acquiring detection data from a third-party data source, a data request may be first sent to the third-party service, wherein the data request includes an agreed key. After receiving the data request, the third-party service verifies the key in the data request. When the key verification is successful, the verification result information is fed back to the electronic device, and data acquisition permissions are set for the electronic device so that the electronic device can acquire data from the third-party data source. When the verification result information is received, the detection data may be acquired from the third-party data source. Accordingly, when acquiring detection data from a third-party data source, the target data table including the log field is first searched in the third-party data source, and then the field data of the log field is extracted from the target data table, and the extracted field data is used as the detection data of the log field.

[0112] Considering that when the business system calls a third-party interface, if the third-party interface cannot return result data, it will cause business problems. At the same time, if there is a problem with the calling method, it may also cause the third-party interface call to fail, thereby causing business problems. Therefore, in this embodiment, for the log field whose field type is the interface type, the field data obtained from the log data includes the interface calling method data and the return result data of the third-party interface. Among them, the interface calling method data refers to the calling method data of the business system calling the third-party interface, and the return result data of the third-party interface refers to the result data returned by the third-party interface after the business system calls the third-party interface.

[0113] In order to achieve all-round detection of the business, in this embodiment, for the log field whose field type is the interface type, its detection data also needs to include the calling method data of the third-party interface and the interface return result data of the third-party interface.

[0114] Optionally, in this embodiment, a registration center platform may be set up, and the interface identifiers of each interface and the calling methods corresponding to each interface identifier may be recorded through the registration center platform. The address of the registration center platform is pre-configured in the electronic device. When obtaining the calling method data of the third-party interface, the registration center may be accessed through the registration center address, and then the interface identifier of the third-party interface may be matched with the interface identifiers of each interface in the registration center platform to obtain the target interface identifier that matches the interface identifier of the third-party interface, and the calling method corresponding to the target interface identifier may be determined as the calling method of the third-party interface, thereby obtaining the interface calling method data of the third-party interface.

[0115] Optionally, in this embodiment, for log fields whose field type is an interface type, a third-party interface can be called through the Hyper Text Transfer Protocol (HTTP) to obtain response data of the third-party interface, and the interface return result data returned by the third-party interface can be obtained from the response data.

[0116] After obtaining the interface calling mode data of the third-party interface and the interface return result data returned by the third-party interface, the interface calling mode data of the third-party interface and the interface return result data returned by the third-party interface can be used as detection data.

[0117] In this embodiment, for a log field whose field type is a delayed type, the expected data of the log field after a preset time length can be determined first, and the expected data of the log field can be determined as the detection data. For example, for a payment field whose field type is a delayed field, the preset time length is 30 minutes. When a payment service is triggered, the payment status of the payment field is changed to paying. It is expected that after 30 minutes, the payment status of the payment field can be changed to payment completed or payment failed. Therefore, for the payment field, after the preset time length, the expected data is that the payment status is changed to payment completed or payment failed.

[0118] The log data processing method provided in this embodiment can obtain comprehensive detection data based on the data channels corresponding to each field type by determining the data channels corresponding to each field type, thereby realizing all-round business detection.

[0119] It can be understood that in this embodiment, for each log field, after obtaining the detection data of the log field according to the data channel corresponding to the field type of the log field, the first field data of the detection field corresponding to the log field can be extracted from the detection data, and the second field data of the log field can be extracted from the log data, and then the detection rule corresponding to the log field is run, and the first field data of the detection field is compared with the second field data of the log field to realize the detection of the log field.

[0120] In this embodiment, since different field types correspond to different detection rules and data channels, the detection contents corresponding to log fields of different field types are also different. In order to better illustrate this embodiment, taking log fields of data table type, interface type, and delay type as examples, the steps of comparing the first field data of the detection field in the detection data with the second field data of the log field based on the detection rules to obtain the detection result are described in detail.

[0121] Step c1: If the field type of the log field is a data table type, the field status of the detection field in the detection data is compared with the field status of the log field.

[0122] Step c2: If the field type of the log field is an interface type, compare the interface call mode data and the interface return result data of the detection field in the detection data with the second field data of the log field.

[0123] Step c3: If the field type of the log field is a delay type, compare the expected data of the detection field in the detection data with the second field data of the log field.

[0124] Among them, for the log field whose field type is a data table type, the field data including the field state of the field is extracted from the data table, and the corresponding detection rule is configured as a comparison detection of the field state. Therefore, in this embodiment, for the log field whose field type is a data table type, based on the detection rule, the first field data of the detection field in the detection data is compared with the second field data of the log field, which can be to extract the field state of the detection field from the detection data, extract the field state of the log field from the log data, and then run the comparison detection rule of the field state, compare the field state of the detection field in the detection data with the field state of the log field, if the field state of the detection field in the detection data is consistent with the field state of the log field, it means that there is no problem with the second field data of the log field in the log data, and then for the log field, a qualified detection result can be obtained. If the field state of the detection field in the detection data is not consistent with the field state of the log field, it means that there is a problem with the second field data of the log field in the log data, and then for the log field, a non-qualified detection result can be obtained.

[0125] For example, for log field A whose field type is a data table type, if the field status of the detection field in the detection data that is the same as the log field is zhuangtai_X1, and the field status of the log field in the log data is zhuangtai_X2, and the field status of the detection field in the detection data is inconsistent with the field status of the log field in the log data, it means that there is a problem with the second field data of the log field in the log data. If the field status of the log field in the log data is zhuangtai_X1, that is, the field status of the detection field in the detection data is consistent with the field status of the log field in the log data, it means that there is no problem with the second field data of the log field in the log data.

[0126] In the present embodiment, for a log field whose field type is an interface type, its detection data includes the interface call method data and the interface return result data of the field, and its corresponding detection rule is configured as the comparison detection of the interface call method and the comparison detection of the interface return result. Therefore, in the present embodiment, for a log field whose field type is an interface type, based on the detection rule, the first field data of the detection field in the detection data is compared with the second field data of the log field, which can be to extract the interface call method and the interface return result of the detection field from the detection data, extract the interface call method and the interface return result of the log field from the log data, and then run the corresponding detection rule, compare the interface call method of the detection field in the detection data with the interface call method of the log field, and compare the interface return result of the detection field in the detection data with the interface return result of the log field.

[0127] If the interface calling method of the detection field in the detection data is inconsistent with the interface calling method of the log field in the log data, and / or the interface return result of the detection field in the detection data is inconsistent with the interface return result of the log field in the log data, it means that there is a problem with the second field data of the log field in the log data, and thus for the log field, a substandard detection result can be obtained. If the interface calling method of the detection field in the detection data is consistent with the interface calling method of the log field in the log data, and the interface return result of the detection field in the detection data is consistent with the interface return result of the log field, it means that there is no problem with the second field data of the log field in the log data, and thus for the log field, a qualified detection result can be obtained.

[0128] In this embodiment, for a log field whose field type is a delay type, its detection data includes the expected data of the field, and its corresponding detection rule is configured as a comparison detection of the expected data. Therefore, in this embodiment, for a log field whose field type is a delay type, based on the detection rule, the first field data of the detection field in the detection data is compared with the second field data of the log field, which can be to extract the expected data corresponding to the detection field from the detection data, extract the second field data of the log field from the log data, and then run the corresponding detection rule to compare the expected data of the detection field in the detection data with the second field data of the log field in the log data.

[0129] If the expected data of the detection field in the detection data is consistent with the second field data of the log field, it means that there is no problem with the second field data of the log field in the log data, and thus a qualified detection result can be obtained for the log field. If the expected data of the detection field in the detection data is not consistent with the second field data of the log field, it means that there is a problem with the second field data of the log field in the log data, and thus a substandard detection result can be obtained for the log field.

[0130] In an optional implementation, in this embodiment, for a log field whose field type is a delayed type, when the business of the log field is triggered, the state of the log field will be monitored and timed. After the set time is reached, the state of the log field after reaching the set time will be collected to obtain the second field data of the log field, and then the second field data of the log field is compared with its expected data to determine whether the expected data of the detection field in the detection data is consistent with the second field data of the log field.

[0131] For example, for a payment field whose field type is a delayed field, the preset duration is 30 minutes. When the payment service is triggered, the payment status of the payment field is paying. After 30 minutes, the payment status of the payment field is obtained and compared with the expected data of the payment field. The expected data of the payment field is set to payment completed or payment failed. If the payment status of the payment field is payment completed or payment failed, it is determined that the second field data of the payment field is consistent with the expected data.

[0132] In an optional implementation, for a log field whose corresponding field type is a delay type, the delay comparison of the log field can also be implemented based on the RocketMQ component, which will not be described in detail here.

[0133] The log data processing method provided in this embodiment can perform comprehensive business detection on the log data based on the detection data and detection rules corresponding to each field type by determining the detection data and detection rules corresponding to each field type.

[0134] In this embodiment, after detecting the second field data of the log field through the detection data and the detection rules, the field with the problem in the log data can be obtained. After obtaining the field with the problem, an alarm can be issued based on the field with the problem. In order to respond to more urgent problems in a timely manner and avoid frequent alarms and increase the workload of operation and maintenance personnel, in this embodiment, when issuing business alarms, alarm classification can be performed. For example, the alarm level of the business system can be divided into emergency alarms, major alarms, and minor alarms according to the severity of the problem.

[0135] Based on this, Figure 3 As shown, after the second field data of the log field is detected according to the detection data and the detection rule, the log data processing method further includes steps S201 to S203.

[0136] Step S201: determining target detection results that do not meet the standards and target detection rules used to obtain the target detection results;

[0137] Step S202: Determine the alarm level of the business system to be monitored according to the target detection rule.

[0138] Step S203: Activate the security warning strategy corresponding to the warning level.

[0139] In this embodiment, for each log field in the log data, a detection result is obtained by detecting the second field data of the log field according to the detection data and detection rule of the log field. If the second field data of the log field is detected to have a problem, the detection result of the log field indicates that the detection of the log field does not meet the standard, and then the detection result of the log field can be determined as the target detection result, and the detection rule corresponding to the log field can be determined as the target detection rule used to obtain the target detection result.

[0140] It can be understood that for a log field in which no problem is detected, the detection result indicates that the log field meets the standard.

[0141] After obtaining the target detection result that is determined to be substandard and the target detection rule used to obtain the target detection result, the alarm level of the business system to be monitored can be determined according to the target detection rule.

[0142] Optionally, an alarm level may be set for each detection rule. After obtaining the target detection rule, the alarm level corresponding to each target detection rule may be determined first, and the highest alarm level may be selected as the alarm level of the business system to be monitored.

[0143] Optionally, in order to accurately obtain the alarm level of the business system to be monitored, in this embodiment, for each detection rule, an alarm score is set for the detection rule, and the alarm level of the business system to be monitored is obtained based on the alarm score corresponding to the detection rule. In detail, the alarm level of the business system to be monitored can be determined by the following steps.

[0144] Step d1: Determine the alarm score corresponding to the target detection rule according to the preset rule score data table.

[0145] Step d3: Determine the alarm level of the business system to be monitored according to the alarm score corresponding to the target detection rule.

[0146] Among them, the rule score data table is used to indicate the alarm score corresponding to each detection rule. After obtaining the target detection rule, the target detection rule can be matched with each detection rule in the rule score data table to obtain the alarm score corresponding to each target detection rule.

[0147] In this embodiment, an alarm score interval is set corresponding to each alarm level. After obtaining the alarm scores corresponding to each target detection rule, the alarm scores corresponding to each target detection rule can be summed to obtain the total alarm score. The total alarm score is then matched with the alarm score intervals corresponding to each alarm level to obtain the target alarm score interval in which the total alarm score is located, and the alarm level corresponding to the target alarm score interval is determined as the alarm level of the business system to be monitored.

[0148] Optionally, after obtaining the alarm scores corresponding to each target detection rule, the alarm scores corresponding to each target detection rule can be averaged to obtain an average alarm score, and then the average alarm score is matched with the alarm score interval corresponding to each alarm level to obtain the target alarm score interval in which the average alarm score is located, and the alarm level corresponding to the target alarm score interval is determined as the alarm level of the business system to be monitored.

[0149] In this embodiment, different security alarm strategies are configured corresponding to different alarm levels. After the alarm level of the business system to be monitored is obtained, the corresponding security alarm strategy can be activated according to the alarm level of the business system to be monitored.

[0150] For example, for the emergency alarm with the highest alarm level, that is, the alarm level with the most serious problem, you can set a security alarm strategy of service downgrade and emergency prompt. In this way, when the monitored business system triggers an emergency alarm, you can first downgrade the business of the monitored business system, suspend the business or reduce the business to avoid the problem from getting worse. Then, you can notify the R&D personnel through various communication methods such as email, SMS, and phone calls, so that the R&D personnel can deal with the problem as soon as possible.

[0151] For example, for major alarms with the second highest alarm level, you can set an emergency warning security alarm strategy. In this way, when the monitoring business system triggers a major alarm, the R&D personnel will be notified through various communication methods such as email, text message, and phone, so that they can deal with the problem as soon as possible.

[0152] For another example, for a minor alarm with the lowest alarm level, a general prompt security alarm policy can be set. In this way, when the monitoring business system triggers a minor alarm, the R&D personnel can be notified via email, text message or phone call so that they can deal with the problem.

[0153] In order to facilitate the tracing of the problem, in this embodiment, after obtaining the target detection rule, the target detection rule and the field detail data corresponding to the target detection rule can be stored. Optionally, it can be stored in a relational database MySQL.

[0154] The log data processing method provided in this embodiment sets a corresponding alarm score for each detection rule, determines the alarm level of the business system based on the score of the detection rule, and then starts the corresponding security alarm strategy based on the alarm level of the business system. This can respond to more urgent problems in a timely manner and ensure the stability of the business system, while avoiding frequent alarms and increasing the workload of operation and maintenance personnel.

[0155] In order to further achieve all-round monitoring of the business system, the log data processing method provided in this embodiment may also include identifying attack behaviors of the business system. Based on this, after detecting the second field data of the log field according to the detection data and the detection rules, the log data processing method provided in this embodiment also includes the following steps:

[0156] Determine the number of times each IP address in the log data accesses the domain name, and the frequency with which each IP address calls the interface of the business system to be monitored.

[0157] The target IP whose access times exceed the preset access threshold and / or whose calling frequency exceeds the preset frequency threshold is marked.

[0158] In this embodiment, the log data may also include domain name data and system interface call data. The domain name data refers to the data of the domain name configured by each IP accessing the business system, including the number of visits and the time of each visit, and the system interface call data refers to the data of the interface of each IP calling the business system, including the number of calls and the time of each call.

[0159] Therefore, after obtaining the log data of the business system to be monitored, the domain name data and system interface call data in the log data are analyzed to obtain the number of times each IP address accesses the domain name, as well as the number of times each IP address calls the interface of the business system to be monitored and the time of each call. Then, based on the number of times each IP address calls the interface of the business system to be monitored and the time of each call, the call frequency of each address calling the interface of the business system to be monitored can be obtained.

[0160] After obtaining the number of times each IP address accesses the domain name and the calling frequency of each IP address calling the interface of the business system to be monitored, the number of times each IP address accesses the domain name is compared with the preset access threshold, and the calling frequency of each IP address calling the interface of the business system to be monitored is compared with the preset frequency threshold.

[0161] For each IP address, if the number of times the IP address accesses the domain name is greater than the preset access threshold, and / or the frequency of the IP address accessing the interface of the monitored business system is greater than the preset frequency threshold, it means that the IP address may have attack behavior or malicious behavior towards the monitored business system, and the IP address can be determined as the target IP address and marked.

[0162] It is understandable that if the number of times the IP address accesses the domain name is not greater than the preset access threshold, and the call frequency of the IP address accessing the interface of the monitored business system is not greater than the preset frequency threshold, there is no need to mark the IP address.

[0163] After marking the target IP whose access times exceed the preset access threshold and / or whose call frequency exceeds the preset frequency threshold, an alarm can be sent to R&D personnel based on the marked IP address to indicate that the marked IP address may have attack or malicious behavior.

[0164] The log data processing method provided in this embodiment can effectively screen out IP addresses with attack or malicious behaviors by detecting the number of times each IP address accesses the domain name and the calling frequency of each IP address calling the interface of the business system, and then timely manage and control the IP addresses with attack or malicious behaviors to ensure the stability and security of the business system operation.

[0165] In order to achieve real-time detection of the business system to be monitored, in an optional implementation, a monitoring platform can be built based on electronic equipment, such as Figure 4 As shown, the monitoring platform includes a log collection unit, a log preprocessing unit, a message sending unit and a computing unit;

[0166] Among them, the log collection unit is connected to each business system, collects log data of the business system based on the Filebeat component, and sends the collected log data to the log preprocessing unit.

[0167] The log preprocessing unit can preprocess the collected log data based on Logstash, and then transmit the preprocessed log data to the message sending unit.

[0168] The message sending unit transmits data based on the Kafka message queue, that is, the log preprocessing unit transmits the preprocessed log data to the Kafka message queue, and the Kafka message queue transmits the preprocessed log data to the computing unit.

[0169] Optionally, the computing unit can be built using the distributed real-time computing system Storm, and the computing processes such as data detection, rule matching, and alarm classification can be performed based on the bolts in Storm.

[0170] If the Kafka message queue pushes the preprocessed log data directly to the bolt, when the data volume suddenly increases, it may cause the bolt to crash. Therefore, spout can be used to obtain log data from the Kafka message queue, and then transmit the log data to the computing unit storm. Based on the bolt, data detection, rule matching, alarm classification and other calculation processes are performed, and finally the alarm information is output to the R&D personnel, and the alarm details and alarm rules are output to the database MySQL for storage.

[0171] This embodiment uses a monitoring platform including a log collection unit, a log preprocessing unit, a message sending unit, and a computing unit to monitor the system's services in all aspects and in real time, and can promptly discover problems and stop losses.

[0172] Based on the above research, please refer to Figure 5 The embodiment of the present invention also provides a log data processing device 10, which is applied to Figure 1 The electronic device shown is executed by the electronic device. Figure 5 As shown, the log data processing device 10 provided in this embodiment includes a field processing module 11 , a mode determination module 12 , a data acquisition module 13 and a data detection module 14 .

[0173] A field processing module 11 is used to perform field analysis on the log data of the business system to be monitored, and obtain the log field in the log data and the field type and second field data of the log field;

[0174] The mode determination module 12 is used to determine the data channel and detection rule of the detection data corresponding to the log field according to the field type of the log field; wherein each field type is correspondingly configured with a data channel and a detection rule;

[0175] A data acquisition module 13, used for acquiring the detection data of the log field based on the data channel;

[0176] The data detection module 14 is used to detect the second field data of the log field according to the detection data and the detection rule.

[0177] In an optional implementation, the data detection module 14 is used to:

[0178] Extracting field data from the detection data to obtain first field data of the detection field in the detection data;

[0179] Based on the detection rule, the first field data of the detection field in the detection data is compared with the second field data of the log field to obtain a detection result.

[0180] In an optional implementation, the data detection module 14 is used to:

[0181] If the field type of the log field is a data table type, the field state of the detection field in the detection data is compared with the field state of the log field;

[0182] If the field type of the log field is an interface type, compare the interface call mode data and the interface return result data of the detection field in the detection data with the second field data of the log field;

[0183] If the field type of the log field is a delay type, the expected data of the detection field in the detection data is compared with the second field data of the log field.

[0184] In an optional implementation manner, the log data processing device provided in this embodiment further includes an alarm module, and after detecting the second field data of the log field according to the detection data and the detection rule, the alarm module is used to:

[0185] Determine target detection results that do not meet the standards and target detection rules used to obtain the target detection results;

[0186] Determine the alarm level of the business system to be monitored based on the target detection rules;

[0187] Start the security alert strategy corresponding to the alert level.

[0188] In an optional implementation manner, the alarm module is used to:

[0189] According to the preset rule score data table, determine the alarm score corresponding to the target detection rule; the rule score data table is used to indicate the alarm score corresponding to each detection rule;

[0190] According to the alarm score corresponding to the target detection rule, the alarm level of the business system to be monitored is determined; each alarm level corresponds to an alarm score range.

[0191] In an optional implementation manner, after detecting the second field data of the log field according to the detection data and the detection rule, the data detection module 14 is further used to:

[0192] Determine the number of times each IP address in the log data accesses the domain name, and the frequency of each IP address calling the interface of the business system to be monitored;

[0193] The target IP whose access times exceed the preset access threshold and / or whose calling frequency exceeds the preset frequency threshold is marked.

[0194] In an optional implementation manner, the data acquisition module 13 is used to:

[0195] If the field type of the log field is a data table type, the field data of the log field is obtained from the database through a pre-configured database interface, and / or the field data of the log field is obtained from a third-party data source based on a preset key, and the detection data is obtained according to the obtained field data;

[0196] If the field type of the log field is an interface type, the third-party interface is called to obtain the interface return result data, and the interface call method data is obtained according to the interface identifier of the third-party interface, and the detection data is obtained according to the interface return result data and the interface call method data;

[0197] If the field type of the log field is a delayed type, the expected data of the log field after a preset time length is determined, and the detection data is obtained according to the expected data of the log field.

[0198] The log data processing device provided by the embodiment of the present invention performs field analysis on the log data of the monitored business system to obtain the log field in the log data and the field type and second field data of the log field, determines the data channel and detection rule of the detection data corresponding to the log field according to the field type of the log field, obtains the detection data of the log field based on the data channel, and detects the second field data of the log field according to the detection data and the detection rule. In this way, by determining the data channel and detection rule of the detection data corresponding to the log field, all-round detection data of the log data can be obtained, and then comprehensive detection of the business can be achieved based on the detection data and the detection rule, and problems can be discovered and handled in a timely manner, thereby improving the stability of the business system.

[0199] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the log data processing device described above can refer to the corresponding process in the aforementioned method, and will not be described in detail here.

[0200] Based on the above, this embodiment provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the log data processing method of any of the aforementioned implementations is implemented.

[0201] The readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0202] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the readable storage medium described above can refer to the corresponding process in the aforementioned method, and will not be described in detail here.

[0203] In summary, the log data processing method, device, electronic device and readable storage medium provided by the embodiments of the present invention perform field analysis on the log data of the monitored business system to obtain the log field in the log data and the field type and second field data of the log field, determine the data channel and detection rule of the detection data corresponding to the log field according to the field type of the log field, obtain the detection data of the log field based on the data channel, and detect the second field data of the log field according to the detection data and the detection rule. In this way, by determining the data channel of the detection data corresponding to the log field, all-round detection data of the log data can be obtained, and then comprehensive detection of the business can be achieved based on the detection data and the detection rules, problems can be discovered and handled in a timely manner, and the stability of the business system is improved.

[0204] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0205] The above is a detailed introduction to a log data processing method, device, electronic device and readable storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the technical solution and its core idea of ​​the present application. Ordinary technicians in this field should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some of the technical features therein with equivalents; and these modifications or replacements do not make the essence of the corresponding technical solution deviate from the scope of the technical solution of the embodiments of the present application.

Claims

1. A log data processing method, characterized in that: The method comprises: Perform field parsing on the log data of the business system to be monitored to obtain the log field in the log data and the field type and second field data of the log field; Determine, according to the field type of the log field, a data channel and a detection rule for the detection data corresponding to the log field; wherein each field type is correspondingly configured with a data channel and a detection rule; Based on the data channel, acquiring detection data of the log field; The second field data of the log field is detected according to the detection data and the detection rule.

2. The log data processing method according to claim 1, characterized in that: The detecting the second field data of the log field according to the detection data and the detection rule includes: Extracting field data from the detection data to obtain first field data of a detection field in the detection data; Based on the detection rule, the first field data of the detection field in the detection data is compared with the second field data of the log field to obtain a detection result.

3. The log data processing method according to claim 2, characterized in that: The comparing, based on the detection rule, the first field data of the detection field in the detection data with the second field data of the log field includes: If the field type of the log field is a data table type, comparing the field state of the detection field in the detection data with the field state of the log field; If the field type of the log field is an interface type, comparing the interface call mode data and the interface return result data of the detection field in the detection data with the second field data of the log field; If the field type of the log field is a delay type, the expected data of the detection field in the detection data is compared with the second field data of the log field.

4. The log data processing method according to claim 1, characterized in that: After detecting the second field data of the log field according to the detection data and the detection rule, the method further includes: Determining target detection results that do not meet the standards and target detection rules used to obtain the target detection results; Determining the alarm level of the business system to be monitored according to the target detection rule; A security alert policy corresponding to the alert level is initiated.

5. The log data processing method according to claim 4, characterized in that: Determining the alarm level of the service system to be monitored according to the target detection rule includes: Determine the alarm score corresponding to the target detection rule according to a preset rule score data table; the rule score data table is used to indicate the alarm score corresponding to each detection rule; According to the alarm score corresponding to the target detection rule, the alarm level of the service system to be monitored is determined; each alarm level corresponds to an alarm score range.

6. The log data processing method according to claim 1, characterized in that: After detecting the second field data of the log field according to the detection data and the detection rule, the method further includes: Determine the number of times each IP address in the log data accesses the domain name, and the frequency of each IP address calling the interface of the business system to be monitored; The target IP whose access times exceed the preset access threshold and / or whose calling frequency exceeds the preset frequency threshold is marked.

7. The log data processing method according to any one of claims 1 to 6, characterized in that: The acquiring the detection data of the log field based on the data channel includes: If the field type of the log field is a data table type, the field data of the log field is obtained from the database through a preconfigured database interface, and / or the field data of the log field is obtained from a third-party data source based on a preset key, and the detection data is obtained according to the obtained field data; If the field type of the log field is an interface type, a third-party interface is called to obtain interface return result data, and interface call method data is obtained according to an interface identifier of the third-party interface, and detection data is obtained according to the interface return result data and the interface call method data; If the field type of the log field is a delayed type, the expected data of the log field after a preset time period is determined, and the detection data is obtained according to the expected data of the log field.

8. A log data processing device, characterized in that: The log data processing device includes a field processing module, a mode determination module, a data acquisition module and a data detection module; A field processing module, used for performing field analysis on the log data of the business system to be monitored, obtaining the log field in the log data and the field type and second field data of the log field; A method determination module, used to determine the data channel and detection rule of the detection data corresponding to the log field according to the field type of the log field; wherein each field type is correspondingly configured with a data channel and a detection rule; A data acquisition module, used for acquiring the detection data of the log field based on the data channel; The data detection module is used to detect the second field data of the log field according to the detection data and the detection rule.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the log data processing method according to any one of claims 1 to 7 when executing the computer program.

10. A readable storage medium, characterized in that: The readable storage medium includes a computer program, and when the computer program is executed, the electronic device where the readable storage medium is located is controlled to execute the log data processing method according to any one of claims 1 to 7.