Abnormal log analysis method and device, electronic equipment and product

Abstract

The invention discloses an abnormal log analysis method and device, electronic equipment and a product, and relates to the technical field of data analysis. The abnormal log analysis method comprises the following steps: establishing an abnormal log query request based on a business module type in a business system and a to-be-analyzed abnormal type; searching a plurality of abnormal logs corresponding to the abnormal log query request from a database based on the abnormal log query request; identifying the exception type of each exception log in the plurality of exception logs; and determining a processing strategy of each abnormal log in the plurality of abnormal logs based on the abnormal type of each abnormal log in the plurality of abnormal logs and a pre-established abnormal processing rule base. According to the abnormal log analysis method and device, the electronic equipment and the product disclosed by the invention, the efficiency and accuracy of daily abnormal processing can be improved, and the analysis and processing requirements of massive abnormal logs are met.

Description

Technical Field

[0001] The present invention belongs to the technical field of data analysis, and specifically relates to an abnormal log analysis method, device, electronic equipment and product. Background Art

[0002] In complex software and system environments, logs are an important tool for recording and tracking system operation status, events, and errors. By analyzing logs, developers and system administrators can understand the health status of the system, diagnose and solve problems, and optimize system performance.

[0003] At present, traditional system log analysis is mostly done by operation and maintenance personnel checking system task error logs and returning them to developers for manual analysis. With the increase in software scale and complexity, traditional exception log analysis methods can no longer meet the needs of massive exception log analysis and processing.

[0004] Therefore, how to provide an effective solution to facilitate the analysis and processing of abnormal logs has become a difficult problem to be solved in the prior art. Summary of the invention

[0005] The purpose of the present invention is to provide an abnormal log analysis method, device, electronic equipment and product to solve the above-mentioned problems existing in the prior art.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides an abnormal log analysis method, comprising:

[0008] Establish an exception log query request based on the business module type in the business system and the exception type to be analyzed;

[0009] Based on the abnormal log query request, searching a plurality of abnormal logs corresponding to the abnormal log query request from a database;

[0010] Identifying the abnormality type of each abnormality log in the plurality of abnormality logs;

[0011] Based on the exception type of each exception log in the plurality of exception logs and a pre-established exception handling rule base, a handling strategy for each exception log in the plurality of exception logs is determined.

[0012] Based on the above disclosed content, the present invention establishes an abnormal log query request based on the business module type and the abnormal type to be analyzed in the business system; finds out multiple abnormal logs corresponding to the abnormal log query request from the database based on the abnormal log query request; identifies the abnormal type of each abnormal log in the multiple abnormal logs; and determines the processing strategy of each abnormal log in the multiple abnormal logs based on the abnormal type of each abnormal log in the multiple abnormal logs and the pre-established abnormal processing rule base. In this way, the abnormal type can be automatically identified and classified, which solves the tedious work of developers needing to manually capture and judge the abnormal type, improves the efficiency and accuracy of abnormal daily processing, meets the massive abnormal log analysis and processing requirements, and developers can also customize the abnormal processing rule base to meet specific needs, so that developers do not need to process the abnormal logs from scratch every time, but process based on predefined rules, which improves development efficiency. In addition, for the identified abnormal logs, corresponding processing strategies can be provided to help developers quickly locate and solve problems, which reduces the time for debugging and troubleshooting, helps to timely discover and solve potential problems, and improves the stability and reliability of the system.

[0013] In a possible design, identifying the abnormality type of each abnormal log in the plurality of abnormal logs includes:

[0014] Extracting key entities and semantic relationships of each of the plurality of abnormal logs through natural speech processing;

[0015] The key entities and semantic relations of each of the plurality of exception logs are used as inputs of a pre-trained classification model to perform operations, so as to obtain an exception type of each of the plurality of exception logs.

[0016] In a possible design, identifying the abnormality type of each abnormal log in the plurality of abnormal logs includes:

[0017] Parsing the exception message and stack information of each exception log in the plurality of exception logs;

[0018] Based on the exception message and stack information of each exception log in the plurality of exception logs, the exception type of each exception log in the plurality of exception logs is determined.

[0019] In one possible design, after establishing an exception log query request based on the business module type and the exception type to be analyzed in the business system, the method further includes:

[0020] Expanding the query condition in the exception log query request in combination with the semantics and the context to obtain an expanded exception log query request;

[0021] The step of searching a database for a plurality of abnormal logs corresponding to the abnormal log query request based on the abnormal log query request includes:

[0022] Based on the expanded exception log query request, multiple exception logs corresponding to the expanded exception log query request are searched from the database through fuzzy query, wildcard query and / or phrase matching query.

[0023] In one possible design, before establishing an exception log query request based on the business module type and the exception type to be analyzed in the business system, the method further includes:

[0024] Based on the load, network latency and / or node health of each node in the distributed system, the target node for abnormal log analysis is selected from the distributed cluster through load balancing;

[0025] The step of searching a database for a plurality of abnormal logs corresponding to the abnormal log query request based on the abnormal log query request includes:

[0026] Based on the abnormal log query request, multiple abnormal logs corresponding to the abnormal log query request are searched from the database of the target node.

[0027] In one possible design, before establishing an exception log query request based on the business module type and the exception type to be analyzed in the business system, the method further includes:

[0028] The log management subsystem collects exception logs during the operation of the business system, and the exception logs include system logs with an error level attribute and system logs with a warning level attribute that are sampled according to a preset ratio.

[0029] In a possible design, after searching a database for a plurality of abnormal logs corresponding to the abnormal log query request based on the abnormal log query request, the method further includes:

[0030] The multiple abnormal logs found are stored in the cache.

[0031] In a second aspect, the present invention provides an abnormal log analysis device, comprising:

[0032] An establishment unit, used to establish an exception log query request based on a business module type in a business system and an exception type to be analyzed;

[0033] A searching unit, configured to search a plurality of abnormal logs corresponding to the abnormal log query request from a database based on the abnormal log query request;

[0034] an identification unit, used to identify the abnormality type of each abnormality log in the plurality of abnormality logs;

[0035] The determination unit is used to determine a processing strategy for each of the plurality of exception logs based on the exception type of each of the plurality of exception logs and a pre-established exception processing rule base.

[0036] In a third aspect, the present invention provides an electronic device comprising a memory, a processor and a transceiver which are communicatively connected in sequence, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the abnormal log analysis method as described in the first aspect or any possible design of the first aspect.

[0037] In a fourth aspect, the present invention provides a computer-readable storage medium having instructions stored thereon. When the instructions are executed on a computer, the abnormal log analysis method described in the first aspect or any possible design of the first aspect is executed.

[0038] In a fifth aspect, the present invention provides a computer program product comprising instructions, which, when executed on a computer, causes the computer to execute the abnormal log analysis method as described in the first aspect or any possible design of the first aspect.

[0039] Beneficial effects:

[0040] The exception log analysis method, device, electronic device and product provided by the present invention can automatically identify and classify exception types, solve the tedious work of developers who need to manually capture and judge exception types, improve the efficiency and accuracy of daily exception processing, and meet the needs of massive exception log analysis and processing. At the same time, developers can also customize the exception processing rule library to meet specific needs, so that developers do not need to process exception logs from scratch every time, but process them based on predefined rules, which improves development efficiency. In addition, for the identified exception logs, corresponding processing strategies can be provided to help developers quickly locate and solve problems, which reduces the time for debugging and troubleshooting, helps to timely discover and solve potential problems, improves the stability and reliability of the system, and facilitates practical application and promotion. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flowchart of the abnormal log analysis method provided in an embodiment of the present application;

[0042] Figure 2 A schematic block diagram of an abnormal log analysis device provided in an embodiment of the present application;

[0043] Figure 3A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in combination with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structure of the drawings is only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0045] It should be understood that although the terms first, second, etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another unit. For example, a first unit can be referred to as a second unit, and similarly, a second unit can be referred to as a first unit without departing from the scope of the exemplary embodiments of the present invention.

[0046] It should be understood that the term "and / or" that may appear in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can represent three situations: A exists alone, B exists alone, and A and B exist at the same time. The term " / and" that may appear in this article describes another type of association object relationship, indicating that two relationships may exist. For example, A / and B can represent two situations: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this article generally indicates that the previous and next associated objects are in an "or" relationship.

[0047] In order to facilitate the analysis and processing of exception logs, the embodiments of the present application provide an exception log analysis method, device, electronic device and product, which can improve the efficiency and accuracy of daily exception processing and meet the needs of massive exception log analysis and processing.

[0048] The abnormal log analysis method provided in the embodiment of the present application can be applied to any node in a distributed system, which includes multiple nodes. The database of each node records the logs of the business system to be analyzed and can identify abnormal logs in the logs recorded in its database through the log management subsystem.

[0049] It can be understood that the execution entity does not constitute a limitation on the embodiments of the present application.

[0050] The abnormal log analysis method provided in the embodiment of the present application will be described in detail below.

[0051] like Figure 1 As shown, it is a flowchart of the abnormal log analysis method provided in the first aspect of the embodiment of the present application. The abnormal log analysis method may include but is not limited to the following steps S101-S104.

[0052] Step S101: Establish an exception log query request based on the business module type in the business system and the exception type to be analyzed.

[0053] The business module type and the type of anomaly to be analyzed may be manually requested and entered by the user, or may be a default option pre-set by the node of the distributed system. In addition, when establishing an abnormal log query request, an abnormal log query request may also be established according to a set query time range, so that only abnormal logs within the query time range can be queried in subsequent queries.

[0054] In one or more embodiments, before establishing an exception log query request based on the business module type in the business system and the exception type to be analyzed, the exception logs during the operation of the business system can also be collected through the log management subsystem. The exception logs include system logs with an error level attribute and system logs with a warning level attribute that are sampled according to a preset ratio.

[0055] The embodiment of the present application can pre-establish a log management subsystem. The core function of the log management subsystem is mainly responsible for collecting, formatting and storing exception logs during the operation of the business system, ensuring the integrity and traceability of the log information, and providing a data basis for subsequent exception analysis and processing.

[0056] Specifically, when establishing a log management subsystem, you can write a log framework file (such as logback-spring.xml) to dynamically configure the log file storage address according to the system environment variable (such as APP_ENV). For example, in the development environment, it is stored in the . / logs / development / directory, in the test environment, it is stored in the / test / logs / directory, and in the production environment, it is stored in the / prod / logs / directory of the distributed system. Then define the environment in which the project is started (development environment, test environment or production environment), and define the console output format, including timestamp, log level, thread name, class name, log message, and node load information (which can be obtained through the embedded system monitoring tool), etc. The node load information can be but is not limited to CPU (Central Processing Unit) usage and memory usage, etc. You can configure database-related properties, including connection timeout (such as 5000 milliseconds), socket timeout (such as 8000 milliseconds), enable SSL (Secure Sockets Layer) encrypted connection, and reasonable connection pool parameters (such as maximum number of connections 30, minimum number of idle connections 5) to ensure stable communication with the database. In addition, you can specify the log level attributes of the recorded logs, such as logs with error level attributes, and logs with warning level attributes (resource warnings that may cause system instability) can be sampled and stored in the database according to a certain ratio (such as 1 / 5) to facilitate subsequent analysis of potential risks.

[0057] The abnormal log analysis method provided in the embodiment of the present application can be applied to any one of the nodes in the distributed system. Therefore, before establishing an abnormal log query request based on the business module type in the business system and the abnormal type to be analyzed, it is also possible to select a target node for abnormal log analysis from a distributed cluster through load balancing based on the load of each node in the distributed system (such as CPU utilization, memory utilization, disk I / O busyness, etc.), network delay and / or node health status (such as hardware failure, software error, etc.), and when searching for abnormal logs, multiple abnormal logs corresponding to the abnormal log query request can be found from the database of the target node based on the abnormal log query request. When the target node fails or the performance is degraded, a new target node for abnormal log analysis can also be re-determined through load balancing.

[0058] Step S102: Based on the abnormal log query request, multiple abnormal logs corresponding to the abnormal log query request are searched from the database.

[0059] In one or more embodiments, after establishing an abnormal log query request based on the business module type in the business system and the abnormal type to be analyzed, the query conditions in the abnormal log query request can also be expanded in combination with semantics and context to obtain an expanded abnormal log query request. For example, for a financial system, the query conditions in the abnormal log query request can be expanded in combination with transaction type, amount range, time interval, and user credit rating, etc. For an e-commerce system, the query conditions in the abnormal log query request can be expanded in combination with factors such as product category, order status, and user region.

[0060] When multiple exception logs corresponding to the exception log query request are found from the database, based on the expanded exception log query request, fuzzy query, wildcard query and / or phrase matching query can be used to find multiple exception logs corresponding to the expanded exception log query request from the database to cope with the uncertainty of exception information and improve the search recall rate and accuracy.

[0061] In one or more embodiments, during the search execution process, paging query parameters can be reasonably set (such as 100-500 results per page), and a result caching mechanism can be enabled to store multiple exception logs found in the cache. The cache mechanism can be set to cache the results of the most recent 10-20 queries, which can reduce the performance impact of repeated queries on the database.

[0062] Step S103: Identify the abnormal type of each abnormal log in the plurality of abnormal logs.

[0063] In one or more embodiments, when identifying the abnormal type of an abnormal log, the abnormal log can be firstly subjected to grammatical and syntactic analysis through natural language processing (NLP), and key entities and semantic relationships of each abnormal log in multiple abnormal logs can be extracted, and then the key entities and semantic relationships of each abnormal log in multiple abnormal logs can be used as inputs of a pre-trained classification model for operation to obtain the abnormal type of each abnormal log in the multiple abnormal logs. Among them, the classification model can be obtained by training with the key entities and semantic relationships of the sample abnormal log as sample inputs and the abnormal type of the sample abnormal log as sample output.

[0064] In one or more embodiments, when identifying the exception type of an exception log, the exception message and stack information of each exception log in the multiple exception logs can be parsed first, and the exception type of each exception log in the multiple exception logs can be determined based on the exception message and stack information of each exception log in the multiple exception logs.

[0065] Step S104: Based on the exception type of each exception log in the plurality of exception logs and a pre-established exception processing rule base, a processing strategy for each exception log in the plurality of exception logs is determined.

[0066] In an embodiment of the present application, an exception handling rule base is pre-established, which is used to define and manage exception log processing rules, realize the mapping of exception types and processing logic, and support the customization and expansion of rules, thereby improving the efficiency and consistency of exception log processing.

[0067] Specifically, when establishing an exception handling rule base, you can comprehensively cover all kinds of exception logs in the rule file (such as exception-rules.drl), such as database connection timeout exception log, transaction deadlock exception log, file read permission exception log, network communication protocol exception log, memory overflow exception log, etc., and define the trigger conditions in detail. The processing logic of the exception handling rule base performs different operations according to the severity of the exception log and the scope of business impact. For example, for exception logs that seriously affect the business, automatic repair operations can be performed and verified immediately. For partially recoverable exception logs, guidance information can be provided to help repair and record the process. For serious exception logs that cannot be automatically repaired, the business process can be suspended, administrators and experts can be notified, and emergency plans can be initiated. At the same time, the priority level can be defined for each rule (such as 1-5 levels, 1 is the highest priority, and 5 is the lowest priority) so that when multiple exceptions occur at the same time, they can be processed according to priority.

[0068] When determining the processing strategy for exception logs, you can formulate and execute appropriate processing strategies based on the identified exception types and the processing rules in the exception processing rule base, and record the processing process and results for subsequent analysis and optimization.

[0069] In summary, the abnormal log analysis method provided by the present invention establishes an abnormal log query request based on the business module type and the abnormal type to be analyzed in the business system; finds out multiple abnormal logs corresponding to the abnormal log query request from the database based on the abnormal log query request; identifies the abnormal type of each abnormal log in the multiple abnormal logs; and determines the processing strategy of each abnormal log in the multiple abnormal logs based on the abnormal type of each abnormal log in the multiple abnormal logs and the pre-established abnormal processing rule base. In this way, the abnormal type can be automatically identified and classified, which solves the tedious work of developers needing to manually capture and judge the abnormal type, improves the efficiency and accuracy of daily abnormal processing, meets the needs of massive abnormal log analysis and processing, and developers can also customize the abnormal processing rule base to meet specific needs, so that developers do not need to process the abnormal logs from scratch every time, but process them based on predefined rules, which improves development efficiency. In addition, for the identified abnormal logs, corresponding processing strategies can be provided to help developers quickly locate and solve problems, which reduces the time for debugging and troubleshooting, helps to timely discover and solve potential problems, improves the stability and reliability of the system, and is convenient for practical application and promotion.

[0070] See also Figure 2 According to a second aspect of an embodiment of the present application, there is provided an abnormal log analysis device, the abnormal log analysis device comprising:

[0071] An establishment unit, used to establish an exception log query request based on a business module type in a business system and an exception type to be analyzed;

[0072] A searching unit, configured to search a plurality of abnormal logs corresponding to the abnormal log query request from a database based on the abnormal log query request;

[0073] an identification unit, used to identify the abnormality type of each abnormality log in the plurality of abnormality logs;

[0074] The determination unit is used to determine a processing strategy for each of the plurality of exception logs based on the exception type of each of the plurality of exception logs and a pre-established exception processing rule base.

[0075] The working process, working details and technical effects of the abnormal log analysis device provided in the second aspect of this embodiment can be found in the first aspect of the embodiment and will not be described in detail here.

[0076] like Figure 3As shown, the third aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor and a transceiver that are communicatively connected in sequence, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the abnormal log analysis method as described in the first aspect of the embodiment.

[0077] For specific examples, the memory may include but is not limited to random access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out memory (FIFO) and / or first-in-last-out memory (FILO), etc.; the processor may be but is not limited to a microprocessor of the STM32F105 series, an ARM (Advanced RISC-Machines), an X86 or other architecture processor, or a processor with an integrated NPU (neural-network processing units); the transceiver may be but is not limited to a WiFi (Wireless Fidelity) wireless transceiver, a Bluetooth wireless transceiver, a General Packet Radio Service (GPRS) wireless transceiver, a ZigBee protocol (a low-power local area network protocol based on the IEEE802.15.4 standard, ZigBee) wireless transceiver, a 3G transceiver, a 4G transceiver and / or a 5G transceiver, etc.

[0078] The fourth aspect of this embodiment provides a computer-readable storage medium storing instructions including the abnormal log analysis method described in the first aspect of the embodiment, that is, the computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the abnormal log analysis method described in the first aspect is executed. The computer-readable storage medium refers to a carrier for storing data, which may include but is not limited to a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash drive, and / or a memory stick, etc., and the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0079] The fifth aspect of this embodiment provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the abnormal log analysis method as described in the first aspect of the embodiment, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0080] It should be understood that certain details are provided in the following description to facilitate a complete understanding of the example embodiments. However, it should be understood by those of ordinary skill in the art that the example embodiments can be implemented without these certain details. For example, the system can be shown in a block diagram to avoid obscuring the example with unnecessary details. In other examples, well-known processes, structures, and techniques may not be shown in unnecessary detail to avoid obscuring the example embodiments.

[0081] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for analyzing abnormal logs, characterized in that: include: Establish an exception log query request based on the business module type in the business system and the exception type to be analyzed; Based on the abnormal log query request, searching a plurality of abnormal logs corresponding to the abnormal log query request from a database; Identifying the abnormality type of each abnormality log in the plurality of abnormality logs; Based on the exception type of each exception log in the plurality of exception logs and a pre-established exception handling rule base, a handling strategy for each exception log in the plurality of exception logs is determined.

2. The abnormal log analysis method according to claim 1, characterized in that: The identifying the abnormal type of each abnormal log in the plurality of abnormal logs includes: Extracting key entities and semantic relationships of each of the plurality of abnormal logs through natural speech processing; The key entities and semantic relations of each of the plurality of exception logs are used as inputs of a pre-trained classification model to perform operations, so as to obtain an exception type of each of the plurality of exception logs.

3. The abnormal log analysis method according to claim 1, characterized in that: The identifying the abnormal type of each abnormal log in the plurality of abnormal logs includes: Parsing the exception message and stack information of each exception log in the plurality of exception logs; Based on the exception message and stack information of each exception log in the plurality of exception logs, the exception type of each exception log in the plurality of exception logs is determined.

4. The abnormal log analysis method according to claim 1, characterized in that: After establishing an exception log query request based on the business module type in the business system and the exception type to be analyzed, the method further includes: Expanding the query condition in the exception log query request in combination with the semantics and the context to obtain an expanded exception log query request; The step of searching a database for a plurality of abnormal logs corresponding to the abnormal log query request based on the abnormal log query request includes: Based on the expanded exception log query request, multiple exception logs corresponding to the expanded exception log query request are searched from the database through fuzzy query, wildcard query and / or phrase matching query.

5. The abnormal log analysis method according to claim 1, characterized in that: Before establishing an exception log query request based on the business module type and the exception type to be analyzed in the business system, the method further includes: Based on the load, network latency and / or node health of each node in the distributed system, the target node for abnormal log analysis is selected from the distributed cluster through load balancing; The step of searching a database for a plurality of abnormal logs corresponding to the abnormal log query request based on the abnormal log query request includes: Based on the abnormal log query request, multiple abnormal logs corresponding to the abnormal log query request are searched from the database of the target node.

6. The abnormal log analysis method according to claim 1, characterized in that: Before establishing an exception log query request based on the business module type and the exception type to be analyzed in the business system, the method further includes: The log management subsystem collects exception logs during the operation of the business system, and the exception logs include system logs with an error level attribute and system logs with a warning level attribute that are sampled according to a preset ratio.

7. The abnormal log analysis method according to claim 1, characterized in that: After searching a plurality of abnormal logs corresponding to the abnormal log query request from a database based on the abnormal log query request, the method further includes: The multiple abnormal logs found are stored in the cache.

8. An abnormal log analysis device, characterized in that: include: An establishment unit, used to establish an exception log query request based on a business module type in a business system and an exception type to be analyzed; A searching unit, configured to search a plurality of abnormal logs corresponding to the abnormal log query request from a database based on the abnormal log query request; an identification unit, used to identify the abnormality type of each abnormality log in the plurality of abnormality logs; The determining unit is used to determine a processing strategy for each of the plurality of exception logs based on the exception type of each of the plurality of exception logs and a pre-established exception processing rule base.

9. An electronic device, characterized in that: It comprises a memory, a processor and a transceiver which are communicatively connected in sequence, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the abnormal log analysis method as described in any one of claims 1 to 7.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or the instruction is executed by a computer, the abnormal log analysis method according to any one of claims 1 to 7 is implemented.

Images