Abnormal log information processing method, device, equipment and storage medium

Through the collaborative work of the aspect-oriented programming system, log file acquisition system and full-text search engine, the problem of low efficiency in distributed exception log information processing is solved, and efficient log information processing and analysis is achieved.

CN112711520BActive Publication Date: 2024-11-12PING AN TECH (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011611184.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-30
Publication Date
2024-11-12
Estimated Expiration
2040-12-30

AI Technical Summary

Technical Problem

In large Internet enterprise applications, as the scale of system logs expands, the collection, analysis and retrieval of distributed exception log information becomes complicated and inefficient, resulting in low processing efficiency.

Method used

By sending exception interception instructions to the aspect-oriented programming system for abnormal detection and file writing, sending monitoring and acquisition instructions to the log file acquisition system for preset period reading and storage, and sending monitoring and extraction instructions to the log management system based on the full-text search engine for Monitor and extract real-time, and filter, index creation and sharding of the exception log information to be processed to obtain the target exception log information.

Benefits of technology

It improves the accuracy and efficiency of exception log file generation, simplifies the performance and application decoupling of log collection, improves the processing efficiency of distributed exception log information, and simplifies collection, analysis and retrieval operations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112711520B_ABST
    Figure CN112711520B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of big data, and provides a method, device, equipment and storage medium for processing abnormal log information, which are used to improve the processing efficiency of distributed abnormal log information. The method for processing abnormal log information includes: performing abnormal detection, reading and file writing on the system application log information of each system application through an aspect-oriented programming system to obtain an abnormal log file; reading and storing the abnormal information in the abnormal log file according to a preset time period through a log file collection system to obtain candidate abnormal log information; monitoring and extracting the candidate abnormal log information in the log file collection system in real time through a log management system based on a full-text search engine to obtain the abnormal log information to be processed; filtering, indexing and sharding the abnormal log information to be processed to obtain the target abnormal log information. In addition, the present invention also relates to blockchain technology, and the target abnormal log information can be stored in the blockchain.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of big data processing, and in particular to a method, device, equipment and storage medium for processing abnormal log information. Background Art

[0002] For large-scale Internet enterprise application services, as the log scale of each system continues to expand, it is difficult to analyze the system status or other system analysis operations. Effectively using the large amount of log information generated during the operation of the system to analyze the system status is of certain significance to the development of system performance, scalability and other requirements. At present, it is usually obtained from each application node to obtain the abnormal log information generated during the operation of the system, and analyze and process the abnormal log information.

[0003] As the number of application nodes increases, the collection, analysis and retrieval of distributed exception log information becomes complicated and difficult, resulting in low processing efficiency of distributed exception log information. Summary of the invention

[0004] The present invention provides a method, device, equipment and storage medium for processing abnormal log information, which are used to improve the processing efficiency of distributed abnormal log information.

[0005] A first aspect of the present invention provides a method for processing abnormal log information, comprising:

[0006] Sending an exception interception instruction to a preset aspect-oriented programming system, so that the aspect-oriented programming system performs exception detection, reading and file writing on the system application log information of each system application according to the exception interception instruction to obtain an exception log file;

[0007] Sending a monitoring and collection instruction to a preset log file collection system, so that the log file collection system reads and stores the abnormal information in the abnormal log file according to the monitoring and collection instruction and according to a preset time period to obtain candidate abnormal log information;

[0008] Sending a monitoring and extraction instruction to a preset log management system based on a full-text search engine, so that the log management system based on the full-text search engine performs real-time monitoring and extraction of candidate abnormal log information in the log file collection system according to the monitoring and extraction instruction, and obtains the abnormal log information to be processed;

[0009] The to-be-processed exception log information is filtered, indexed, and fragmented to obtain target exception log information.

[0010] Optionally, in a first implementation of the first aspect of the present invention, the sending of the exception interception instruction to a preset aspect-oriented programming system so that the aspect-oriented programming system performs exception detection, reading, and file writing on the system application log information of each system application according to the exception interception instruction to obtain an exception log file, including:

[0011] Sending an exception interception instruction to a preset aspect-oriented programming system, so that the aspect-oriented programming system calls a pre-configured interceptor chain according to the exception interception instruction, performs exception detection and exception information reading on the system application log information of each system application, and obtains initial exception log information;

[0012] Classify the initial abnormal log information by type to obtain classified log information, and obtain a target error log file corresponding to the classification type of the classified log information;

[0013] According to a preset writing period, the initial abnormal log information is written into the target error log file to obtain an abnormal log file.

[0014] Optionally, in a second implementation of the first aspect of the present invention, the sending of the monitoring and extraction instruction to a preset log management system based on a full-text search engine, so that the log management system based on the full-text search engine performs real-time monitoring and extraction of candidate abnormal log information in the log file collection system according to the monitoring and extraction instruction to obtain abnormal log information to be processed, including:

[0015] Sending a monitoring and extraction instruction to a preset log management system based on a full-text search engine, so that the log management system based on the full-text search engine monitors and analyzes the increment, reading time and storage time of the candidate abnormal log information in the log file collection system according to the monitoring and extraction instruction and based on a preset monitoring mechanism, and obtains abnormal log information that meets the preset monitoring conditions;

[0016] The abnormal log information that meets the preset monitoring conditions is extracted from the log file collection system into a full-text search engine corresponding to a pre-configured cluster service address to obtain the abnormal log information to be processed.

[0017] Optionally, in a third implementation of the first aspect of the present invention, filtering, indexing, and sharding the exception log information to be processed to obtain target exception log information includes:

[0018] Through the full-text search engine and the preset log key information decision tree, the to-be-processed abnormal log information is subjected to information matching, information extraction and keyword extraction to obtain the filtered abnormal log information and the target keyword corresponding to the filtered abnormal log information;

[0019] Creating a clustered index according to the filtering exception log information and the target keyword, and performing sharding processing on the data in the clustered index to obtain sharded index data;

[0020] According to a preset storage strategy, the shard index data is stored in a preset storage space to obtain target abnormal log information.

[0021] Optionally, in a fourth implementation of the first aspect of the present invention, the sending of the monitoring and collection instruction to a preset log file collection system so that the log file collection system reads and stores the abnormal information in the abnormal log file according to the monitoring and collection instruction and according to a preset time period to obtain candidate abnormal log information, including:

[0022] Sending a monitoring and collecting instruction to a preset log file collection system, so that the log file collection system reads the abnormal information in the abnormal log file according to the monitoring and collecting instruction and according to a preset time period;

[0023] Acquire historical read position information and target position offset information of the abnormal log file, wherein the historical position is used to indicate the position offset information of the abnormal log file read before the start time of the preset time period, and the target position offset information is used to indicate the position offset information read at the end time of the preset time period;

[0024] Determining whether the historical read position information is consistent with the target position offset information;

[0025] If the historical read position information is inconsistent with the target position offset information, the exception information, the historical read position information and the target position offset information are written into a registration file in a preset storage space to obtain candidate exception log information.

[0026] Optionally, in a fifth implementation of the first aspect of the present invention, after filtering, indexing and sharding the exception log information to be processed to obtain target exception log information, the method further includes:

[0027] A search instruction sent by a preset visual user interface is received, and based on the search instruction, the target abnormal log information is queried and a report is generated to obtain a statistical analysis report, and the statistical analysis report is sent to the visual user interface.

[0028] Optionally, in a sixth implementation of the first aspect of the present invention, the receiving a retrieval instruction sent by a preset visual user interface, querying and generating a report on the target abnormal log information based on the retrieval instruction, obtaining a statistical analysis report, and sending the statistical analysis report to the visual user interface includes:

[0029] Receiving a search instruction sent by a preset visual user interface, and based on the search instruction, searching and filtering the target abnormal log information by using a preset routing algorithm and a genetic algorithm to obtain target search information;

[0030] Perform statistical analysis on the target search information to obtain statistical analysis information, generate a statistical analysis report based on the statistical analysis information, and send the statistical analysis report to the visual user interface.

[0031] A second aspect of the present invention provides a device for processing abnormal log information, comprising:

[0032] An exception interception module is used to send an exception interception instruction to a preset aspect-oriented programming system, so that the aspect-oriented programming system performs exception detection, reading and file writing on the system application log information of each system application according to the exception interception instruction to obtain an exception log file;

[0033] A monitoring and collection module, used for sending a monitoring and collection instruction to a preset log file collection system, so that the log file collection system reads and stores the abnormal information in the abnormal log file according to the monitoring and collection instruction and according to a preset time period, and obtains candidate abnormal log information;

[0034] A monitoring and extraction module, used for sending a monitoring and extraction instruction to a preset log management system based on a full-text search engine, so that the log management system based on the full-text search engine performs real-time monitoring and extraction of candidate abnormal log information in the log file collection system according to the monitoring and extraction instruction, and obtains the abnormal log information to be processed;

[0035] The exception handling module is used to filter, create indexes and perform sharding on the exception log information to be processed to obtain target exception log information.

[0036] Optionally, in a first implementation of the second aspect of the present invention, the exception interception module is specifically used to:

[0037] Sending an exception interception instruction to a preset aspect-oriented programming system, so that the aspect-oriented programming system calls a pre-configured interceptor chain according to the exception interception instruction, performs exception detection and exception information reading on the system application log information of each system application, and obtains initial exception log information;

[0038] Classify the initial abnormal log information by type to obtain classified log information, and obtain a target error log file corresponding to the classification type of the classified log information;

[0039] According to a preset writing period, the initial abnormal log information is written into the target error log file to obtain an abnormal log file.

[0040] Optionally, in a second implementation of the second aspect of the present invention, the monitoring and extraction module is specifically used to:

[0041] Sending a monitoring and extraction instruction to a preset log management system based on a full-text search engine, so that the log management system based on the full-text search engine monitors and analyzes the increment, reading time and storage time of the candidate abnormal log information in the log file collection system according to the monitoring and extraction instruction and based on a preset monitoring mechanism, and obtains abnormal log information that meets the preset monitoring conditions;

[0042] The abnormal log information that meets the preset monitoring conditions is extracted from the log file collection system into a full-text search engine corresponding to a pre-configured cluster service address to obtain the abnormal log information to be processed.

[0043] Optionally, in a third implementation of the second aspect of the present invention, the exception handling module is specifically used to:

[0044] Through the full-text search engine and the preset log key information decision tree, the to-be-processed abnormal log information is subjected to information matching, information extraction and keyword extraction to obtain the filtered abnormal log information and the target keyword corresponding to the filtered abnormal log information;

[0045] Creating a clustered index according to the filtering exception log information and the target keyword, and performing sharding processing on the data in the clustered index to obtain sharded index data;

[0046] According to a preset storage strategy, the shard index data is stored in a preset storage space to obtain target abnormal log information.

[0047] Optionally, in a fourth implementation of the second aspect of the present invention, the monitoring and collection module is specifically used to:

[0048] Sending a monitoring and collecting instruction to a preset log file collection system, so that the log file collection system reads the abnormal information in the abnormal log file according to the monitoring and collecting instruction and according to a preset time period;

[0049] Acquire historical read position information and target position offset information of the abnormal log file, wherein the historical position is used to indicate the position offset information of the abnormal log file read before the start time of the preset time period, and the target position offset information is used to indicate the position offset information read at the end time of the preset time period;

[0050] Determining whether the historical read position information is consistent with the target position offset information;

[0051] If the historical read position information is inconsistent with the target position offset information, the exception information, the historical read position information and the target position offset information are written into a registration file in a preset storage space to obtain candidate exception log information.

[0052] Optionally, in a fifth implementation of the second aspect of the present invention, the device for processing abnormal log information further includes:

[0053] A generating and sending module is used to receive a retrieval instruction sent by a preset visual user interface, query and generate a report on the target abnormal log information based on the retrieval instruction, obtain a statistical analysis report, and send the statistical analysis report to the visual user interface.

[0054] Optionally, in a sixth implementation manner of the second aspect of the present invention, the generating and sending module is specifically used to:

[0055] Receiving a search instruction sent by a preset visual user interface, and based on the search instruction, searching and filtering the target abnormal log information by using a preset routing algorithm and a genetic algorithm to obtain target search information;

[0056] Perform statistical analysis on the target search information to obtain statistical analysis information, generate a statistical analysis report based on the statistical analysis information, and send the statistical analysis report to the visual user interface.

[0057] The third aspect of the present invention provides a device for processing abnormal log information, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the device for processing abnormal log information executes the above-mentioned method for processing abnormal log information.

[0058] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned method for processing abnormal log information.

[0059] In the technical solution provided by the present invention, an exception interception instruction is sent to a preset aspect-oriented programming system, so that the aspect-oriented programming system performs exception detection, reading and file writing on the system application log information of each system application according to the exception interception instruction, and obtains an exception log file; a monitoring and collection instruction is sent to a preset log file collection system, so that the log file collection system reads and stores the exception information in the exception log file according to the monitoring and collection instruction and according to a preset time period, and obtains candidate exception log information; a monitoring and extraction instruction is sent to a preset log management system based on a full-text search engine, so that the log management system based on the full-text search engine performs real-time monitoring and extraction on the candidate exception log information in the log file collection system according to the monitoring and extraction instruction, and obtains the exception log information to be processed; the exception log information to be processed is filtered, indexed and fragmented, and the target exception log information is obtained. In the embodiment of the present invention, by using the aspect programming system, the system application log information of each system application is subjected to exception detection, reading and file writing, thereby solving the problem of long application startup time and low efficiency in obtaining exception log information, and improving the accuracy and efficiency of exception log file generation. By using the log file collection system, the exception information in the exception log file is read and stored according to a preset time period, and the exception log information to be processed is filtered, indexed and fragmented, which facilitates subsequent query and positioning, improves the performance of log collection and the decoupling of applications, simplifies and facilitates the collection, analysis and retrieval of distributed exception log information, and thus improves the processing efficiency of distributed exception log information. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A schematic diagram of an embodiment of a method for processing abnormal log information in an embodiment of the present invention;

[0061] Figure 2 A schematic diagram of another embodiment of a method for processing abnormal log information in an embodiment of the present invention;

[0062] Figure 3 A schematic diagram of an embodiment of a device for processing abnormal log information in an embodiment of the present invention;

[0063] Figure 4 A schematic diagram of another embodiment of a device for processing abnormal log information in an embodiment of the present invention;

[0064] Figure 5 The figure is a schematic diagram of an embodiment of a device for processing abnormal log information in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The embodiments of the present invention provide a method, device, equipment and storage medium for processing abnormal log information, which are used to improve the processing efficiency of distributed abnormal log information.

[0066] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0067] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of a method for processing abnormal log information in an embodiment of the present invention includes:

[0068] 101. Send an exception interception instruction to a preset aspect-oriented programming system, so that the aspect-oriented programming system performs exception detection, reading, and file writing on system application log information of each system application according to the exception interception instruction to obtain an exception log file.

[0069] It is understandable that the execution subject of the present invention may be a device for processing abnormal log information, or may be a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0070] Among them, the preset aspect-oriented programming framework is the aspect-oriented programming framework (AOP) of Java, and can also be the aspect-oriented programming framework. Spring AOP. The server pre-builds and configures the aspect-oriented programming system connected to each system application, sends an exception interception instruction to the aspect-oriented programming system to start the aspect-oriented programming system, and monitors the system application log information generated by each system application according to the exception interception instruction, reads the system application log information, and performs data cleaning and data conversion (converting to the data format corresponding to the aspect-oriented programming framework AOP, etc.) on the system application log information after data cleaning and data conversion, intercepts the exception information in the system application log information after data cleaning and data conversion, and writes the intercepted exception information into a pre-generated separate error log file, thereby obtaining an exception log file, solving the problem of long startup time and low efficiency of the application that intercepts the exception log information, and improving the accuracy and efficiency of the generation of the exception log file.

[0071] 102. Send a monitoring and collection instruction to a preset log file collection system, so that the log file collection system reads and stores the abnormal information in the abnormal log file according to the monitoring and collection instruction and a preset time period to obtain candidate abnormal log information.

[0072] For example, the preset log file collection system can be Filebeat, and the server sends a monitoring and collection instruction to the log file collection system Filebeat, so that the log file collection system monitors the generation time of the exception log file in the aspect-oriented programming system according to the monitoring and collection instruction and the preset time period, and determines whether the generation time is later than the starting time of the preset time period. If so, read the exception information written in the exception log file later than the starting time of the preset time period to obtain the collected exception log information; if not, continue to monitor the generation time of the exception log file in the aspect-oriented programming system until the collected exception log information written in the exception log file later than the starting time of the preset time period is obtained. After obtaining the collected exception log information, the log file collection system obtains the exception type of the collected exception log information, marks the exception log information with label information according to the exception type, obtains the marked exception log information, stores the marked exception log information in the storage space corresponding to the exception type, or writes the marked exception log information into the file corresponding to the exception type, thereby obtaining candidate exception log information.

[0073] 103. Send a monitoring and extraction instruction to a preset log management system based on a full-text search engine, so that the log management system based on a full-text search engine performs real-time monitoring and extraction of candidate abnormal log information in the log file collection system according to the monitoring and extraction instruction to obtain abnormal log information to be processed.

[0074] The server pre-builds the log collection system Logstash and the distributed full-text retrieval system Elasticsearch, obtains the config configuration file and the ElasticSearch cluster service address information in the log collection system Logstash, and configures the ElasticSearch cluster service address in the config configuration file to connect and integrate the log collection system Logstash and the distributed full-text retrieval system Elasticsearch, thereby obtaining a log management system based on the full-text search engine.

[0075] The server sends a monitoring and extraction instruction to the log management system based on the full-text search engine, so that the log management system based on the full-text search engine monitors the update of the candidate abnormal log information in the log file collection system in real time according to the monitoring and extraction instruction. When the log collection system Logstash in the log management system based on the full-text search engine detects that the log file collection system updates the candidate abnormal log information, it extracts the candidate abnormal log information and sends the candidate abnormal log information to the distributed full-text retrieval system Elasticsearch, thereby obtaining the abnormal log information to be processed.

[0076] 104. Filter, create indexes, and perform sharding on the exception log information to be processed to obtain target exception log information.

[0077] After the server obtains the exception log information to be processed through the log management system based on the full-text search engine, the server filters the exception log information to be processed according to the preset filtering rules through the distributed full-text retrieval system Elasticsearch to obtain the filtered information. The preset filtering rules include the identification, encryption and conversion of sensitive fields, empty fields, and preset unnecessary packages and logs. The preset unnecessary packages and logs include: things that can be placed in one log are output in multiple logs, a pile of useless stacks printed for exceptions that are expected to occur and can be handled normally, and "temporary" logs added for the convenience of development and debugging. The filtered information is indexed and created through the distributed full-text retrieval system Elasticsearch to obtain index data, and multiple sub-tables with associated and corresponding relationships are generated according to the index data to realize sharding processing of the index data, thereby obtaining the target exception log information.

[0078] In the embodiment of the present invention, by using the aspect programming system, the system application log information of each system application is subjected to exception detection, reading and file writing, thereby solving the problem of long application startup time and low efficiency in obtaining exception log information, and improving the accuracy and efficiency of exception log file generation. By using the log file collection system, the exception information in the exception log file is read and stored according to a preset time period, and the exception log information to be processed is filtered, indexed and fragmented, which facilitates subsequent query and positioning, improves the performance of log collection and the decoupling of applications, simplifies and facilitates the collection, analysis and retrieval of distributed exception log information, and thus improves the processing efficiency of distributed exception log information.

[0079] See also Figure 2 Another embodiment of the method for processing abnormal log information in the embodiment of the present invention includes:

[0080] 201. Send an exception interception instruction to a preset aspect-oriented programming system, so that the aspect-oriented programming system performs exception detection, reading, and file writing on system application log information of each system application according to the exception interception instruction to obtain an exception log file.

[0081] Specifically, the server sends an exception interception instruction to the preset aspect-oriented programming system, so that the aspect-oriented programming system calls the pre-configured interceptor chain according to the exception interception instruction, performs exception detection and exception information reading on the system application log information of each system application, and obtains initial exception log information; classifies the initial exception log information by type to obtain classified log information, and obtains the target error log file corresponding to the classification type of the classified log information; writes the initial exception log information into the target error log file according to the preset writing period to obtain the exception log file.

[0082] The server sends an exception interception instruction to a preset aspect-oriented programming system, so that the aspect-oriented programming system receives the target object sent by the event sender for processing based on the system application log information according to the exception interception instruction, triggers the call of the pre-configured interceptor chain based on the event object, performs exception detection and exception information reading on the system application log information of each system application according to the preset execution order, obtains the initial exception log information, and classifies the initial exception log information according to the exception type through a preset classifier or classification network to obtain the classified log information, the classification network includes but is not limited to the residual network ResNext, the convolutional neural network Inception V4 and the squeeze-and-excitation network (SENet), etc., retrieves the preset database according to the classification type (i.e., the exception type) of the classified log information, matches the corresponding target error log file from the preset database, and writes the initial exception log information into the target error log file according to the preset writing period through the aspect-oriented programming framework to obtain the exception log file, which is conducive to the classified management of the initial exception log information and improves the efficiency of the classified management of the initial exception log information.

[0083] 202. Send a monitoring and collection instruction to a preset log file collection system, so that the log file collection system reads and stores the abnormal information in the abnormal log file according to the monitoring and collection instruction and a preset time period to obtain candidate abnormal log information.

[0084] Specifically, the server sends a monitoring and collection instruction to a preset log file collection system, so that the log file collection system reads the abnormal information in the abnormal log file according to the monitoring and collection instruction and according to the preset time period; obtains the historical reading position information and target position offset information of the abnormal log file, the historical position is used to indicate the position offset information of the abnormal log file read before the start time of the preset time period, and the target position offset information is used to indicate the position offset information read at the end time of the preset time period; determines whether the historical reading position information is consistent with the target position offset information; if the historical reading position information is inconsistent with the target position offset information, writes the abnormal information, the historical reading position information and the target position offset information into the registration file of the preset storage space to obtain candidate abnormal log information.

[0085] For example, the server sends a monitoring and collection instruction to the preset log file collection system Filebeat, so that the log file collection system Filebeat calls the preset monitoring and recording component according to the monitoring and collection instruction, and monitors the update time and write time of the abnormal log file in the aspect-oriented programming system according to the preset time period in days, obtains the abnormal log file written and updated before the preset time period, collects the abnormal log file, and obtains the historical reading position information and target position offset information of the abnormal log file reading, and the historical position is used to indicate the position offset information of the abnormal log file read before the starting time of the preset time period The target position offset information is used to indicate the position offset information read at the end of the preset time period to prevent repeated reading of the exception log information, and call the preset collection tool harvester to read the exception information written in the exception log file before the preset time period, and determine whether the historical read position information is consistent with the target position offset information. If so, the historical read position information and the target position offset information are written into the registration file of the preset storage space to obtain the stored exception log information. If not, the exception information, the historical read position information and the target position offset information are written into the registration file of the preset storage space to obtain the candidate exception log information.

[0086] 203. Send a monitoring and extracting instruction to a preset log management system based on a full-text search engine, so that the log management system based on a full-text search engine performs real-time monitoring and extraction of candidate abnormal log information in the log file collection system according to the monitoring and extracting instruction to obtain abnormal log information to be processed.

[0087] Specifically, the server sends a monitoring and extraction instruction to a preset log management system based on a full-text search engine, so that the log management system based on the full-text search engine monitors and analyzes the increment, reading time and storage time of the candidate abnormal log information in the log file collection system based on the monitoring and extraction instruction and the preset monitoring mechanism, and obtains the abnormal log information that meets the preset monitoring conditions; from the log file collection system, the abnormal log information that meets the preset monitoring conditions is extracted into the full-text search engine corresponding to the pre-configured cluster service address to obtain the abnormal log information to be processed.

[0088] For example, the increment includes the size of the file or space increased after the candidate abnormal log information is stored. The server sends a monitoring and extraction instruction to the log management system based on the full-text search engine, so that the log management system based on the full-text search engine monitors the reading time, storage time and size of the read content after the log file collection system reads the abnormal information in the abnormal log file, as well as the size of the file or space increased (i.e., the increment) after the candidate abnormal log information is stored, based on the monitoring and extraction instruction and according to the preset monitoring mechanism, and analyzes the reading time, storage time and size of the read content, as well as the increment according to preset monitoring conditions to obtain abnormal log information that meets the monitoring conditions. The preset monitoring conditions include that the reading time and storage time are within a preset time period, the size of the read content is larger than the size before the monitoring time, and an increment is generated. The log management system based on the full-text search engine extracts the abnormal log information that meets the preset monitoring conditions from the log file collection system, and stores the abnormal log information that meets the preset monitoring conditions in the full-text search engine corresponding to the pre-configured cluster service address, thereby obtaining the abnormal log information to be processed.

[0089] 204. Filter, create indexes, and perform sharding on the exception log information to be processed to obtain target exception log information.

[0090] Specifically, the server uses a full-text search engine and a preset log key information decision tree to perform information matching, information extraction and keyword extraction on the exception log information to be processed, and obtains filtered exception log information and target keywords corresponding to the filtered exception log information; creates a clustered index based on the filtered exception log information and the target keywords, and performs sharding on the data in the clustered index to obtain sharded index data; stores the sharded index data in a preset storage space according to a preset storage strategy to obtain the target exception log information.

[0091] For example, after the server obtains the to-be-processed exception log information through a log management system based on a full-text search engine, the server matches and extracts information from the to-be-processed exception log information through the full-text search engine and a preset log key information decision tree to obtain the extracted exception log information, encrypts and converts the sensitive information in the extracted exception log information to obtain the filtered exception log information, extracts keywords from the filtered exception log information to obtain the corresponding target keywords, and the log key information decision tree includes but is not limited to sensitive information, log field information related to business information, and log information required by the exception log analysis rule, etc. A clustered index of the target keywords and the filtered exception log information is established through the full-text search engine, and the data in the clustered index is sharded to obtain sharded index data, thereby achieving the effect of both ensuring retrieval accuracy and improving retrieval efficiency, wherein the preset storage strategy includes storing the sharded index data in the corresponding preset storage space (with a relatively small resource proportion) according to the resource proportion of the preset storage space, and / or storing the sharded index data in the preset storage space corresponding to the preset theme according to the theme of the cluster, thereby improving the security and storage availability of the target exception log information.

[0092] 205. Receive a search instruction sent by a preset visual user interface, query and generate a report on target abnormal log information based on the search instruction, obtain a statistical analysis report, and send the statistical analysis report to the visual user interface.

[0093] For example, a user inputs abnormal log retrieval information in a preset visual user interface, which can be built by the log analysis platform Kibana. The visual user interface generates a retrieval instruction based on the abnormal log retrieval information and sends the retrieval instruction to the server. When the server receives the retrieval instruction sent by the visual user interface, it parses the retrieval instruction to obtain parsing information, and generates a query language based on the parsing information. Through the query language, the target abnormal log information is queried to obtain query information, and the target retrieval information is statistically analyzed and a report is generated to obtain a statistical analysis report, which is sent to the visual user interface for rendering to the visual user interface. By querying the target abnormal log information and generating a report to obtain a statistical analysis report, and sending the statistical analysis report to the visual user interface, the automatic retrieval, automatic analysis and display of distributed abnormal log information is realized, and the accuracy and efficiency of generating statistical analysis reports are improved, thereby improving the processing efficiency of distributed abnormal log information and increasing the automation of processing distributed abnormal log information.

[0094] Specifically, the server receives a retrieval instruction sent by a preset visual user interface, and based on the retrieval instruction, retrieves and filters the target exception log information through a preset routing algorithm and genetic algorithm to obtain target retrieval information; performs statistical analysis on the target retrieval information to obtain statistical analysis information, and generates a statistical analysis report based on the statistical analysis information, and sends the statistical analysis report to the visual user interface.

[0095] For example, when the server receives a retrieval instruction sent by a preset visual user interface, the preset routing algorithm is called based on the retrieval instruction, path selection analysis and log information reading are performed on the target exception log information, thereby obtaining initial retrieval information, and initial population generation, fitness calculation, and iterative crossover and mutation processing are performed on the initial retrieval information in sequence through a preset genetic algorithm, thereby obtaining target retrieval information, and the preset statistical analysis tool is called to perform statistical analysis on the target retrieval tool according to business needs, and a statistical analysis report is generated, which is sent to the visual user interface and rendered on the visual user interface.

[0096] In the embodiments of the present invention, not only the problems of long application startup time and low efficiency in obtaining exception log information are solved, the accuracy and efficiency of exception log file generation are improved, subsequent query and positioning are facilitated, the performance of log collection and the decoupling of applications are improved, and distributed operations such as collection, analysis and retrieval of exception log information are simplified and facilitated, thereby improving the processing efficiency of distributed exception log information. Statistical analysis reports are obtained by querying and generating reports on target exception log information, and the statistical analysis reports are sent to a visual user interface, thereby realizing automatic retrieval, automatic analysis and display of distributed exception log information, and improving the accuracy and efficiency of generating statistical analysis reports, thereby improving the processing efficiency of distributed exception log information and increasing the automation of processing distributed exception log information.

[0097] The above describes the method for processing abnormal log information in the embodiment of the present invention. The following describes the device for processing abnormal log information in the embodiment of the present invention. Figure 3 , an embodiment of a device for processing abnormal log information in an embodiment of the present invention includes:

[0098] The exception interception module 301 is used to send an exception interception instruction to a preset aspect-oriented programming system, so that the aspect-oriented programming system performs exception detection, reading and file writing on the system application log information of each system application according to the exception interception instruction to obtain an exception log file;

[0099] The monitoring and collection module 302 is used to send a monitoring and collection instruction to a preset log file collection system, so that the log file collection system reads and stores the abnormal information in the abnormal log file according to the monitoring and collection instruction and the preset time period to obtain candidate abnormal log information;

[0100] The monitoring and extraction module 303 is used to send a monitoring and extraction instruction to a preset log management system based on a full-text search engine, so that the log management system based on the full-text search engine performs real-time monitoring and extraction of candidate abnormal log information in the log file collection system according to the monitoring and extraction instruction to obtain abnormal log information to be processed;

[0101] The exception processing module 304 is used to filter, create indexes and perform sharding processing on the exception log information to be processed to obtain target exception log information.

[0102] The functional implementation of each module in the above-mentioned abnormal log information processing device corresponds to each step in the above-mentioned abnormal log information processing method embodiment, and its functions and implementation processes are no longer repeated here.

[0103] In the embodiment of the present invention, by using the aspect programming system, the system application log information of each system application is subjected to exception detection, reading and file writing, thereby solving the problem of long application startup time and low efficiency in obtaining exception log information, and improving the accuracy and efficiency of exception log file generation. By using the log file collection system, the exception information in the exception log file is read and stored according to a preset time period, and the exception log information to be processed is filtered, indexed and fragmented, which facilitates subsequent query and positioning, improves the performance of log collection and the decoupling of applications, simplifies and facilitates the collection, analysis and retrieval of distributed exception log information, and thus improves the processing efficiency of distributed exception log information.

[0104] See also Figure 4 Another embodiment of the device for processing abnormal log information in the embodiment of the present invention includes:

[0105] The exception interception module 301 is used to send an exception interception instruction to a preset aspect-oriented programming system, so that the aspect-oriented programming system performs exception detection, reading and file writing on the system application log information of each system application according to the exception interception instruction to obtain an exception log file;

[0106] The monitoring and collection module 302 is used to send a monitoring and collection instruction to a preset log file collection system, so that the log file collection system reads and stores the abnormal information in the abnormal log file according to the monitoring and collection instruction and the preset time period to obtain candidate abnormal log information;

[0107] The monitoring and extraction module 303 is used to send a monitoring and extraction instruction to a preset log management system based on a full-text search engine, so that the log management system based on the full-text search engine performs real-time monitoring and extraction of candidate abnormal log information in the log file collection system according to the monitoring and extraction instruction to obtain abnormal log information to be processed;

[0108] The exception processing module 304 is used to filter, create indexes and perform sharding on the exception log information to be processed to obtain target exception log information;

[0109] The generation and sending module 305 is used to receive a search instruction sent by a preset visual user interface, query and generate a report on the target abnormal log information based on the search instruction, obtain a statistical analysis report, and send the statistical analysis report to the visual user interface.

[0110] Optionally, the exception interception module 301 may also be specifically used for:

[0111] Sending an exception interception instruction to a preset aspect-oriented programming system, so that the aspect-oriented programming system calls a pre-configured interceptor chain according to the exception interception instruction, performs exception detection and exception information reading on the system application log information of each system application, and obtains initial exception log information;

[0112] Classify the initial abnormal log information into different types to obtain classified log information, and obtain the target error log file corresponding to the classification type of the classified log information;

[0113] According to the preset writing period, the initial abnormal log information is written into the target error log file to obtain the abnormal log file.

[0114] Optionally, the monitoring and extraction module 303 may also be specifically used for:

[0115] Sending a monitoring and extraction instruction to a preset log management system based on a full-text search engine, so that the log management system based on the full-text search engine monitors and analyzes the increment, reading time and storage time of candidate abnormal log information in the log file collection system based on the monitoring and extraction instruction and a preset monitoring mechanism, and obtains abnormal log information that meets the preset monitoring conditions;

[0116] From the log file collection system, the abnormal log information that meets the preset monitoring conditions is extracted to the full-text search engine corresponding to the pre-configured cluster service address to obtain the abnormal log information to be processed.

[0117] Optionally, the exception handling module 304 may also be specifically used for:

[0118] Through the full-text search engine and the preset log key information decision tree, the abnormal log information to be processed is matched, information extracted and keyword extracted to obtain the filtered abnormal log information and the target keyword corresponding to the filtered abnormal log information;

[0119] Create a clustered index based on the filtered abnormal log information and target keywords, and shard the data in the clustered index to obtain sharded index data;

[0120] According to the preset storage strategy, the shard index data is stored in the preset storage space to obtain the target exception log information.

[0121] Optionally, the monitoring and collection module 302 may also be specifically used for:

[0122] Sending a monitoring and collection instruction to a preset log file collection system, so that the log file collection system reads the abnormal information in the abnormal log file according to the monitoring and collection instruction and according to a preset time period;

[0123] Obtaining historical read position information and target position offset information of the abnormal log file, where the historical position is used to indicate the position offset information of the abnormal log file read before the start time of the preset time period, and the target position offset information is used to indicate the position offset information read at the end time of the preset time period;

[0124] Determine whether the historical read position information is consistent with the target position offset information;

[0125] If the historical read position information is inconsistent with the target position offset information, the exception information, the historical read position information and the target position offset information are written into a registration file in the preset storage space to obtain candidate exception log information.

[0126] Optionally, the generating and sending module 305 may also be specifically used for:

[0127] Receiving a search instruction sent by a preset visual user interface, based on the search instruction, searching and filtering the target abnormal log information through a preset routing algorithm and a genetic algorithm to obtain the target search information;

[0128] Perform statistical analysis on the target retrieval information to obtain statistical analysis information, generate a statistical analysis report based on the statistical analysis information, and send the statistical analysis report to the visual user interface.

[0129] The functional implementation of each module and each unit in the above-mentioned abnormal log information processing device corresponds to each step in the above-mentioned abnormal log information processing method embodiment, and its functions and implementation processes are no longer repeated here.

[0130] In the embodiments of the present invention, not only the problems of long application startup time and low efficiency in obtaining exception log information are solved, the accuracy and efficiency of exception log file generation are improved, subsequent query and positioning are facilitated, the performance of log collection and the decoupling of applications are improved, and distributed operations such as collection, analysis and retrieval of exception log information are simplified and facilitated, thereby improving the processing efficiency of distributed exception log information. Statistical analysis reports are obtained by querying and generating reports on target exception log information, and the statistical analysis reports are sent to a visual user interface, thereby realizing automatic retrieval, automatic analysis and display of distributed exception log information, and improving the accuracy and efficiency of generating statistical analysis reports, thereby improving the processing efficiency of distributed exception log information and increasing the automation of processing distributed exception log information.

[0131] above Figure 3 and Figure 4 The apparatus for processing abnormal log information in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The device for processing abnormal log information in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0132] Figure 5 1 is a schematic diagram of the structure of a device for processing abnormal log information provided by an embodiment of the present invention. The device 500 for processing abnormal log information may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 510 (for example, one or more processors) and a memory 520, and one or more storage media 530 (for example, one or more mass storage devices) storing application programs 533 or data 532. Among them, the memory 520 and the storage medium 530 may be short-term storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the device 500 for processing abnormal log information. Furthermore, the processor 510 may be configured to communicate with the storage medium 530 to execute a series of instruction operations in the storage medium 530 on the device 500 for processing abnormal log information.

[0133] The abnormal log information processing device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and / or one or more operating systems 531, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that Figure 5The structure of the abnormal log information processing device shown does not constitute a limitation on the abnormal log information processing device, and may include more or less components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0134] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the method for processing abnormal log information.

[0135] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0136] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0137] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0138] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0139] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing abnormal log information, characterized in that: The method for processing abnormal log information includes: Sending an exception interception instruction to a preset aspect-oriented programming system, so that the aspect-oriented programming system performs exception detection, reading and file writing on the system application log information of each system application according to the exception interception instruction to obtain an exception log file; The sending of the exception interception instruction to the preset aspect-oriented programming system so that the aspect-oriented programming system performs exception detection, reading and file writing on the system application log information of each system application according to the exception interception instruction to obtain the exception log file includes: Sending an exception interception instruction to a preset aspect-oriented programming system, so that the aspect-oriented programming system calls a pre-configured interceptor chain according to the exception interception instruction, performs exception detection and exception information reading on the system application log information of each system application, and obtains initial exception log information; Classify the initial abnormal log information by type to obtain classified log information, and obtain a target error log file corresponding to the classification type of the classified log information; Writing the initial exception log information into the target error log file according to a preset writing period to obtain an exception log file; Sending a monitoring and collection instruction to a preset log file collection system, so that the log file collection system reads and stores the abnormal information in the abnormal log file according to the monitoring and collection instruction and according to a preset time period to obtain candidate abnormal log information; Sending a monitoring and extraction instruction to a preset log management system based on a full-text search engine, so that the log management system based on the full-text search engine performs real-time monitoring and extraction of candidate abnormal log information in the log file collection system according to the monitoring and extraction instruction, and obtains the abnormal log information to be processed; The sending of the monitoring and extraction instruction to a preset log management system based on a full-text search engine, so that the log management system based on a full-text search engine performs real-time monitoring and extraction of candidate abnormal log information in the log file collection system according to the monitoring and extraction instruction, and obtains the abnormal log information to be processed, including: Sending a monitoring and extraction instruction to a preset log management system based on a full-text search engine, so that the log management system based on the full-text search engine monitors and analyzes the increment, reading time and storage time of the candidate abnormal log information in the log file collection system according to the monitoring and extraction instruction and based on a preset monitoring mechanism, and obtains abnormal log information that meets the preset monitoring conditions; Extracting the abnormal log information that meets the preset monitoring conditions from the log file collection system into a full-text search engine corresponding to a pre-configured cluster service address to obtain the abnormal log information to be processed; Filter, index and fragment the to-be-processed exception log information to obtain target exception log information; The filtering, index creation and sharding of the to-be-processed exception log information to obtain target exception log information includes: Through the full-text search engine and the preset log key information decision tree, the to-be-processed abnormal log information is subjected to information matching, information extraction and keyword extraction to obtain the filtered abnormal log information and the target keyword corresponding to the filtered abnormal log information; Creating a clustered index according to the filtering exception log information and the target keyword, and performing sharding processing on the data in the clustered index to obtain sharded index data; According to a preset storage strategy, the shard index data is stored in a preset storage space to obtain target abnormal log information.

2. The method for processing abnormal log information according to claim 1, characterized in that: The sending of the monitoring and collection instruction to the preset log file collection system, so that the log file collection system reads and stores the abnormal information in the abnormal log file according to the monitoring and collection instruction and according to the preset time period, and obtains the candidate abnormal log information, including: Sending a monitoring and collecting instruction to a preset log file collection system, so that the log file collection system reads the abnormal information in the abnormal log file according to the monitoring and collecting instruction and according to a preset time period; Acquire historical read position information and target position offset information of the abnormal log file, wherein the historical read position is used to indicate the position offset information of the abnormal log file read before the start time of the preset time period, and the target position offset information is used to indicate the position offset information read at the end time of the preset time period; Determining whether the historical read position information is consistent with the target position offset information; If the historical read position information is inconsistent with the target position offset information, the exception information, the historical read position information and the target position offset information are written into a registration file in a preset storage space to obtain candidate exception log information.

3. The method for processing abnormal log information according to any one of claims 1-2, characterized in that: After filtering, indexing and sharding the exception log information to be processed to obtain target exception log information, the method further includes: A search instruction sent by a preset visual user interface is received, and based on the search instruction, the target abnormal log information is queried and a report is generated to obtain a statistical analysis report, and the statistical analysis report is sent to the visual user interface.

4. The method for processing abnormal log information according to claim 3, characterized in that: The receiving of a retrieval instruction sent by a preset visual user interface, querying and generating a report on the target abnormal log information based on the retrieval instruction, obtaining a statistical analysis report, and sending the statistical analysis report to the visual user interface includes: Receiving a search instruction sent by a preset visual user interface, and based on the search instruction, searching and filtering the target abnormal log information by using a preset routing algorithm and a genetic algorithm to obtain target search information; Perform statistical analysis on the target search information to obtain statistical analysis information, generate a statistical analysis report based on the statistical analysis information, and send the statistical analysis report to the visual user interface.

5. A device for processing abnormal log information, characterized in that: The abnormal log information processing device includes: An exception interception module is used to send an exception interception instruction to a preset aspect-oriented programming system, so that the aspect-oriented programming system performs exception detection, reading and file writing on the system application log information of each system application according to the exception interception instruction to obtain an exception log file; wherein, the sending of the exception interception instruction to the preset aspect-oriented programming system, so that the aspect-oriented programming system performs exception detection, reading and file writing on the system application log information of each system application according to the exception interception instruction to obtain an exception log file, includes: sending the exception interception instruction to the preset aspect-oriented programming system, so that the aspect-oriented programming system calls a pre-configured interceptor chain according to the exception interception instruction, performs exception detection and exception information reading on the system application log information of each system application, and obtains initial exception log information; classifies the initial exception log information into a type to obtain classified log information, and obtains a target error log file corresponding to the classification type of the classified log information; writes the initial exception log information into the target error log file according to a preset writing period to obtain an exception log file; A monitoring and collection module, used for sending a monitoring and collection instruction to a preset log file collection system, so that the log file collection system reads and stores the abnormal information in the abnormal log file according to the monitoring and collection instruction and according to a preset time period to obtain candidate abnormal log information; A monitoring and extraction module, used for sending monitoring and extraction instructions to a preset log management system based on a full-text search engine, so that the log management system based on the full-text search engine performs real-time monitoring and extraction of candidate abnormal log information in the log file collection system according to the monitoring and extraction instructions, and obtains abnormal log information to be processed; wherein, the sending of monitoring and extraction instructions to a preset log management system based on a full-text search engine, so that the log management system based on the full-text search engine performs real-time monitoring and extraction of candidate abnormal log information in the log file collection system according to the monitoring and extraction instructions, and obtains abnormal log information to be processed, includes: sending monitoring and extraction instructions to a preset log management system based on a full-text search engine, so that the log management system based on the full-text search engine performs monitoring and analysis of increment, reading time and storage time of candidate abnormal log information in the log file collection system according to the monitoring and extraction instructions and based on a preset monitoring mechanism, and obtains abnormal log information that meets preset monitoring conditions; extracting the abnormal log information that meets the preset monitoring conditions from the log file collection system into a full-text search engine corresponding to a pre-configured cluster service address, and obtains abnormal log information to be processed; The exception handling module is used to filter, create indexes and perform sharding on the exception log information to be processed to obtain target exception log information; wherein, the filtering, indexing and sharding on the exception log information to be processed to obtain target exception log information includes: performing information matching, information extraction and keyword extraction on the exception log information to be processed through the full-text search engine and a preset log key information decision tree to obtain filtered exception log information and target keywords corresponding to the filtered exception log information; creating a clustered index based on the filtered exception log information and the target keywords, and sharding the data in the clustered index to obtain sharded index data; storing the sharded index data in a preset storage space according to a preset storage strategy to obtain target exception log information.

6. A device for processing abnormal log information, characterized in that: The abnormal log information processing device includes: a memory and at least one processor, wherein the memory stores instructions; The at least one processor calls the instruction in the memory so that the abnormal log information processing device executes the abnormal log information processing method according to any one of claims 1 to 4.

7. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instruction is executed by the processor, the method for processing abnormal log information as described in any one of claims 1 to 4 is implemented.