Abnormal log collection method, device, equipment, medium and product

Abstract

The present disclosure provides an exception log collection method, which can be applied to the field of system monitoring. The exception log collection method includes: generating a configuration file, the generating configuration file includes configuring the producer of the message, the producer of the message includes a log collector and a log outputter; collecting and outputting the operation log of the system according to the configuration file, filtering out the exception log and writing it into the message middleware; pulling the exception log in the message middleware for the message consumer to monitor whether the preset condition is triggered, wherein the preset condition includes that the number of exception logs output corresponding to the same abnormal event reaches a preset threshold; when the preset condition is triggered, the exception log is filtered to reduce the number of exception logs output corresponding to the same abnormal event, and the filtered exception log is written into the storage medium. The present disclosure also provides an exception log collection device, equipment, storage medium and program product.

Description

Technical Field

[0001] The present disclosure relates to the field of data transmission and processing, specifically to the field of system monitoring, and more specifically to an abnormal log collection method, device, equipment, medium and program product. Background Art

[0002] It is very necessary to monitor the abnormal logs during system operation, and the corresponding operation and maintenance or developers must be informed in time when abnormal events occur. The popular open source ELK+ZABBIX framework can collect and analyze abnormal logs, visualize and alarm abnormalities, but the framework involves the deployment of multiple component services, which is an additional deployment and maintenance cost for small and medium-sized single or distributed systems.

[0003] Therefore, how to collect abnormal logs in small and medium-sized systems is an urgent problem to be solved by those skilled in the art. Summary of the invention

[0004] In view of the above problems, the present disclosure provides an abnormal log collection method, device, equipment, medium and program product.

[0005] According to a first aspect of the present disclosure, a method for collecting exception logs is provided, including: generating a configuration file, wherein the generating of the configuration file includes configuring a message producer, wherein the message producer includes a log collector and a log outputter; collecting and outputting a system operation log according to the configuration file, filtering out the exception log and writing it into a message middleware; pulling the exception log in the message middleware for message consumers to monitor whether a preset condition is triggered, wherein the preset condition includes that the number of exception logs output corresponding to the same exception event reaches a preset threshold; when the preset condition is triggered, filtering the exception log to reduce the number of exception logs output corresponding to the same exception event, and writing the filtered exception log to a storage medium.

[0006] According to an embodiment of the present disclosure, the configuration of the message producer includes: configuring the log collector to clarify relevant information of the abnormal log output; configuring the log outputter to clarify the destination of the abnormal log output.

[0007] According to an embodiment of the present disclosure, the collection and output of the system's operation log according to the configuration file, filtering out the abnormal log and writing it into the message middleware, includes: collecting the operation log according to the log collector, filtering the abnormal log in the operation log based on the output level; and outputting it to the corresponding message middleware through the log outputter.

[0008] According to an embodiment of the present disclosure, pulling the exception log in the message middleware for message consumers to monitor whether a preset condition is triggered includes: extracting the classification identifier in each exception log, and classifying the exception log according to the classification identifier to determine the same exception event; judging whether the number of occurrences of the same exception event within a specified time reaches a preset threshold; if so, only retaining a preset number of the exception logs for the same exception event; wherein the preset number is less than the preset threshold.

[0009] According to an embodiment of the present disclosure, the classification processing of the exception log according to the classification identifier to determine the same exception event includes: converting the format of the same classification identifier into a unique Key value; using the exception data corresponding to the Key value as the Value value to establish a Key-Value relationship.

[0010] According to an embodiment of the present disclosure, the log collector includes: at least one of the log collector name, specific class, system identifier, exception occurrence timestamp, exception summary and exception detailed stack; the log outputter includes: at least one of the log exporter name, system identifier and destination port.

[0011] According to an embodiment of the present disclosure, the classification identifier includes: a system identifier, a specific class, and an exception summary.

[0012] The second aspect of the present disclosure provides an exception log collection device, including: a first module: used to generate a configuration file, the generation of the configuration file includes configuring a message producer, and the message producer includes a log collector and a log outputter; a second module: used to collect and output the system operation log according to the configuration file, filter out the exception log and write it into the message middleware; a third module: used to pull the exception log in the message middleware for the message consumer to monitor whether a preset condition is triggered, wherein the preset condition includes that the number of exception logs output corresponding to the same exception event reaches a preset threshold; a fourth module: used to filter the exception log when the preset condition is triggered to reduce the number of exception logs output corresponding to the same exception event, and write the filtered exception log to a storage medium.

[0013] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned abnormal log collection method.

[0014] A fourth aspect of the present disclosure further provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned abnormal log collection method.

[0015] A fifth aspect of the present disclosure further provides a computer program product, including a computer program, which implements the above-mentioned abnormal log collection method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0017] Figure 1 The application scenario diagram of the abnormal log collection method, device, equipment, medium and program product according to the embodiment of the present disclosure is schematically shown;

[0018] Figure 2 The flowchart of the abnormal log collection method according to the embodiment of the present disclosure is schematically shown;

[0019] Figure 3 A flowchart of abnormal log filtering according to an embodiment of the present disclosure is schematically shown;

[0020] Figure 4 A flowchart of abnormal log classification according to an embodiment of the present disclosure is schematically shown;

[0021] Figure 5 The schematic diagram shows the architecture of abnormal log collection for small and medium-sized systems according to an embodiment of the present disclosure;

[0022] Figure 6 A diagram schematically shows an abnormal log collection device according to an embodiment of the present disclosure; and

[0023] Figure 7 A block diagram of an electronic device suitable for implementing the abnormal log collection method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0024] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0025] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0026] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0027] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0028] An embodiment of the present disclosure provides an exception log collection method, which collects and outputs the system's operation logs according to a configuration file and filters the logs of the exception type to obtain the exception logs, wherein the configuration file includes configuring the message producer; the exception logs are written into a message middleware for the message consumer to pull and monitor whether a preset condition is triggered, wherein the preset condition includes that the number of exception logs output corresponding to the same exception event reaches a preset threshold; when the preset condition is triggered, the exception logs are filtered, and the filtered exception logs are written into a storage medium.

[0029] Figure 1 The application scenario diagram of abnormal log collection according to an embodiment of the present disclosure is schematically shown.

[0030] like Figure 1 As shown, the application scenario 100 according to this embodiment may include abnormal log collection. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0031] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0032] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0033] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0034] It should be noted that the abnormal log collection method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the abnormal log collection device provided in the embodiment of the present disclosure can generally be set in the server 105. The abnormal log collection method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the abnormal log collection device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0035] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0036] The following will be based on Figure 1 The scene described by Figure 2 to Figure 6 The abnormal log collection method of the disclosed embodiment is described in detail.

[0037] Figure 2 The flowchart of the abnormal log collection method according to the embodiment of the present disclosure is schematically shown.

[0038] like Figure 2 As shown, the abnormal log collection method of this embodiment includes operations S210 to S240, and the transaction processing method can be executed by the server 105.

[0039] In operation S210, a configuration file is generated, wherein generating the configuration file includes configuring a message producer, wherein the message producer includes a log collector and a log outputter.

[0040] In operation S220, the operation log of the system is collected and output according to the configuration file, and the abnormal log is screened out and written into the message middleware.

[0041] It should be noted that the configuration file includes configuration based on the log framework. There are many types of log frameworks, and the embodiments of the present disclosure are not specifically limited. Taking the field of java (an object-oriented programming language) as an example, commonly used log frameworks include common-logging, Log4j, Logback or jdk-logging. Among them, the Log4j framework is a log management tool provided by Apache for Java, which is used to track, debug and maintain programs. There are three major components in Log4j: 1. Log collector (logger): responsible for the class called by the client code to output log messages, which can output messages of different levels, such as error messages, warning messages, etc.; 2. Log output (Appender): responsible for the output of logs, which can output logs to files, output logs to consoles, and output logs to sockets. 3. Formatter (Layout): format the output messages, such as adding dates or levels to messages. Log messages in Log4j are divided into five levels: FATAL (major error), ERROR (error), WARN (warning), INFO (information) and DEBUG (debugging). By changing the level of the log message, we can change the output volume of the log to get the information we want.

[0042] It should also be noted that AppenderSkeleton inherits from the Appender class and implements the common functions of Appender, but does not implement some interfaces inherited from Appender, so it is still an abstract class and cannot be instantiated. All functions of AppenderSkeleton are thread-safe. Therefore, custom Appenders can be derived from AppenderSkeleton. In the Java environment, the access to the log framework is based on the pre-installed Monitor.jar, which encapsulates log collection including MonitorAppender under the Log4j framework and message middleware kafka or rabbitmq.

[0043] According to an embodiment of the present disclosure, the log framework is inherited and rewritten to obtain a configuration file.

[0044] According to an embodiment of the present disclosure, the configuration of the message producer includes: configuring the log collector, including collecting at least one of the following information: log collector name, specific class, system identifier, exception occurrence timestamp, exception summary, and exception detailed stack; configuring the log outputter, including adding the following information: at least one of the log outputter name, system identifier, and destination port.

[0045] For example, the configuration file includes a custom Appender. Here, we name the custom Appender MonitorAppender. MonitorAppender implements the Appender interface in disguise by inheriting the AppenderSkeleton under the Log4j framework, and rewrites the Appender to only retain logs whose specific class is ERROR, and filter the logs whose specific class is non-ERROR, so as to obtain logs of only ERROR type, that is, exception logs. The ERROR log includes complete information such as the stack. Of course, the exception log can also be defined as one or more combinations of other specific class logs according to actual needs. Specifically, for the log collector, modify the relevant configuration in the custom Appender, including at least one of the logger name, specific class, system identifier, exception occurrence timestamp, exception summary, and exception detailed stack; for the log outputter, add relevant configuration items in the custom Appender, including at least one of the outputter name, system identifier, and destination port.

[0046] For another example, the message middleware may use a message system that supports distributed messaging, such as kafka or rabbitmq, and transmit the data of the ERROR log to the message middleware kafka or rabbitmq.

[0047] In the embodiment of the present disclosure, when the service is started, when the program code segment in the software development kit under the set directory is triggered to be executed, the rewritten log framework begins to be executed and runs the abnormal log collection in the background to continuously produce abnormal logs. Zero intrusion to the original code is achieved and it is easy to reuse, avoiding the problem of damaging the integrity of the code in the service. At the same time, the use of message middleware has the advantages of decoupling, asynchronous processing, and traffic peak shaving, especially during high concurrency periods, it can also prevent business interruptions.

[0048] In operation S230, the abnormal log in the message middleware is pulled for the message consumer to monitor whether a preset condition is triggered, and the preset condition includes that the number of abnormal logs output corresponding to the same abnormal event reaches a preset threshold;

[0049] According to an embodiment of the present disclosure, after a consumer pulls an exception log, the exception log is classified. The classification includes clarifying which exception logs represent the same exception event but are output at different times, and recording the number of exception logs corresponding to the exception event.

[0050] In operation S240, when a preset condition is triggered, the exception log is filtered to reduce the number of exception logs output corresponding to the same exception event, and the filtered exception log is written into a storage medium.

[0051] According to an embodiment of the present disclosure, when the number of exception logs corresponding to the same exception event reaches a preset threshold, most of the exception logs issued for the same exception event are filtered out to prevent redundant exception logs from entering the database.

[0052] According to the embodiments of the present disclosure, the filtering rules for filtering the abnormal logs can be determined according to the specific situation. For example, if a certain abnormality is triggered under high concurrency, we can set an upper limit such as only storing the same abnormality 100 times in 1 minute (the preset threshold is configurable).

[0053] In the embodiments of the present disclosure, consumers monitor message queues and filter abnormal logs before storing them in a database, thereby preventing duplicate information from overwhelming the database. This can also avoid storing each message one by one in a database, resulting in the consumption speed being seriously unable to keep up with the message production speed, which causes other subsequent abnormal alarms to be blocked.

[0054] Figure 3 The flowchart of abnormal log filtering according to an embodiment of the present disclosure is schematically shown.

[0055] like Figure 3 As shown, the abnormal log filtering method of this embodiment includes operations S310 to S330, and the transaction processing method can be executed by the server 105.

[0056] In operation S310, a classification identifier is extracted from each abnormal log, and the abnormal log is classified according to the classification identifier to determine the same abnormal event;

[0057] According to an embodiment of the present disclosure, the classification identifier includes: a system identifier, a specific class, and an exception summary. The exception log is classified based on the classification identifier.

[0058] For example, there are often multiple host servers in a distributed system, and the system identifier is used to locate the exception log corresponding to the system of a specific host server. The system identifier has been configured in the configuration file, that is, the system identifier has been added when rewriting the Appender. After locating the specific system, the specific class is located, and then a more precise location is performed based on the exception summary. When the exception summary is located, it is possible to distinguish whether the received exception logs reflect the same exception event.

[0059] It should be noted that Java has the concept of class, which is a template used to describe the behavior and state of a class of objects. A Java class can contain class variables, member variables, and local variables. In the embodiments of the present disclosure, the location where the exception occurs in the exception log is located based on the system identifier, the specific class, and the exception summary, and the exception logs belonging to the same exception event are classified.

[0060] In operation S320, it is determined whether the number of occurrences of the same abnormal event within a specified time reaches a preset threshold;

[0061] According to the embodiments of the present disclosure, the number of the above-mentioned abnormal logs is not a cumulative number, but refers to the number output within a certain period of time. The certain period of time here can be adjusted by the operation and maintenance personnel, and can be one minute or three minutes, and is not limited here.

[0062] For example, when using memory cache Redis as a counter consumer to get exception logs from Kafka, the exception logs of the same exception event are first identified through a classification algorithm, and the number of occurrences of the exception logs of the same exception event in the current three minutes (configurable) is obtained from Redis (set to 1 for the first time). If it exceeds the preset threshold (for example, 100 times), the number exceeding the threshold is directly ignored and does not need to be written to storage media such as Mysql. If it does not exceed the threshold, 1 is added to the original value and the exception information is written to the storage medium corresponding to the Mysql database.

[0063] In operation S330, if yes, only a preset number of the abnormal logs are retained for the same abnormal event; wherein the preset number is much smaller than the preset threshold.

[0064] According to the embodiments of the present disclosure, the establishment of a preset threshold corresponds to the number of abnormal logs corresponding to the same abnormal event within a specified time, and the establishment of a preset number corresponds to the number of abnormal logs corresponding to the same abnormal event stored in a storage medium. Its purpose is only for alarm and display, so the number of corresponding abnormal logs stored in the storage medium for the same abnormal event should not be too large.

[0065] For example, when there is a high concurrency of exception logs, it can be stipulated that only one of 100 identical logs is stored within a period of time; or, it can be stipulated that only one of 300 exception logs corresponding to the same exception time is stored within a period of time.

[0066] Figure 4 The flowchart of abnormal log classification according to an embodiment of the present disclosure is schematically shown.

[0067] like Figure 4 As shown, the abnormal log classification method of this embodiment includes operations S410 to S420, and the transaction processing method can be executed by the server 105.

[0068] In operation S410, the same classification identifier is formatted into a unique Key value;

[0069] In operation S420, the abnormal data corresponding to the Key value is used as the Value value to establish a Key-Value relationship.

[0070] According to an embodiment of the present disclosure, the above-mentioned exception data includes an exception detailed stack.

[0071] For example, based on the MD5 algorithm, the above classification identifiers: system identifier, output class, and exception summary are transcoded and converted into a unique Key value, that is, a Key value corresponds to a unique exception event. In addition, a Key-Value relationship between the Key value and the exception detail stack is established to access data while ensuring that the data Key value is encrypted.

[0072] It should be noted that the principle of the MD5 algorithm is to generate a random number and combine it with the MD5 generated string. The randomness and uncertainty of the algorithm lead to an extremely high level of confidentiality.

[0073] Figure 5 The schematic diagram shows the architecture of abnormal log collection for small and medium-sized systems according to an embodiment of the present disclosure.

[0074] like Figure 5 As shown in the figure, the producer of messages in the distributed system, MonitorAppender, collects exception logs, where each MonitorAppender corresponds to a system. The collected exception logs are sent to the blocking queue of the message middleware kafka for consumers to pull, monitor and filter. Then, the filtered exception logs are stored in the storage medium mysql for message storage. Finally, the background exception analysis and alarm process regularly summarizes the alarm program and sends it to the external interface. For example, the exception analysis and alarm process can summarize all the exception classification data within 1 minute or configure a specified time according to the abnormal log name and exception summary information to send an alarm message once, and inform the corresponding system operation and maintenance or development person in charge through SMS, email or access to notification interfaces such as enterprise WeChat.

[0075] Based on the above abnormal log collection method, the present disclosure also provides an abnormal log collection device. Figure 6 The device is described in detail.

[0076] Figure 6 The structural block diagram of the abnormal log collection device according to an embodiment of the present disclosure is schematically shown.

[0077] like Figure 6 As shown, the abnormal log collection device 600 of this embodiment includes a first module 610 , a second module 620 , a third module 630 and a fourth module 640 .

[0078] The first module 610 is used to generate a configuration file, wherein the configuration file generation includes configuring a message producer, wherein the message producer includes a log collector and a log outputter;

[0079] In one embodiment, the module 610 may be used to perform the operation S210 described above, which will not be described in detail here.

[0080] The second module 620 is used to collect and output the system operation log according to the configuration file, filter out the abnormal log and write it into the message middleware.

[0081] In one embodiment, the second module 820 may be used to execute the operation S220 described above, which will not be described in detail herein.

[0082] The third module 630 is used to pull the exception log in the message middleware for the message consumer to monitor whether a preset condition is triggered, wherein the preset condition includes that the number of exception logs output corresponding to the same abnormal event reaches a preset threshold.

[0083] In one embodiment, the third module 630 may be used to execute the operation S230 described above, which will not be described in detail herein.

[0084] The fourth module 640 is used to filter the abnormal logs when a preset condition is triggered to reduce the number of abnormal logs output corresponding to the same abnormal event, and write the filtered abnormal logs into a storage medium.

[0085] According to an embodiment of the present disclosure, any multiple modules in the first module 610, the second module 620, the third module 630 and the fourth module 640 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the first module 610, the second module 620, the third module 630 and the fourth module 640 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the first module 610, the second module 620, the third module 630 and the fourth module 640 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be performed.

[0086] Figure 7 A block diagram of an electronic device suitable for implementing the abnormal log collection method according to an embodiment of the present disclosure is schematically shown.

[0087] like Figure 7As shown, the electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include an onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0088] In RAM 703, various programs and data required for the operation of electronic device 700 are stored. Processor 701, ROM 702 and RAM 703 are connected to each other via bus 704. Processor 701 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 702 and / or RAM 703. It should be noted that the program can also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0089] According to an embodiment of the present disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to the bus 704. The electronic device 700 may further include one or more of the following components connected to the I / O interface 705: an input portion 706 including a keyboard, a mouse, etc.; an output portion 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 708 including a hard disk, etc.; and a communication portion 709 including a network interface card such as a LAN card, a modem, etc. The communication portion 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage portion 708 as needed.

[0090] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0091] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 702 and / or RAM 703 described above and / or one or more memories other than ROM 702 and RAM 703.

[0092] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the item recommendation method provided by the embodiment of the present disclosure.

[0093] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 701. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0094] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 709, and / or installed from the removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0095] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0096] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0097] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0098] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0099] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A method for collecting abnormal logs, characterized in that: include: Generate a configuration file, wherein the generating of the configuration file includes configuring a message producer, wherein the message producer includes a log collector and a log outputter; Collect and output the system's operation logs according to the configuration file, filter out abnormal logs and write them into the message middleware; Pull the exception log in the message middleware for the message consumer to monitor whether the preset condition is triggered, wherein the preset condition includes that the number of exception logs output corresponding to the same abnormal event reaches a preset threshold; When a preset condition is triggered, the abnormal log is filtered to reduce the number of abnormal logs output corresponding to the same abnormal event, and the filtered abnormal log is written into a storage medium; The configuration of the message producer includes: configuring the log collector to clarify the relevant information of the abnormal log output; configuring the log outputter to clarify the destination of the abnormal log output; The collecting and outputting the operation log of the system according to the configuration file, filtering out the abnormal log and writing it into the message middleware, comprises: collecting the operation log according to the log collector, filtering out the abnormal log in the operation log based on the output level; and outputting to the corresponding message middleware through the log outputter; Pulling the exception log in the message middleware for the message consumer to monitor whether the preset condition is triggered includes: extracting the classification identifier in each exception log, classifying the exception log according to the classification identifier to determine the same exception event; judging whether the number of occurrences of the same exception event within a specified time reaches a preset threshold; if so, only retaining a preset number of the exception logs for the same exception event; wherein the preset number is less than the preset threshold; The classifying process of the abnormal log according to the classification identifier to determine the same abnormal event includes: converting the same classification identifier format into a unique Key value; taking the abnormal data corresponding to the Key value as the Value value, and establishing a Key-Value relationship.

2. The method according to claim 1, characterized in that The log collector includes: at least one of the log collector name, specific class, system identifier, exception occurrence timestamp, exception summary and exception detailed stack; the log outputter includes: at least one of the log outputter name, system identifier and destination port.

3. The method according to claim 1, characterized in that The classification identifier includes: system identifier, specific class and exception summary.

4. An abnormal log collection device, characterized in that: include: The first module is used to generate a configuration file, wherein the configuration file generation includes configuring a message producer, wherein the message producer includes a log collector and a log outputter; The second module is used to collect and output the system operation log according to the configuration file, filter out the abnormal log and write it into the message middleware; The third module is used to pull the exception log in the message middleware for the message consumer to monitor whether the preset condition is triggered, wherein the preset condition includes that the number of exception logs output corresponding to the same abnormal event reaches a preset threshold; The fourth module is used for filtering the abnormal logs when a preset condition is triggered to reduce the number of abnormal logs output corresponding to the same abnormal event, and writing the filtered abnormal logs into a storage medium; The configuration of the message producer includes: configuring the log collector to clarify the relevant information of the abnormal log output; configuring the log outputter to clarify the destination of the abnormal log output; The collecting and outputting the operation log of the system according to the configuration file, filtering out the abnormal log and writing it into the message middleware, comprises: collecting the operation log according to the log collector, filtering the abnormal log in the operation log based on the output level; and outputting to the corresponding message middleware through the log outputter; Pulling the exception log in the message middleware for the message consumer to monitor whether the preset condition is triggered includes: extracting the classification identifier in each exception log, classifying the exception log according to the classification identifier to determine the same exception event; judging whether the number of occurrences of the same exception event within a specified time reaches a preset threshold; if so, only retaining a preset number of the exception logs for the same exception event; wherein the preset number is less than the preset threshold; The classifying process of the abnormal log according to the classification identifier to determine the same abnormal event includes: converting the same classification identifier format into a unique Key value; taking the abnormal data corresponding to the Key value as the Value value, and establishing a Key-Value relationship.

5. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to execute the method according to any one of claims 1 to 3.

6. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 3.

7. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images