Intelligent log problem diagnosis method and device in application operation and maintenance and medium

Abstract

The invention discloses an intelligent log problem diagnosis method and device in application operation and maintenance and a medium. The method comprises the steps that log data are collected from an application system, a server and network equipment, and the formats of the log data comprise a text format, an XML format and a JSON format; cleaning, formatting and standardizing the collected log data to obtain effective log data; performing data analysis on the effective log data by adopting a pre-trained data analysis algorithm to obtain abnormal behaviors of the effective log data; and performing problem diagnosis on the abnormal behavior according to a preset knowledge base, and obtaining a solution of the abnormal behavior, the knowledge base including an application operation and maintenance problem, a symptom of the problem and a solution method.

Description

Technical Field

[0001] The present invention relates to the field of log analysis technology, and more specifically, to an intelligent log problem diagnosis method, device and medium in application operation and maintenance. Background Art

[0002] In today's complex application environment, the amount of log data generated by application systems is huge and complex. Traditional log analysis methods mainly rely on manual viewing and simple text search, which is inefficient, error-prone, and difficult to promptly discover potential problems from massive logs. With the continuous development of information technology, a system that can automatically and intelligently process log data and quickly diagnose problems is needed to improve the efficiency and quality of application operation and maintenance. Summary of the invention

[0003] In view of the deficiencies in the prior art, the present invention provides a method, device and medium for intelligent log problem diagnosis in application operation and maintenance.

[0004] According to one aspect of the present invention, there is provided an intelligent log problem diagnosis method in application operation and maintenance, comprising:

[0005] Collect log data from application systems, servers, and network devices, where the log data is in text, XML, and JSON formats;

[0006] Clean, format and standardize the collected log data to obtain valid log data;

[0007] Use pre-trained data analysis algorithms to analyze valid log data and obtain abnormal behaviors of valid log data;

[0008] Diagnose abnormal behaviors and obtain solutions to abnormal behaviors based on a pre-set knowledge base, where the knowledge base includes application operation and maintenance issues, symptoms of the problems, and solutions.

[0009] Optionally, collect log data from application systems, servers, and network devices, including:

[0010] Based on the multi-threaded collection mechanism, log data is collected from application systems, servers and network devices through network protocols and file systems. The network protocols include TCP / IP and UDP, and the log data is stored in a pre-set buffer pool.

[0011] Optionally, the collected log data is cleaned, formatted, and standardized to obtain valid log data, including:

[0012] A rule-based cleaning algorithm removes noise data from log data to obtain noise-free log data, where the noise data includes duplicate data and invalid characters;

[0013] The formatting algorithm based on machine learning converts logs of different formats into a pre-set format to obtain log data in a standard format;

[0014] Standardize the standard format log data to obtain valid log data.

[0015] Optionally, a pre-trained data analysis algorithm is used to perform data analysis on the valid log data to obtain abnormal behaviors of the valid log data, including:

[0016] Use cluster analysis algorithm to perform cluster analysis on valid log data to obtain log clustering results;

[0017] Use association rule mining algorithm to perform association analysis on log clustering results to obtain the association relationship between different clusters;

[0018] Based on the association relationship, obtain the patterns and rules of valid log data;

[0019] Determine abnormal behavior of valid log data based on the patterns and regularities of valid logs.

[0020] Optionally, the abnormal behavior is diagnosed based on a pre-set knowledge base to obtain a solution to the abnormal behavior, including:

[0021] Based on Bayesian networks and fuzzy logic, abnormal behaviors are diagnosed according to a pre-set knowledge base to obtain solutions to abnormal behaviors.

[0022] According to another aspect of the present invention, there is provided an intelligent log problem diagnosis device in application operation and maintenance, comprising:

[0023] The collection module is used to collect log data from application systems, servers and network devices, wherein the log data is in text format, XML format and JSON format;

[0024] The processing module is used to clean, format and standardize the collected log data to obtain valid log data;

[0025] An analysis module is used to analyze valid log data using a pre-trained data analysis algorithm to obtain abnormal behavior of the valid log data;

[0026] The diagnosis module is used to diagnose abnormal behaviors and obtain solutions to abnormal behaviors based on a pre-set knowledge base, where the knowledge base includes application operation and maintenance problems, symptoms of the problems, and solutions.

[0027] Optionally, the collection module includes:

[0028] The collection submodule is used to collect log data from application systems, servers and network devices through network protocols and file systems based on a multi-threaded collection mechanism, where the network protocols include TCP / IP and UDP, and the log data is stored in a pre-set buffer pool.

[0029] Optionally, the processing module includes:

[0030] A removal submodule is used to remove noise data in the log data based on a rule-based cleaning algorithm to obtain noise-free log data, wherein the noise data includes repeated data and invalid characters;

[0031] The conversion submodule is used to convert logs in different formats into a preset format based on a machine learning formatting algorithm to obtain log data in a standard format;

[0032] The processing submodule is used to standardize the log data in the standard format and obtain valid log data.

[0033] Optionally, the analysis module includes:

[0034] The cluster analysis submodule is used to perform cluster analysis on valid log data using a cluster analysis algorithm to obtain log clustering results;

[0035] The association analysis submodule is used to perform association analysis on the log clustering results using the association rule mining algorithm to obtain the association relationship between different clusters;

[0036] The acquisition submodule is used to obtain the patterns and rules of valid log data based on the association relationship;

[0037] The determination submodule is used to determine the abnormal behavior of the valid log data according to the patterns and rules of the valid logs.

[0038] Optionally, the diagnostic module comprises:

[0039] The diagnosis submodule is used to diagnose the abnormal behavior based on the Bayesian network and fuzzy logic according to the pre-set knowledge base and obtain the solution to the abnormal behavior.

[0040] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the method described in any one of the above aspects of the present invention.

[0041] According to another aspect of the present invention, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in any one of the above aspects of the present invention.

[0042] Beneficial effects of the present invention:

[0043] 1. Improve the accuracy of problem diagnosis

[0044] Through intelligent log analysis and a rich knowledge base, problems in application operation and maintenance can be diagnosed more accurately. Compared with traditional manual diagnosis methods, the impact of human factors is reduced and the accuracy of diagnosis is improved. For example, in a complex distributed system, it can quickly and accurately locate which node or service has a problem.

[0045] 2. Improve operation and maintenance efficiency

[0046] The automated log analysis and problem diagnosis process greatly reduces the time and workload of manual intervention. Operation and maintenance personnel can quickly obtain solutions to problems, thereby improving the efficiency of operation and maintenance. For example, when processing a large amount of log data, it is no longer necessary to manually check each one, but the system can automatically analyze and give the results.

[0047] 3. Enhance system stability

[0048] Because problems in the application system can be discovered and solved in a timely manner, the impact of the problem on the system is reduced, thereby enhancing the stability of the system. For example, some potential performance problems can be discovered and solved before the problem worsens. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] A more complete understanding of exemplary embodiments of the present invention may be obtained by referring to the following drawings:

[0050] Figure 1 It is a flowchart of an intelligent log problem diagnosis method in application operation and maintenance provided by an exemplary embodiment of the present invention;

[0051] Figure 2 It is a structural diagram of an intelligent log problem diagnosis device in application operation and maintenance provided by an exemplary embodiment of the present invention;

[0052] Figure 3 This is a structure of an electronic device provided by an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0053] Below, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described here.

[0054] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

[0055] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present invention are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0056] It should also be understood that, in the embodiments of the present invention, “plurality” may refer to two or more than two, and “at least one” may refer to one, two or more than two.

[0057] It should also be understood that any component, data or structure mentioned in the embodiments of the present invention can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0058] In addition, the term "and / or" in the present invention is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects before and after are in an "or" relationship.

[0059] It should also be understood that the description of the various embodiments of the present invention focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0060] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0061] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0062] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0063] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0064] Embodiments of the present invention can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0065] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0066] Exemplary Methods

[0067] Figure 1 FIG. 1 is a flow chart of an intelligent log problem diagnosis method in application operation and maintenance provided by an exemplary embodiment of the present invention. This embodiment can be applied to electronic devices, such as Figure 1 As shown, the intelligent log problem diagnosis method 100 in application operation and maintenance includes the following steps:

[0068] Step 101, collecting log data from application systems, servers and network devices, wherein the log data is in a text format, an XML format and a JSON format;

[0069] Step 102, cleaning, formatting and standardizing the collected log data to obtain valid log data;

[0070] Step 103, using a pre-trained data analysis algorithm to perform data analysis on the valid log data to obtain abnormal behavior of the valid log data;

[0071] Step 104 , diagnose the abnormal behavior according to a preset knowledge base and obtain a solution to the abnormal behavior, wherein the knowledge base includes application operation and maintenance problems, symptoms of the problems, and solutions.

[0072] Specifically, the intelligent log analysis and problem diagnosis system of the present invention mainly includes a log collection module, a log preprocessing module, a log analysis module, a knowledge base module and a problem diagnosis module. The specific implementation process is as follows:

[0073] 1. Log collection module

[0074] Responsible for collecting log data from various application systems, servers, network devices and other sources. It supports multiple log formats, such as text format, XML format, JSON format, etc., and can be collected through network protocols (such as TCP / IP, UDP, etc.), file systems and other methods. For example, you can configure the network listening port to receive the log data sent by the application system in real time, or you can periodically read the log file from the specified file directory.

[0075] Furthermore, in order to ensure timely and comprehensive acquisition of log data, a collection mechanism based on multi-threading and buffer pool is designed. Multi-threading can collect data from multiple data sources at the same time to improve collection efficiency; the buffer pool can temporarily store the collected data to avoid data loss. For example, when collecting logs from multiple servers, each server corresponds to a collection thread, and the collected data is first placed in the buffer pool before subsequent processing.

[0076] 2. Log preprocessing module

[0077] Clean, format and standardize the collected log data. Remove noise data in the log, such as duplicate data and invalid characters, and convert logs of different formats into a unified format for subsequent analysis and processing. For example, for logs containing timestamps, the timestamps can be uniformly converted into a standard time format for easy analysis in chronological order.

[0078] Furthermore, the log preprocessing algorithm includes a rule-based cleaning algorithm and a machine learning-based formatting algorithm. The rule-based cleaning algorithm removes noise data according to preset rules, such as deleting duplicate log entries, filtering out specific invalid characters, etc. The machine learning-based formatting algorithm automatically identifies the format characteristics of the log by learning a large amount of sample log data, and performs corresponding formatting. For example, for a new log source, the machine learning algorithm can determine the meaning and format of each field in the log by analyzing the sample data, thereby realizing automatic formatting.

[0079] 3. Log analysis module

[0080] Advanced data analysis algorithms, such as clustering algorithms, association rule mining algorithms, and deep learning algorithms, are used to analyze preprocessed log data. It can mine patterns and rules in log data and discover abnormal behaviors. For example, clustering algorithms can be used to cluster similar log data together, making it easier to quickly identify abnormal log groups; association rule mining algorithms can be used to discover the correlation between different log events.

[0081] Furthermore, a method combining convolutional neural network (CNN) and recurrent neural network (RNN) in deep learning is used for log analysis. CNN can effectively extract local features in log data, while RNN can model the sequence features of log data. For example, when analyzing network logs, CNN can extract the features of each data packet, while RNN can discover the pattern of network attacks based on the sequential relationship of data packets.

[0082] 4. Knowledge Base Module

[0083] Build a rich knowledge base that contains common application problems, symptoms of problems, solutions, and other information. The knowledge base can be continuously enriched and improved through manual entry, learning from historical cases, etc. For example, when encountering a specific application error code, you can query the knowledge base for the corresponding problem and solution for the error code.

[0084] Furthermore, the knowledge base is constructed using knowledge graph technology, which represents various application problems, symptoms, and solutions in the form of graphs, facilitating quick query and reasoning. At the same time, through machine learning algorithms, knowledge is automatically learned from new problem cases and updated to the knowledge base. For example, when encountering a new problem, the system can automatically analyze the characteristics of the problem and merge it with the existing knowledge graph to enrich the content of the knowledge base.

[0085] 5. Problem diagnosis module

[0086] Combine the results of log analysis with the information in the knowledge base to diagnose problems in application operation and maintenance. It can quickly locate the root cause of the problem and provide corresponding solutions. For example, if log analysis finds that the response time of a service is abnormally prolonged, the problem diagnosis module can combine the relevant cases about the prolonged service response time in the knowledge base to determine whether it is network congestion, excessive server load or a problem with the application itself, and give corresponding solution suggestions.

[0087] Furthermore, the reasoning mechanism of problem diagnosis is based on Bayesian networks and fuzzy logic. Bayesian networks can reason based on known probability information and calculate the probability of different causes of problems; fuzzy logic can handle uncertainty and ambiguity. For example, when multiple factors may cause a problem, Bayesian networks can calculate the probability of each factor causing the problem based on the prior probability and conditional probability of these factors, and fuzzy logic can judge some vague symptoms, such as the vague description of "the response time is a bit long".

[0088] In one embodiment of the present invention, the specific implementation is as follows:

[0089] 1. System deployment

[0090] Deploy the intelligent log analysis and problem diagnosis system in the application operation and maintenance environment. The system can be installed on a single server or deployed on multiple servers in a distributed manner to improve the system's performance and processing capabilities. For example, in the application operation and maintenance center of a large enterprise, a cluster of multiple servers can be deployed to process massive amounts of log data.

[0091] 2. Data Collection

[0092] Configure the log collection module according to the actual situation of the application system. Determine the data source for collecting logs, such as application servers, database servers, network devices, etc., and set the corresponding collection methods and parameters. For example, for a Java-based application server, you can configure the collection of log files generated during its runtime, and implement regular collection by setting the file path and collection frequency.

[0093] 3. Log preprocessing

[0094] After collecting log data, the log preprocessing module processes it according to preset rules and algorithms. First, the noise data is removed, and then the format is converted and standardized. For example, for a data set containing a large number of duplicate logs, after removing the duplicate data, the timestamp format is unified into the ISO 8601 standard format.

[0095] 4. Log Analysis

[0096] The log analysis module selects the appropriate analysis algorithm based on the application requirements. For large-scale log data, clustering algorithms can be used for preliminary classification, and then association rule mining algorithms or deep learning algorithms can be used for in-depth analysis of different categories. For example, when analyzing network attack logs, clustering algorithms are used to classify logs into different attack types, and then deep learning algorithms are used to mine attack patterns and rules for each type.

[0097] 5. Knowledge base construction and update

[0098] In the early stage of system operation, the basic content of the knowledge base is constructed by manual entry. As the system runs, it continuously learns from new problem cases and updates the knowledge base. For example, when a new application error is encountered, the symptoms, causes, and solutions of the error are entered into the knowledge base, and the knowledge base is optimized through machine learning algorithms.

[0099] 6. Problem diagnosis

[0100] When an abnormality is found in the log, the problem diagnosis module combines the results of log analysis with the information in the knowledge base to diagnose the problem. Different reasoning mechanisms can be used depending on the complexity of the problem. For simple problems, solutions can be directly queried from the knowledge base; for complex problems, Bayesian networks and fuzzy logic need to be used for reasoning. For example, when diagnosing a problem of service performance degradation, if there are similar cases in the knowledge base, the solution in the case can be directly referred to; if not, the possible causes are analyzed through Bayesian networks and fuzzy logic, and corresponding solutions are given.

[0101] Beneficial effects of the present invention:

[0102] 1. Improve the accuracy of problem diagnosis

[0103] Through intelligent log analysis and a rich knowledge base, problems in application operation and maintenance can be diagnosed more accurately. Compared with traditional manual diagnosis methods, the impact of human factors is reduced and the accuracy of diagnosis is improved. For example, in a complex distributed system, it can quickly and accurately locate which node or service has a problem.

[0104] 2. Improve operation and maintenance efficiency

[0105] The automated log analysis and problem diagnosis process greatly reduces the time and workload of manual intervention. Operation and maintenance personnel can quickly obtain solutions to problems, thereby improving the efficiency of operation and maintenance. For example, when processing a large amount of log data, it is no longer necessary to manually check each one, but the system can automatically analyze and give the results.

[0106] 3. Enhance system stability

[0107] Because problems in the application system can be discovered and solved in a timely manner, the impact of the problem on the system is reduced, thereby enhancing the stability of the system. For example, some potential performance problems can be discovered and solved before the problem worsens.

[0108] Exemplary Devices

[0109] Figure 2FIG. 1 is a schematic diagram of the structure of an intelligent log problem diagnosis device in application operation and maintenance provided by an exemplary embodiment of the present invention. Figure 2 As shown, the device 200 includes:

[0110] The collection module 210 is used to collect log data from application systems, servers and network devices, wherein the log data is in a text format, an XML format and a JSON format;

[0111] The processing module 220 is used to clean, format and standardize the collected log data to obtain valid log data;

[0112] An analysis module 230 is used to perform data analysis on valid log data using a pre-trained data analysis algorithm to obtain abnormal behavior of the valid log data;

[0113] The diagnosis module 240 is used to diagnose the abnormal behavior according to a preset knowledge base and obtain a solution to the abnormal behavior, wherein the knowledge base includes application operation and maintenance problems, symptoms of the problems, and solutions.

[0114] Optionally, the collection module 210 includes:

[0115] The collection submodule is used to collect log data from application systems, servers and network devices through network protocols and file systems based on a multi-threaded collection mechanism, where the network protocols include TCP / IP and UDP, and the log data is stored in a pre-set buffer pool.

[0116] Optionally, the processing module 220 includes:

[0117] A removal submodule is used to remove noise data in the log data based on a rule-based cleaning algorithm to obtain noise-free log data, wherein the noise data includes repeated data and invalid characters;

[0118] The conversion submodule is used to convert logs in different formats into a preset format based on a machine learning formatting algorithm to obtain log data in a standard format;

[0119] The processing submodule is used to standardize the log data in the standard format and obtain valid log data.

[0120] Optionally, the analysis module 230 includes:

[0121] The cluster analysis submodule is used to perform cluster analysis on valid log data using a cluster analysis algorithm to obtain log clustering results;

[0122] The association analysis submodule is used to perform association analysis on the log clustering results using the association rule mining algorithm to obtain the association relationship between different clusters;

[0123] The acquisition submodule is used to obtain the patterns and rules of valid log data based on the association relationship;

[0124] The determination submodule is used to determine the abnormal behavior of the valid log data according to the patterns and rules of the valid logs.

[0125] Optionally, the diagnosis module 240 includes:

[0126] The diagnosis submodule is used to diagnose the abnormal behavior based on the Bayesian network and fuzzy logic according to the pre-set knowledge base and obtain the solution to the abnormal behavior.

[0127] Exemplary Electronic Devices

[0128] Figure 3 This is a structure of an electronic device provided by an exemplary embodiment of the present invention. Figure 3 As shown, the electronic device 30 includes one or more processors 31 and a memory 32 .

[0129] The processor 31 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0130] The memory 32 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 31 may run the program instructions to implement the methods of the software programs of the various embodiments of the present invention described above and / or other desired functions. In one example, the electronic device may also include: an input device 33 and an output device 34, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0131] In addition, the input device 33 may also include, for example, a keyboard, a mouse, and the like.

[0132] The output device 34 can output various information to the outside. The output device 34 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto.

[0133] Of course, to simplify, Figure 3Only some of the components related to the present invention in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device may further include any other appropriate components according to specific application conditions.

[0134] Exemplary computer program products and computer-readable storage media

[0135] In addition to the above-mentioned methods and devices, an embodiment of the present invention may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present invention described in the above-mentioned "Exemplary Method" section of this specification.

[0136] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the embodiments of the present invention, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0137] In addition, an embodiment of the present invention may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present invention described in the above “Exemplary Method” section of this specification.

[0138] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, system or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0139] The basic principle of the present invention is described above in conjunction with specific embodiments. However, it should be pointed out that the advantages, strengths, effects, etc. mentioned in the present invention are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present invention. In addition, the specific details disclosed above are only for the purpose of illustration and facilitation of understanding, rather than limitation, and the above details do not limit the present invention to being implemented by adopting the above specific details.

[0140] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0141] The block diagrams of the devices, systems, equipment, and systems involved in the present invention are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, systems, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The word "such as" used here refers to the phrase "such as but not limited to", and can be used interchangeably with it.

[0142] The method and system of the present invention may be implemented in many ways. For example, the method and system of the present invention may be implemented by software, hardware, firmware or any combination of software, hardware, firmware. The above order of steps for the method is only for illustration, and the steps of the method of the present invention are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present invention may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present invention. Thus, the present invention also covers a recording medium storing a program for executing the method according to the present invention.

[0143] It should also be noted that in the system, device and method of the present invention, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present invention. The above description of the disclosed aspects is provided to enable any technician in the field to make or use the present invention. Various modifications to these aspects are very obvious to those skilled in the art, and the general principles defined here can be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown here, but in accordance with the widest range consistent with the principles and novel features disclosed here.

[0144] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present invention to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. An intelligent log problem diagnosis method in application operation and maintenance, characterized in that: include: Collect log data from application systems, servers and network devices, wherein the log data is in a text format, an XML format and a JSON format; Cleaning, formatting and standardizing the collected log data to obtain valid log data; Performing data analysis on the valid log data using a pre-trained data analysis algorithm to obtain abnormal behavior of the valid log data; The abnormal behavior is diagnosed according to a preset knowledge base to obtain a solution to the abnormal behavior, wherein the knowledge base includes application operation and maintenance problems, symptoms of the problems, and solutions.

2. The method according to claim 1, characterized in that Collect log data from application systems, servers, and network devices, including: Based on a multi-threaded collection mechanism, the log data is collected from application systems, servers and network devices through network protocols and file systems, wherein the network protocols include TCP / IP and UDP, and the log data is stored in a pre-set buffer pool.

3. The method according to claim 1, characterized in that The collected log data is cleaned, formatted and standardized to obtain valid log data, including: Using a rule-based cleaning algorithm to remove noise data from the log data to obtain noise-free log data, wherein the noise data includes repeated data and invalid characters; The formatting algorithm based on machine learning converts logs of different formats into a pre-set format to obtain log data in a standard format; The standard format log data is standardized to obtain the valid log data.

4. The method according to claim 1, characterized in that: The valid log data is analyzed using a pre-trained data analysis algorithm to obtain abnormal behavior of the valid log data, including: Using a cluster analysis algorithm to perform cluster analysis on the valid log data to obtain log clustering results; Using an association rule mining algorithm to perform association analysis on the log clustering results to obtain association relationships between different clusters; According to the association relationship, the pattern and regularity of the valid log data are obtained; The abnormal behavior of the valid log data is determined according to the pattern and regularity of the valid log.

5. The method according to claim 1, characterized in that Diagnose the abnormal behavior according to a preset knowledge base and obtain a solution to the abnormal behavior, including: Based on Bayesian network and fuzzy logic, the abnormal behavior is diagnosed according to a preset knowledge base to obtain a solution to the abnormal behavior.

6. An intelligent log problem diagnosis device in application operation and maintenance, characterized in that: include: A collection module is used to collect log data from application systems, servers and network devices, wherein the log data is in a text format, an XML format and a JSON format; A processing module, used for cleaning, formatting and standardizing the collected log data to obtain valid log data; An analysis module, used to perform data analysis on the valid log data using a pre-trained data analysis algorithm to obtain abnormal behavior of the valid log data; The diagnosis module is used to diagnose the abnormal behavior according to a preset knowledge base and obtain a solution to the abnormal behavior, wherein the knowledge base includes application operation and maintenance problems, symptoms of the problems and solutions.

7. The device according to claim 6, characterized in that Collection modules, including: The collection submodule is used to collect the log data from the application system, server and network equipment through the network protocol and the file system based on the multi-threaded collection mechanism, wherein the network protocol includes TCP / IP and UDP, and the log data is stored in a pre-set buffer pool.

8. The device according to claim 6, characterized in that Processing modules, including: A removal submodule, used for removing noise data in the log data based on a rule-based cleaning algorithm to obtain noise-free log data, wherein the noise data includes repeated data and invalid characters; The conversion submodule is used to convert logs in different formats into a preset format based on a machine learning formatting algorithm to obtain log data in a standard format; The processing submodule is used to perform standardization processing on the standard format log data to obtain the valid log data.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the method according to any one of claims 1 to 5.

10. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1 to 5.

Images