Multi-dimensional log management and anomaly analysis method and device, medium and equipment

Abstract

The invention discloses a multi-dimensional log management and anomaly analysis method and device, a medium and equipment, and the method comprises the steps: collecting pipeline log data printed by a system container, and feeding back the collected pipeline log data to a log collector; after a preset condition is met, reading the pipeline log data from a log collector, and performing data cleaning processing on the pipeline log data to obtain log tag data and log text data; respectively collecting the log tag data and the log text data, and respectively storing the log tag data and the log text data in a log file database; and based on an analysis result of the log query instruction, querying the log file database to obtain log query data, obtaining abnormal aggregation data after secondary screening, and displaying the abnormal aggregation data. According to the abnormal clustering analysis method and device, the abnormal clustering analysis function based on multiple dimensions is achieved, the abnormal aggregated data obtained after secondary screening has good readability, and the efficiency and accuracy of abnormal clustering analysis are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet, and in particular to a multi-dimensional log management and abnormality analysis method, device, medium and equipment. BACKGROUND

[0002] Abnormal aggregation data refers to aggregating link data determined as abnormal by a user, so as to obtain historical information related to abnormal alarm according to the aggregated link data, and then analyze and optimize abnormal services.

[0003] At present, abnormal aggregation data can usually be obtained through log data analysis. The log data is a record file automatically generated when a system, application or device runs, and is stored and managed in a log management system. The log data includes timestamp, event description, error information and other key data.

[0004] Existing various log management systems only include basic log query functions, lack log query management functions from more dimensional perspectives such as business and customer perspectives, and often need logs to analyze and locate abnormal problems when troubleshooting. Abnormal aggregation data can only be analyzed by manually collecting data one by one, and there is a lack of more direct and effective troubleshooting abnormality analysis auxiliary tools. Therefore, there is an urgent need for a multi-dimensional log management and abnormality analysis method or device. SUMMARY

[0005] In view of the above problems, the present application is proposed to provide a multi-dimensional log management and abnormality analysis method, device, medium and equipment which can overcome the above problems or at least partially solve the above problems.

[0006] Other characteristics and advantages of the present application will become apparent from the following detailed description, or will be learned by practice of the present application.

[0007] According to a first aspect of an embodiment of the present application, a multi-dimensional log management and abnormality analysis method is provided, which comprises:

[0008] Collecting flow log data printed by a system container, and feeding the collected flow log data to a log collector;

[0009] After a preset condition is met, reading the flow log data from the log collector, performing data cleaning processing on the flow log data, and obtaining log label data and log text data;

[0010] Collecting the log label data and the log text data respectively, and storing them in a log file database respectively;

[0011] In response to the log query instruction, log query data is queried from the log file database based on a parsing result of the log query instruction, and abnormal aggregation data is obtained after secondary screening, and the abnormal aggregation data is displayed.

[0012] In some embodiments of the present application, the data cleaning processing on the flow log data comprises:

[0013] The flow log data is parsed to extract feature data of the flow log data;

[0014] After the extracted feature data is respectively processed by de-duplication, abnormal data deletion and format normalization, it is stored in a predetermined storage format to obtain log label data and log text data.

[0015] In some embodiments of the present application, the feature data of the flow log data comprises system label data, business label data, interface label data, abnormal error label data, system attribution information, business information, interface information and abnormal error information.

[0016] In some embodiments of the present application, in response to the log query instruction, log query data is queried from the log file database based on a parsing result of the log query instruction:

[0017] In response to the log query instruction, the log query instruction is parsed to determine log query parameters in the log query instruction to obtain a parsing result of the log query instruction;

[0018] Based on the parsing result of the log query instruction, corresponding log label data and log text data are called from the log file database to obtain log query data.

[0019] In some embodiments of the present application, the method comprises: in response to a customer-dimension log query instruction, parsing the log query instruction to determine log query parameters in the log query instruction, the log query parameters comprising a time parameter and a business number of a customer.

[0020] In some embodiments of the present application, the method comprises: in response to a business-dimension log query instruction, parsing the log query instruction to determine log query parameters in the log query instruction, the log query parameters comprising a time parameter and a business index.

[0021] In some embodiments of the present application, the method comprises: in response to an interface link log query instruction, parsing the log query instruction to determine log query parameters in the log query instruction, the log query parameters comprising a time parameter and a flow number.

[0022] According to a second aspect of the embodiments of the present application, a multi-dimensional log management and abnormality analysis device is provided, which comprises:

[0023] A log collection module is configured to collect flow log data printed by a system container and feed the collected flow log data to a log collector;

[0024] A log cleaning module is configured to read the flow log data from the log collector after a preset condition is met, perform data cleaning processing on the flow log data, and obtain log label data and log text data;

[0025] A log storage module is configured to collect the log label data and the log text data respectively and store them in a log file database;

[0026] A log query module is configured to, in response to a log query instruction, obtain log query data from the log file database based on an analysis result of the log query instruction, obtain abnormal aggregation data after secondary screening, and display the abnormal aggregation data.

[0027] According to a third aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores computer program instructions, the computer program instructions are loaded and executed by a processor to implement the operations performed by the method according to any one of the above.

[0028] According to a fourth aspect of the embodiments of the present application, an electronic device is provided, which comprises a processor and a memory, the memory stores computer program instructions executable by the processor, and the processor executes the computer program instructions to implement the instructions of the method according to any one of the above.

[0029] The technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0030] The multi-dimensional log management and abnormality analysis method, device, medium and equipment provided by the embodiments of the present application collect flow log data in a system container, perform data cleaning on the flow log data to obtain log label data and log text data, call corresponding log query data according to log query instructions of different dimensions, obtain abnormal aggregation data after secondary screening, and display the abnormal aggregation data, thereby realizing abnormal clustering analysis function based on multiple dimensions, the abnormal aggregation data after secondary screening has better readability, and the efficiency and accuracy of abnormal clustering analysis are improved.

[0031] The above description is only a summary of the technical solutions of the present application, in order to make the technical means of the present application more clearly understood, and can be implemented according to the content of the specification, and in order to make the above and other purposes, characteristics and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described below. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0033] Figure 1 Fig. 1 shows a flow diagram of a multi-dimensional log management and abnormality analysis method provided by an embodiment of the present application;

[0034] Figure 2 Fig. 2 shows a log query reference interface of a customer dimension;

[0035] Figure 3 Fig. 3 shows a log query reference interface of a business dimension;

[0036] Figure 4 Fig. 4 shows a log query reference interface of an interface link;

[0037] Figure 5 Fig. 5 shows a data display reference diagram of an operation interface of the system;

[0038] Figure 6 Fig. 6 shows a principle structure diagram of a multi-dimensional log management and abnormality analysis device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0039] Exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0040] In the drawings, various structural diagrams according to embodiments of the present disclosure are shown. These drawings are not drawn to scale, in which certain details are exaggerated for the purpose of clear expression, and certain details can be omitted. The shapes of various regions, layers shown in the drawings, and their relative size, positional relationship may deviate in actuality due to manufacturing tolerances or technical limitations, and a person skilled in the art can additionally design regions / layers with different shapes, sizes, relative positions according to actual needs.

[0041] In the context of the present disclosure, when one layer / element is referred to as being located "on" another layer / element, the layer / element can be directly on the other layer / element, or there can be an intervening layer / element therebetween. In addition, if one layer / element is located "on" another layer / element in one orientation, the layer / element can be located "under" the other layer / element when the orientation is reversed. In the context of the present disclosure, similar or identical components can be denoted by the same or similar reference numerals.

[0042] In order to better understand the above technical solutions, the above technical solutions will be described in detail below in conjunction with specific embodiments. It should be understood that the embodiments of the present disclosure and the specific features in the embodiments are detailed descriptions of the technical solutions of the present disclosure, and not limitations of the technical solutions of the present disclosure. In the case of no conflict, the technical features in the embodiments of the present disclosure and the embodiments can be combined with each other.

[0043] Figure 1 is a flow diagram of a multi-dimensional log management and abnormality analysis method provided by an embodiment of the present disclosure, as shown in Figure 1 The multi-dimensional log management and abnormality analysis method comprises the following steps:

[0044] S1, collecting flow log data printed by a system container, and feeding the collected flow log data into a log collector of a log management system;

[0045] In the embodiment of the present disclosure, the system container is a collection of applications installed on various operating systems or working systems, including code, dependencies, libraries, etc. The embodiment of the present disclosure connects the system container through the log management system to collect the record files automatically generated by the system container during operation, i.e. the flow log data, which contains timestamp, event description, error information and other key data.

[0046] After a certain amount of flow log data or flow log data within a fixed time is collected, the embodiment of the present disclosure feeds the collected flow log data into the log collector of the log management system; the amount may, for example, be 100 or other amounts, and the fixed time may, for example, be one day or one week. The actual application requirements can be determined by the user, and the embodiment of the present disclosure has no limitation on this. Of course, in other application scenarios, the embodiment of the present disclosure can also feed the real-time collected flow log data into the log collector of the log management system.

[0047] S2, after the preset condition is met, reading the flow log data from the log collector, performing data cleaning processing on the flow log data, and obtaining log label data and log text data;

[0048] In order to reduce the consumption of system operation resources, the embodiment of the present application reads the flow log data from the log collector after meeting the preset condition, for example, the preset condition can be that the log collector obtains new storage data or a predetermined time period is passed.

[0049] Since the collected flow log data is original log data and does not conform to visualization after user query, the readability of the data is poor, therefore, the embodiment of the present application performs data cleaning on the collected flow log data; the data cleaning of the flow log data in the embodiment of the present application includes: analyzing the flow log data to extract feature data of the flow log data; after performing deduplication processing, abnormal data deletion processing and format normalization processing on the extracted feature data, storing according to a predetermined storage format to obtain log label data and log text data.

[0050] In the embodiment of the present application, the deduplication processing, abnormal data deletion processing and format normalization processing corresponding to the data cleaning processing all adopt a conventional data cleaning manner, in other embodiments of the present application, the data cleaning processing can also adopt different technical means according to actual application requirements, as long as the flow log data is arranged into the log label data and the log text data conforming to the actual application requirements.

[0051] Among them, the feature data of the flow log data extracted by the embodiment of the present application includes: system label data, business label data, interface label data, abnormal error label data, system ownership information, business information, interface information and abnormal error information; the system label data, business label data, interface label data and abnormal error label data constitute the log label data, and the system ownership information, business information, interface information and abnormal error information constitute the log text data.

[0052] Specifically, the embodiment of the present application stores the feature data after data cleaning processing according to a predetermined format, for example: system ownership information+interface information+business information+abnormal error information, wherein the log label data and the log text data correspond one by one, for example, the system label data corresponds to the system ownership information, the business label data corresponds to the business information, the interface label data corresponds to the interface information, and the abnormal error label data corresponds to the abnormal error information.

[0053] In the embodiment of the present application, the system home information includes, but is not limited to, the calling level relationship or the order relationship of each log label data and the log text data in the system; the interface information includes, but is not limited to, the interface calling serial number or the interface link data corresponding to each log label data and the log text data; the business information includes, but is not limited to, the channel code, the business index code, etc.; and the abnormal error information includes, but is not limited to, the ERROR information and the WARN abnormal information, etc.

[0054] S3, collecting the log label data and the log text data respectively, and storing them in a log file database;

[0055] S4, in response to a log query instruction, obtaining log query data from the log file database based on the analysis result of the log query instruction, and obtaining abnormal aggregation data after secondary screening, and displaying the abnormal aggregation data.

[0056] After the log label data and the log text data are obtained by performing data cleaning processing on the serial log data, in the embodiment of the present application, in response to a log query instruction, log query data is obtained from the log file database based on the analysis result of the log query instruction: in response to a log query instruction, the log query instruction is analyzed, log query parameters in the log query instruction are determined, and the analysis result of the log query instruction is obtained; based on the analysis result of the log query instruction, corresponding log label data and log text data are called from the log file database, and log query data is obtained.

[0057] In the embodiment of the present application, the log query parameters are used to call corresponding log label data and log text data from the log file database, so different log query parameters correspond to different log data to be called; at the same time, the log query instruction has different query instruction sources according to different application scenarios; for example, when it is necessary to view all calling request logs of users of a client and track user behaviors and problems of the client, the log query instruction is a log query instruction based on the client dimension; for example, when it is necessary to view all processing logs of a corresponding business and troubleshoot business-related problems, the log query instruction is a log query instruction based on the business dimension; when it is necessary to deeply view abnormal details in the business calling link and view upstream and downstream access rationality on the link, the log query instruction is an interface link log query instruction.

[0058] For the log query instruction based on the client dimension, refer to Figure 2As shown in the figure, it is a log query reference interface of the customer dimension, and the embodiment of the application comprises: in response to a log query instruction of the customer dimension, analyzing the log query instruction, determining log query parameters in the log query instruction, the log query parameters comprising a time parameter and a business number (i.e. customer number) of the customer. In the scenario of the log query instruction of the customer dimension, the embodiment of the application can call corresponding log data by taking the business number of the customer as the log query parameter, view all call request logs of the user, and track the behavior and problems of the user.

[0059] For a log query instruction of the business dimension, refer to Figure 3 As shown in the figure, it is a log query reference interface of the business dimension, and the embodiment of the application comprises: in response to a log query instruction of the business dimension, analyzing the log query instruction, determining log query parameters in the log query instruction, the log query parameters comprising a time parameter and a business index. In the scenario of the log query instruction of the business dimension, the embodiment of the application can query all processing logs of the business according to the business index, and can quickly query when a fault occurs.

[0060] For a log query instruction of the interface link, refer to Figure 4 As shown in the figure, it is a log query reference interface of the interface link, and the embodiment of the application comprises: in response to a log query instruction of the interface link, analyzing the log query instruction, determining log query parameters in the log query instruction, the log query parameters comprising a time parameter and a serial number. In the scenario of the log query instruction of the interface link, the embodiment of the application can select the serial number of the interface call as the log query parameter, filter and query all log data in the call request according to the serial number, according to the log type and the call object, concatenate the call level relationship and the front and rear order between each log data, and finally display the complete log detail information of the call link in the hierarchical order.

[0061] The embodiment of the application selects log query instructions of different dimensions based on different application requirements, queries log query data from the log file database based on the analysis result of the log query instruction, and finally displays the log query data on the operation interface of the system. The log query data is different according to the analysis result of the log query instruction, and the corresponding log query data is also different.

[0062] Meanwhile, in order to further improve the readability of the log query data, refer to Figure 5As shown, the data display reference diagram of the operation interface of the system, the embodiment of the present application further embeds a selection interface of the log query parameter on the operation interface of the system, and the user can directly re-input, adjust or select new log query parameters (or corresponding log label data) on the operation interface of the system, so as to obtain log query data more in line with actual needs through secondary screening, compare the log query data after secondary screening with historical log data, and display in the form of a column chart, a bar chart or a chart to obtain the abnormal aggregation data. Figure 5 As shown, according to the time parameter in the log query parameter, the embodiment of the present application can generate multi-dimensional comparison data on the operation interface of the system, so that the user can quickly confirm the abnormality in the log data through the log query data displayed on the visual interface, and realize effects such as comparison and analysis display of abnormal log quantity and historical log quantity.

[0063] The multi-dimensional log management and abnormal analysis method collects flow log data in a system container, and obtains log label data and log text data after data cleaning of the flow log data, respectively calls corresponding log query data according to log query instructions of different dimensions, and obtains abnormal aggregation data after secondary screening, and displays the abnormal aggregation data, realizes the abnormal clustering analysis function based on multi-dimensions, and the abnormal aggregation data after secondary screening has better readability, the log query based on the business perspective, the calling link analysis, improves the efficiency and accuracy of the abnormal clustering analysis, provides technical support for fault abnormal positioning, and through multi-dimensional log query and interface calling link analysis, investigates and governs system operation problems, and confirms the upstream and downstream access rationality.

[0064] On the basis of the above embodiment, as to the above Figure 1 As an implementation of the above Figure 1 The method embodiment corresponds to the device embodiment, and the device can be applied to various electronic devices, as shown in Figure 6 The multi-dimensional log management and abnormal analysis device includes:

[0065] The log collection module 100 is used for collecting flow log data printed by a system container, and feeding the collected flow log data to a log collector;

[0066] The log cleaning module 200 is used for reading the flow log data from the log collector after meeting a preset condition, performing data cleaning processing on the flow log data, and obtaining log label data and log text data;

[0067] The log storage module 300 is configured to collect the log label data and the log text data respectively and store them in a log file database.

[0068] The log query module 400 is configured to, in response to a log query instruction, query log query data from the log file database based on a result of analysis of the log query instruction, obtain abnormal aggregation data after secondary screening, and display the abnormal aggregation data.

[0069] The multi-dimensional log management and abnormal analysis device provided by the embodiments of the present application can execute the multi-dimensional log management and abnormal analysis method provided by the above embodiments, and has the corresponding functional steps and beneficial effects of the multi-dimensional log management and abnormal analysis method provided by the above embodiments. For details, please refer to the embodiments of the multi-dimensional log management and abnormal analysis method. The embodiments of the present application will not be described here again.

[0070] The embodiments of the present application also provide an electronic device, which can include a processor and a memory. The processor and the memory can be connected through a bus or other means. The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. or combinations of the above. The memory, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the multi-dimensional log management and abnormal analysis method in the embodiments of the present application. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, implements the multi-dimensional log management and abnormal analysis method in the above method embodiments.

[0071] The memory can include a program storage area and a data storage area, where the program storage area can store an operating system, application programs required by at least one function, and the like, and the data storage area can store data created by the processor and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. The one or more modules are stored in the memory and, when executed by the processor, perform the multi-dimensional log management and exception analysis method as in the above method embodiments. The above electronic device specific details can correspond to the corresponding related description and effects in the above method embodiments, and will not be described here. Those skilled in the art can understand that all or part of the processes in the above embodiments can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above embodiments. The storage medium can be a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc. The storage medium can also include a combination of the above types of memories.

[0072] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order to not obscure the understanding of this description.

[0073] Similarly, it is to be understood that the embodiments of the present application can be used in the exact form disclosed herein, or with minor modifications, and the present disclosure is not limited to the exact form disclosed herein. It is also to be understood that the features of the various aspects of the present disclosure can be combined, exchanged, or removed, and the present disclosure encompasses all such possibilities.

[0074] It should be noted that the above-mentioned embodiments illustrate the application rather than limit the application, and alternative embodiments can be designed by those skilled in the art without departing from the scope of the appended claims.

Claims

1. A multi-dimensional log management and anomaly analysis method, characterized in that, The multi-dimensional log management and anomaly analysis method includes: The system collects the log data printed by the container and feeds the collected log data back to the log collector. After the preset conditions are met, the log data is read from the log collector, and the log data is cleaned to obtain log tag data and log text data. The log tag data and log text data are collected separately and stored in the log file database respectively; In response to a log query command, log query data is obtained from the log file database based on the parsing result of the log query command, and abnormal aggregate data is obtained after secondary filtering and displayed.

2. The multi-dimensional log management and anomaly analysis method according to claim 1, characterized in that, The data cleaning process for the log data includes: The transaction log data is parsed, and the feature data of the transaction log data is extracted; After the extracted feature data is deduplicated, abnormal data is deleted, and format is normalized, it is stored according to a predetermined storage format to obtain log tag data and log text data.

3. The multi-dimensional log management and anomaly analysis method according to claim 2, characterized in that, The feature data extracted from the log data includes: system tag data, business tag data, interface tag data, exception and error tag data, system affiliation information, business information, interface information, and exception and error information.

4. The multi-dimensional log management and anomaly analysis method according to claim 1, characterized in that, In response to a log query command, based on the parsing result of the log query command, log query data is retrieved from the log file database: In response to a log query command, the log query command is parsed to determine the log query parameters in the log query command and obtain the parsing result of the log query command. Based on the parsing result of the log query command, the corresponding log tag data and log text data are retrieved from the log file database to obtain the log query data.

5. The multi-dimensional log management and anomaly analysis method according to claim 4, characterized in that, The method includes: responding to a log query command at the customer level, parsing the log query command, and determining the log query parameters in the log query command, wherein the log query parameters include a time parameter and the customer's business number.

6. The multi-dimensional log management and anomaly analysis method according to claim 4, characterized in that, The method includes: responding to a log query command in response to a business dimension, parsing the log query command, and determining the log query parameters in the log query command, wherein the log query parameters include time parameters and business metrics.

7. The multi-dimensional log management and anomaly analysis method according to claim 4, characterized in that, The method includes: responding to a log query command from an interface link, parsing the log query command, and determining the log query parameters in the log query command, wherein the log query parameters include a time parameter and a serial number.

8. A multi-dimensional log management and anomaly analysis device, applied to the method described in any one of claims 1-7, characterized in that, The multi-dimensional log management and anomaly analysis device includes: The log collection module is used to collect the log data printed by the system container and feed the collected log data back to the log collector. The log cleaning module is used to read the log data from the log collector after meeting preset conditions, and to perform data cleaning processing on the log data to obtain log tag data and log text data. The log storage module is used to collect the log tag data and log text data respectively, and store them in the log file database respectively; The log query module is used to respond to a log query command, retrieve log query data from the log file database based on the parsing result of the log query command, obtain abnormal aggregate data after secondary filtering, and display the abnormal aggregate data.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that are loaded and executed by a processor to perform the operations described in any one of claims 1-7.

10. An electronic device comprising a processor and a memory, characterized in that, The memory stores computer program instructions that can be executed by the processor, and when the processor executes the computer program instructions, it implements the instructions of the method as described in any one of claims 1-7.

Images