Log analysis method and apparatus, electronic device, and storage medium

Abstract

The application relates to the technical field of Internet communication, and discloses a log analysis method and device, an electronic device and a storage medium. The method comprises the following steps: acquiring a first log query instruction; wherein the first log query instruction contains a first keyword; acquiring a result data set according to the first log query instruction; wherein the result data set is generated according to basic data containing the first keyword and log data containing a second keyword associated with the first keyword; and the result data set is sent to a visual analysis platform, so that the visual analysis platform completes log analysis according to the result data set. According to the method, the middleware system generates a result data set aggregated by log data containing different keywords and basic data according to the first log query instruction, so that the visual analysis platform completes log analysis, thereby avoiding the problems of writing basic data into log data or customizing data collection components, and the influence of a large amount of data on log analysis efficiency.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of Internet communication technology, and in particular to a log analysis method, device, electronic device, and storage medium. Background Art

[0002] With the continuous development of communications technology and the internet, log analysis has become an indispensable tool for obtaining accurate information on user demand trends and preferences, as well as the speed of new product adoption. To achieve accurate log analysis, the technology stack of most log analysis systems uses the data collection component Logstash to collect log data, the search engine Elasticsearch to provide search engine support, and the visualization platform Grafana to provide visualization tools for log analysis. Elasticsearch is a distributed, highly scalable, and highly real-time search and data analysis engine. To improve search performance, Elasticsearch encourages flattening data when creating indexes. Grafana is an open source data visualization tool developed in the Go language that can perform data monitoring and statistics, and includes alerting capabilities. Logstash acts as a data collector, collecting and parsing data from various formats and channels, and then formatting and outputting it to Elasticsearch.

[0003] In order to combine basic data and log data for log analysis, there are three implementation solutions: the first solution is to write basic data into log data and directly analyze the log data and basic data at the same time. However, this solution has poor scalability and requires maintaining a large amount of duplicate log data; the second solution is to customize Logstash to meet special requirements. When parsing log data, the basic data is stored in Elasticsearch. Since Logstash is a general data collector, customized parsing requirements are difficult to implement; the third solution is to use Grafana to implement data bridging through the front-end, obtaining basic data and log data for analysis separately. However, Grafana is a pure front-end project, and data processing is located at the front-end page level. The performance is poor when the data volume is large.

[0004] Therefore, how to simply and efficiently implement a visual analysis platform to conduct targeted analysis and display of log data and basic data, and improve the efficiency and effectiveness of log analysis is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] The main purpose of the embodiments of the present application is to propose a log analysis method, device, electronic device and storage medium, aiming to avoid the data maintenance caused by the need to write basic data into the log during the log analysis process, as well as the difficulty of customizing data acquisition components, thereby improving the efficiency of log analysis while ensuring the accuracy of log analysis.

[0006] To achieve the above-mentioned purpose, an embodiment of the present application provides a log analysis method, including: obtaining a first log query instruction; wherein the first log query instruction contains a first keyword; obtaining a result data set according to the first log query instruction; wherein the result data set is generated based on basic data containing the first keyword and log data containing a second keyword associated with the first keyword; sending the result data set to a visualization analysis platform, for the visualization analysis platform to complete log analysis based on the result data set.

[0007] To achieve the above-mentioned purpose, an embodiment of the present application also proposes a log analysis device, including: an acquisition module, used to obtain a first log query instruction; wherein the first log query instruction contains a first keyword; a query module, used to obtain a result data set according to the first log query instruction; wherein the result data set is generated based on basic data containing the first keyword and log data containing a second keyword associated with the first keyword; a sending module, used to send the result data set to a visualization analysis platform, so that the visualization analysis platform can complete log analysis based on the result data set.

[0008] To achieve the above-mentioned objectives, an embodiment of the present application also proposes an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the log analysis method as described above.

[0009] To achieve the above-mentioned objectives, an embodiment of the present application further proposes a computer-readable storage medium storing a computer program, which implements the log analysis method described above when executed by a processor.

[0010] The log analysis method provided by the embodiment of the present application is that the middleware system receives a first log query instruction containing a first keyword sent by the visualization analysis platform when performing log analysis, obtains the basic data containing the first keyword and the log data containing the second keyword required for the current log analysis according to the first log query instruction, and then filters and aggregates the obtained basic data and log data to generate a result data set, and sends the generated result data set to the visualization analysis platform for the visualization analysis platform to complete the log analysis according to the received result data set. By utilizing the middleware system to generate a result data set aggregated from log data containing different keywords and basic data according to the first log query instruction of the visualization analysis platform for the visualization analysis platform to complete the log analysis, the problem of needing to write the basic data into the log data or customize the data acquisition component when performing targeted analysis on the basic data is avoided; by sending the integrated result data set directly to the visualization analysis platform, the influence of excessive data volume on the efficiency of log analysis is avoided, and while ensuring the accuracy of log analysis, the efficiency of log analysis is greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] One or more embodiments are exemplarily described by the figures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments.

[0012] Figure 1 is a flow chart of the log analysis method in an embodiment of the present application;

[0013] Figure 2 is a structural diagram of a log analysis device in another embodiment of the present application;

[0014] Figure 3 It is a structural diagram of an electronic device in another embodiment of the present application. DETAILED DESCRIPTION

[0015] As we can see from the background, existing log analysis methods, in order to achieve targeted analysis of basic data, often require maintaining large amounts of duplicate log data, struggle to customize data collection components, and suffer from poor log analysis efficiency and performance. Therefore, a pressing technical challenge is to simply and efficiently implement a visual analysis platform that analyzes and displays log data and basic data in a targeted manner, thereby improving the efficiency and effectiveness of log analysis.

[0016] In order to solve the above problems, some embodiments of the present application provide a log analysis method, including: obtaining a first log query instruction; wherein the first log query instruction contains a first keyword; according to the first log query instruction, obtaining a result data set; wherein the result data set is generated based on basic data containing the first keyword and log data containing a second keyword associated with the first keyword; sending the result data set to a visualization analysis platform, for the visualization analysis platform to complete log analysis based on the result data set.

[0017] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, each embodiment of the present application will be described in detail below with reference to the accompanying drawings. However, it will be understood by those skilled in the art that in each embodiment of the present application, many technical details are proposed to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present application can be implemented. The division of the following embodiments is for convenience of description and should not constitute any limitation on the specific implementation of the present application. The various embodiments can be combined and referenced with each other under the premise of no contradiction.

[0018] The implementation details of the log analysis method recorded in this application will be specifically described below in conjunction with specific embodiments. The following content is only for the convenience of understanding the implementation details and is not necessary for implementing this solution.

[0019] The first aspect of the embodiment of the present application provides a log analysis method, the process of the log analysis method is referenced Figure 1 In some embodiments, the log analysis method is applied to a middleware system or a terminal deployed with the middleware system, such as a computer or other electronic device. This embodiment is described using the application in the middleware system as an example. The log analysis includes the following steps:

[0020] Step 101: Obtain a first log query instruction; wherein the first log query instruction includes a first keyword.

[0021] Specifically, when performing log analysis, the visual analysis platform acquires a log analysis task input by an operation and maintenance personnel according to an administrator, determines a specific log analysis chart to be generated, for example, a chart corresponding to an increase in the number of visits to a specified website in a period of time. After determining the specified chart to be generated, the visual analysis platform generates a first log query instruction containing a first keyword according to a customized syntax and the first keyword of the basic data to be queried, and sends the first log query instruction to a middleware system pre-configured through a specified data transmission interface. The middleware system performs syntax detection on the data message in the data transmission interface to acquire the first log query instruction sent by the visual analysis platform, wherein the first log query instruction contains the first keyword. The data request is performed through the query instruction containing the first keyword, so that the data required in the log analysis process can be accurately acquired.

[0022] It should be noted that before performing log analysis, the source of the middleware system needs to be pre-configured for the visual analysis platform to ensure normal source configuration. For example, the communication interface of the middleware system, the syntax specification used in the communication process, the address information configuration of the middleware system, and the like are pre-configured. The communication interface can be search, msearch, mapping, and the like. When generating the first log query instruction containing the first keyword, the visual analysis platform can encapsulate the first keyword in the first log query instruction through a specific field according to the customized syntax, for example, encapsulating the first keyword XXX in the first log query instruction through the specified field DATABASE-XXX generated according to the customized syntax. In specific applications, the communication interface and the customized syntax selected when the visual analysis platform communicates with the middleware system can be set according to specific needs, and the embodiment does not limit the communication interface and the customized syntax selected when the visual analysis platform communicates with the middleware system.

[0023] In step 102, the result data set is acquired according to the first log query instruction; wherein the result data set is generated according to the basic data containing the first keyword and the log data containing the second keyword associated with the first keyword.

[0024] Specifically, after receiving the first log query instruction sent by the visualization analysis platform, the middleware system needs to obtain the first keyword contained in the basic data and the second keyword contained in the log data when determining the result data set generated according to the first log query instruction, and then obtains the specified basic data and log data according to the first keyword and the second keyword, and integrates the obtained basic data and log data to generate the result data set. The middleware system generates the result data set integrated by the basic data and the log data containing different keywords according to the first log query instruction, so that the visualization analysis platform can directly obtain the information corresponding to the field not contained in the log data, and improve the efficiency and accuracy of the analysis.

[0025] In one example, the first keyword is determined according to target basic data needed to complete log analysis, and the second keyword is determined according to target log data associated with the target basic data. That is, the first keyword is determined by the visualization analysis platform according to the target basic data needed when generating the log analysis chart corresponding to the current log analysis task. For example, a statistical table of new users in a certain time period needs to be generated, and user login data in a certain time period needs to be used, so the first keyword can be the login creation time. However, when log analysis is performed only by relying on the login creation time as the keyword, the same user's login at different login times may be repeatedly counted, so it is necessary to eliminate the repeated login of the user. However, the login name and login operation at each login time can be counted in the log data, so the target log data with the login name as the keyword associated with the target basic data with the login creation time as the keyword can be obtained, and the repeated data can be eliminated by combining the obtained target log data. That is, the first keyword is a key field existing in the basic data, the second keyword is a key field existing in the log data associated with the basic data, and the log data generally does not contain data containing the first keyword. In order to ensure the efficiency of log analysis, the association relationship between the keywords similar to the above example can be stored in the middleware system in advance, and after the first keyword contained in the basic data needed is obtained, the second keyword contained in the log data needed is obtained according to the preset association relationship, and then the basic data needed for log analysis and the log data associated with the basic data are obtained.

[0026] In one example, the middleware system obtains a result data set according to the first log query instruction, including: obtaining the basic data containing the first keyword and the log data containing the second keyword according to the first log query instruction; and selecting a corresponding syntax converter to perform deduplication and / or grouped statistics on the basic data and the log data according to the statement type of the first log query instruction, to generate the result data set. After obtaining the basic data containing the first keyword and the log data containing the second keyword according to the first log query instruction, the middleware system detects the statement type used by the first log query instruction, and determines the specific statement type used by the first log query instruction. That is, when the visual analysis platform generates the first log query instruction, it will select the query statement type used when generating the first log query instruction according to the specific type of the log analysis result to be generated, such as a chart or a list. After obtaining the basic data and the log data that need to be integrated, the middleware system selects a corresponding syntax converter for data integration according to the determined statement type and the pre-set correspondence between the statement type and the syntax converter, from the pre-customized and saved syntax converters. Through the selected syntax converter, the basic data and the log data are deduplicated and / or grouped according to the first keyword and the second keyword, and the result data set is generated according to the filtering and aggregation results. By selecting the appropriate syntax converter to filter and aggregate the basic data and the log data according to the statement type of the first log query instruction to generate the result data set, the result data set obtained by the visual analysis platform can accurately meet the log analysis needs. And greatly reduce the data processing pressure of the visual analysis platform, so that the visual analysis platform can quickly perform intuitive display of data reports or charts, greatly improve the log analysis efficiency while ensuring the accuracy of log analysis.

[0027] In another example, the middleware system obtains basic data containing a first keyword according to a first log query instruction, including: performing keyword analysis on the first log query instruction to obtain the first keyword; and obtaining the basic data according to the first keyword. When the middleware system obtains the basic data containing the first keyword, it first performs keyword analysis on the first log query instruction to detect the specified field generated using the customized syntax, obtains the first keyword encapsulated in the first log query instruction, and then searches and extracts the basic data containing the first keyword according to the obtained first keyword. By performing keyword analysis on the first log query instruction, the keyword contained in the basic data to be obtained is accurately obtained, ensuring the accuracy of the basic data acquisition and facilitating subsequent accurate log analysis.

[0028] Further, the middleware system acquires the basic data according to the first keyword, including: acquiring the basic data in the preset relational database according to the first keyword. After the middleware system parses the first keyword contained in the basic data that needs to be acquired, the relational database query statement is generated according to the first keyword, and the data request is initiated to the preset relational database, for example, the basic data containing the first keyword is acquired in the preset relational database. By writing the basic data into the preset relational database, the query and acquisition in the relational database are performed when the basic data containing the first keyword is acquired, which avoids the problem of maintaining a large amount of repeated log data caused by writing the basic data into the log data.

[0029] Further, the preset relational database in which the middleware system acquires the basic data containing the first keyword is a Mysql database. The strong consistency of the Mysql database is used to store the basic data in the preset Mysql database, so that the stored basic data is the basic data updated in real time according to the actual situation, and the consistency and timeliness of the basic data acquired according to the first keyword are ensured.

[0030] It should be noted that the preset relational database including the Mysql database can be set by the configuration tool of the software after storing the basic data, and the basic data synchronization or real-time synchronization of the basic data is performed at a certain time interval. In specific applications, the synchronization interval can be set as needed, and the specific interval of the middleware system for basic data synchronization of the relational database is not limited.

[0031] In another example, after the middleware system acquires the basic data containing the first keyword according to the first log query instruction, it further includes: caching the basic data into the data structure server; before acquiring the result data set, it further includes: reading the basic data in the data structure server. After the middleware system acquires the basic data containing the first keyword, the basic data set corresponding to the first keyword is generated according to the acquired basic data, and then the generated basic data set is stored in the data structure server. When the result data set is generated, the basic data set is extracted from the data structure server, which avoids the storage burden caused by storing the basic data set in the middleware system, and the influence of real-time acquisition of the basic data on the generation speed of the result data set. By using the data structure server to cache the basic data set acquired according to the first keyword, the efficiency of data calling and querying is greatly improved, and the generation efficiency of the result data set is ensured.

[0032] In another example, after the middleware system obtains basic data based on the first keyword, it also includes: obtaining a second keyword associated with the first keyword based on the preset association relationship between the keywords and the first keyword; and obtaining log data based on the second keyword. After the middleware system parses the first keyword and obtains the specified basic data based on the first keyword, it obtains the second keyword contained in the log data to be obtained based on the preset management relationship between the pre-set keywords and the parsed first keyword. Then, based on the obtained second keyword, the specified log data to be obtained is searched or read. By pre-setting the association relationship between the keywords, the middleware system can determine the second keyword contained in the log data to be obtained based on the parsed first keyword after receiving the first log query instruction, thereby accurately obtaining the log data needed for the log analysis process, thereby ensuring the accuracy of subsequent log analysis.

[0033] Furthermore, the middleware system obtains log data based on the second keyword, including: generating a second log query instruction based on the second keyword, and sending the second log query instruction to the search engine; receiving the data query result of the search engine, and obtaining log data containing the second keyword. After obtaining the second keyword, the middleware system generates a second log query instruction for querying log data based on the second keyword, and then sends the generated second log query instruction to the search engine, so that the search engine can search and query the log data containing the second keyword based on the received second log query instruction, and generate data query results based on the searched log data and feed them back to the middleware system, so that the middleware system can obtain log data containing the second keyword. By generating a second log query instruction for the search engine to query log data based on the second keyword, so that the search engine can search and feed back the log data set containing the second keyword, the log data needed in the log analysis process can be accurately obtained, thereby ensuring the accuracy of the log analysis.

[0034] It is worth mentioning that when the middleware system generates the second log query instruction, it can also encapsulate the first keyword field or identifier contained in the generated basic data set in the second log query instruction, for example, replace the field value of the term under the aggs node in the second log query instruction with the first keyword; so that when the search engine feeds back the data query result, it directly carries the first keyword field or identifier in the query result. When the middleware system receives the data query result and integrates the basic data and log data, it parses the query result, obtains the first keyword in the array object under the aggregate node in the query result, and replaces it with the basic data set obtained in advance based on the first keyword. Then, according to the statement type of the first log query instruction, the basic data set and the log data set are filtered and aggregated, and encapsulated into a result data set.

[0035] Step 103, send the result data set to the visualization analysis platform, so that the visualization analysis platform completes log analysis according to the result data set.

[0036] Specifically, after the middleware system generates the result data set according to the basic data containing the first keyword and the log data containing the second keyword, the middleware system sends the result data set to the visualization analysis platform through the preset communication interface. After receiving the result data set, the visualization analysis platform converts the basic data and the log data integrated in the result data set, generates specific analysis charts, and displays the log analysis result on the display page of the visualization analysis platform.

[0037] In addition, it should be understood that the division of steps of the above methods is only for the purpose of clear description. In implementation, one step can be combined or some steps can be split and decomposed into multiple steps, as long as the same logical relationship is included, which is within the protection scope of the patent. Adding irrelevant modifications or introducing irrelevant designs in the algorithm or process, but not changing the core design of the algorithm and process, are within the protection scope of the patent.

[0038] Another aspect of the embodiment of the present application relates to a log analysis device, referring to Figure 2 , comprising:

[0039] The acquisition module 201 is configured to acquire a first log query instruction. The first log query instruction contains a first keyword.

[0040] The query module 202 is configured to acquire a result data set according to the first log query instruction. The result data set is generated according to basic data containing the first keyword and log data containing a second keyword associated with the first keyword.

[0041] The sending module 203 is configured to send the result data set to the visualization analysis platform, so that the visualization analysis platform completes log analysis according to the result data set.

[0042] It is not difficult to find that the present embodiment is a device embodiment corresponding to the method embodiment. The present embodiment can be implemented in cooperation with the method embodiment. The related technical details mentioned in the method embodiment are still valid in the present embodiment. In order to reduce repetition, they will not be described here. Accordingly, the related technical details mentioned in the present embodiment can also be applied to the method embodiment.

[0043] It is worth noting that all modules involved in this embodiment are logical modules. In actual applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, to highlight the innovations of the present invention, this embodiment does not include units that are not closely related to solving the technical problems proposed by the present invention. However, this does not mean that other units do not exist in this embodiment.

[0044] Another aspect of the present application embodiment further provides an electronic device, Figure 3 , including: at least one processor 301; and a memory 302 communicatively connected to the at least one processor 301; wherein the memory 302 stores instructions that can be executed by the at least one processor 301, and the instructions are executed by the at least one processor 301 so that the at least one processor 301 can execute the log analysis method described in any of the above method embodiments.

[0045] The memory 302 and processor 301 are connected using a bus. The bus can include any number of interconnected buses and bridges, connecting various circuits of one or more processors 301 and memory 302. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits. These are all well known in the art and are therefore not described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor 301 is transmitted over a wireless medium via an antenna. Furthermore, the antenna receives data and transmits it to the processor 301.

[0046] The processor 301 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. The memory 302 can be used to store data used by the processor 301 when performing operations.

[0047] Another aspect of the present invention provides a computer-readable storage medium storing a computer program that implements the above method embodiment when executed by a processor.

[0048] That is, a person skilled in the art can understand that all or part of the steps in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a program stored in a storage medium, including a plurality of instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0049] A person of ordinary skill in the art can understand that the above-mentioned embodiments are specific embodiments for implementing the present application, and in actual applications, various changes can be made in form and details without departing from the spirit and scope of the present application.

Claims

1. A log analysis method, characterized in that: include: Obtain a first log query instruction; wherein the first log query instruction includes a first keyword; Obtaining a result data set according to the first log query instruction; wherein the result data set is generated based on basic data containing the first keyword and log data containing a second keyword associated with the first keyword; Sending the result data set to the visual analysis platform for the visual analysis platform to complete log analysis based on the result data set; Wherein, obtaining the result data set according to the first log query instruction includes: According to the first keyword, obtaining the basic data in a preset relational database; According to the preset association relationship between keywords and the first keyword, obtaining the second keyword associated with the first keyword; According to the second keyword, obtaining the log data; Filter and aggregate basic data and log data to generate result datasets.

2. The log analysis method according to claim 1, characterized in that: The obtaining of a result data set according to the first log query instruction includes: According to the first log query instruction, acquiring the basic data containing the first keyword and the log data containing the second keyword; According to the statement type of the first log query instruction, a corresponding syntax converter is selected to perform deduplication and / or group statistics on the basic data and the log data to generate the result data set.

3. The log analysis method according to claim 2, characterized in that: The acquiring, according to the first log query instruction, the basic data containing the first keyword, includes: Perform keyword parsing on the first log query instruction to obtain the first keyword; The basic data is acquired according to the first keyword.

4. The log analysis method according to claim 3, characterized in that: The preset relational database is a Mysql database.

5. The log analysis method according to any one of claims 3 to 4, characterized in that: After obtaining the basic data containing the first keyword, the method further includes: caching the basic data in a data structure server; Before obtaining the result data set, the method further includes: The basic data is read from the data structure server.

6. The log analysis method according to claim 1, characterized in that: The acquiring the log data according to the second keyword includes: generating a second log query instruction according to the second keyword, and sending the second log query instruction to the search engine; Receive the data query result of the search engine, and obtain the log data containing the second keyword.

7. The log analysis method according to any one of claims 1 to 4, characterized in that: The first keyword is determined based on target basic data required to complete the log analysis, and the second keyword is determined based on target log data associated with the target basic data.

8. A log analysis device, characterized in that: include: An acquisition module, configured to acquire a first log query instruction; wherein the first log query instruction includes a first keyword; a query module, configured to obtain a result data set according to the first log query instruction; wherein the result data set is generated based on basic data containing the first keyword and log data containing a second keyword associated with the first keyword; A sending module, configured to send the result data set to a visual analysis platform, so that the visual analysis platform can complete log analysis based on the result data set; Wherein, the query module is further used for: According to the first keyword, obtaining the basic data in a preset relational database; According to the preset association relationship between keywords and the first keyword, obtaining the second keyword associated with the first keyword; According to the second keyword, obtaining the log data; Filter and aggregate basic data and log data to generate result datasets.

9. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the log analysis method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the log analysis method according to any one of claims 1 to 7 is implemented.

Images