Multi-cluster log data processing method and device, electronic device and storage medium

Abstract

The invention relates to a multi-cluster log data processing method and device, an electronic device and a storage medium. The method comprises the following steps: in response to a query request of log data, converting a query statement carried by the query request into query information in a unified query format; the query request is triggered based on a unified query interface, a query grammar corresponding to the query statement is a preset query grammar configured by the unified query interface, and the unified query interface is used for uniformly querying log data of a plurality of heterogeneous clusters; based on the query information, determining at least one first target cluster needing to be queried by the query request from a plurality of heterogeneous clusters; querying the grammar based on the target corresponding to each first target cluster; converting the query information into target query statements corresponding to the first target clusters; and performing log data query in the corresponding first target cluster by using the target query statement to obtain a log query result. According to the technical scheme, cross-heterogeneous cluster log query can be supported, and the query efficiency is high.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a multi-cluster log data processing method. Background Art

[0002] With the development of Internet applications, Internet-based businesses are becoming more and more abundant. Accordingly, each business will also generate a large amount of log data. If the business party wants to query the log data within a period of time, since the log data may be stored in multiple heterogeneous clusters, it is necessary to query according to the query syntax of the corresponding cluster. In this way, the query interface needs to be customized according to each cluster, which is costly. In addition, the query methods of different types of clusters are different. If multi-cluster queries are required, multiple queries are required, resulting in low query efficiency and unable to meet the needs of high-performance analytical queries of large-scale data sets. In addition, when adding new clusters, the query interface needs to be redesigned, which has poor scalability and leads to high development and maintenance costs. Summary of the invention

[0003] The present application provides a multi-cluster log data processing method that can support cross-cluster log query, has high query efficiency, good scalability, and can flexibly select different types of clusters for log storage. The technical solution of the present application is as follows:

[0004] According to a first aspect of an embodiment of the present application, a multi-cluster log data processing method is provided, including:

[0005] In response to a query request for log data, converting a query statement carried in the query request into query information in a unified query format; the query request is triggered based on a unified query interface, the query syntax corresponding to the query statement is a preset query syntax configured by the unified query interface, and the unified query interface is used to uniformly query the log data of multiple heterogeneous clusters;

[0006] Based on the query information, determining at least one first target cluster that needs to be queried by the query request from the multiple heterogeneous clusters;

[0007] Obtaining a target query grammar corresponding to each of the at least one first target cluster;

[0008] Based on the target query grammar, converting the query information into a target query statement corresponding to each of the at least one first target cluster;

[0009] The target query statement is used to perform a log data query in the corresponding first target cluster to obtain a log query result.

[0010] According to a second aspect of an embodiment of the present application, a multi-cluster log data processing system is provided, including:

[0011] A log query system, for responding to a query request for log data, converting a query statement carried in the query request into query information in a unified query format; the query request is triggered based on a unified query interface, the query syntax corresponding to the query statement is a preset query syntax configured by the unified query interface, and the unified query interface is used to uniformly query the log data of multiple heterogeneous clusters;

[0012] A backend service for storing a query routing table; the query routing table stores the corresponding relationship between the account number, log type, time range, and routing identifier;

[0013] The log query system is used to call the query routing table, extract the query log type and the query time range from the query information; obtain the first account that triggers the query request; based on the first account, the query log type and the query time range, find at least one matching first target routing identifier in the query routing table; and use the routing identifiers pre-configured by the multiple heterogeneous clusters to determine the cluster corresponding to each of the at least one first target routing identifier from the multiple heterogeneous clusters as the at least one first target cluster; and based on the target query syntax corresponding to each of the at least one first target cluster, convert the query information into a target query statement corresponding to each of the at least one first target cluster; use the target query statement to query log data in the corresponding first target cluster to obtain a log query result.

[0014] According to a third aspect of an embodiment of the present application, a multi-cluster log data processing device is provided, including:

[0015] A query response module, configured to respond to a query request for log data and convert a query statement carried in the query request into query information in a unified query format; the query request is triggered based on a unified query interface, the query syntax corresponding to the query statement is a preset query syntax configured by the unified query interface, and the unified query interface is used to uniformly query the log data of multiple heterogeneous clusters;

[0016] A first target cluster determination module, configured to determine, based on the query information, at least one first target cluster to be queried by the query request from among the multiple heterogeneous clusters;

[0017] A target query grammar acquisition module, used to acquire the target query grammar corresponding to each of the at least one first target cluster;

[0018] A query statement conversion module, configured to convert the query information into a target query statement corresponding to each of the at least one first target cluster based on the target query grammar;

[0019] The log query result acquisition module is used to use the target query statement to perform log data query in the corresponding first target cluster to obtain a log query result.

[0020] According to a fourth aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement a method as described in any one of the first aspects above.

[0021] According to the fifth aspect of the embodiments of the present application, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute any method described in the first aspect of the embodiments of the present application.

[0022] According to a sixth aspect of an embodiment of the present application, a computer program product is provided, comprising computer instructions, which, when executed by a processor, enable a computer to execute any one of the methods described in the first aspect of the embodiment of the present application.

[0023] The technical solution provided by the embodiments of the present application brings at least the following beneficial effects:

[0024] By setting a unified query interface and configuring a preset query grammar for the unified query interface, the unified query interface is used to uniformly query the log data of multiple heterogeneous clusters, so that users can trigger a unified query request for multiple heterogeneous clusters based on the query statement of the preset query grammar, and can convert the query statement carried by the query request into query information in a unified query format, and determine at least one first target cluster that the query request needs to query from multiple heterogeneous clusters based on the query information, so that the query information can be converted into a target query statement corresponding to at least one first target cluster based on the target query grammar corresponding to each first target cluster, and the target query statement is used to query log data in the corresponding first target cluster to obtain a log query result. Users can perform cross-cluster unified log queries on multiple heterogeneous clusters based on the query statement of the preset query grammar, without having to query multiple heterogeneous clusters separately, thereby eliminating the need for multiple queries, and having high query efficiency, which can meet the needs of high-performance analysis queries of large-scale data sets, and has low query resource consumption; and when adding a new cluster, there is no need to redesign the query interface, which has good scalability, and has low development and maintenance costs for log queries.

[0025] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application, and do not constitute an improper limitation on the present application.

[0027] Figure 1 A schematic diagram of the architecture of a multi-cluster log data processing system provided according to an embodiment of the present application is shown.

[0028] Figure 2 The figure is a flowchart of a multi-cluster log data processing method according to an exemplary embodiment.

[0029] Figure 3 The figure is a flow diagram of a log query according to an exemplary embodiment.

[0030] Figure 4 The figure is a flow chart of a log storage method according to an exemplary embodiment.

[0031] Figure 5 The diagram is a structural diagram of a log storage system according to an exemplary embodiment.

[0032] Figure 6 The figure is a flow diagram of log storage according to an exemplary embodiment.

[0033] Figure 7 The figure is a block diagram of a multi-cluster log data processing device according to an exemplary embodiment.

[0034] Figure 8 The invention is a block diagram of an electronic device for multi-cluster log data processing according to an exemplary embodiment. DETAILED DESCRIPTION

[0035] Various exemplary embodiments, features and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0036] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0037] In addition, in order to better illustrate the present application, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present application can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present application.

[0038] It should be noted that in the specific implementation of this application, user-related data is involved. When the following embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0039] Before introducing the method embodiments provided in the present application, a brief introduction is first given to the application scenarios, relevant terms or nouns that may be involved in the method embodiments of the present application to facilitate understanding by technicians in the field of the present application.

[0040] ES: Elasticsearch is a distributed full-text search and analysis engine built on the Lucene syntax search engine.

[0041] ETL: Extract-Transform-Load, refers to the process of extracting data from one system, converting and cleaning it, and then loading it into another system.

[0042] API: Application Programming Interface, which is a set of rules, protocols, and tools that define the interactions between software components.

[0043] CH: ClickHouse is a column-oriented database management system.

[0044] Antlr4: A syntax parsing tool library that can parse custom DSL to generate syntax data for easy application processing

[0045] DSL: Domain Specific Language, a custom language defined to handle specific scenario problems, such as SQL.

[0046] SQL: Structured Query Language, a database query and programming language.

[0047] See also Figure 1 , Figure 1 The following is a schematic diagram showing the architecture of a multi-cluster log data processing system provided according to an embodiment of the present application. The log data processing system can be used in the multi-cluster log data processing method of the present application. Figure 1 As shown, the log data processing system may at least include a log query system (corresponding to Figure 1 API part in ), log storage system (corresponding to Figure 1The ETL part in the log data processing system) and the backend service (backend Server), as well as multiple external heterogeneous clusters. Each business can use the log data processing system to provide users with log storage and log query functions across multiple heterogeneous clusters. Among them, multiple heterogeneous clusters may include clusters of multiple different types, for example, including but not limited to clusters of types such as CH, ES, Mysql, etc., which are not limited in this application. Each business may include but is not limited to cloud firewalls (cloud defense), WAF (Web Application Firewall, Web application protection system), security operation centers, etc., which can be applied to log auditing, log analysis, report statistics and other functions. For example, users of each business can trigger log query and storage functions through the front end or HttpClient (programming toolkit) on the terminal side.

[0048] Among them, the log query system may include a query API (QueryAPI) for responding to query requests, and a billing API (BillingAPI) for responding to billing requests. QueryAPI can receive query statements input in Json (JavaScript Object Notation, JS object notation, a lightweight data exchange format) format and using preset query syntax, so that the query statement can be converted into an SQL statement supported by the corresponding cluster according to routing information (for example, querying the routing table) to execute the query. It can also merge multi-cluster query data and finally return the log query results. BillingAPI can be used to synchronize user billing information and return user log usage information.

[0049] The log storage system may include an ETL program (ETL for short), which may mainly include Kafka and Loader. ETL can read data from different data sources according to configuration information and routing information, and write it to each storage cluster after processing. Usually, the business side can input log data into the system through Kafka delivery, and the ETL program consumes the log data in Kafka based on Loader to finally write the log data to the target storage cluster, such as at least one second target cluster.

[0050] Specifically, the background server can include log lifecycle management (Lifecycle Manager), scheduled task management (Job Manager), routing (Router) and external API.

[0051] Router can include read routing and write routing (or query routing and storage routing). The read routing is set with a query routing table, which is used to determine the cluster to be queried based on the account (user ID), log type and time range; the write routing is set with a storage routing table, which is used to determine which cluster the current log should be written to based on the account, log type, and log timestamp (timestamp).

[0052] LifecycleManager can be responsible for log lifecycle management. For example, it can associate and manage the lifecycles of CH and ES:

[0053] 1. Automatically create CH table or ES index according to database configuration and predefined table template.

[0054] 2. Call the management interface as needed to achieve CH log migration, archiving, isolation, destruction, etc.

[0055] 3. Roll the ES index as needed, and delete or cold-backup expired ES logs.

[0056] JobManager can create ETL tasks for data import and migration based on the current job configuration, and manage the task lifecycle. Other modules can create ETL tasks through Job Manager.

[0057] Reference Figure 1 The architecture of the above-mentioned log data processing system is introduced. Specifically, a multi-cluster log data processing system provided by the present application may include:

[0058] A log query system, which is used to respond to a query request for log data and convert the query statement carried in the query request into query information in a unified query format; the query request may be triggered based on a unified query interface, and the query syntax corresponding to the query statement is a preset query syntax configured by the unified query interface, and the unified query interface is used to uniformly query the log data of multiple heterogeneous clusters;

[0059] Backend service, used to save the query routing table; the query routing table stores the correspondence between the account, log type, time range and routing identifier;

[0060] The above-mentioned log query system can also be used to call the query routing table, extract the query log type and the query time range from the query information; obtain the first account that triggers the query request; based on the first account, the query log type and the query time range, find at least one matching first target routing identifier in the query routing table; and use the routing identifiers pre-configured by the multiple heterogeneous clusters to determine the cluster corresponding to each of the at least one first target routing identifier from the multiple heterogeneous clusters as the at least one first target cluster; and based on the target query syntax corresponding to each of the at least one first target cluster, convert the query information into a target query statement corresponding to each of the at least one first target cluster; use the target query statement to query log data in the corresponding first target cluster to obtain a log query result.

[0061] Optionally, the log data processing system may further include:

[0062] The log storage system is used to obtain the original log data to be stored; perform data preprocessing on the original log data to obtain preprocessed log data;

[0063] The above-mentioned background service can also be used to save and store routing tables; the routing table stores the corresponding relationship between the account number, log type, timestamp and routing identifier;

[0064] The above-mentioned log storage system can also be used to determine the second account, storage log type and log timestamp corresponding to the preprocessed log data; based on the second account, the storage log type and the log timestamp, find at least one matching second target routing identifier in the storage routing table; and use the routing identifiers pre-configured by the multiple heterogeneous clusters to determine the cluster corresponding to each of the at least one second target routing identifier from the multiple heterogeneous clusters as the at least one second target cluster; and send the preprocessed log data to the write buffer corresponding to the second target cluster; when the write buffer meets the write condition, write the preprocessed log data to the corresponding second target cluster.

[0065] A unified query interface is provided through a log data processing system and a preset query grammar is configured for the unified query interface. The unified query interface is used to uniformly query the log data of multiple heterogeneous clusters, so that users can trigger a unified query request for multiple heterogeneous clusters based on the query statement of the preset query grammar, and the query statement carried by the query request can be converted into query information in a unified query format, and at least one first target cluster that the query request needs to query is determined from multiple heterogeneous clusters based on the query information, so that the query information can be converted into a target query statement corresponding to at least one first target cluster based on the target query grammar corresponding to each first target cluster, and the target query statement is used to query log data in the corresponding first target cluster to obtain a log query result. The user can perform a unified log query across multiple heterogeneous clusters based on the query statement of the preset query grammar, without having to query multiple heterogeneous clusters separately, thereby not having to query multiple times, with high query efficiency, being able to meet the needs of high-performance analysis query of large-scale data sets, and having low query resource consumption; and when adding a new cluster, there is no need to redesign the query interface, with good scalability, so that the development and maintenance costs of log queries are low.

[0066] The specific processing of the log query system and the log storage system to respectively implement the log query and log storage functions can be referred to the following introduction and will not be repeated here.

[0067] Figure 2 FIG. 1 is a flow chart of a multi-cluster log data processing method according to an exemplary embodiment. Figure 2 As shown, the following steps may be included.

[0068] S201, in response to a query request for log data, converting a query statement carried in the query request into query information in a unified query format.

[0069] In the embodiments of this specification, the query request may be triggered based on a unified query interface, which may be used to uniformly query the log data of multiple heterogeneous clusters to achieve unified query of logs across clusters. For example, a query statement may be entered in a query interface corresponding to the unified query interface to trigger the query request. The query interface may be a page in an application, or may be a web page, which is not limited in this application.

[0070] The query syntax corresponding to the query statement can be a preset query syntax configured in the unified query interface, that is, the log data of multiple heterogeneous clusters can be queried based on a unified preset query syntax. The preset query syntax can include but is not limited to Lucene syntax, SQL syntax, ESDSL syntax, etc.

[0071] Log data may refer to log data generated by various businesses, and these log data are stored in multiple heterogeneous clusters.

[0072] In the embodiments of the present specification, in response to a query request for log data, a query statement carried in the query request can be converted into query information in a unified query format. Log query indication information can be extracted from the query statement, and the log query indication information can be filled into the unified query format to obtain query information in the unified query format, thereby realizing unified format conversion of the query statement.

[0073] The unified query format may be pre-set, and the query information may refer to the information obtained by extracting and filling the log query indication information in the query statement into the unified query format. The log query indication information may include at least one of the following: query log type, query time range, number of returned logs, filtering conditions, and query result sorting conditions. Accordingly, the query information may include at least one of the following: query log type, query time range, number of returned logs, filtering conditions, and query result sorting conditions.

[0074] In a possible implementation, the above-mentioned conversion of the query statement carried in the query request into query information in a unified query format may include:

[0075] Use a preset syntax parsing tool to construct a syntax tree of the query statement; illustratively, the preset syntax parsing tool may be Antlr4 or the like.

[0076] Traverse the syntax tree and extract log query indication information from the syntax tree;

[0077] The log query indication information is converted into a data format according to a unified query format to obtain query information, that is, query information in a unified query format.

[0078] In the embodiments of the present specification, a preset syntax parsing tool, such as Antlr4, can be used to construct a syntax tree for a query statement; thereby, the syntax tree can be traversed to extract log query indication information from the syntax tree, and the log query indication information can refer to information used to indicate a log query. For example, it may include the query log type, query time range, number of returned logs, filtering conditions, query result sorting conditions, etc. Further, the log query indication information can be converted into a data format according to a unified query format to obtain query information, that is, the log query indication information is filled into the unified query format to obtain query information structured in the unified query format. In this way, the query information can be unified structured information, providing a basis for subsequent conversion into target query statements corresponding to each cluster.

[0079] S203: Based on the query information, determine at least one first target cluster that needs to be queried by the query request from multiple heterogeneous clusters.

[0080] In the embodiment of the present specification, at least one first target cluster to be queried by the query request can be determined from multiple heterogeneous clusters based on the query information. In other words, it can be determined first in which clusters the log data of the query request is stored.

[0081] In a possible implementation, the determining, based on the query information, from a plurality of heterogeneous clusters, at least one first target cluster to be queried by the query request may include:

[0082] Extract the query log type and query time range from the query information; the query time range may include the query start time and the query end time. The query log type may include but is not limited to network logs, user operation logs, etc., which are not limited in this application.

[0083] Obtain a first account that triggers the query request; illustratively, the first account may be carried in the query request, so that the first account may be extracted from the query request.

[0084] Furthermore, at least one matching first target routing identifier can be found in the query routing table based on the first account, query log type, and query time range. The first target routing identifier is used to identify the corresponding cluster; the query routing table may store the correspondence between the account, log type, time range, and routing identifier. Based on this, the account, log type, and time range that match the first account, query log type, and query time range can be found in the correspondence, so that the routing identifier corresponding to the matching account, log type, and time range can be used as at least one first target routing identifier. The match here may mean that the first account, query log type, and query time range are respectively the same as the account, log type, and time range; or it may mean that the first account and query log type are respectively the same as the account and log type, and the query time range is within the time range.

[0085] Thus, the routing identifiers preconfigured in multiple heterogeneous clusters can be used to determine the cluster corresponding to at least one first target routing identifier from the multiple heterogeneous clusters as at least one first target cluster to find at least one first target cluster to be queried. The at least one first target cluster can refer to one or more first target clusters.

[0086] S205: Obtain a target query grammar corresponding to each of the at least one first target cluster.

[0087] S207: Convert the query information into target query statements corresponding to at least one first target cluster based on the target query grammar.

[0088] In the embodiments of this specification, a cluster may refer to a service cluster, and a heterogeneous cluster may refer to clusters of different types, such as clusters with different log storage methods or query methods. In order to perform queries according to the syntax of different clusters to achieve unified log queries across clusters, the target query syntax corresponding to each of the at least one first target cluster may be obtained, so that the query information may be converted into a target query statement corresponding to each of the at least one first target cluster based on the target query syntax.

[0089] S209: Use the target query statement to query log data in the corresponding first target cluster to obtain a log query result.

[0090] In the embodiment of the present specification, a target query statement may be used to query log data in the corresponding first target cluster to obtain a log query result. The log query result may include log data obtained by querying at least one first target cluster.

[0091] In a possible implementation, the at least one first target cluster may be one first target cluster; accordingly, the log query result may be directly the log data obtained by querying the one first target cluster.

[0092] In another possible implementation, the at least one first target cluster may be multiple first target clusters; accordingly, the log query result may be a merged result of the log data obtained by querying the multiple first target clusters. Based on this, the log data query is performed in the corresponding first target cluster using the target query statement to obtain the log query result, which may include:

[0093] Use the target query statement to concurrently query multiple first target clusters to obtain multiple query return results; concurrent query here can effectively improve query efficiency; for example, the corresponding first target cluster can be queried using the filtering condition in the target query statement to obtain multiple query return results, and the filtering condition includes the field and value to be queried, so that the log data under the value in the queried field can be extracted from the first target cluster as the query return result.

[0094] The query result sorting conditions and the number of returned log entries are extracted from the query information; the query result sorting conditions are used to indicate the sorting method of the query results, such as sorting by the timestamp of the log; the filtering conditions can be used to indicate the filtering method of the query results, such as filtering out duplicate log data; the number of returned log entries can be used to indicate the number of returned log data, which can be pre-set or specified by the account that triggers the query request, and this application does not limit this.

[0095] Furthermore, the filter conditions and query result sorting conditions can be used to filter and sort multiple query return results to obtain filtered and sorted query results; and the query result of the number of returned log entries can be extracted from the filtered and sorted query results as the log query results.

[0096] As an example, see Figure 3 , taking the preset syntax parsing tool as Antlr4 and the preset query syntax as Lucene syntax as an example, the process of log query for heterogeneous clusters based on query requests is introduced as follows.

[0097] 1. Users can input Lucene syntax query statements through the front end or HttpClient, and construct query requests to the query API;

[0098] 2. The query API receives the query request. The server corresponding to the query API can determine whether the query statement is in Lucene syntax. If so, the query request can be parsed, parameter verification can be performed, and query information can be obtained based on Antlr4. If not, it means that the query information directly input is in a unified query format, and query request parsing is not necessary. The specific parsing process can be as follows:

[0099] 1) Use Antlr4 to build a syntax tree based on the input query statement.

[0100] 2) Traverse the syntax tree to extract the log query indication information therein, which may include, for example: query time range, number of data to be returned, logical expression of query filter condition, query sorting condition, etc.

[0101] 3) The above log query indication information is converted into a data format according to a unified query format to obtain query information in a unified query format. The unified query format may include multiple structured fields, such as the query time range corresponding to the start time StartTime and end time EndTime fields, the number of data to be returned corresponding to the Limit and Offset fields, the query filter condition corresponding to the Filters field, and the query sorting condition corresponding to the Order and By fields, wherein the Filters field includes the field Name and Values ​​in the data table to be queried.

[0102] Optionally, each field of the query information may be subjected to a validity check and an SQL injection check. If an abnormality is found, an error may be returned and a log may be recorded.

[0103] 3. Query routing: Based on the account that triggers the query request, the query log type, and the query time range, search in the query routing table to obtain at least one matching first target cluster, that is, the cluster to be queried.

[0104] 4. Traverse the target query syntax corresponding to each cluster to be queried, and convert the query information into the target query statement corresponding to each cluster to be queried.

[0105] 5. Concurrently query the clusters to be queried. After all queries are returned, you can determine whether the number of the first target cluster is multiple. If it is one, you can directly return the log query result. If it is multiple, you need to merge the returned results to get the log query result.

[0106] Exemplarily, the query information converted into the unified query format may be as follows:

[0107]

[0108]

[0109] Exemplarily, if the first target cluster is CH, the above query information is converted into the SQL query statement corresponding to CH (target query statement), that is, the CH SQL statement can be: SELECT*FROM netflow WHERE protocol like%UDP%or protocol like'%TCP%'AND direction IN(1,0,3)AND timestamp>'2022-07-27 16:00:00'AND timestamp<'2022-07-27 17:00:00'ORDER BY start_time LIMIT2OFFSET 0.

[0110] By setting a unified query interface and configuring a preset query grammar for the unified query interface, the unified query interface is used to uniformly query the log data of multiple heterogeneous clusters, so that users can trigger a unified query request for multiple heterogeneous clusters based on the query statement of the preset query grammar, and can convert the query statement carried by the query request into query information in a unified query format, and determine at least one first target cluster that the query request needs to query from multiple heterogeneous clusters based on the query information, so that the query information can be converted into a target query statement corresponding to at least one first target cluster based on the target query grammar corresponding to each first target cluster, and the target query statement is used to query log data in the corresponding first target cluster to obtain a log query result. Users can perform cross-cluster unified log queries on multiple heterogeneous clusters based on the query statement of the preset query grammar, without having to query multiple heterogeneous clusters separately, thereby eliminating the need for multiple queries, and having high query efficiency, which can meet the needs of high-performance analysis queries of large-scale data sets, and has low query resource consumption; and when adding a new cluster, there is no need to redesign the query interface, which has good scalability, and has low development and maintenance costs for log queries.

[0111] The above is an introduction to the query function of log data. The following is an introduction to the storage of log data. Based on this, the method may also include:

[0112] S401, obtaining original log data to be stored.

[0113] like Figure 5 As shown, the original log data may refer to the log data delivered by the business party or user to an intermediate storage system such as Kafka, ES, or MySQL.

[0114] Reference Figure 5 , log storage can be implemented based on ETL, which can include extraction (Extractor), transformation (Transformer) and loading (Loader), as well as some external dependencies, such as routing (Router), custom transformation function (TransFunc), TCP client (TCPClient) and TCP service (TCPServer) and other functional modules.

[0115] Among them, Extractor can be responsible for reading raw log data from intermediate storage systems (such as Kafka, MySQL, ES, etc.) where different data sources are located, and can pass the read raw log data to Transformer for further processing. Correspondingly, Transformer can be responsible for pre-processing such as cleaning and normalization of the data. For example, it can be customized through custom transformation functions, and data can also be processed by implementing local TCP services and through RPC calls. Loader can be responsible for writing pre-processed log data to storage clusters, such as ES and CH, according to Router routing information. Optionally, ETL can also include the above-mentioned intermediate storage system that provides the data source, such as Figure 1 Part of the introduction, as the consumption of log data provided by ETL, this application does not limit this.

[0116] S403: performing data preprocessing on the original log data to obtain preprocessed log data.

[0117] In the embodiments of the present specification, data preprocessing may be performed on the original log data, such as data cleaning, filtering, etc., to obtain preprocessed log data.

[0118] In a possible implementation, the above-mentioned data preprocessing of the original log data to obtain the preprocessed log data may include: performing data cleaning on the original log data to obtain the first log data; and supplementing the first log data with associated data according to a preset data supplementation strategy to obtain the second log data; illustratively, the preset data supplementation strategy may be supplementing geographic location information, etc., which is not limited in this application. And the second log data may be normalized to obtain the preprocessed log data.

[0119] S405 , determining at least one second target cluster for storing the preprocessed log data; the at least one second target cluster belongs to the multiple heterogeneous clusters.

[0120] In an embodiment of the present specification, at least one second target cluster for storing preprocessed log data may be determined from multiple heterogeneous clusters. In a possible implementation, the above-mentioned determination of at least one second target cluster for storing preprocessed log data may include:

[0121] Determine the second account, storage log type, and log timestamp corresponding to the preprocessed log data;

[0122] Based on the second account, storage log type and log timestamp, at least one matching second target routing identifier is found in the storage routing table; the storage routing table may store a correspondence between the account, log type, timestamp and routing identifier.

[0123] Furthermore, the routing identifiers preconfigured in the multiple heterogeneous clusters may be used to determine a cluster corresponding to each of the at least one second target routing identifier from the multiple heterogeneous clusters as the at least one second target cluster.

[0124] S407, sending the preprocessed log data to a write buffer corresponding to the second target cluster;

[0125] S409: When the write buffer satisfies the write condition, write the preprocessed log data into the corresponding second target cluster.

[0126] In the embodiment of this specification, the preprocessed log data can be sent to the write buffer corresponding to the second target cluster, so that it can be monitored whether the write buffer meets the write condition. If the write buffer meets the write condition, the preprocessed log data is written to the corresponding second target cluster.

[0127] In an example, the write condition may include, but is not limited to, the write buffer being full, reaching a specified time, etc., which is not limited in the present application.

[0128] Reference Figure 5 and Figure 6 , the storage process of log data can be as follows:

[0129] 1. Business parties or users deliver data to intermediate storage systems such as Kafka, ES, and MySQL.

[0130] 2. The Extractor module reads the original log data from the above intermediate storage system. Different intermediate storage systems have different reading methods. For example:

[0131] 1) When reading Kafka, you can consume Kafka message queue data in batches through the Kafka Client.

[0132] 2) When reading ES, you can traverse the ES index through the ES scroll interface.

[0133] 3. The Extractor can send the acquired raw log data to the downstream Transformer module for processing one by one.

[0134] 4. The Transformer module processes the received raw log data one by one and can perform the following preprocessing steps:

[0135] 1) According to the preset basic filtering rules, filter out abnormal log data with problems such as missing fields and field value overflow to clean the original log data.

[0136] 2) Supplement data according to the preset data supplement strategy, such as supplementing IP geographic location information, querying data related information such as the machine where the data is located, etc., to supplement the original log data.

[0137] 3) Normalize data to ensure that the data format is unified, each field complies with the predefined specifications, can be correctly written into the storage cluster, and facilitate subsequent unified queries.

[0138] Optionally, the log data may be filtered based on filtering rules configured by the user or administrator and matching data content.

[0139] Furthermore, the preprocessed log data can be sent to the Loader.

[0140] 5. Loader receives the pre-processed log data sent by Transformer and can call the Router interface to obtain routing information, such as the storage routing table, so that it can determine which storage cluster or clusters the log data should be written to based on the storage routing table query, and then send the log data to the write buffer of the corresponding cluster. Accordingly, when the write buffer is full or after waiting for a given time, each cluster will uniformly write the log data in the buffer to the corresponding cluster.

[0141] Figure 7 FIG. 1 is a block diagram of a multi-cluster log data processing device according to an exemplary embodiment. Figure 7 , the device may include:

[0142] A query response module 701 is used to respond to a query request for log data and convert the query statement carried in the query request into query information in a unified query format; the query request is triggered based on a unified query interface, the query syntax corresponding to the query statement is a preset query syntax configured by the unified query interface, and the unified query interface is used to uniformly query the log data of multiple heterogeneous clusters;

[0143] A first target cluster determination module 703, configured to determine, based on the query information, at least one first target cluster to be queried by the query request from the multiple heterogeneous clusters;

[0144] A target query grammar acquisition module 705 is used to acquire a target query grammar corresponding to each of the at least one first target cluster;

[0145] A query statement conversion module 707, configured to convert the query information into a target query statement corresponding to each of the at least one first target cluster based on the target query grammar;

[0146] The log query result acquisition module 709 is used to use the target query statement to perform log data query in the corresponding first target cluster to obtain a log query result.

[0147] According to the log data processing device of the present application, a unified query interface is set and a preset query grammar is configured for the unified query interface. The unified query interface is used to uniformly query the log data of multiple heterogeneous clusters, so that the user can trigger a unified query request for multiple heterogeneous clusters based on the query statement of the preset query grammar, and the query statement carried by the query request can be converted into query information in a unified query format, and at least one first target cluster that the query request needs to query is determined from multiple heterogeneous clusters based on the query information, so that the query information can be converted into a target query statement corresponding to each of the at least one first target cluster based on the target query grammar corresponding to each first target cluster, and the target query statement is used to query the log data in the corresponding first target cluster to obtain the log query result. The user can perform a unified log query across clusters for multiple heterogeneous clusters based on the query statement of the preset query grammar, without having to query multiple heterogeneous clusters separately, thereby not having to query multiple times, with high query efficiency, being able to meet the needs of high-performance analysis query of large-scale data sets, and having low query resource consumption; and when adding a new cluster, there is no need to redesign the query interface, with good scalability, so that the development and maintenance costs of log queries are low.

[0148] In a possible implementation, the query response module 701 may include:

[0149] A syntax tree construction unit, used to construct a syntax tree of the query statement using a preset syntax parsing tool;

[0150] A first extraction unit, configured to traverse the syntax tree and extract log query indication information from the syntax tree;

[0151] a format conversion unit, configured to convert the log query indication information into a data format according to the unified query format to obtain the query information;

[0152] The query information includes at least one of the following: query log type, query time range, number of returned logs, filtering conditions, and query result sorting conditions.

[0153] In a possible implementation, the first target cluster determining module 703 may include:

[0154] A log type and time range extraction unit, used to extract the query log type and query time range from the query information;

[0155] A first account acquisition unit, used to acquire a first account that triggers the query request;

[0156] A query routing matching unit, configured to find at least one first target routing identifier that matches the query routing table based on the first account, the query log type, and the query time range; the first target routing identifier is used to identify the corresponding cluster; the query routing table stores the corresponding relationship between the account, log type, time range, and routing identifier;

[0157] The query cluster determination unit is configured to use the preconfigured routing identifiers of the multiple heterogeneous clusters to determine, from the multiple heterogeneous clusters, clusters corresponding to each of the at least one first target routing identifiers as the at least one first target cluster.

[0158] In a possible implementation, the at least one first target cluster is a plurality of first target clusters; the log query result acquisition module 709 may include:

[0159] A query return result acquisition unit, configured to concurrently query the multiple first target clusters using the target query statement to obtain multiple query return results;

[0160] A second extraction unit, used to extract the query result sorting conditions and the number of returned log entries from the query information;

[0161] A sorting unit, configured to sort the plurality of query return results using the query result sorting condition to obtain sorted query results;

[0162] The log query result acquisition unit is used to extract the query result of the number of returned log entries from the sorted query results as the log query result.

[0163] In a possible implementation, the log data processing device may further include:

[0164] The original log data acquisition module is used to acquire the original log data that needs to be stored;

[0165] A preprocessing module, used to perform data preprocessing on the original log data to obtain preprocessed log data;

[0166] A storage cluster determination module, configured to determine at least one second target cluster for storing the preprocessed log data; the at least one second target cluster belongs to the plurality of heterogeneous clusters;

[0167] A write cache processing module, used for sending the preprocessed log data to a write buffer corresponding to the second target cluster;

[0168] The log storage module is used to write the preprocessed log data into the corresponding second target cluster when the write buffer satisfies the write condition.

[0169] In a possible implementation, the storage cluster determination module may include:

[0170] A determination unit, configured to determine a second account number, a storage log type, and a log timestamp corresponding to the preprocessed log data;

[0171] A storage route matching unit, configured to find at least one second target route identifier that matches in a storage route table based on the second account, the storage log type and the log timestamp; the storage route table stores a correspondence between the account, log type, timestamp and route identifier;

[0172] The storage cluster determination unit is configured to use the preconfigured routing identifiers of the multiple heterogeneous clusters to determine, from the multiple heterogeneous clusters, clusters corresponding to each of the at least one second target routing identifiers as the at least one second target cluster.

[0173] In a possible implementation, the preprocessing module may include:

[0174] A data cleaning unit, configured to clean the original log data to obtain first log data;

[0175] A data supplement unit, configured to supplement the first log data with associated data according to a preset data supplement strategy to obtain second log data;

[0176] The normalization processing unit is used to perform normalization processing on the second log data to obtain the preprocessed log data.

[0177] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0178] Figure 8 is a block diagram of an electronic device for multi-cluster log data processing according to an exemplary embodiment. The electronic device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8As shown. The electronic device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for multi-cluster log data processing is implemented.

[0179] Those skilled in the art will understand that Figure 8 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0180] In an exemplary embodiment, an electronic device is also provided, including: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the multi-cluster log data processing method in the embodiment of the present application.

[0181] In an exemplary embodiment, a computer-readable storage medium is also provided, and when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the multi-cluster log data processing method in the embodiment of the present application. The computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0182] In an exemplary embodiment, a computer program product including instructions is also provided. When the computer program product is executed on a computer, the computer is enabled to execute the method for processing log data of multiple clusters in the embodiment of the present application.

[0183] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0184] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0185] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A multi-cluster log data processing method, characterized in that: include: In response to a query request for log data, converting a query statement carried in the query request into query information in a unified query format; The query request is triggered based on a unified query interface, the query syntax corresponding to the query statement is a preset query syntax configured by the unified query interface, and the unified query interface is used to uniformly query the log data of multiple heterogeneous clusters; Based on the query information, determining at least one first target cluster that needs to be queried by the query request from the multiple heterogeneous clusters; Obtaining a target query grammar corresponding to each of the at least one first target cluster; Based on the target query grammar, converting the query information into a target query statement corresponding to each of the at least one first target cluster; The target query statement is used to perform a log data query in the corresponding first target cluster to obtain a log query result.

2. The method according to claim 1, characterized in that The converting the query statement carried in the query request into query information in a unified query format includes: Using a preset syntax parsing tool to construct a syntax tree of the query statement; Traversing the syntax tree, and extracting log query indication information from the syntax tree; Convert the log query indication information into a data format according to the unified query format to obtain the query information; The query information includes at least one of the following: query log type, query time range, number of returned logs, filtering conditions, and query result sorting conditions.

3. The method according to claim 1, characterized in that The determining, based on the query information, at least one first target cluster to be queried by the query request from the multiple heterogeneous clusters includes: Extracting the query log type and the query time range from the query information; Obtaining a first account that triggers the query request; Based on the first account, the query log type and the query time range, at least one matching first target routing identifier is found in the query routing table; the first target routing identifier is used to identify the corresponding cluster; the query routing table stores the correspondence between the account, log type, time range and routing identifier; Using the preconfigured routing identifiers of the multiple heterogeneous clusters, clusters corresponding to the at least one first target routing identifier are determined from the multiple heterogeneous clusters as the at least one first target cluster.

4. The method according to claim 1, characterized in that: The at least one first target cluster is a plurality of first target clusters; The using the target query statement to query log data in the corresponding first target cluster to obtain a log query result includes: Concurrently query the multiple first target clusters using the target query statement to obtain multiple query return results; Extracting query result sorting conditions and the number of returned log entries from the query information; Using the query result sorting condition, sorting the multiple query return results to obtain sorted query results; The query result of the number of returned log entries is extracted from the sorted query results as the log query result.

5. The method according to claim 1, characterized in that The method further comprises: Get the original log data that needs to be stored; Performing data preprocessing on the original log data to obtain preprocessed log data; Determine at least one second target cluster for storing the preprocessed log data; the at least one second target cluster belongs to the multiple heterogeneous clusters; Sending the preprocessed log data to a write buffer corresponding to the second target cluster; When the write buffer satisfies the write condition, the preprocessed log data is written to the corresponding second target cluster.

6. The method according to claim 5, characterized in that The determining at least one second target cluster for storing the preprocessed log data includes: Determine a second account number, a storage log type, and a log timestamp corresponding to the preprocessed log data; Based on the second account, the storage log type and the log timestamp, find at least one second target routing identifier that matches in the storage routing table; the storage routing table stores a correspondence between the account, log type, timestamp and routing identifier; Using the preconfigured routing identifiers of the multiple heterogeneous clusters, clusters corresponding to the at least one second target routing identifier are determined from the multiple heterogeneous clusters as the at least one second target cluster.

7. The method according to claim 5, characterized in that The performing data preprocessing on the original log data to obtain preprocessed log data includes: Performing data cleaning on the original log data to obtain first log data; Supplementing the first log data with associated data according to a preset data supplementation strategy to obtain second log data; The second log data is normalized to obtain the preprocessed log data.

8. A multi-cluster log data processing system, characterized in that: include: A log query system, used to respond to a query request for log data and convert the query statement carried in the query request into query information in a unified query format; The query request is triggered based on a unified query interface, the query syntax corresponding to the query statement is a preset query syntax configured by the unified query interface, and the unified query interface is used to uniformly query the log data of multiple heterogeneous clusters; Backend service, used to save query routing table; The query routing table stores the correspondence between the account number, log type, time range and routing identifier; The log query system is further used to call the query routing table, extract the query log type and the query time range from the query information; obtain the first account that triggers the query request; based on the first account, the query log type and the query time range, find at least one matching first target routing identifier in the query routing table; and using the routing identifiers preconfigured by the multiple heterogeneous clusters, determining, from the multiple heterogeneous clusters, a cluster corresponding to each of the at least one first target routing identifiers as the at least one first target cluster; and converting the query information into a target query statement corresponding to each of the at least one first target cluster based on the target query grammar corresponding to each of the at least one first target cluster; The target query statement is used to perform a log data query in the corresponding first target cluster to obtain a log query result.

9. The system according to claim 8, characterized in that The log data processing system also includes: The log storage system is used to obtain the original log data to be stored; perform data preprocessing on the original log data to obtain preprocessed log data; The background service is used to save and store the routing table; the routing table stores the corresponding relationship between the account number, log type, timestamp and routing identifier; The log storage system is also used to determine the second account, storage log type and log timestamp corresponding to the preprocessed log data; based on the second account, the storage log type and the log timestamp, find at least one matching second target routing identifier in the storage routing table; and use the routing identifiers pre-configured by the multiple heterogeneous clusters to determine the cluster corresponding to each of the at least one second target routing identifiers from the multiple heterogeneous clusters as the at least one second target cluster; and send the preprocessed log data to the write buffer corresponding to the second target cluster; if the write buffer meets the write condition, write the preprocessed log data to the corresponding second target cluster.

10. A multi-cluster log data processing device, characterized in that: include: A query response module, configured to respond to a query request for log data and convert a query statement carried in the query request into query information in a unified query format; The query request is triggered based on a unified query interface, the query syntax corresponding to the query statement is a preset query syntax configured by the unified query interface, and the unified query interface is used to uniformly query the log data of multiple heterogeneous clusters; A first target cluster determination module, configured to determine, based on the query information, at least one first target cluster to be queried by the query request from among the multiple heterogeneous clusters; A target query grammar acquisition module, used to acquire the target query grammar corresponding to each of the at least one first target cluster; A query statement conversion module, configured to convert the query information into a target query statement corresponding to each of the at least one first target cluster based on the target query grammar; The log query result acquisition module is used to use the target query statement to perform log data query in the corresponding first target cluster to obtain a log query result.

11. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the log data processing method according to any one of claims 1 to 7.

12. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the log data processing method as claimed in any one of claims 1 to 7.

Images