Multi-source data query analysis method, query engine, electronic device and storage medium

Abstract

The application provides a multi-source data query analysis method, a query engine, an electronic device and a storage medium, and belongs to the technical field of big data query and analysis. In the method, the query is directly pushed to a data source for execution, the pulling of full data is avoided, memory calculation is reduced, server resource occupation is reduced, and the parser interface is diversified. The parser interface of the fast query syntax supports a custom extended syntax, the compatibility of multiple query syntaxs is realized, the custom extended syntax is also supported, the customization of a query process is met, complex business rules can be adapted, the parser interface of the pipeline syntax can be connected to a pipeline syntax query statement, is compatible with a standardized pipeline, each interface in the query engine is pluggable design, any interface therein can be customized according to business requirements, and only interface semantics needs to be met, and the flexibility is high.

Description

Technical Field

[0001] This invention relates to the technical field of big data query and analysis, and in particular to a multi-source data query and analysis method, query engine, electronic device, and storage medium. Background Technology

[0002] In multi-source data analysis scenarios, the data is massive and comes from diverse sources. Traditional query engines (such as Presto and Impala) often first pull the data from each data source into memory when performing multi-source data (i.e., database) query analysis, and then perform correlation calculations and analysis in memory.

[0003] The above process relies on large memory computing, which incurs high resource consumption. When faced with large-scale data, it is prone to problems such as memory exhaustion and decreased computing efficiency. At the same time, traditional query engines are difficult to meet the requirements of modern query process customization (such as integrating enterprise-specific data filtering rules and business logic) and lightweight engine deployment (such as embedding into third-party systems). In addition, they cannot be compared with standardized pipelines (efficient tools such as Splunk and Kusto).

[0004] In summary, traditional multi-source data query and analysis processes suffer from high resource consumption, difficulty in handling high concurrency and massive data scenarios, inability to meet the customization requirements of query processes and adapt to complex business rules, and incompatibility with standardized pipelines such as Splunk and Kusto. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a multi-source data query and analysis method, query engine, electronic device and storage medium to alleviate the technical problems of traditional multi-source data query and analysis processes, such as high resource consumption, difficulty in dealing with high concurrency and massive data scenarios, inability to meet the customization of query processes, inability to adapt to complex business rules, and incompatibility with standardized pipelines such as Splunk and Kusto.

[0006] In a first aspect, embodiments of the present invention provide a multi-source data query and analysis method, applied to a query engine used as an SDK or running as a standalone service. The query engine includes: a pluggable parser interface, a pluggable executor interface, a pluggable normalizer interface, a pluggable cache interface, and a pluggable formatter interface. The method includes:

[0007] The parser interface parses the user-input query statement into a native database query statement adapted to the database. The parser interface includes: a parser interface for fast query syntax, a parser interface for advanced query SQL syntax, a parser interface for pipeline syntax, and a parser interface for natural language query. The parser interface for fast query syntax supports custom extended syntax.

[0008] The executor interface executes the corresponding database query based on the native database query statement to obtain the data query result;

[0009] The standardizer interface performs standardization processing on the data query results to obtain standardized data query results.

[0010] The caching interface caches the standardized data query results.

[0011] The formatter interface formats the standardized data query results output by the standardizer interface and / or all standardized data query results in the cache interface, and returns the formatted data query results to the front end for display.

[0012] Furthermore, the parser interface is used separately.

[0013] Furthermore, the syntax of the custom extension is used to express the query requirements of the front end; the syntax of the custom extension adopts JSON format, and the syntax of the custom extension supports query result set restrictions.

[0014] Furthermore, the restrictions on the query result set include:

[0015] If the user does not provide result restriction parameters, the query result set restrictions will remain unchanged;

[0016] If the original query information in the native database query statement does not have a limit on the number of results, and the user provides the result limit parameter, then the query result set limit of the result limit parameter is added to the native database query statement;

[0017] If the information in the original query contains a limit on the number of results, and the user provides the result limit parameter, then the smaller of the result limit and the result limit parameter will be used as the query result set limit for the native database query statement.

[0018] Furthermore, the parser interface of the pipeline syntax is modeled after the Kusto syntax. The original database query statement parsed by the pipeline syntax parser is an optimized original database query statement, which is a multi-level nested SQL subquery statement.

[0019] Furthermore, when the query statement is a pipe syntax query statement, the optimization methods used by the pipe syntax parser when parsing the pipe syntax query statement include:

[0020] The pipeline syntax query statement is parsed into an abstract syntax tree, wherein the abstract syntax tree includes multiple pipeline tasks;

[0021] Traverse the multiple pipeline tasks from back to front;

[0022] Obtain the current pipeline task and its adjacent pipeline tasks, and optimize the obtained current pipeline task and the adjacent pipeline tasks using the corresponding executor interface;

[0023] Determine whether the optimization was successful;

[0024] If successful, the current pipeline task is updated to obtain the updated pipeline task;

[0025] Determine whether the updated pipeline tasks have been reduced;

[0026] If the number is reduced, the updated pipeline task is treated as multiple pipeline tasks, and the step of traversing multiple pipeline tasks from back to front is returned.

[0027] If unsuccessful, or if the pointer is not reduced, move it forward;

[0028] Determine if the pointer is valid;

[0029] If valid, return to the step of obtaining the current pipeline task and the adjacent pipeline tasks of the current pipeline task;

[0030] If it is invalid, clear the current parsing cache result and use the finally obtained updated pipeline task as the optimized pipeline task;

[0031] The optimized native database query statement is determined based on the optimized pipeline task.

[0032] Furthermore, there are multiple actuator interfaces, and each actuator interface corresponds to a backend database.

[0033] Secondly, embodiments of the present invention also provide a query engine that can be used as an embedded SDK or run as a standalone service. The query engine includes: a pluggable parser interface, a pluggable executor interface, a pluggable normalizer interface, a pluggable cache interface, and a pluggable formatter interface.

[0034] The parser interface is used to parse the user-input query statement into a native database query statement adapted to the database. The parser interface includes: a parser interface for fast query syntax, a parser interface for advanced query SQL syntax, a parser interface for pipeline syntax, and a parser interface for natural language query. The parser interface for fast query syntax supports custom extended syntax.

[0035] The executor interface is used to execute the corresponding database query based on the native database query statement to obtain the data query result;

[0036] The standardizer interface is used to standardize the data query results to obtain standardized data query results.

[0037] The caching interface is used to cache the data query results after the standardization process;

[0038] The formatter interface is used to format the standardized data query results output by the standardizer interface and / or all standardized data query results in the cache interface, and return the formatted data query results to the front end for display.

[0039] Thirdly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described in any of the first aspects above.

[0040] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing machine-executable instructions, which, when invoked and executed by a processor, cause the processor to perform the method described in any of the first aspects above.

[0041] In this embodiment of the invention, a multi-source data query and analysis method is provided, applied to a query engine used as an SDK or running as a standalone service. The query engine includes: a pluggable parser interface, a pluggable executor interface, a pluggable normalizer interface, a pluggable cache interface, and a pluggable formatter interface. The method includes: the parser interface parses the user-input query statement into a native database query statement adapted to the database, wherein the parser interface includes: a parser interface for fast query syntax, a parser interface for advanced query SQL syntax, a parser interface for pipeline syntax, and a parser interface for natural language query; the parser interface for fast query syntax supports custom extended syntax; the executor interface executes the corresponding database query according to the native database query statement to obtain the data query result; the normalizer interface performs normalization processing on the data query result to obtain normalized data query result; the cache interface caches the normalized data query result; and the formatter interface formats the normalized data query result output by the normalizer interface and / or all normalized data query results in the cache interface, and returns the formatted data query result to the front end for display. As described above, the multi-source data query and analysis method of this invention executes the corresponding database query directly in the native database based on the native database query statement, that is, the query is directly pushed down to the data source for execution, avoiding the retrieval of the entire data, reducing memory computation, and lowering server resource consumption. In addition, the parser interface includes a parser interface for fast query syntax, a parser interface for advanced query SQL syntax, a parser interface for pipeline syntax, and a parser interface for natural language query. Among them, the parser interface for fast query syntax supports custom extended syntax, realizing compatibility with multiple query syntaxes, and also supports custom extended syntax to meet the customization of the query process and adapt to complex queries. In addition to business rules, the parser interface for pipeline syntax can interface with pipeline syntax query statements, ensuring compatibility with standardized pipelines. Furthermore, all interfaces in the query engine are pluggable, allowing for customization of any interface according to business needs, as long as the interface semantics are met. This provides high flexibility. Moreover, the query engine can be embedded as an SDK or run as a standalone service. The overall design is lightweight and easy to deploy, alleviating the technical problems of traditional multi-source data query and analysis processes, such as high resource consumption, difficulty in handling high concurrency and massive data scenarios, inability to meet the customization of query processes, adaptability to complex business rules, and incompatibility with standardized pipelines such as Splunk and Kusto. Attached Figure Description

[0042] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0043] Figure 1 A flowchart illustrating a multi-source data query and analysis method provided in an embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram of the structure of the query engine provided in an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of the interface semantic design of the parser interface provided in an embodiment of the present invention;

[0046] Figure 4 A schematic diagram of the interface semantic design of the actuator interface provided in an embodiment of the present invention;

[0047] Figure 5 A schematic diagram of the interface semantic design of the standardizer interface provided in an embodiment of the present invention;

[0048] Figure 6 A schematic diagram of the interface semantic design of the caching interface provided in an embodiment of the present invention;

[0049] Figure 7 A schematic diagram of the interface semantic design of the formatter interface provided in an embodiment of the present invention;

[0050] Figure 8 This is a schematic diagram illustrating the deployment of the query engine provided in an embodiment of the present invention;

[0051] Figure 9 A schematic diagram of JSON format provided for an embodiment of the present invention;

[0052] Figure 10 A schematic diagram of the optimizer process provided in an embodiment of the present invention;

[0053] Figure 11 This is a schematic diagram illustrating the registration of the actuator interface to the controller according to an embodiment of the present invention.

[0054] Figure 12 A schematic diagram of the actuator interface provided in an embodiment of the present invention;

[0055] Figure 13 A schematic diagram illustrating the definition of the logic implementation provided in this embodiment of the invention;

[0056] Figure 14This is a schematic diagram of AST information extraction provided in an embodiment of the present invention;

[0057] Figure 15 A schematic diagram of the structure sequence provided in an embodiment of the present invention;

[0058] Figure 16 This is a schematic diagram of the SqlQuery structure definition provided in an embodiment of the present invention;

[0059] Figure 17 This is a schematic diagram of the Rebuild merging structure provided in an embodiment of the present invention;

[0060] Figure 18 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0061] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] Traditional multi-source data query and analysis processes are resource-intensive, making it difficult to handle high concurrency and massive data scenarios, unable to meet the customization of query processes, adapt to complex business rules, and incompatible with standardized pipelines such as Splunk and Kusto.

[0063] Based on this, the multi-source data query and analysis method of this invention executes the corresponding database query directly in the native database according to the native database query statement, that is, the query is directly pushed down to the data source for execution, avoiding the retrieval of the full data, reducing memory computation, and reducing server resource consumption. In addition, the parser interface includes a parser interface for fast query syntax, a parser interface for advanced query SQL syntax, a parser interface for pipeline syntax, and a parser interface for natural language query. The parser interface for fast query syntax supports custom extended syntax, realizing compatibility of multiple query syntaxes, and also supports custom extended syntax to meet the customization of the query process and adapt to complex business rules. In addition, the parser interface for pipeline syntax can connect to pipeline syntax query statements and is compatible with standardized pipelines. Furthermore, all interfaces in the query engine are pluggable, and any interface can be customized according to business needs, as long as the interface semantics are met, resulting in high flexibility. Moreover, this query engine can be used as an SDK or run as an independent service. The overall design is lightweight and easy to deploy.

[0064] To facilitate understanding of this embodiment, a multi-source data query and analysis method disclosed in this embodiment of the invention will first be described in detail.

[0065] Example 1:

[0066] According to an embodiment of the present invention, an embodiment of a multi-source data query and analysis method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0067] Figure 1 This is a flowchart of a multi-source data query and analysis method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0068] Step S102: The parser interface parses the user-input query statement into a native database query statement adapted to the database. The parser interface includes: a parser interface for fast query syntax, a parser interface for advanced query SQL syntax, a parser interface for pipeline syntax, and a parser interface for natural language query. The parser interface for fast query syntax supports custom extended syntax.

[0069] In embodiments of the present invention, the above-described multi-source data query and analysis method can be applied to query engines used as an SDK embedding or running as a standalone service, see reference. Figure 2 The query engine includes: a pluggable parser interface, a pluggable executor interface, a pluggable normalizer interface, a pluggable cache interface, and a pluggable formatter interface. These interfaces can be customized as needed to meet specified interface semantics, offering good flexibility. Figure 2 In this context, query type custom parameters are present when the database table is created. Different query types correspond to different custom parameters. Field metadata is used to describe the distribution of the database, such as the included fields and field types.

[0070] The aforementioned parser interfaces (such as the Doris SQL converter) can mask storage differences and generate native database query statements adapted to different databases, which are then pushed down to each database for execution, as long as the specified interface semantics are met. For commonly used syntaxes, the built-in parser interfaces include: a parser interface for fast query syntax, a parser interface for advanced query SQL syntax, a parser interface for pipeline syntax, and a parser interface for natural language queries. The parser interface for fast query syntax supports custom extended syntax, and their corresponding query statements are custom syntax query statements, SQL syntax query statements, pipeline syntax query statements, and natural language query statements, respectively.

[0071] In the parser interface, the parser interfaces for advanced SQL query syntax, pipeline syntax, and natural language query can directly pass their own business parameters, while the parser interface for fast query syntax can implement customized parsing optimizations for specific business needs. The interface semantic design of the parser interface is as follows: Figure 3 As shown.

[0072] Step S104: The executor interface executes the corresponding database query based on the native database query statement to obtain the data query result;

[0073] Specifically, different executor interface implementations correspond to different backend database / big data systems. The interface semantic design of the executor interface is as follows: Figure 4 As shown.

[0074] Step S106: The normalizer interface performs standardization processing on the data query results to obtain standardized data query results;

[0075] Specifically, the standardized data query results facilitate subsequent standardization operations, such as sorting, and can be used with common formats like JSON / CSV. The standardization process can be customized using a standardizer interface, incorporating additional enrichment information, such as business details. The interface semantics of the standardizer interface are designed as follows: Figure 5 As shown.

[0076] Step S108: The cache interface caches the standardized data query results;

[0077] Specifically, this invention introduces a caching interface to uniformly optimize performance, especially in pagination scenarios. For large data storage, it prioritizes batch queries, caching the results in the caching interface, eliminating the need to request the database during pagination. Various storage interfaces can be implemented via the caching interface, such as memory, MongoDB, and PostgreSQL. The interface semantics of the caching interface are designed as follows: Figure 6 As shown.

[0078] In step S110, the formatter interface formats the standardized data query results output by the standardizer interface and / or all standardized data query results in the cache interface, and returns the formatted data query results to the front end for display.

[0079] Specifically, formatting processing is customized on demand, and the interface semantics of the formatter interface are designed as follows: Figure 7 As shown.

[0080] In this embodiment of the invention, a multi-source data query and analysis method is provided, applied to a query engine used as an SDK or running as a standalone service. The query engine includes: a pluggable parser interface, a pluggable executor interface, a pluggable normalizer interface, a pluggable cache interface, and a pluggable formatter interface. The method includes: the parser interface parses the user-input query statement into a native database query statement adapted to the database, wherein the parser interface includes: a parser interface for fast query syntax, a parser interface for advanced query SQL syntax, a parser interface for pipeline syntax, and a parser interface for natural language query; the parser interface for fast query syntax supports custom extended syntax; the executor interface executes the corresponding database query according to the native database query statement to obtain the data query result; the normalizer interface performs normalization processing on the data query result to obtain normalized data query result; the cache interface caches the normalized data query result; and the formatter interface formats the normalized data query result output by the normalizer interface and / or all normalized data query results in the cache interface, and returns the formatted data query result to the front end for display. As described above, the multi-source data query and analysis method of this invention executes the corresponding database query directly in the native database based on the native database query statement, that is, the query is directly pushed down to the data source for execution, avoiding the retrieval of the entire data, reducing memory computation, and lowering server resource consumption. In addition, the parser interface includes a parser interface for fast query syntax, a parser interface for advanced query SQL syntax, a parser interface for pipeline syntax, and a parser interface for natural language query. Among them, the parser interface for fast query syntax supports custom extended syntax, realizing compatibility with multiple query syntaxes, and also supports custom extended syntax to meet the customization of the query process and adapt to complex queries. In addition to business rules, the parser interface for pipeline syntax can interface with pipeline syntax query statements, ensuring compatibility with standardized pipelines. Furthermore, all interfaces in the query engine are pluggable, allowing for customization of any interface according to business needs, as long as the interface semantics are met. This provides high flexibility. Moreover, the query engine can be embedded as an SDK or run as a standalone service. The overall design is lightweight and easy to deploy, alleviating the technical problems of traditional multi-source data query and analysis processes, such as high resource consumption, difficulty in handling high concurrency and massive data scenarios, inability to meet the customization of query processes, adaptability to complex business rules, and incompatibility with standardized pipelines such as Splunk and Kusto.

[0081] The above provides a brief overview of the multi-source data query and analysis method of the present invention. The specific details involved are described in detail below.

[0082] In an alternative embodiment of the invention, the parser interface is used by a separate call.

[0083] Specifically, the query engine is designed to be very lightweight, such as... Figure 8 As shown, the supported deployment modes include: the parser interface is called separately; the query engine is used as an embedded SDK; and the query engine runs as a standalone service.

[0084] In an optional embodiment of the present invention, the custom extension syntax is used to express the query requirements of the front end; the custom extension syntax adopts JSON format, and the custom extension syntax supports query result set restrictions.

[0085] Specifically, queries on the client's front end typically involve simple combinations of conditions, lacking fixed business rules. Here, we customize a DSL (Domain Specific Language) to meet client query needs. This allows users to express query conditions in a structured and flexible way without having to write complex SQL or other database query code. The DSL can use JSON for quick queries of common functions (because it is easy to generate and parse, and can well represent hierarchical query conditions). It directly implements a built-in parser interface. Common JSON formats include... Figure 9 As shown.

[0086] In an optional embodiment of the present invention, the query result set limitation includes:

[0087] (1) If the user does not provide result restriction parameters, the query result set restrictions will remain unchanged;

[0088] (2) If the original query information in the native database query statement does not set a limit on the number of results, and the user provides a result limit parameter, then add a query result set limit to the native database query statement with a result limit parameter (that is, use the result limit parameter as a limit on the number of results in the original query information in the native database query statement).

[0089] (3) If the information in the original query contains a limit on the number of results and the user provides a result limit parameter, then the smaller of the result limit and the result limit parameter shall be used as the query result set limit of the native database query statement.

[0090] Specifically, the above-mentioned query result set limitation can reduce network and memory overhead. Specifically: The `result_limit` parameter can be passed to specify the maximum number of results returned by the query to prevent system overload. This requires the following conditions to be met: 1. If no limit is passed, the query remains unchanged; 2. If the current query does not have a limit, the passed `result_limit` is added by default; 3. If the current query has a limit, it is compared with `result_limit`, and the smaller value is taken.

[0091] The above process will be explained as follows:

[0092] 1. If no result_limit is passed, the query remains unchanged.

[0093] This means that if the caller (user) does not provide a `result_limit` parameter (i.e., no maximum desired number of results is specified), the system's default behavior is to not impose any limit on the number of query results. In other words, the query will execute as usual and may return any number of results.

[0094] 2. The current query does not have a limit; the query will add the passed-in result_limit by default.

[0095] If the current query does not specify a limit (meaning the original query itself does not limit the number of results returned), and the caller provides a `result_limit` parameter, then the system should automatically add a limit clause to the query with a value equal to the passed `result_limit`. This ensures that even if the original query was not designed to control the number of results, the passed parameter protects the system from overload.

[0096] 3. The current query has a limit; compare it with the result_limit and take the smaller value.

[0097] In some cases, the original query may already include a `LIMIT` clause specifying the maximum number of results to return. If the caller also provides a `result_limit`, then the stricter (i.e., smaller) limit should be chosen between the two. This is to ensure that regardless of how the original query is designed, the number of records returned to the user will not exceed `result_limit`, thus further preventing problems caused by excessive data volume.

[0098] In an optional embodiment of the present invention, the parser interface of the pipeline syntax is modeled after the Kusto syntax. The original database query statement parsed by the pipeline syntax parser is an optimized original database query statement, which is a multi-level nested SQL subquery statement.

[0099] Specifically, the pipeline syntax converts multi-level nested SQL subqueries and pushes them down to the data source for execution. The pipeline syntax is as follows:

[0100] select src_ip, dst_ip, datatime, log_type, dst_port from qt_eventwhere activity='net_connect'

[0101] | datatime>"now-1d"

[0102] | dst_port in (7001, 7002, 22) and log_type = 0

[0103] | group by src_ip, dst_ip

[0104] | group by src_ip

[0105] | order by cnt desc

[0106] The following is a multi-level nested SQL subquery statement (i.e., the optimized native database query statement):

[0107] SELECT src_ip, count(1) AS cnt

[0108] FROM (SELECT src_ip, dst_ip, count(1) AS cnt

[0109] FROM (SELECT src_ip, dst_ip, datatime, log_type, dst_port

[0110] FROM qt_event

[0111] WHERE activity = 'net_connect'

[0112] AND datatime>"2025-03-12 20:24:28.000"

[0113] AND dst_port IN (7001, 7002, 22)

[0114] AND log_type = 0) AS sub

[0115] GROUP BY src_ip, dst_ip) AS sub

[0116] GROUP BY src_ip

[0117] ORDER BY cnt DESC

[0118] This invention supports pipeline syntax while also achieving lightweight pipeline optimization. The core of this approach lies in using a multi-level merging strategy to combine adjacent pipelines and conditions as much as possible, ultimately transforming the pipeline into multi-level nested SQL subquery statements. In an optional embodiment of this invention, when the query statement is a pipeline syntax query statement, the optimization method used by the pipeline syntax parser when parsing the pipeline syntax query statement includes the following steps:

[0119] (1) Parse the pipeline syntax query statement into an abstract syntax tree, wherein the abstract syntax tree includes multiple pipeline tasks;

[0120] (2) Traverse multiple pipeline tasks from back to front;

[0121] (3) Obtain the current pipeline task and the adjacent pipeline tasks of the current pipeline task, and optimize the obtained current pipeline task and adjacent pipeline tasks using the corresponding executor interface;

[0122] (4) Determine whether the optimization was successful;

[0123] (5) If successful, update the current pipeline task and obtain the updated pipeline task;

[0124] (6) Determine whether the updated pipeline tasks have decreased;

[0125] (7) If the number of pipeline tasks is reduced, the updated pipeline tasks will be treated as multiple pipeline tasks, and the steps of traversing multiple pipeline tasks from back to front will be returned.

[0126] (8) If unsuccessful, or if the pointer is not reduced, move the pointer forward;

[0127] (9) Determine if the pointer is valid;

[0128] (10) If valid, return to the step of obtaining the current pipeline task and the adjacent pipeline tasks of the current pipeline task;

[0129] (11) If invalid, clear the current parsing cache result and use the finally obtained updated pipeline task as the optimized pipeline task;

[0130] (12) Determine the optimized native database query statement based on the optimized pipeline task.

[0131] refer to Figure 10 The controller manages the entire optimization process, iterating through all registered executor interfaces and merging adjacent pipeline logic. This process is polling, continuing until all executor interfaces have finished executing or optimization is no longer possible (all pipelines are merged together). The detailed process is as follows:

[0132] (1) Input and initialization:

[0133] Input AST (Abstract Syntax Tree) task list: The input to the optimizer process is an AST task list.

[0134] Controller initialization: Initialize the controller to prepare it for subsequent processing.

[0135] (2) Traversal and processing:

[0136] Traversing pipeline tasks from back to front: The optimizer starts from the back of the list and traverses the pipeline tasks in the pipeline backward.

[0137] (3) Task optimization:

[0138] Get adjacent pipeline tasks: During the traversal, get the current and adjacent pipeline tasks.

[0139] Try all executor interface optimizations: Try optimizing the obtained tasks using different executor interfaces (such as LimitExecutor, other executors, WhereExecutor, GroupByExecutor, SelectExecutor, OrderExecutor, etc.).

[0140] Determine if the optimization was successful: After each executor interface performs an optimization, it is necessary to determine whether the optimization was successful.

[0141] (4) Update and judgment:

[0142] Update Pipeline Tasks: If the optimization is successful, update the current pipeline tasks.

[0143] Task simplicity judgment: Determine whether the optimized task has become simpler (e.g., whether the task complexity has been reduced). If the task simplicity has not decreased, move the pointer forward to the next round of optimization.

[0144] (5) Loop and End:

[0145] Proceed to the next round of optimization: If the task simplicity decreases, continue optimizing; otherwise, move the pointer forward and proceed to the next round of optimization.

[0146] Is the pointer valid?: Check if the pointer is valid. If invalid, clean up the empty structure.

[0147] Return to optimized task: Finally, return the optimized pipeline task.

[0148] By dynamically selecting and executing the optimal executor interface, intelligent optimization of data processing tasks is achieved, improving resource utilization and execution efficiency. Furthermore, a back-to-foreign traversal approach ensures maximum optimization effectiveness.

[0149] In the above process, the controller coordinates the entire optimization process, using a strategy of traversing the pipeline tasks from back to front, attempting to optimize adjacent pipeline tasks. All executor interfaces to be optimized must be registered with the Controller for the optimization to take effect. For example... Figure 11 As shown.

[0150] Actuator interface such as Figure 12 As shown, multiple executors (i.e., executor interfaces) are supported, each tailored to a specific optimization scenario. All executors implement the above-described interface. Figure 12 The interface.

[0151] WhereExecutor: Merges the filtering conditions of adjacent pipelines, supporting the merging of complex condition trees.

[0152] GroupExecutor: Pushes grouping operations down to the data source and handles HAVING conditions.

[0153] OrderByExecutor: Pushes the sorting operation down to the data source and verifies that the sorting field exists in the projection.

[0154] LimitExecutor: Pushes limit operations down to the data source, optimizing paginated queries.

[0155] The logic implementation (ReBuild) involves controlling the merging logic within each executor. The basic definition of ReBuild is as follows: Figure 13 As shown.

[0156] Taking multi-stage pipeline merging as an example, the corresponding process is as follows:

[0157] AST information extraction, such as Figure 14 As shown, the above multi-level filtering conditions will be extracted into the following form through the parser interface: Figure 15A sequence of structures, where the SqlQuery structure is defined as follows: Figure 16 As shown, the Rebuild merge structure is as follows: Figure 17 As shown, the multi-level SqlQuery structure sequence, after passing through the FilterRebuild internal process, will eventually be merged into a single-level sequence. The final adjusted SqlQuery will then be output.

[0158] In an optional embodiment of the present invention, there are multiple executor interfaces, and each executor interface corresponds to a backend database.

[0159] The method of this invention has the following characteristics: a lightweight and scalable query engine; multiple syntax and rule expansion mechanisms; and a lightweight pipeline implementation algorithm adapted to industry standards.

[0160] The method of this invention directly sends user query filtering and aggregation conditions to the data source (e.g., database, big data platform) for execution, avoiding the need to fetch large amounts of data over the network for local computation and reducing memory computation. It has the following advantages:

[0161] Resource efficiency: Query pushdown reduces memory computation and lowers server resource consumption;

[0162] High flexibility: Supports customized business rules and adapts to diverse business processes and syntax;

[0163] Lightweight and compatible: The minimalist architecture enables lightweight deployment, is compatible with industry standard syntax, and improves versatility.

[0164] Example 2:

[0165] This invention also provides a query engine that can be used as an embedded SDK or run as a standalone service. This query engine is mainly used to execute the multi-source data query and analysis method provided in Embodiment 1 of this invention. The following is a detailed description of the query engine provided in this invention that can be used as an embedded SDK or run as a standalone service.

[0166] The query engine includes: a pluggable parser interface, a pluggable executor interface, a pluggable normalizer interface, a pluggable cache interface, and a pluggable formatter interface;

[0167] The parser interface is used to parse user-input query statements into native database query statements adapted to the database. The parser interface includes: a parser interface for fast query syntax, a parser interface for advanced query SQL syntax, a parser interface for pipeline syntax, and a parser interface for natural language query. The parser interface for fast query syntax supports custom extended syntax.

[0168] The executor interface is used to execute the corresponding database query based on the native database query statement and obtain the data query results.

[0169] The standardizer interface is used to standardize data query results to obtain standardized data query results.

[0170] The caching interface is used to cache the standardized data query results.

[0171] The formatter interface is used to format the standardized data query results output by the standardizer interface and / or all standardized data query results in the cache interface, and then return the formatted data query results to the front end for display.

[0172] In this embodiment of the invention, a query engine is provided that can be embedded as an SDK or run as a standalone service. The query engine includes: a pluggable parser interface, a pluggable executor interface, a pluggable normalizer interface, a pluggable cache interface, and a pluggable formatter interface. The parser interface parses the user-input query statement into a native database query statement adapted to the database. The parser interface includes: a parser interface for fast query syntax, a parser interface for advanced query SQL syntax, a parser interface for pipeline syntax, and a parser interface for natural language query. The parser interface for fast query syntax supports custom extended syntax. The executor interface executes the corresponding database query based on the native database query statement to obtain the data query result. The normalizer interface standardizes the data query result to obtain a standardized data query result. The cache interface caches the standardized data query result. The formatter interface formats the standardized data query result output by the normalizer interface and / or all standardized data query results in the cache interface, and returns the formatted data query result to the front end for display. As described above, the query engine of this invention executes the corresponding database query directly in the native database based on the native database query statement, that is, the query is directly pushed down to the data source for execution, avoiding the retrieval of the entire dataset, reducing memory computation, and lowering server resource consumption. Furthermore, the parser interface includes a parser interface for fast query syntax, a parser interface for advanced query SQL syntax, a parser interface for pipeline syntax, and a parser interface for natural language queries. The parser interface for fast query syntax supports custom extended syntax, achieving compatibility with multiple query syntaxes. It also supports custom extended syntax, satisfying the customization of the query process and adapting to complex business rules. Furthermore, the parser interface for pipeline syntax can interface with pipeline syntax query statements, ensuring compatibility with standardized pipelines. Additionally, all interfaces within the query engine are pluggable, allowing for customization based on business needs, as long as the interface semantics are met. This provides high flexibility. Moreover, the query engine can be embedded as an SDK or run as a standalone service. Its lightweight design facilitates easy deployment and alleviates the technical challenges of traditional multi-source data query and analysis processes, such as high resource consumption, difficulty in handling high concurrency and massive data scenarios, inability to customize query processes, failure to adapt to complex business rules, and incompatibility with standardized pipelines like Splunk and Kusto.

[0173] Alternatively, the parser interface can be used by calling it separately.

[0174] Optionally, the custom extension syntax is used to express the query requirements of the front end; the custom extension syntax adopts JSON format, and the custom extension syntax supports query result set restrictions.

[0175] Optionally, query result set restrictions include: if the user does not provide a result restriction parameter, the query result set restriction remains unchanged; if the information in the original query of the native database query statement does not set a result quantity restriction, and the user provides a result restriction parameter, then a query result set restriction with a result restriction parameter is added to the native database query statement; if the information in the original query contains a result quantity restriction, and the user provides a result restriction parameter, then the smaller of the result quantity restriction and the result restriction parameter is used as the query result set restriction of the native database query statement.

[0176] Optionally, the parser interface of the pipeline syntax is modeled after the Kusto syntax. The original database query statement parsed by the pipeline syntax parser is an optimized original database query statement, which is a multi-level nested SQL subquery statement.

[0177] Optionally, when the query statement is a pipeline syntax query statement, the optimizations employed by the pipeline syntax parser during parsing include: parsing the pipeline syntax query statement into an abstract syntax tree, where the abstract syntax tree includes multiple pipeline tasks; traversing multiple pipeline tasks from back to front; obtaining the current pipeline task and its adjacent pipeline tasks, and optimizing the obtained current pipeline task and adjacent pipeline tasks using the corresponding executor interface; determining whether the optimization is successful; if successful, updating the current pipeline task to obtain the updated pipeline task; determining whether the updated pipeline task has decreased; if decreased, treating the updated pipeline task as multiple pipeline tasks and returning to the step of traversing multiple pipeline tasks from back to front; if unsuccessful, or not decreased, moving the pointer forward; determining whether the pointer is valid; if valid, returning to the step of obtaining the current pipeline task and its adjacent pipeline tasks; if invalid, clearing the current parsing cache result and using the finally obtained updated pipeline task as the optimized pipeline task; and determining the optimized native database query statement based on the optimized pipeline task.

[0178] Optionally, there may be multiple executor interfaces, each corresponding to a backend database.

[0179] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0180] like Figure 18As shown in the embodiment of this application, an electronic device 600 includes a processor 601, a memory 602, and a bus. The memory 602 stores machine-readable instructions that can be executed by the processor 601. When the electronic device is running, the processor 601 communicates with the memory 602 via the bus, and the processor 601 executes the machine-readable instructions to perform the steps of the multi-source data query and analysis method described above.

[0181] Specifically, the memory 602 and processor 601 mentioned above can be general-purpose memory and processor, without any specific limitations. When the processor 601 runs the computer program stored in the memory 602, it can execute the above-mentioned multi-source data query and analysis method.

[0182] The processor 601 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 601 or by instructions in software form. The processor 601 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 602, and processor 601 reads the information from memory 602 and, in conjunction with its hardware, completes the steps of the above method.

[0183] Corresponding to the above-described multi-source data query and analysis method, this application embodiment also provides a computer-readable storage medium storing machine-executable instructions. When the machine-executable instructions are invoked and executed by a processor, the machine-executable instructions cause the processor to perform the steps of the above-described multi-source data query and analysis method.

[0184] The query engine provided in this application embodiment, whether used as an embedded SDK or running as a standalone service, can be specific hardware on a device or software or firmware installed on the device. The implementation principle and technical effects of the query engine provided in this application embodiment are the same as those in the foregoing method embodiments. For the sake of brevity, any parts not mentioned in the query engine embodiment can be referred to the corresponding content in the foregoing method embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the system, query engine, and units described above can all be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0185] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0186] For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0187] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0188] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0189] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the multi-source data query and analysis method described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0190] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0191] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A multi-source data query and analysis method, characterized in that, A query engine, applicable as an SDK embedded or running as a standalone service, includes: a pluggable parser interface, a pluggable executor interface, a pluggable normalizer interface, a pluggable cache interface, and a pluggable formatter interface; the method includes: The parser interface parses the user-input query statement into a native database query statement adapted to the database. The parser interface includes: a parser interface for fast query syntax, a parser interface for advanced query SQL syntax, a parser interface for pipeline syntax, and a parser interface for natural language query. The parser interface for fast query syntax supports custom extended syntax. The executor interface executes the corresponding database query based on the native database query statement to obtain the data query result; The standardizer interface performs standardization processing on the data query results to obtain standardized data query results. The caching interface caches the standardized data query results. The formatter interface formats the standardized data query results output by the standardizer interface and / or all standardized data query results in the cache interface, and returns the formatted data query results to the front end for display. The parser interface of the pipeline syntax is modeled after the Kusto syntax. The original database query statement parsed by the pipeline syntax parser is an optimized original database query statement, which is a multi-level nested SQL subquery statement.

2. The method according to claim 1, characterized in that, The parser interface is used independently.

3. The method according to claim 1, characterized in that, The syntax of the custom extension is used to express the query requirements of the front end; the syntax of the custom extension adopts JSON format, and the syntax of the custom extension supports query result set restrictions.

4. The method according to claim 3, characterized in that, The restrictions on the query result set include: If the user does not provide result restriction parameters, the query result set restrictions will remain unchanged; If the original query information in the native database query statement does not have a limit on the number of results, and the user provides the result limit parameter, then the query result set limit of the result limit parameter is added to the native database query statement; If the information in the original query contains a limit on the number of results, and the user provides the result limit parameter, then the smaller of the result limit and the result limit parameter will be used as the query result set limit for the native database query statement.

5. The method according to claim 1, characterized in that, When the query statement is a pipe syntax query statement, the optimization methods used by the pipe syntax parser when parsing the pipe syntax query statement include: The pipeline syntax query statement is parsed into an abstract syntax tree, wherein the abstract syntax tree includes multiple pipeline tasks; Traverse multiple pipeline tasks from back to front; Obtain the current pipeline task and its adjacent pipeline tasks, and optimize the obtained current pipeline task and the adjacent pipeline tasks using the corresponding executor interface; Determine whether the optimization was successful; If successful, the current pipeline task is updated to obtain the updated pipeline task; Determine whether the updated pipeline tasks have been reduced; If the number is reduced, the updated pipeline task is treated as multiple pipeline tasks, and the step of traversing multiple pipeline tasks from back to front is returned. If unsuccessful, or if the pointer is not reduced, move it forward; Determine if the pointer is valid; If valid, return to the step of obtaining the current pipeline task and the adjacent pipeline tasks of the current pipeline task; If it is invalid, clear the current parsing cache result and use the finally obtained updated pipeline task as the optimized pipeline task; The optimized native database query statement is determined based on the optimized pipeline task.

6. The method according to claim 1, characterized in that, There are multiple actuator interfaces, and each actuator interface corresponds to a backend database.

7. A query engine that can be used as an embedded SDK or run as a standalone service, characterized in that, The query engine includes: a pluggable parser interface, a pluggable executor interface, a pluggable normalizer interface, a pluggable cache interface, and a pluggable formatter interface; The parser interface is used to parse the user-input query statement into a native database query statement adapted to the database. The parser interface includes: a parser interface for fast query syntax, a parser interface for advanced query SQL syntax, a parser interface for pipeline syntax, and a parser interface for natural language query. The parser interface for fast query syntax supports custom extended syntax. The executor interface is used to execute the corresponding database query based on the native database query statement to obtain the data query result; The standardizer interface is used to standardize the data query results to obtain standardized data query results. The caching interface is used to cache the data query results after the standardization process; The formatter interface is used to format the standardized data query results output by the standardizer interface and / or all standardized data query results in the cache interface, and return the formatted data query results to the front end for display. The parser interface of the pipeline syntax is modeled after the Kusto syntax. The original database query statement parsed by the pipeline syntax parser is an optimized original database query statement, which is a multi-level nested SQL subquery statement.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores machine-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

Images