Query processing methods, devices, equipment, storage media and program products

Abstract

This application relates to a query processing method, apparatus, device, storage medium, and program product, and pertains to the field of data management technology. The method includes: obtaining a query request; the query request indicating query information, the query information indicating at least two indexes and the corresponding data sources; obtaining table information based on the query request; the table information indicating configuration information corresponding to each of the at least two indexes; the configuration information indicating the attributes of the data corresponding to the indexes; obtaining query statements corresponding to each of the at least two indexes based on the query request and the table information; and executing a query operation based on the at least two query statements to obtain the query results of the query request. This application can improve the validity and completeness of the query results.

Description

Technical Field

[0001] This application relates to the field of data management technology, and in particular to a query processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] Log data is a type of semi-structured data that stores key information about system operation status, business processes, and other aspects.

[0003] Log data is distributed across different indexes using a distributed storage approach. Each index is responsible for storing a portion of the data, and these indexes can be distributed across different nodes in the data source cluster. In this technology, as business needs change, users can adjust the index configuration information, including adding, deleting, or modifying index fields. Each such change triggers a new index creation process—a rolling addition of a new index—ensuring that historical log data is not affected by new index configurations, while also providing the latest index structure for new log data.

[0004] However, each time the index configuration is changed, a new index is added on a rolling basis. Different indexes may have different schemes (i.e., table mapping) information, which may be inconsistent. Historical log data may not be able to support new queries, especially when it comes to complex queries, thus affecting the accuracy of the queries. Summary of the Invention

[0005] This application provides a query processing method, apparatus, device, storage medium, and program product, which can improve the effectiveness and completeness of query results; the technical solution is as follows:

[0006] According to one aspect of this application, a query processing method is provided, the method comprising:

[0007] Obtain a query request; the query request is used to indicate query information, the query information indicating at least two indexes and the data source corresponding to the indexes;

[0008] Based on the query request, table information is obtained; the table information is used to indicate the configuration information corresponding to the at least two indexes respectively; the configuration information is used to indicate the attributes of the data corresponding to the indexes;

[0009] Based on the query request and the table information, obtain the query statements corresponding to the at least two indexes respectively;

[0010] Execute a query operation based on at least two of the query statements to obtain the query results of the query request.

[0011] According to one aspect of this application, a query processing apparatus is provided, the apparatus comprising:

[0012] The first acquisition module is used to acquire a query request; the query request is used to indicate query information, and the query information indicates at least two indexes and the data source corresponding to the indexes;

[0013] The second acquisition module is used to acquire table information based on the query request; the table information is used to indicate the configuration information corresponding to the at least two indexes respectively; the configuration information is used to indicate the attributes of the data corresponding to the indexes;

[0014] The third acquisition module is used to acquire the query statements corresponding to the at least two indexes based on the query request and the table information.

[0015] The query module is used to perform a query operation based on at least two of the query statements to obtain the query results of the query request.

[0016] In some embodiments, the configuration information includes at least the field type corresponding to the index;

[0017] The third acquisition module is used to acquire the field types corresponding to the at least two indexes based on the table information;

[0018] The third acquisition module is used to acquire the query statements corresponding to the at least two indexes based on the query request and the field types corresponding to the at least two indexes respectively.

[0019] In some embodiments, the third acquisition module is configured to perform a query operation to acquire the query information based on the query request in response to the fact that the field types corresponding to the at least two indexes are different.

[0020] The third acquisition module is used to, in response to the fact that the field types corresponding to the at least two indexes do not satisfy the query operation, acquire the first query statements corresponding to the at least two indexes based on the query request; the query content in the first query statements of different indexes is different;

[0021] The third acquisition module is used to, in response to the query operation being satisfied by the field types corresponding to the at least two indexes, acquire the second query statements corresponding to the at least two indexes based on the query request; the query content in the second query statements of different indexes is the same.

[0022] In some embodiments, the query processing apparatus further includes: a fourth acquisition module, configured to, in response to the fact that the field types corresponding to the at least two indexes are the same, acquire a third query statement corresponding to the at least two indexes based on the query request; the query content in the third query statement of different indexes is the same.

[0023] In some embodiments, the query module is configured to send at least two query statements to the data source to obtain at least two query data output by the data source;

[0024] The query module is configured to perform type conversion processing on the at least two query data in response to the fact that the query content in at least two query statements is different, so as to obtain the at least two query data after processing.

[0025] The query module is used to obtain the query result of the query request based on the processed at least two query data.

[0026] In some embodiments, the query module is configured to, in response to the fact that the query content in at least two of the query statements is the same, obtain the query result of the query request based on the at least two query data.

[0027] According to another aspect of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one computer instruction, the at least one computer instruction being loaded and executed by the processor to implement the query processing method as described above.

[0028] According to another aspect of this application, a computer-readable storage medium is provided, wherein at least one computer instruction is stored therein, the at least one computer instruction being loaded and executed by a processor to implement the query processing method as described above.

[0029] According to another aspect of this application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium, wherein a processor reads from the computer-readable storage medium and executes the computer instructions to implement the query processing method described above.

[0030] The technical solutions provided in this application embodiment may have the following beneficial effects:

[0031] When the index configuration information in the data source changes, for a user-submitted query request for multiple heterogeneous indexes, the computer device can parse the query request statement to obtain the configuration information (such as data table information and field information), and then generate a query statement that matches the configuration information. The query statement is then used to execute the query operation on the user-specified data source to obtain the query results. The above solution considers the relevance between the configuration information corresponding to each heterogeneous index and the query statement, and can generate a query statement that matches the configuration information corresponding to each heterogeneous index, thereby improving the effectiveness and completeness of the query results. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is an architectural diagram of a computer system provided in an exemplary embodiment of this application;

[0034] Figure 2 This is a flowchart of a query processing method provided in an exemplary embodiment of this application;

[0035] Figure 3 This is an example diagram of an SQL query request provided in an exemplary embodiment of this application;

[0036] Figure 4 This is a flowchart of a query processing method provided in an exemplary embodiment of this application;

[0037] Figure 5 This is a flowchart of an exemplary embodiment of the SQL analysis method provided in this application;

[0038] Figure 6 This is a flowchart of an exemplary embodiment of the SQL analysis method provided in this application;

[0039] Figure 7 This is a block diagram illustrating a query processing apparatus according to an exemplary embodiment of this application;

[0040] Figure 8 This is a structural block diagram of a computer device provided in an exemplary embodiment of this application.

[0041] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0043] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0044] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0045] In this application embodiment, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0046] It should be understood that although the terms first, second, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, a first query statement may also be referred to as a second query statement, and similarly, a second query statement may also be referred to as a first query statement. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0047] The following is a definition of some terms used in this application:

[0048] 1) Index: An index is a data structure that helps improve the speed of specific types of queries in a database or log system. For log services, indexes enable quick location of log entries that meet certain criteria, thereby greatly improving retrieval efficiency.

[0049] 2) Abstract Syntax Tree (AST): A syntax tree is an abstract representation of the parsed query data from Structured Query Language (SQL). It is a hierarchical data structure that describes the syntactic structure of SQL statements through a tree-like structure. Each node represents an SQL operator or expression, such as table name, column name, condition, etc. In distributed databases, the syntax tree is the first step in query processing, generated through lexical and syntactic analysis.

[0050] 3) Logical Execution Plan: The execution plan, generated based on the syntax tree, describes the execution steps of a query. It is typically a directed acyclic graph (DAG) representing the data flow and operation sequence. The execution plan does not concern itself with specific physical storage locations but focuses on how to efficiently complete the query. The optimizer performs various optimizations at this stage, such as predicate pushdown and pruning. In a distributed environment, the execution plan can be further broken down into multiple sub-plans (i.e., distributed execution plans) for parallel execution on different nodes. The distributed execution plan transforms the logical execution plan into a physical plan that can be executed in parallel on distributed nodes. This process includes mapping logical operations to physical operators on specific distributed nodes and distributing them according to the cluster topology.

[0051] Please refer to Figure 1 This illustration shows an architectural diagram of a computer system provided in an exemplary embodiment of this application. This computer system can be implemented as an architecture for a query processing configuration system. Figure 1 As shown, the computer system may include a terminal device 100 and a server 200. The terminal device 100 and the server 200 may be directly or indirectly connected via wired or wireless communication, and this application does not impose any restrictions on this connection.

[0052] Optionally, the terminal device 100 may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. A client application for the target application may be installed and run on the terminal device 100. This target application may be an application with query processing capabilities. This application does not limit the form of the target application; the target application includes, but is not limited to, applications (Apps), mini-programs, etc., installed on the terminal device 100, and may also be in web page form.

[0053] Optionally, server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services such as cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms. Among these, the cloud server for big data and artificial intelligence platforms can provide artificial intelligence cloud services. Server 200 can be a backend server for the aforementioned target application, used to provide backend services to the clients of the target application.

[0054] For example, a user can submit a query request through terminal device 100, which then sends the query request to server 200. Accordingly, server 200 receives the query request, which indicates query information, including at least two indexes and their corresponding data sources. Then, based on the query request, server 200 obtains table information, which indicates configuration information corresponding to each of the at least two indexes, including at least the field types corresponding to the indexes. Finally, based on the query request and the table information, server 200 obtains the query statements corresponding to each of the at least two indexes. Specifically, if the field types corresponding to the at least two indexes are different and do not satisfy the query operation, the query content in the query statements corresponding to the at least two indexes is different; if the field types corresponding to the at least two indexes are different and satisfy the query operation, the query content in the query statements corresponding to the at least two indexes is the same; and if the field types corresponding to the at least two indexes are the same, the query content in the query statements corresponding to the at least two indexes is the same. Then, server 200 executes a query operation based on at least two query statements to obtain the query results of the query request, and returns the query results to terminal device 100.

[0055] Please refer to Figure 2 The diagram illustrates a flowchart of a query processing method provided in an exemplary embodiment of this application. This method is executed by a computer device, which may optionally be a computer device... Figure 1 The terminal device 110 in the system shown; or, it could be server 120, or both terminal device 110 and server 120. For example... Figure 2 As shown, the method may include steps 210, 220, 230 and 240.

[0056] Step 210: Obtain the query request; the query request is used to indicate query information, which indicates at least two indexes and the data source corresponding to the indexes.

[0057] The aforementioned query request is a user-submitted query request for multiple heterogeneous indexes, used to query or retrieve specified data from a specified data source. The query information refers to the content the user wants to query, such as the value of a field, the number of records, or data in a field that meets a specified range.

[0058] The "at least two indexes" mentioned above refer to one or more indexes related to the content the user wants to query. Each index corresponds to at least one shard. The "data source" mentioned above refers to a data source associated with one or more indexes, where the data corresponding to the indexes is stored. For example, the data source may contain at least one cluster, a cluster may contain at least one data node, and a data node may contain at least one shard.

[0059] Optionally, the above query request is an SQL query request. Please refer to [link / reference]. Figure 3 The diagram illustrates an example of an SQL query request provided in an exemplary embodiment of this application.

[0060] like Figure 3 As shown, Example 1 of the user-submitted query request is: `select count(*)from es1.default.index1,es2.default.index2,es1.default.index3`, where `count(*)` indicates the number of records the user wants to query, `es1` and `es2` represent the data source, and `index1`, `index2`, and `index3` represent the multiple indexes involved in the query. Example 2 of the user-submitted query request is: `select *from es1.default.index1,index2,index3 where *>1`, where `*` indicates the value of a field the user wants to query, `where *>1` represents the query operation (calculation rule), `es1` represents the data source, and `index1`, `index2`, and `index3` represent the multiple indexes involved in the query.

[0061] Step 220: Based on the query request, obtain table information; the table information is used to indicate the configuration information corresponding to at least two indexes; the configuration information is used to indicate the attributes of the data corresponding to the indexes.

[0062] For example, a computer device can call a specified interface to obtain the above table information based on at least two indexes in the query request and the data source corresponding to the indexes.

[0063] For example, the table information mentioned above may include connection information for the data source and metadata information for the data corresponding to the index. For instance, the connection information for the data source includes basic parameters for accessing the data source, such as IP address, port number, username, and password. The metadata information for the data corresponding to the index may include data size and data distribution; specifically, for example, an index may contain 1 billion records, divided into 100 parts and stored on different data nodes.

[0064] The computer device can obtain the aforementioned configuration information from the data source through the table information. The attributes of the data corresponding to the index refer to the field information (such as field type) in the data (such as a table) corresponding to the index. For example, the aforementioned configuration information also indicates the storage location of the data corresponding to the index, including row storage, column storage, etc.

[0065] Step 230: Based on the query request and table information, obtain at least two query statements corresponding to each index.

[0066] The aforementioned query statements are used to retrieve the data required by the user from the data source corresponding to the query request. In this embodiment, the computer device generates query statements corresponding to each index based on the query information in the query request, i.e., the content the user wants to query. For example, the computer device constructs the aforementioned query statements using the query language supported by the aforementioned data source.

[0067] The query statement mentioned above corresponds to at least two indexes, and the table information is used to indicate the query content in the query statement. That is, one index corresponds to one query statement, and the query content in the query statements corresponding to different indexes can be the same or different. For example, if the data corresponding to at least two indexes has the same attributes, the query content in the query statements corresponding to at least two indexes can be the same; if the data corresponding to at least two indexes has different attributes, the query content in the query statements corresponding to at least two indexes can be different.

[0068] Step 240: Execute a query operation based on at least two query statements to obtain the query results of the query request.

[0069] The above query results refer to the data that the user wants to retrieve through the submitted query request, that is, the content corresponding to the query information.

[0070] For example, a computer device may send at least two query statements to the data source indicated in the query request to perform a query operation, and then obtain the query results of the query request based on the data output by the data source.

[0071] In some embodiments, to improve the efficiency of obtaining query results, the computer device can determine an execution plan for performing the query operation based on the amount of data to be queried. The execution plan indicates the task scheduling of at least two query statements. The task scheduling refers to the relationship between the at least two query statements and different worker nodes. For example, the computer device can determine the number of tasks responsible for processing an index based on the amount of data in the query corresponding to that index.

[0072] For example, a computer device can parse the above query request to generate a syntax tree corresponding to the query request. For instance, the computer device can first perform lexical analysis and syntax analysis on the above query request through a parser to generate a syntax tree, and then perform semantic analysis through an analyzer to optimize the generated syntax tree.

[0073] Next, the computer device can use a planner to generate a preliminary logical execution plan based on the syntax tree described above. Then, optimizers refine this preliminary execution plan to obtain an optimized logical execution plan, allowing the selection of the optimal path to retrieve the required data. Finally, the computer device can use a distributed planner to split the optimized logical execution plan into multiple distributed execution plans, enabling parallel execution of the query operation on different worker nodes.

[0074] Subsequently, the computer device, through the task scheduler, distributes at least two query statements to the corresponding worker nodes for execution based on the aforementioned task scheduling. Each worker node's task executor sends the corresponding query statement to the data source to obtain the query sub-results for each statement. The computer device then integrates these sub-results to obtain the final query result and returns it to the party that submitted the query request.

[0075] In summary, the solution presented in this application, when the index configuration information in the data source changes, allows the computer device to parse the query request statement to obtain configuration information (such as data table information and field information) for a user-submitted query request for multiple heterogeneous indexes. It then generates a query statement that matches the configuration information and executes the query operation on the user-specified data source to obtain the query results. This solution considers the correlation between the configuration information corresponding to each heterogeneous index and the query statement, enabling the generation of query statements that match the configuration information corresponding to each heterogeneous index, thereby improving the effectiveness and completeness of the query results.

[0076] Based on the above Figure 2 In one possible implementation of the scheme shown in the embodiments, the configuration information includes at least the field type corresponding to the index.

[0077] The field types mentioned above refer to the field types in the mapping, which determine the data type of each field and how it is processed.

[0078] For example, the above field types include at least one of the following:

[0079] Numeric types: used to store numerical data and support mathematical operations; for example, numeric types include integer types (such as INT, BIGINT), floating-point types (such as FLOAT, DOUBLE), and fixed-point types (such as DECIMAL, NUMERIC);

[0080] String type: Used to store text data; for example, string types include fixed-length strings (such as CHAR), variable-length strings (such as VARCHAR), and large text types (such as TEXT);

[0081] Time types: Used to store date and time information, ensuring that timestamps are recorded accurately and for easy retrieval later; for example, time types include date type (DATE), time type (TIME), date and time type (DATETIME), and timestamp type (TIMESTAMP);

[0082] Undefined type: This indicates that the field is missing or in an undefined state; for example, an undefined type is represented as NULL.

[0083] In one possible implementation, the above Figure 2 Step 230 in the illustrated embodiment can be implemented as steps 230a and 230b.

[0084] Step 230a: Based on the table information, obtain the field types corresponding to at least two indexes.

[0085] In this embodiment of the application, in order to determine whether the fields corresponding to at least two indexes have changed, the computer device can obtain the field types corresponding to at least two indexes based on the table information.

[0086] If at least two indexes correspond to fields of different types, it means that the fields corresponding to at least two indexes have changed; if at least two indexes correspond to fields of the same type, it means that the fields corresponding to at least two indexes have not changed.

[0087] Step 230b: Based on the query request and the field types corresponding to at least two indexes, obtain the query statements corresponding to at least two indexes.

[0088] In this embodiment of the application, since data of different field types may lead to different query behaviors, the computer device can determine the appropriate query statement for the field type corresponding to each index.

[0089] For example, the computer device first determines the query address in the query statement corresponding to each index based on at least two indexes in the query request and the data source corresponding to the indexes. Then, based on whether the field types corresponding to the at least two indexes have changed and the content that the user wants to query in the query request, it determines the query content in the query statement corresponding to each index.

[0090] Based on the above embodiments, this embodiment shows how to obtain query statements corresponding to at least two indexes based on query requests and table information. The above scheme can determine whether the fields corresponding to each index have changed by comparing the field types of each index, and can obtain query statements that match the field types of each index, thereby improving the accuracy of the query statements corresponding to each index and ensuring the completeness of the query results.

[0091] Based on the solutions shown in the above embodiments of this application, in one possible implementation, step 230c can be implemented as step 230c1, step 230c2 and step 230c3.

[0092] Step 230c1: In response to the fact that at least two indexes correspond to different field types, a query operation is performed to obtain query information based on the query request.

[0093] If at least two of the aforementioned indexes correspond to fields of different types, it indicates that the fields corresponding to at least two indexes have been changed. This means the user has adjusted the configuration information of one or more of the indexes, including but not limited to adding, deleting, or modifying fields. In this case, the computer needs to determine whether the changes to the fields corresponding to at least two indexes affect the query operation of the user's submitted query.

[0094] In this embodiment, the computer device can parse a user-submitted query request to obtain the aforementioned query operation. Here, the query operation refers to the processing information specified by the user, performed on the data corresponding to the two indexes. For example, the query operation can be a calculation rule, such as the one described above. Figure 3 The query operation can specify at least one field type, or it can specify at least one field type and the calculation rules for the data corresponding to at least one field type.

[0095] Step 230c2: In response to the fact that the field types corresponding to at least two indexes do not satisfy the query operation, based on the query request, obtain the first query statement corresponding to at least two indexes respectively; the query content in the first query statement of different indexes is different.

[0096] Among them, the field types corresponding to at least two of the above indexes do not meet the query operation, indicating the changed content of the fields corresponding to at least two of the indexes respectively, which affects the query operation of the query information submitted by the user. At this time, the computer device can determine the first query statement corresponding to each index according to the content that the user wants to query in the query request, that is, the query statements of each index are different.

[0097] Optionally, the field types corresponding to at least two of the above indexes do not meet the query operation, including at least: the field type corresponding to the first index does not conform to the field type specified by the query operation, and the field type corresponding to the first index is not an undefined type; the first index is at least one of the at least two indexes. That is to say, the field types corresponding to at least two of the indexes include field types not specified by the query operation, and the field type is in a defined state. For example, the field type specified by the query operation is INT, and the field type corresponding to a certain index is DOUBLE. At this time, the field type corresponding to this index does not conform to the field type specified by the query operation.

[0098] Exemplarily, the above first query statement is used to instruct the data source to output the data corresponding to each index indicated by the query operation; then, the processing module of the computer device performs the processing executed by the user on the data corresponding to each index respectively, and obtains the query result of the query information.

[0099] Specifically, for example, the query request A submitted by the user is: select a,b from es1.default.index1,index2 where time>=1737388800000000 AND time<1737475>200000000. Among them, the field a in index1 is of text type, and the field a in index2 is converted to double; the field b in index1 and index2 has not changed. Correspondingly, the query statement corresponding to index1 is:

[0100]

[0101]

[0102] The query statement corresponding to index2 is:

[0103]

[0104]

[0105] In the query statements corresponding to index1 and index2 above, the first line represents the query address, and the subsequent part represents the query content. In other words, the part of the query statement other than the query address corresponds to the query content.

[0106] Step 230c3: In response to the fact that the field types corresponding to at least two indexes satisfy the query operation, based on the query request, obtain the second query statements corresponding to at least two indexes; the query content in the second query statements of different indexes is the same.

[0107] Where at least two of the aforementioned indexes correspond to fields whose types satisfy the query operation, it means that although the fields corresponding to at least two indexes have changed, the changes do not affect the query operation of the user-submitted query information. In this case, the computer device can determine a second query statement corresponding to each index based on the content the user wants to query in the query request; that is, the query statements for each index are the same.

[0108] Optionally, the field types corresponding to at least two of the above indexes must satisfy the query operation, including at least one of the following:

[0109] The query operation did not specify the field type;

[0110] At least two indexes must have fields whose data types match the field types specified in the query operation.

[0111] The field type corresponding to the second index does not conform to the field type specified in the query operation, and the field type corresponding to the second index is an undefined type; the second index is at least one of at least two indexes.

[0112] The statement that the above query operation does not specify the field type means that the query request submitted by the user is unrelated to the field types corresponding to at least two indexes. Therefore, whether the field types corresponding to at least two indexes have changed or not does not affect the query operation.

[0113] The requirement that the field types corresponding to at least two indexes match the field types specified in the query operation means that the field types corresponding to at least two indexes are consistent with the field types specified in the query operation.

[0114] Specifically, the field type corresponding to the second index mentioned above does not match the field type specified in the query operation, and the field type corresponding to the second index is an undefined type. This means that at least two indexes contain field types that are not specified in the query operation, but these field types are in an undefined state and therefore do not affect the query operation. For example, the field type corresponding to the second index is NULL.

[0115] For example, the second query statement described above instructs the data source to output query results that satisfy the query operation based on the data corresponding to each index. In other words, the data source performs the user-executed processing on the data corresponding to each index to obtain the query results, without requiring the computer device's processing module to participate, thus saving computing device resources.

[0116] Based on the above embodiments, this embodiment provides a feasible solution for obtaining query statements that match the field types of at least two indexes when the corresponding field types are different. Specifically, it may include: determining whether the query statements corresponding to the at least two indexes are the same or different based on whether the changes to the fields corresponding to the at least two indexes affect the query operation of the user-submitted query information; if the changes do not affect the query operation, the query content in the query statements corresponding to each index is the same, which can save query resources and improve query speed; if the changes affect the query operation, the query content in the query statements corresponding to each index is different, which can ensure the accuracy of the query statements corresponding to each index and thus guarantee the completeness of the query results.

[0117] Based on the solutions shown in the above embodiments of this application, in one possible implementation, the query processing method further includes:

[0118] In response to the fact that at least two indexes correspond to fields of the same type, based on the query request, obtain the third query statement corresponding to at least two indexes respectively; the query content in the third query statement of different indexes is the same.

[0119] Where at least two of the aforementioned indexes correspond to fields of the same type, it means that the fields corresponding to at least two indexes have not changed. In this case, the computer device can determine a third query statement corresponding to each index based on the content the user wants to query in the query request. That is, the query content in the query statements of each index is the same, which can save query resources and improve query speed.

[0120] For example, the third query statement above is used to instruct the data source to output query results that satisfy the query operation based on the data corresponding to each index, without the need for the computer device's processing module to participate, thereby saving the computing device's operating resources.

[0121] For example, taking the user-submitted query request A above as an example, if fields a and b in index1 and index2 remain unchanged, the query statements corresponding to index1 and index2 are as follows:

[0122]

[0123]

[0124] Based on the solutions shown in the above embodiments of this application, in one possible implementation, the above... Figure 2 Step 240 can be implemented as steps 240a, 240b, and 240c.

[0125] Step 240a: Send at least two query statements to the data source and obtain at least two query data outputs from the data source.

[0126] In this embodiment, the computer device can distribute the query statement corresponding to the index to a specified node in the data source to obtain the query data corresponding to the query statement. Specifically, for example, the computer device determines the data node in the data source where the data corresponding to the index is located based on the partition corresponding to the index, and then sends the query statement corresponding to the index to that data node.

[0127] Step 240b: In response to the fact that the query content in at least two query statements is different, perform type conversion processing on at least two query data to obtain at least two processed query data.

[0128] In this embodiment, since different query statements are designed to adapt to different index configuration information, there may be differences in field types between the query data output by different query statements. To ensure that the field types of each query data are consistent with the field types of the latest data in each query data, the computer device can perform type conversion processing on at least two query data.

[0129] The aforementioned data conversion process refers to unifying the field types of all query data entries. In other words, it converts the field types of each query data entry to the field types of the most recent data within that query. Optionally, the computer device will convert the field types of the query data to an undefined state in at least one of the following situations: the query data type conversion fails; key-value retrieval is not enabled for the query data; or the query data is missing.

[0130] Step 240c: Based on at least two processed query data, obtain the query results of the query request.

[0131] In this embodiment, the computer device performs integration processing on the scattered query data to form a complete query result. This integration processing refers to the computer device combining at least two scattered query data sets into data that conforms to the query operation, thereby obtaining the query result.

[0132] For example, when the query operation is a calculation rule, the computer device will substitute at least two query data into the above calculation rule to obtain the query results of the query information.

[0133] Based on the above embodiments, this embodiment provides that when at least two query statements are different, the computer device can perform type conversion processing on each query data output by the data source to ensure that the field type of each query data is consistent with the field type of the latest data in each query data, thereby ensuring the rationality of the query results.

[0134] Based on the solutions shown in the above embodiments of this application, in one possible implementation, the query processing method further includes:

[0135] If at least two query statements contain the same query content, the query result of the query request is obtained based on at least two query data.

[0136] In this embodiment of the application, the fact that the query content in at least two query statements is the same means that the fields corresponding to at least two indexes have not changed, that is, at least two query data are the same.

[0137] For example, if the query information contains user-specified calculation rules and the query statement instructs the data source to output the data corresponding to each index, the computer device will substitute at least two query data into the above calculation rules to obtain the query results of the query information.

[0138] For example, if the query information contains user-specified calculation rules and the query statement instructs the data source to output query results that satisfy the query operation, the computer device will determine one of the query data as the query result of the query information.

[0139] Based on the above embodiments, this embodiment provides that when at least two query statements are identical, a computer device can directly determine the query result of the query information based on the query data output by the data source, without needing to perform type conversion processing, thereby saving query resources and improving query speed.

[0140] Based on the above embodiments, please refer to Figure 4 This illustrates a flowchart of a query processing method provided in an exemplary embodiment of this application. Figure 4 As shown, the method includes the following steps:

[0141] Step 401: Obtain the query request;

[0142] The query request is used to indicate query information, which indicates at least two indexes and the data source corresponding to the indexes;

[0143] Step 402: Obtain table information based on the query request;

[0144] The table information is used to indicate the configuration information corresponding to at least two indexes; the configuration information is used to indicate the attributes of the data corresponding to the indexes.

[0145] Step 403: Based on the query request, obtain the field types corresponding to at least two indexes;

[0146] Step 404: Determine whether the field types corresponding to at least two indexes are the same;

[0147] If they are the same, proceed to step 405; otherwise, proceed to steps 406 and 407.

[0148] Step 405: Based on the query request, obtain at least two third query statements corresponding to different indexes; the query content in the third query statements of different indexes is the same;

[0149] Then, send at least two third query statements corresponding to the indexes to the data source to obtain at least two query data output by the data source;

[0150] Then, based on at least two sets of query data, obtain the query results for the query request, and the process ends;

[0151] Step 406: Query operation to obtain query information based on the query request;

[0152] Step 407: Determine whether the field types corresponding to at least two indexes satisfy the query operation;

[0153] If the condition is met, proceed to step 408; otherwise, proceed to step 409.

[0154] Step 408: Based on the query request, obtain at least two second query statements corresponding to different indexes; the query content in the second query statements of different indexes is the same;

[0155] Then, send at least two second query statements corresponding to the indexes to the data source to obtain at least two query data output by the data source;

[0156] Then, based on at least two sets of query data, obtain the query results for the query request, and the process ends;

[0157] Step 409: Based on the query request, obtain at least two first query statements corresponding to different indexes; the query content in the first query statements of different indexes is different;

[0158] Then, send at least two first query statements corresponding to the indexes to the data source to obtain at least two query data output by the data source;

[0159] Then, perform type conversion on at least two query data sets to obtain at least two processed query data sets;

[0160] Then, based on at least two processed query data, the query results of the query request are obtained, and the process ends.

[0161] It should be noted that the execution methods of the above steps are the same as those of the above embodiments of this application, and will not be repeated here.

[0162] For example, based on Figure 2 For any one or more embodiments, this application proposes a SQL analysis method for heterogeneous indexes of data sources based on an SQL query engine. For semi-structured data such as logs, users frequently change index configuration information, and the data source needs to provide users with information on adding, deleting, and modifying index fields as needed. Previously, data sources faced the problem of historical data not supporting SQL analysis after changes. This method dynamically generates different Domain Specific Languages ​​(DSLs) based on index configuration information (mainly including field types, row storage, and column storage in the mapping) during the generation of the distributed plan by the SQL query engine. These DSLs are then sent to the data source for querying. After the SQL query engine's worker nodes retrieve the data, they process these different types of logic, transforming the data into a unified target type for analysis.

[0163] The SQL query engine is a tool specifically designed and developed for real-time querying and computation of big data. Its architecture consists of three main parts: a coordinator, a discovery service, and worker nodes. The coordinator is responsible for node management, SQL parsing, execution planning, task scheduling, and result aggregation; the discovery service is responsible for saving and synchronizing worker node states; and the worker nodes are responsible for computation and read / write operations.

[0164] In this context, the data source can be a full-text search engine. A data source cluster consists of multiple nodes, which can store data and participate in the cluster's indexing and search services. The logical concept that the data source provides to the outside world is an index. Each index consists of multiple shards, and each shard consists of multiple independent segments. Each segment is a complete and independent index.

[0165] This method can be used for SQL analysis within different schemas of a SQL query engine's federated queries, generating different query statements based on the structure of different tables. Log data is primarily stored in the data source, distributed across different indexes. Each change to the index configuration results in a new index being added. Different indexes may have different schemas (i.e., table mapping information).

[0166] Specifically, the configuration information mentioned above includes: whether to enable key-value indexes for fields, for example, setting the attribute index to true indicates that key-value indexes are enabled; whether to enable columnar storage, for example, setting the attribute doc_value to true indicates that columnar storage is enabled; and the type information of the fields, such as numeric type, string type, and time type.

[0167] Please refer to Figure 5 This illustrates a flowchart of an exemplary embodiment of the SQL analysis method provided in this application. Figure 5 As shown, firstly, the coordinator of the SQL query engine receives the SQL sent by the client. The SQL is a query request submitted by the user on the client side, specifying the query information the user wants to retrieve. This query information includes at least two indexes and their corresponding data sources. Based on these at least two indexes and their corresponding data sources in the query request, the coordinator calls a specified interface to obtain table information. This table information includes data source connection information and metadata information for the tables corresponding to the indexes.

[0168] The coordinator's parser performs lexical and syntactic analysis on the SQL to generate a syntax tree, while the coordinator's analyzer performs semantic analysis on the generated syntax tree and outputs an optimized syntax tree.

[0169] Next, the coordinator's planner generates a logical execution plan based on the optimized syntax tree. The coordinator's optimizers then optimize the logical execution plan so that the distributed planner can generate a distributed query plan.

[0170] The Distributed Planner generates a DSL corresponding to at least two indexes based on the metadata information of the tables corresponding to at least two indexes. If the field types of the tables corresponding to at least two indexes change, the query content in the DSLs corresponding to at least two indexes will be different; if the field types of the tables corresponding to at least two indexes do not change, the query content in the DSLs corresponding to at least two indexes will be the same.

[0171] Next, the task scheduler executes the distributed query plan, assigning task executors (e.g., task executor 0, task executor 1, task executor n) to each worker node for execution, and sending the corresponding DSL to the data source (e.g., cluster 1, cluster 2, etc.). Each worker node sends the queried data to its designated task executor (e.g., task executor m), which then sends the data back to the coordinator. The coordinator then processes the data into a complete query result and sends it to the client.

[0172] Before each worker node sends the queried data to the designated task executor, each worker node can perform compatibility processing on the queried data. Compatibility processing includes: each worker node performing type conversion on the queried data, filling NULL if the type conversion fails; filling NULL for fields where key-value retrieval is not enabled or for missing fields; in addition, to ensure SQL performance, data that can be calculated down is calculated down to the data source.

[0173] Please refer to Figure 6 This illustrates a flowchart of an exemplary embodiment of the SQL analysis method provided in this application. Figure 6 As shown, the user has three fields: code, level, and the system field __HOSTNAME__. Users will set the fields to different types (long, double, and text) according to different business scenarios. Some of the system fields may be of type keyword or date.

[0174] In the initial stage, key-value analysis was not enabled for the three fields code, level, and __HOSTNAME__ in index1.

[0175] During the first change phase, a new index, index2, is created. Specifically, indexes are enabled for code, level, and __HOSTNAME__.

[0176] In the second modification phase, index3 is generated. During this phase, the `code` type is changed from `text` to `double`, and the `level` type is changed from `long` to `text`.

[0177] For example, if a user's SQL query submitted on the client includes fields that have index changes, such as the SQL query: `select * from index1, index2, index3`, the SQL query engine sends different DSLs to each data node in the data source. Then, compatibility processing is performed on the retrieved data.

[0178] 1) Fill NULL values ​​in index1 where no key-value index is enabled, and retrieve the field: "docvalue_fields":[{"field":"__TIMESTAMP__"};

[0179] 2) Perform type conversion on the data retrieved from index2:

[0180] The `code` field: convert text to double, and fill with NULL if the conversion fails.

[0181] The `level` field: convert `long` to `text`;

[0182] The retrieved fields are: "docvalue_fields":[{"field":"__TIMESTAMP__"},{"field":"__HOSTNAME__"},{"field":"code"},{"field":"level"}];

[0183] 3) Retrieve all fields for index3:

[0184] "docvalue_fields":[{"field":"__TIMESTAMP__"},{"field":

[0185] "__HOSTNAME__"},{"field":"code"},{"field":"level"}].

[0186] If the SQL query submitted by the user on the client does not contain the fields to be changed, for example, the SQL query is: select count() from index1, index2, index3, then the pushdown to the data source calculation method is used. The same DSL is sent to each data node of the data source, and the data output by the data source is used as the SQL query result.

[0187] In summary, the method described in the above embodiments of this application can be applied to a SQL analysis scheme based on different schemas in a SQL query engine's federated query. Users can easily modify index configurations without writing complex SQL statements. On the one hand, this application supports SQL analysis of heterogeneous indexes of data sources without using complex federated queries; on the other hand, this application utilizes the columnar storage of data sources to improve the concurrency of query indexes and improve log SQL analysis performance by 10 times.

[0188] Please refer to Figure 7 The diagram illustrates a block diagram of a query processing apparatus according to an exemplary embodiment of this application, which can be used to perform, for example... Figure 2 In the method shown, all or part of the steps performed by the computer device are as follows: Figure 7 As shown, the device includes:

[0189] The first acquisition module 701 is used to acquire a query request; the query request is used to indicate query information, and the query information indicates at least two indexes and the data source corresponding to the indexes;

[0190] The second acquisition module 702 is used to acquire table information based on the query request; the table information is used to indicate the configuration information corresponding to at least two indexes; the configuration information is used to indicate the attributes of the data corresponding to the indexes;

[0191] The third acquisition module 703 is used to acquire at least two query statements corresponding to the indexes based on the query request and table information;

[0192] The query module 704 is used to perform a query operation based on at least two query statements to obtain the query results of the query request.

[0193] In some embodiments, the configuration information includes at least the field type corresponding to the index;

[0194] The third acquisition module 703 is used to obtain the field types corresponding to at least two indexes based on table information;

[0195] The third acquisition module 703 is used to acquire the query statements corresponding to at least two indexes based on the query request and the field types corresponding to at least two indexes respectively.

[0196] In some embodiments, the third acquisition module 703 is used to perform a query operation to acquire query information based on a query request in response to at least two indexes having different field types.

[0197] The third acquisition module 703 is used to, in response to the fact that the field types corresponding to at least two indexes do not meet the query operation, acquire the first query statement corresponding to at least two indexes based on the query request; the query content in the first query statement of different indexes is different;

[0198] The third acquisition module 703 is used to acquire the second query statement corresponding to at least two indexes based on the query request, in response to the fact that the field types corresponding to at least two indexes satisfy the query operation; the query content in the second query statement of different indexes is the same.

[0199] In some embodiments, the query processing apparatus further includes: a fourth acquisition module, configured to, in response to the fact that at least two indexes correspond to the same field type, acquire, based on a query request, a third query statement corresponding to at least two indexes; the query content in the third query statements of different indexes is the same.

[0200] In some embodiments, the query module 704 is used to send at least two query statements to the data source and obtain at least two query data output by the data source;

[0201] Query module 704 is used to perform type conversion processing on at least two query data in response to the fact that the query content in at least two query statements is different, so as to obtain at least two query data after processing.

[0202] The query module 704 is used to obtain the query results of the query request based on at least two processed query data.

[0203] In some embodiments, the query module 704 is configured to, in response to the fact that the query content in at least two query statements is the same, obtain the query result of the query request based on at least two query data.

[0204] It should be noted that the device provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules according to actual needs, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0205] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments of the relevant method; the technical effects achieved by each module performing its operation are the same as the technical effects in the embodiments of the relevant method, and will not be elaborated here.

[0206] Please refer to Figure 8 This diagram illustrates a structural block diagram of a computer device 800 according to an exemplary embodiment of this application. The computer device can be implemented as a server as described above in this application. The computer device 800 includes a Central Processing Unit (CPU) 801, a system memory 804 including Random Access Memory (RAM) 802 and Read-Only Memory (ROM) 803, and a system bus 805 connecting the system memory 804 and the CPU 801. The computer device 800 also includes a mass storage device 808 for storing an operating system 809, application programs 810, and other program modules 811.

[0207] The mass storage device 808 is connected to the central processing unit 801 via a mass storage controller (not shown) connected to the system bus 805. The mass storage device 808 and its associated computer-readable media provide non-volatile storage for the computer device 800. That is, the mass storage device 808 may include computer-readable media (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0208] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage medium is not limited to the above-mentioned types. The system memory 804 and mass storage device 808 described above can be collectively referred to as memory.

[0209] According to various embodiments of this disclosure, the computer device 800 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 800 can be connected to a network 808 via a network interface unit 807 connected to the system bus 805, or it can use the network interface unit 807 to connect to other types of networks or remote computer systems (not shown).

[0210] The memory also includes at least one computer instruction stored in the memory, and the central processing unit 801 executes the at least one computer instruction to implement all or part of the steps in the methods shown in the above embodiments.

[0211] In an exemplary embodiment, a chip is also provided, the chip including programmable circuitry and program instructions, which, when the chip is run on a computer device, are used to implement the query processing method described above.

[0212] In an exemplary embodiment, a computer program product is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions to implement the query processing method provided in the above-described method embodiments.

[0213] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores computer instructions that are loaded and executed by a processor to implement the query processing methods provided in the above-described method embodiments.

[0214] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0215] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0216] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method of query processing, the method comprising: The method includes: Obtain a query request; the query request is used to indicate query information, the query information indicating at least two indexes and the data source corresponding to the indexes; Based on the query request, table information is obtained; the table information is used to indicate the configuration information corresponding to the at least two indexes respectively; the configuration information is used to indicate the attributes of the data corresponding to the indexes; Based on the query request and the table information, obtain the query statements corresponding to the at least two indexes respectively; Execute a query operation based on at least two of the query statements to obtain the query results of the query request.

2. The method of claim 1, wherein, The configuration information includes at least the field types corresponding to the index; The step of obtaining the query statements corresponding to the at least two indexes based on the query request and the table information includes: Based on the table information, obtain the field types corresponding to the at least two indexes respectively; Based on the query request and the field types corresponding to the at least two indexes, obtain the query statements corresponding to the at least two indexes.

3. The method of claim 2, wherein, The step of obtaining the query statements corresponding to the at least two indexes based on the query request and the field types corresponding to the at least two indexes includes: In response to the fact that the field types corresponding to the at least two indexes are different, a query operation is performed to obtain the query information based on the query request; In response to the fact that the field types corresponding to the at least two indexes do not satisfy the query operation, the first query statement corresponding to the at least two indexes is obtained based on the query request; the query content in the first query statement of different indexes is different; In response to the fact that the field types corresponding to the at least two indexes satisfy the query operation, a second query statement corresponding to the at least two indexes is obtained based on the query request; the query content in the second query statements of different indexes is the same.

4. The method of claim 3, wherein, The method further includes: In response to the fact that the field types corresponding to the at least two indexes are the same, a third query statement corresponding to the at least two indexes is obtained based on the query request; the query content in the third query statement of different indexes is the same.

5. The method according to any one of claims 1 to 4, characterized in that, The step of performing a query operation based on at least two of the query statements to obtain the query results of the query request includes: Send at least two query statements to the data source and obtain at least two query data output by the data source; In response to the fact that the query content in at least two of the query statements is different, a type conversion process is performed on the at least two query data to obtain the at least two query data after processing. Based on the processed at least two query data, obtain the query result of the query request.

6. The method of claim 5, wherein, The method further includes: In response to the fact that the query content in at least two of the query statements is the same, the query result of the query request is obtained based on the at least two query data.

7. A query processing apparatus characterized by comprising: The device includes: The first acquisition module is used to acquire a query request; the query request is used to indicate query information, and the query information indicates at least two indexes and the data source corresponding to the indexes; The second acquisition module is used to acquire table information based on the query request; the table information is used to indicate the configuration information corresponding to the at least two indexes respectively; the configuration information is used to indicate the attributes of the data corresponding to the indexes; The third acquisition module is used to acquire the query statements corresponding to the at least two indexes based on the query request and the table information. The query module is used to perform a query operation based on at least two of the query statements to obtain the query results of the query request.

8. A computer device, comprising: The computer device includes a processor and a memory, the memory storing at least one computer instruction, which is loaded and executed by the processor to implement the query processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer instruction, which is loaded and executed by a processor to implement the query processing method as described in any one of claims 1 to 6.

10. A computer program product, characterised in that, The computer program product includes computer instructions stored in a computer-readable storage medium; the computer instructions are read and executed by a processor of a computer device to implement the query processing method as described in any one of claims 1 to 6.

Images