A query statement conversion method and device, electronic equipment and storage medium

Abstract

The application provides a query statement conversion method and device, electronic equipment and a storage medium. The method comprises the following steps: receiving a lucene query statement based on a custom EBNF; the lucene query statement comprises a plurality of sub-query statements and a Boolean operator; parsing the plurality of sub-query statements and the Boolean operator based on a syntax analysis library and the EBNF to obtain an abstract syntax tree (AST); and converting the AST into a real DSL statement according to an ES mapping. In the application, an attribute capable of converting the lucene query statement into the DSL statement is added in the ES mapping, the lucene query statement is parsed into the AST, and then the AST is converted into the DSL statement based on the ES mapping. Since the lucene query statement is relatively concise when being written, the user can easily write the lucene query statement, and after being converted into the DSL statement, the function originally not supported by the standard lucene can be added through the function of the DSL query statement itself.

Description

Technical Field

[0001] This application relates to the field of data retrieval technology, and more specifically, to a query statement conversion method, apparatus, electronic device, and storage medium. Background Technology

[0002] Elasticsearch (ES) is an excellent distributed search engine widely used in industry. Its Domain Specific Language (DSL) query statements can support a wide variety of search needs; however, the syntax of the DSL is relatively complex and prone to errors, thus reducing search efficiency. Summary of the Invention

[0003] The purpose of this application is to provide a query statement conversion method, apparatus, electronic device, and storage medium to improve retrieval efficiency.

[0004] In a first aspect, embodiments of this application provide a query statement conversion method, comprising: receiving a Lucene query statement based on a custom extended Backus normal form (EBNF); the Lucene query statement includes multiple subquery statements and Boolean operators; parsing the multiple subquery statements and Boolean operators based on a syntax parsing library and EBNF to obtain an abstract syntax tree (AST); and converting the AST into a DSL statement according to an ES mapping; wherein the ES mapping includes an attribute that converts the AST into a DSL statement.

[0005] This application embodiment parses the Lucene query statement into an abstract syntax tree, and then converts the abstract syntax tree into a DSL statement based on ES mapping. Since the Lucene query statement is relatively concise to write, it is easy for users to write. After being converted into a DSL statement, the functions of the DSL query statement itself can make the retrieval more efficient.

[0006] In any embodiment, the subquery statement includes a query form that transforms the AST into a DSL statement based on the ES mapping, including:

[0007] Starting from the leaf nodes, the AST nodes of each layer are converted into corresponding DSL nodes according to the ES mapping and query form, and the DSL nodes are returned to the previous layer AST nodes, until the conversion of all AST nodes is completed and the DSL statement is obtained.

[0008] In this embodiment, since the AST includes multiple layers of AST nodes, the AST nodes can be converted into DSL nodes layer by layer, starting from the bottom layer (i.e., the node layer corresponding to the leaf nodes), and the converted DSL nodes are returned to the parent node of the upper layer until all nodes in the AST have been converted. Finally, the DSL statement is obtained. Since Lucene statements are simple to write and DSL statements have many functions, the efficiency of retrieval is improved by combining the two.

[0009] In any embodiment, the AST includes multiple node layers, each node layer including at least one AST node; the AST node of each layer is converted into a corresponding DSL node according to the ES mapping and query form, including:

[0010] If the AST node corresponding to a node layer is a leaf node, obtain the field name corresponding to the leaf node;

[0011] If the field name is a field other than the preset field, then the target attribute corresponding to the field name is obtained from the ES mapping, and the corresponding target transformation function is obtained based on the target attribute;

[0012] Based on the target transformation function and query form, the leaf nodes are converted into intermediate DSL nodes, and the intermediate DSL nodes are stored in the cache;

[0013] Generate the corresponding DSL nodes for the node layer based on the intermediate DSL nodes in the cache.

[0014] This application embodiment converts the leaf nodes of the AST into intermediate DSL nodes. When converting to intermediate DSL nodes, the target attribute corresponding to the field name of the leaf node is obtained from the ES mapping. The target attribute provides a target conversion function, which can be used to convert the leaf node into a DSL node. Finally, the Lucene statement is converted into a DSL statement, which improves the efficiency of retrieval.

[0015] In any embodiment, the preset fields include _id and _exists_; if the field name is _id, the corresponding leaf node is converted to IdsQuery; if the field name is _exists_, the corresponding leaf node is converted to ExistsQuery.

[0016] In this embodiment of the application, when converting leaf nodes into DSL statements, if the field is a preset field, the conversion into DSL statements can be performed directly, which improves the conversion efficiency.

[0017] In any embodiment, storing intermediate DSL nodes in a cache includes:

[0018] If the cache contains a converted DSL node with the same field name as the intermediate DSL node, and the field name in ESmapping contains an aggregable attribute, then the intermediate DSL node and the converted DSL node are aggregated according to the logical operator corresponding to the intermediate DSL node and the aggregation method corresponding to the aggregable attribute, and the aggregated node is stored in the cache.

[0019] This application embodiment improves retrieval efficiency by expanding the fields in ES mapping to aggregate field names containing aggregateable attributes and intermediate DSL nodes with the same field names.

[0020] In any embodiment, the AST includes multiple node layers, each node layer including at least one AST node; the AST node of each layer is converted into a corresponding DSL node according to the ES mapping and query form, including:

[0021] If the AST node corresponding to a node layer is an operator node, then the DSL node sent by the AST node of the next layer is converted into a Bool Query; where the Bool Query is the DSL node corresponding to the node layer.

[0022] In this embodiment, by converting operator nodes into Bool Queries, a DSL query statement is finally obtained, and the DSL query statement is used for retrieval, thereby improving retrieval efficiency.

[0023] In any embodiment, the Boolean operators defined in the EBNF include lowercase forms. In this embodiment, a custom EBNF is used to allow Lucene queries conforming to this EBNF to use lowercase Boolean operators. Users do not need to frequently switch between uppercase and lowercase when writing Lucene queries, thus improving the efficiency of writing Lucene queries.

[0024] Secondly, embodiments of this application provide a query statement conversion device, including:

[0025] The receiving module is used to receive Lucene query statements based on a custom Extended Backus Normal Form (EBNF); the Lucene query statement includes multiple subqueries and Boolean operators.

[0026] The parsing module is used to parse multiple subquery statements and Boolean operators based on the syntax parsing library and EBNF to obtain an abstract syntax tree (AST).

[0027] The conversion module is used to convert the AST into DSL statements based on the ES mapping; wherein the ES mapping includes attributes for converting the lAST into DSL statements.

[0028] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a bus, wherein,

[0029] The processor and the memory communicate with each other via the bus;

[0030] The memory stores program instructions that can be executed by the processor, and the processor can execute the method of the first aspect by calling the program instructions.

[0031] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium, comprising:

[0032] The non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform the method of the first aspect.

[0033] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0034] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a flowchart illustrating a query statement conversion method provided in an embodiment of this application.

[0036] Figure 2 This is a flowchart illustrating a method for converting a leaf node into a DSL node, as provided in an embodiment of this application.

[0037] Figure 3 A schematic diagram of an AST structure provided in an embodiment of this application;

[0038] Figure 4 This is a schematic diagram of a query statement conversion device provided in an embodiment of this application;

[0039] Figure 5 This is a schematic diagram of a query statement conversion device provided in an embodiment of this application;

[0040] Figure 6This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0041] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0043] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0044] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0045] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0046] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0047] In the description of the embodiments of this application, the technical terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of this application and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application.

[0048] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0049] To facilitate understanding of the solutions in this application, the relevant concepts involved in this application are explained as follows:

[0050] ES is an abbreviation for Elasticsearch, referring to a distributed full-text search engine. It provides distributed, real-time file storage where every field is indexed and searchable; a distributed search engine for real-time analysis; and can scale to hundreds of servers to handle petabytes of structured or unstructured data.

[0051] ES Mapping: Defines the type and characteristics of each field in an ES document. It defines how a document and its contained fields are stored and indexed. A mapping consists of one or more analyzers, and each analyzer consists of one or more filters. When ES indexes a document, it passes the content of the fields to the corresponding analyzer, which then passes it to its respective filters.

[0052] The function of a filter is easy to understand: a filter is a method that transforms data. It takes a string as input and returns another string, such as a method that converts a string to lowercase.

[0053] An analyzer consists of a set of sequentially arranged filters. The analysis process involves calling each filter in turn, and Elasticsearch stores and indexes the final results.

[0054] Therefore, the role of mapping is to execute a series of instructions to transform the input data into searchable index items.

[0055] Lucene is a query language that supports two types of query terms: single query terms, such as "hello", and phrases, such as "hello world". The query can specify which field to search for the query term in; if not specified, it searches from the default field. However, it has the following problems: 1. Logical operators (AND / OR / NOT) are only available in uppercase, while most search terms are lowercase, requiring frequent case switching; 2. It does not support queries for non-existent fields; 3. Input compatibility is insufficient; for example, when searching for a boolean value, it only recognizes true / false, even though 1 or 0 should also represent true or false; 4. Range queries are slow when written as two sub-statements (e.g., x:>1 AND x:<2 is slower than x:{1TO 2}), as it does not aggregate sub-statements; 5. Fuzzy queries cannot customize the fuzziness level; 6. It filters unreasonable queries (e.g., querying fields without indexes, or using strings to query numeric fields).

[0056] DSL stands for Domain Specific Language, referring to a computer language focused on a specific application domain. In this context, it specifically refers to the query statements used by Elasticsearch. DSL queries interact with Elasticsearch using a JSON-formatted request body, enabling a wide variety of query requirements.

[0057] DSL queries mainly include two types of query statements:

[0058] (1) Leaf query statement: used to query special values ​​of special fields, such as match, term, range, etc.

[0059] (2) Compound query statement: It can combine other leaf queries or compound queries to achieve very complex query logic.

[0060] EBNF (Extended Backus–Naur Form) is a metasyntax notation used to describe the context-free syntax of formal languages ​​such as computer programming languages. In short, it is a language for describing languages. It is an extension of the Basic Backus Form (BNF) metasyntax notation.

[0061] An AST (Abstract Syntax Tree) is a tree-like representation of the abstract syntactic structure of source code. Here, it specifically refers to the source code of a programming language. Each node in the tree represents a structure within the source code. It's described as abstract because the AST doesn't represent every detail of the actual syntax; for example, nested parentheses are implicit in the tree structure and not presented as nodes.

[0062] Abstract syntax trees (ASTs) do not depend on the syntax of the source language, i.e., the context-agnostic grammar used in the parsing phase. This is because when writing grammars, equivalent transformations are frequently performed (eliminating left recursion, backtracking, ambiguity, etc.), which introduces redundant components into grammar analysis, negatively impacting subsequent phases and potentially causing confusion throughout the entire process. Therefore, many compilers often construct ASTs independently to establish a clear interface between the front-end and back-end.

[0063] While Lucene queries offer the advantage of concise writing, they also suffer from the aforementioned drawbacks. Therefore, using Lucene queries for retrieval cannot meet users' requirements in complex scenarios. DSL queries, on the other hand, offer powerful functionality, but their writing is more difficult and has a higher learning curve, resulting in low retrieval efficiency.

[0064] Through long-term research, the inventors of this application discovered that by analyzing the advantages and disadvantages of Lucene query statements and DSL query statements, the two can be combined. That is, users write query statements using Lucene, and the method of this application converts the Lucene query statements into DSL query statements, thereby satisfying the requirements of simple writing and the ability to use the powerful functions of DSL.

[0065] It is understood that the query statement conversion method provided in this application embodiment can be applied to electronic devices, which can be terminals or servers; the terminals can specifically be smartphones, tablets, computers, personal digital assistants (PDAs), etc.; the servers can specifically be application servers or web servers.

[0066] Figure 1This is a flowchart illustrating a query statement conversion method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes:

[0067] Step 101: Receive the Lucene query statement based on the custom Extended Backus Normal Form (EBNF); the Lucene query statement includes multiple subqueries and Boolean operators.

[0068] EBNF is a predefined syntax, which can be understood as the syntax of Lucene query statements. When writing Lucene query statements, users must meet the syntax requirements of the predefined EBNF.

[0069] As an example, the definition of EBNF is as follows:

[0070]

[0071]

[0072] Based on the custom EBNF described above, the Lucene query statements provided in this application embodiment support lowercase input of Boolean operators.

[0073] A Lucene query statement consists of multiple subqueries, which are then concatenated into a string using Boolean operators.

[0074] In addition, Lucene query statements can be entered by the user through the input port provided by the electronic device, or by the user through other input terminals that are connected to the electronic device and transmitted to the electronic device by the input terminal. This application embodiment does not specifically limit the way the electronic device receives Lucene query statements.

[0075] Step 102: Based on the syntax parsing library and EBNF, parse multiple subquery statements and Boolean operators to obtain an abstract syntax tree (AST).

[0076] In specific implementation, the parsing library can be Participle or other existing parsing libraries; this application does not specifically limit this. By inputting the Lucene query statement and EBNF into the parsing library, the parsing library can parse the Lucene query statement into an Abstract Syntax Tree (AST) based on the EBNF.

[0077] Step 103: Convert the AST into DSL statements based on the ES mapping; wherein the ES mapping includes attributes for converting the AST into DSL statements.

[0078] In the specific implementation process, attributes were pre-extended in the original ES mapping. Specifically, attributes were added under each field name in the ES mapping to guide the conversion of AST nodes into DSL queries. For example, an `ext_properties` structure could be added, where the key is a string representing an additional attribute, and the value is the value of that additional attribute. (Additional attributes are user-defined attributes beyond those defined by ES and various types; they are primarily used to guide the subsequent AST to DSL conversion). Below is an example with `ext_properties`:

[0079] {"mapping":{"properties":{ "foo":{ "type":"te xt", "ext_properties":{ "filter":true, / / Use filtering logic instead of query logic"only_upper_case":true, / / Whether uppercase is supported}

[0080] "input_field_change":"bar", / / Converts the input field to bar.

[0081] "enable_range":false / / The foo field does not support range queries.

[0082] In the example above, the ext_properties structure includes four additional properties, which, from top to bottom, represent the following: the sub-statement used for transformation using the filter DSL sub-statement (filter query); converting the passed terms to lowercase; converting the passed foo field to the bar field for querying; and the foo field does not support range queries, so if a range query is passed, it will be filtered in advance.

[0083] It should be noted that other additional attributes can be added to the ext_properties structure according to the actual situation, and one of the above additional attributes can also be deleted. The specifics can be determined according to the actual situation, and this application embodiment does not make specific limitations on this.

[0084] After parsing the Lucene query into an AST, the electronic device uses Elasticsearch mapping to convert the AST into a DSL statement. Based on the extended mapping above, if the input Lucene query is: foo:TERM, the conversion will result in the following DSL statement:

[0085] {"query":{"bool":{ "filter":{ "match":{

[0086] "bar":{ "query":"term"}}}

[0087] }}}

[0088] This application embodiment parses the Lucene query statement into an abstract syntax tree, and then converts the abstract syntax tree into a DSL statement based on ES mapping. Since the Lucene query statement is relatively concise to write, it is easy for users to write. After being converted into a DSL statement, the functions of the DSL query statement itself can make the retrieval more efficient.

[0089] Furthermore, by customizing EBNF, Lucene queries can support queries like `foo:null` for documents where the field `foo` does not exist, and queries like `not foo:null`, `foo:>1 OR foo:<=1`, and `foo:*` for documents where the `foo` field exists. Additionally, existing query requests can be modified according to business needs, for example: 1. If a field is numeric, it can support queries that accept strings containing numbers; 2. Querying is disabled for sensitive words; 3. Term content replacement.

[0090] Based on the above embodiments, the AST is converted into DSL statements according to the ES mapping, including:

[0091] Starting from the leaf nodes, the AST nodes of each layer are converted into corresponding DSL nodes according to the ES mapping with extended fields, and the DSL nodes are returned to the previous layer AST nodes, until the conversion of all AST nodes is completed and the DSL statement is obtained.

[0092] In the specific implementation process, since the AST includes parent nodes and leaf nodes, the topmost parent node can also be called the root node. When converting the AST into DSL statements, the electronic device can start traversing from the root node. Each time an AST node is traversed, it can be determined whether the AST node is a leaf node. If it is not a leaf node, the traversal continues until a leaf node is reached.

[0093] After traversing to the leaf node, starting from the leaf node, the AST node of each layer is converted into a DSL node, and the DSL node of that layer is uploaded to its parent node above it. The parent node then converts the node again based on the node uploaded from the lower layer, until the conversion of the root node is completed, and the DSL statement is obtained.

[0094] It is understandable that one can also directly traverse and transform from the leaf node upwards, and this application embodiment does not specifically limit this.

[0095] In this embodiment, since the AST includes multiple layers of AST nodes, the AST nodes can be converted into DSL nodes layer by layer, starting from the bottom layer (i.e., the node layer corresponding to the leaf nodes), and the converted DSL nodes are returned to the parent node of the upper layer until all nodes in the AST have been converted. Finally, the DSL statement is obtained. Since Lucene statements are simple to write and DSL statements have many functions, the efficiency of retrieval is improved by combining the two.

[0096] Based on the above embodiments, Figure 2 This is a flowchart illustrating a method for converting a leaf node into a DSL node, as provided in an embodiment of this application. Figure 2 As shown. The transformation of AST nodes in the node layer containing the leaf nodes in the above embodiments can be achieved through the following steps:

[0097] Step 201: Obtain the field names corresponding to the leaf nodes; It is understood that each subquery statement includes its corresponding query form, which represents the query type of the Lucene query statement, such as: single-term query (TermQuery), range query (RangeQuery), and fuzzy query (FuzzyQuery), etc. Fuzzy queries may have the form foo:bar~3, and regular expressions may have foo: / regular expression / . It is understood that subquery statements may also include other types, and this embodiment does not specifically limit them.

[0098] In the AST, leaf nodes correspond to the query field names in the Lucene query statement, and parent nodes correspond to the operators in the Lucene query statement, which are referred to as operator nodes in this embodiment. Therefore, leaf nodes include the corresponding field names.

[0099] Step 202: If the field name is a field other than a preset field, retrieve the target attribute corresponding to the field name from the ES mapping, and obtain the corresponding target transformation function based on the target attribute. Here, a preset field refers to a field whose name can be directly converted into a DSL statement, such as _id and _exists_. For preset fields, this embodiment can directly convert the corresponding leaf node into a DSL query statement. For example, when the preset field is _id, it can be converted into IdsQuery; when the preset field is _exists_, it can be converted into ExistsQuery.

[0100] For fields other than the preset fields, the target attribute corresponding to the field name needs to be obtained from the ES mapping. It can be understood that this target attribute is an extension of the ES mapping provided in the above embodiments. Furthermore, each target attribute has its corresponding executable file, which includes the transformation function corresponding to that target attribute. The transformation function represents the method for converting AST nodes into DSL nodes.

[0101] Step 203: Convert leaf nodes into intermediate DSL nodes according to the target transformation function and query form, and store the intermediate DSL nodes in the cache. In the specific implementation, a cache, also called a container, can be created for each layer of AST nodes. The purpose of this cache is to store the intermediate DSL nodes converted from the AST nodes of that layer. During the transformation, AST nodes are converted into intermediate DSL nodes according to the target transformation function and query form. Specifically, if the query form is an equality query, it can be converted to TermQuery; if the query form is a range query, it can be converted to RangeQuery; if the query form is a fuzzy query, it can be converted to FuzzyQuery, etc.

[0102] Step 204: Generate the DSL nodes corresponding to the node layer based on the intermediate DSL nodes in the cache.

[0103] After completing the transformation of the AST nodes for this layer, the intermediate DSL nodes in the cache are generated into Boolean type DSL nodes, i.e., BoolQuery. It should be noted that if the AST node of this layer contains a NOT operator, then the NOT operator needs to be combined when generating the DSL node.

[0104] This application embodiment converts the leaf nodes of the AST into intermediate DSL nodes. When converting to intermediate DSL nodes, the target attribute corresponding to the field name of the leaf node is obtained from the ES mapping. The target attribute provides a target conversion function, which can be used to convert the leaf node into a DSL node. Finally, the Lucene statement is converted into a DSL statement, which improves the efficiency of retrieval.

[0105] Based on the above embodiments, storing intermediate DSL nodes in a cache includes:

[0106] If the cache contains a converted DSL node with the same field name as the intermediate DSL node, and the field name in ESmapping contains aggregateable attributes, then the intermediate DSL node and the converted DSL node are aggregated according to the logical operator corresponding to the intermediate DSL node and the aggregation method corresponding to the aggregateable attributes, and the aggregated node is stored in the cache.

[0107] In practical implementation, if the database stores {"foo":[1,4]}, and the query is foo:>2 AND foo:<2, logically, it would aggregate into {"query":{"bool":{"must":{"range":{"foo":{"lt":2,"gt":2}}}}}}}. However, in reality, there is no number in the database that satisfies the requirement of being both greater than 2 and less than 2, so the stored data cannot be retrieved. To solve this problem, this application provides a method for aggregating AST nodes with the same field name. Specifically, it adds an attribute indicating whether aggregation is possible in the ES mapping. This attribute corresponds to the aggregation transformation function. If a field name can be aggregated, then the ES mapping contains an attribute indicating whether aggregation is possible. This attribute can be represented by is_conpact, for example: is_conpact=1 indicates aggregation is possible, and is_conpact=0 indicates aggregation is not possible. It is understood that other characters can also be used to indicate whether aggregation is possible, and this application does not specifically limit this. Additionally, if a field in the ES mapping does not have this attribute, it means that aggregation can be performed according to the existing aggregation method, i.e., the aggregation for the "foo" field mentioned above.

[0108] Therefore, if sibling nodes at the same level contain leaf nodes with the same field name, that is, if the cache contains a converted DSL node with the same field name as the intermediate DSL node to be stored in the cache, and the field name contains an aggregateable attribute in the ES mapping, then the intermediate DSL node and the converted DSL node are aggregated using the node aggregation method of this application embodiment and the logical operator corresponding to the intermediate DSL node, and then stored in the cache after aggregation. It can be understood that the cache will no longer contain the converted DSL node before aggregation. In addition, the logical operator can be determined according to the parent node of the intermediate DSL node. After aggregating the above query statement for foo, it can become: {"query":{"bool":{"must":[{"range":{"foo":{"gt":2}}},{"range":{"foo":{"lt":2}}}]}}}).

[0109] This application embodiment improves retrieval efficiency by expanding the fields in ES mapping to aggregate field names containing aggregateable attributes and intermediate DSL nodes with the same field names.

[0110] Experiments conducted in this application demonstrate that, when dealing with 1 billion documents, for range queries involving multiple identical fields (such as writing `foo:>1 AND foo:<3`), the DSL transformed using this solution is approximately 9.2 times faster than the traditional query using Lucene. (In ES 5.6.2 testing, the original Lucene query took 397ms, while the DSL statement transformed using the method provided in this application's embodiment takes 43ms).

[0111] Building upon the above embodiments, for the case where the AST node corresponding to a certain node layer is an operator node, it is explained that all AST nodes in that layer are parent nodes, and each AST node contains at least one child node. This AST node receives DSL nodes uploaded by its child nodes in the next lower layer, packages the received DSL nodes into a BoolQuery type DSL node, and continues uploading it to the upper-level AST nodes until that layer becomes the root node. The BoolQuery generated by the root node is the DSL statement.

[0112] Figure 3 A schematic diagram of an AST structure provided for an embodiment of this application is shown below. Figure 3 As shown, this AST contains three levels of nodes. The root node is node A, which has two child nodes, B and C. Node B has two leaf nodes, B1 and B2, and node C has two leaf nodes, C1 and C2. When converting the AST to a DSL statement, the traversal can start from the root node. First, traverse node A. After determining that node A is not a leaf node, continue traversing node B. After determining that node B is not a leaf node, traverse node B1. At this point, node B1 is determined to be a leaf node, and cache 1 for this level is established. Then, based on the target attribute under the field name corresponding to node B1 in the ES mapping, the transformation is performed to obtain intermediate DSL nodes. It is then checked whether cache 1 contains a transformed DSL node with the same field name as the intermediate DSL node. If not, the intermediate DSL node is stored in cache 1. The intermediate DSL node stored in cache 1 is called the transformed DSL node.

[0113] Next, node B2 is traversed and transformed using a similar transformation method as node B1 to obtain the intermediate DSL node corresponding to node B2. It is then checked whether cache 1 contains a transformed DSL node with the same field name as the intermediate DSL node. If the field name of the transformed DSL node in cache 1 is the same as the field name of the intermediate DSL node, and the corresponding ES mapping contains an aggregateable attribute, the transformed DSL node is retrieved from cache 1. The transformed DSL node and the intermediate DSL node are then aggregated according to their corresponding aggregation method and logical operator, and the resulting DSL node is stored in cache 1. After transforming both B1 and B2, since B is an operator node, it will be one of the three operators NOT / AND / OR. The transformed DSL node in cache 1, along with the operator nodes of node B1 and node B2, are used to generate a DSL node of type BoolQuery. This process continues with node A, which performs the corresponding transformation operation based on its operator type.

[0114] For example:

[0115]

[0116]

[0117]

[0118] Figure 4 This is a schematic diagram of the AST structure provided in the embodiments of this application, such as... Figure 4 As shown, the transformed AST in the above example can be found here. Figure 4 As shown.

[0119] The DSL statement generated based on the above AST is as follows:

[0120]

[0121]

[0122]

[0123] Similarly, the transformation process of nodes C, C1, and C2 is similar to that described above, and will not be repeated here.

[0124] Finally, node A generates the final DSL statement based on the BoolQuery type DSL nodes uploaded by nodes B and C, and the corresponding logical operators of node A.

[0125] In this embodiment, by converting operator nodes into Bool Queries, a DSL query statement is finally obtained, and the DSL query statement is used for retrieval, thereby improving retrieval efficiency.

[0126] Figure 5 This is a schematic diagram of a query statement conversion device provided in an embodiment of this application. The device can be a module, program segment, or code on an electronic device. It should be understood that this device is similar to the one described above. Figure 1 The method implementation corresponds to this and can be executed. Figure 1 The specific functions of the device involved in the method embodiment can be found in the description above; to avoid repetition, detailed descriptions are omitted here. The device includes: a receiving module 501, a parsing module 502, and a conversion module 503, wherein:

[0127] The receiving module 501 is used to receive Lucene query statements based on a custom extended Backus normal form (EBNF); the Lucene query statement includes multiple subquery statements and Boolean operators;

[0128] Parsing module 502 is used to parse multiple subquery statements and Boolean operators based on the syntax parsing library and EBNF to obtain an abstract syntax tree (AST);

[0129] The conversion module 503 is used to convert the AST into DSL statements based on the ES mapping; wherein the ES mapping includes attributes for converting the AST into DSL statements.

[0130] Based on the above embodiments, the subquery statement includes a query form, and the conversion module 503 is specifically used for:

[0131] Starting from the leaf nodes, the AST nodes of each layer are converted into corresponding DSL nodes according to the ES mapping and query form, and the DSL nodes are returned to the previous layer AST nodes, until the conversion of all AST nodes is completed and the DSL statement is obtained.

[0132] Based on the above embodiments, the AST includes multiple node layers, and each node layer includes at least one AST node; the conversion module 503 is specifically used for:

[0133] If the AST node corresponding to a node layer is a leaf node, obtain the field name corresponding to the leaf node;

[0134] If the field name is a field other than the preset field, then the target attribute corresponding to the field name is obtained from the ES mapping, and the corresponding target transformation function is obtained based on the target attribute;

[0135] Based on the target transformation function and query form, the leaf nodes are converted into intermediate DSL nodes, and the intermediate DSL nodes are stored in the cache;

[0136] Generate the corresponding DSL nodes for the node layer based on the intermediate DSL nodes in the cache.

[0137] Based on the above embodiments, the preset fields include _id and _exists_;

[0138] If the field name is _id, then the corresponding leaf node will be converted to IdsQuery;

[0139] If the field name is _exists_, then the corresponding leaf node will be converted to ExistsQuery.

[0140] Based on the above embodiments, the conversion module 503 is specifically used for:

[0141] If the cache contains a converted DSL node with the same field name as the intermediate DSL node, and the field name in ESmapping contains an aggregable attribute, then the intermediate DSL node and the converted DSL node are aggregated according to the logical operator corresponding to the intermediate DSL node and the aggregation method corresponding to the aggregable attribute, and the aggregated node is stored in the cache.

[0142] Based on the above embodiments, the AST includes multiple node layers, and each node layer includes at least one AST node; the conversion module 503 is specifically used for:

[0143] If the AST node corresponding to a node layer is an operator node, then the DSL node sent by the AST node of the next layer is converted into a Bool Query; where the Bool Query is the DSL node corresponding to the node layer.

[0144] Based on the above embodiments, the Boolean operators defined in the above EBNF include lowercase forms.

[0145] Figure 6 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of this application, such as... Figure 6 As shown, the electronic device includes: a processor 601, a memory 602, and a bus 603; wherein,

[0146] The processor 601 and the memory 602 communicate with each other through the bus 603;

[0147] The processor 601 is used to call program instructions in the memory 602 to execute the methods provided in the above-described method embodiments, such as: receiving a Lucene query statement based on a custom extended Backus normal form (EBNF); the Lucene query statement includes multiple subqueries and Boolean operators; parsing the multiple subqueries and Boolean operators based on a syntax parsing library and EBNF to obtain an abstract syntax tree (AST); converting the AST into a DSL statement according to the ES mapping; wherein the ES mapping includes attributes for converting the AST into a DSL statement.

[0148] Processor 601 can be an integrated circuit chip with signal processing capabilities. The processor 601 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.

[0149] The memory 602 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0150] This embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the methods provided in the above-described method embodiments, such as: receiving a Lucene query statement based on a custom extended Backus normal form (EBNF); the Lucene query statement includes multiple subqueries and Boolean operators; parsing the multiple subqueries and Boolean operators based on a syntax parsing library and EBNF to obtain an abstract syntax tree (AST); and converting the AST into a DSL statement according to an ES mapping; wherein the ES mapping includes an attribute that converts the AST into a DSL statement.

[0151] This embodiment provides a non-transitory computer-readable storage medium storing computer instructions that cause the computer to execute the methods provided in the above-described method embodiments, such as: receiving a Lucene query statement based on a custom extended Backus normal form (EBNF); the Lucene query statement includes a query form; parsing the Lucene query statement using a syntax parsing library to obtain an abstract syntax tree (AST); and converting the AST into a DSL statement according to ES mapping; wherein the ES mapping includes an attribute that converts the AST into a DSL statement.

[0152] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0153] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0154] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0155] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0156] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A query statement conversion method, characterized in that, include: Receive Lucene query statements based on a custom Extended Backus Normal Form (EBNF); the Lucene query statement includes multiple subqueries and Boolean operators; Based on the syntax parsing library and the EBNF, the multiple subquery statements and Boolean operators are parsed to obtain the Abstract Syntax Tree (AST); The AST is converted into DSL statements according to the ES mapping; wherein the ES mapping includes attributes that convert the AST into the DSL statements; The subquery statement includes a query form, and the step of converting the AST into a DSL statement according to the ES mapping includes: Starting from the leaf node, the AST node of each layer is converted into the corresponding DSL node according to the ES mapping and the query form, and the DSL node is returned to the previous layer AST node until the conversion of all AST nodes is completed and the DSL statement is obtained. The AST comprises multiple node layers, each node layer including at least one AST node; the process of converting the AST node of each layer into a corresponding DSL node according to the ES mapping and the query form includes: If the AST node corresponding to a node layer is a leaf node, obtain the field name corresponding to the leaf node; If the field name is a field other than the preset field, then the target attribute corresponding to the field name is obtained from the ES mapping, and the corresponding target transformation function is obtained according to the target attribute; The leaf node is converted into an intermediate DSL node according to the target transformation function and the query form, and the intermediate DSL node is stored in the cache; The DSL nodes corresponding to the node layer are generated based on the intermediate DSL nodes in the cache.

2. The method according to claim 1, characterized in that, The preset fields include _id and _exists_; If the field name is _id, then the corresponding leaf node will be converted to IdsQuery; If the field name is _exists_, then the corresponding leaf node will be converted to ExistsQuery.

3. The method according to claim 1, characterized in that, The step of storing the intermediate DSL node in the cache includes: If the cache contains a converted DSL node with the same field name as the intermediate DSL node, and the field name in the ES mapping contains an aggregable attribute, then the intermediate DSL node and the converted DSL node are aggregated according to the logical operator corresponding to the intermediate DSL node and the aggregation method corresponding to the aggregable attribute, and the aggregated node is stored in the cache.

4. The method according to claim 1, characterized in that, The AST comprises multiple node layers, each node layer including at least one AST node; the process of converting the AST node of each layer into a corresponding DSL node according to the ES mapping and the query form includes: If the AST node corresponding to a node layer is an operator node, then the DSL node sent by the AST node of the next layer is converted into a Bool Query; wherein the Bool Query is the DSL node corresponding to the node layer.

5. The method according to any one of claims 1-4, characterized in that, The Boolean operators defined in EBNF include lowercase forms.

6. A query statement conversion device, characterized in that, include: The receiving module is used to receive Lucene query statements based on a custom Extended Backus Normal Form (EBNF); the Lucene query statement includes multiple subquery statements and Boolean operators. The parsing module is used to parse the multiple subquery statements and Boolean operators based on the syntax parsing library and the EBNF to obtain the abstract syntax tree (AST). A conversion module is used to convert the AST into DSL statements according to the ES mapping; wherein the ES mapping includes attributes for converting the AST into the DSL statements; The subquery statement includes a query format, and the conversion module is specifically used for: Starting from the leaf node, the AST node of each layer is converted into the corresponding DSL node according to the ES mapping and the query form, and the DSL node is returned to the previous layer AST node until the conversion of all AST nodes is completed and the DSL statement is obtained. The AST comprises multiple node layers, and each node layer includes at least one AST node; the conversion module is specifically used for: If the AST node corresponding to a node layer is a leaf node, obtain the field name corresponding to the leaf node; If the field name is a field other than the preset field, then the target attribute corresponding to the field name is obtained from the ES mapping, and the corresponding target transformation function is obtained according to the target attribute; The leaf node is converted into an intermediate DSL node according to the target transformation function and the query form, and the intermediate DSL node is stored in the cache; The DSL nodes corresponding to the node layer are generated based on the intermediate DSL nodes in the cache.

7. An electronic device, characterized in that, include: Processor, memory, and bus, among which, The processor and the memory communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor can execute the method as described in any one of claims 1-5 by calling the program instructions.

8. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions, which, when executed by a computer, cause the computer to perform the method as described in any one of claims 1-5.

Images