Method and system for reserving query sequence based on ElasticSearch

Through hashing algorithm and script rewriting technology, the document primary key is distributed to sharded parallel query in ElasticSearch, which solves the problem of performance degradation in complex queries, realizes flexible rewriting and performance optimization of query results, and improves the customization of the system and user experience.

CN118939858APending Publication Date: 2024-11-12BEIJING ZHIGUAGUA TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411427280.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-14
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In ElasticSearch, as the number of query clauses increases, especially in complex queries, the performance problems of boosting query become obvious, resulting in extended search response times, especially in large data sets or high concurrency scenarios.

Method used

The hash algorithm is used to distribute the document primary keys evenly in each shard of ElasticSearch, query each shard in parallel, and rewrite the scoring part of ElasticSearch using scripts or plug-ins to ensure that the results are returned in the order of the primary keys of the user query, and design the data structure. Serialize the query results, keeping their original sort order.

Benefits of technology

While ensuring the order of query results, it improves the customizability and flexibility of the system, significantly improves user experience and system performance, and avoids the performance degradation or resource consumption problems that may be caused by traditional scripting methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939858A_ABST
    Figure CN118939858A_ABST
Patent Text Reader

Abstract

The invention discloses a method and system for reserving a query sequence based on ElasticSearch, and the method comprises the steps: storing and transmitting an ElasticSearch query condition in a format capable of reserving an original sorting sequence through a specific serialization method; the query rewriting mechanism provides an effective script or algorithm, and can rewrite the query process to meet specific requirements or optimization purposes on the premise of not damaging the original query process; according to the performance guarantee scheme, when query rewriting is implemented, the problem of performance reduction or resource consumption possibly caused by a traditional script mode is effectively avoided in a mode of optimizing an algorithm or a data structure.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of document recognition processing, and in particular to a method and system for retaining query order based on ElasticSearch. Background Art

[0002] In modern information retrieval systems, ElasticSearch (search data analysis engine) is a popular open source search engine and distributed document storage solution, which is widely used in real-time search, analysis, and data storage. However, the sorting order of current ElasticSearch query results is usually determined by its internal scoring algorithm. Although this sorting method is effective, the sorting order of the results is difficult to rewrite directly, especially when performance needs to be guaranteed.

[0003] Currently, Elasticsearch provides a way to enhance the importance of specific query clauses through boosting (boosting query), which is very useful in many search scenarios. By setting the boost parameter in the query DSL, you can significantly affect the score of each clause and thus affect the ranking result of the entire query.

[0004] However, as the number of clauses in a query increases, especially in complex queries, such as compound queries with multiple Boolean logic or fuzzy queries with multiple fields, performance issues with boosting queries may become apparent. This is because the boost parameters for each clause need to be calculated during the scoring phase, and Elasticsearch needs to perform relevance scoring for each query clause separately. When the number of clauses is too large, these calculations can have a negative impact on query performance, resulting in longer search response times, especially in large data sets or high concurrency scenarios. Summary of the invention

[0005] In view of this, the present invention provides a method and system for retaining query order based on ElasticSearch, which can ensure the sorting order of query results while allowing flexible rewriting of queries. This solution not only improves the customizability and flexibility of the system, but also significantly improves user experience and system performance.

[0006] The present invention provides a method for retaining query order based on ElasticSearch, comprising:

[0007] Step S1, receiving a query request submitted by a user;

[0008] Step S2, obtaining a query request and reconstructing the query conditions according to user requirements;

[0009] Step S3, use ElasticSearch to perform a query and obtain the original sorted query results;

[0010] Step S4, serialize the results according to preset rules and return them to the user.

[0011] Preferably, the query is performed using ElasticSearch, and the document primary key is evenly distributed in each shard of ElasticSearch using a hash algorithm to ensure that the data of each shard is aligned with the query data during the query. The specific steps are as follows:

[0012] Hash sharding: Use a hash function to calculate the hash value of the document primary key and map the hash value to each shard of ElasticSearch;

[0013] Query optimization: When executing a query, a hash function is used to determine the shard where the document primary key is located, and then each related shard is queried in parallel to reduce the query load on a single shard and improve query efficiency.

[0014] More preferably, the query using ElasticSearch also includes a query rewriting part, comparing the primary key part in the current text with the primary key of the user query part, and sorting the user query part according to the order-preserving flow, and the specific steps are as follows:

[0015] Intrusive code implementation: Use a script or plug-in to rewrite the scoring part of ElasticSearch;

[0016] Primary key comparison: Compare the document primary key in the user query with the primary key of the actual document to ensure that the results are returned in the order of the primary keys in the user query.

[0017] Preferably, the steps of serializing the results according to preset rules include:

[0018] Data structure design: Design a data structure to store the document primary key and corresponding content of each query result item;

[0019] Serialization process: After the query results are returned, the primary key and content of each document are serialized into a predetermined object and organized into a data stream in an order-preserving manner.

[0020] Preferably, the data structure is a JSON array, and the predetermined object is a JSON object.

[0021] The present invention also provides a system for retaining query order based on ElasticSearch, comprising:

[0022] A receiving module, used for receiving a query request submitted by a user;

[0023] Acquisition module,The module is used to obtain query requests and reconstruct query conditions according to user needs;

[0024] The query module uses ElasticSearch to query and obtain the original sorted query results;

[0025] Output module,The module is used to serialize the results according to preset rules and return them to the user.

[0026] Preferably, the query module uses ElasticSearch to perform queries, and uses a hash algorithm to evenly distribute the document primary key in each shard of ElasticSearch to ensure that the data of each shard is aligned with the queried data during the query. The specific steps are as follows:

[0027] Hash sharding: Use a hash function to calculate the hash value of the document primary key and map the hash value to each shard of ElasticSearch;

[0028] Query optimization: When executing a query, a hash function is used to determine the shard where the document primary key is located, and then each related shard is queried in parallel to reduce the query load on a single shard and improve query efficiency.

[0029] The present invention provides a method and system for retaining query order based on ElasticSearch, and the beneficial technical effects achieved include the following aspects:

[0030] (I) Query result serialization scheme: Through a specific serialization method, ElasticSearch query conditions are stored and transmitted in a format that can preserve their original sorting order.

[0031] (ii) Query rewriting mechanism: It provides an effective script or algorithm that can rewrite the query process to meet specific needs or optimization purposes without destroying the original query process.

[0032] (III) Performance guarantee plan: When implementing query rewriting, the performance degradation or resource consumption problems that may be caused by traditional scripting methods can be effectively avoided by optimizing algorithms or data structures. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the drawings required for the implementation methods or the description of the prior art are briefly introduced below. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0034] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0035] Figure 1 Schematic diagram of query processing flow before transformation in the prior art;

[0036] Figure 2 Schematic diagram of query processing flow after transformation in the prior art;

[0037] Figure 3 A flow chart of a method for preserving query order provided in an embodiment of the present invention;

[0038] Figure 4 A schematic diagram of a system architecture for preserving query order provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] In order to achieve the technical effects of the above invention, the embodiments of the present invention involve the following steps or modules:

[0041] 1) Data structure optimization: Design and implement an efficient data structure for storing and processing serialized query results to preserve their original sort order.

[0042] 2) Query rewriting module: Develop a script or algorithm module to rewrite query statements through specific logic and rules to achieve performance optimization and specific needs.

[0043] 3) Performance evaluation and optimization: Through systematic performance evaluation and testing, verify that the query rewriting solution provided does not reduce query efficiency and ensure that performance is effectively guaranteed.

[0044] An embodiment of the present invention provides a method for retaining query order based on ElasticSearch, comprising:

[0045] Step S1, receiving a query request submitted by a user;

[0046] Step S2, obtaining a query request and reconstructing the query conditions according to user requirements;

[0047] Step S3, use ElasticSearch to perform a query and obtain the original sorted query results;

[0048] Step S4, serialize the results according to preset rules and return them to the user.

[0049] In step S3, ElasticSearch is used to perform a query, and a hash algorithm is used to evenly distribute the primary key of the document in each shard of ElasticSearch to ensure that the data of each shard is aligned with the queried data during the query. The specific steps are as follows:

[0050] Hash sharding: Use a hash function to calculate the hash value of the document primary key and map the hash value to each shard of ElasticSearch;

[0051] Query optimization: When executing a query, a hash function is used to determine the shard where the document primary key is located, and then each related shard is queried in parallel to reduce the query load on a single shard and improve query efficiency.

[0052] Among them, in step S3, query is performed using ElasticSearch, and query is performed using ElasticSearch, which also includes a query rewriting part, comparing the primary key part in the current text with the primary key of the user query part, and sorting the user query part according to the order-preserving flow. The specific steps are as follows:

[0053] Intrusive code implementation: Use a script or plug-in to rewrite the query on ElasticSearch;

[0054] Primary key comparison: Compare the document primary key in the user query with the primary key of the actual document to ensure that the results are returned in the order of the primary keys in the user query.

[0055] Among them, in step S4, ElasticSearch is used to query, and the results are serialized according to preset rules. The specific steps include:

[0056] Data structure design: Design a data structure to store the document primary key and corresponding content of each query result item;

[0057] Serialization process: After the query results are returned, the primary key and content of each document are serialized into a predetermined object and organized into a data stream in an order-preserving manner.

[0058] A specific implementation example is provided below, which shows how to implement a query rewriting and result retention method based on ElasticSearch according to the teachings of the present invention.

[0059] 1. Query result serialization method

[0060] To serialize the query results, you can use an order-preserving data stream similar to an array object. The specific steps are as follows:

[0061] Data structure design: Design a data structure, such as a JSON array, to store the document primary key and corresponding content of each query result item.

[0062] Serialization process: After the query results are returned, the primary key and content of each document are serialized into a JSON object and organized into a data stream in an order-preserving manner.

[0063] Example:

[0064] const doc_list = ["doc001","doc002","doc003","doc005","......"]

[0065] 2. Query rewriting part

[0066] Implement an intrusive code method to compare the primary key part of the current text with the primary key of the user query part, and sort it according to the order-preserving flow of the user query part. The specific steps are as follows:

[0067] Intrusive code implementation: Develop a script or plug-in to rewrite the query on ElasticSearch.

[0068] Primary key comparison: Compare the document primary key in the user query with the primary key of the actual document to ensure that the results are returned in the order of the primary keys in the user query.

[0069] Sample code (pseudo-code example):

[0070] function_score: {

[0071] query: dsl,

[0072] boost_mode: 'replace',

[0073] functions:

[0074] {

[0075] script_score: {

[0076] script: {

[0077] params: { list: doc_list},

[0078] source: 'params.list.length - params.list.indexOf(doc[\'_id\'].value)',

[0079] },

[0080] },

[0081] },

[0082] ],

[0083] },

[0084] 3. Performance optimization

[0085] To optimize performance, use a hash algorithm to evenly distribute the document primary key across the shards of ElasticSearch to ensure that the data in each shard is aligned with the queried data. The specific steps are as follows:

[0086] Hash sharding: Use a hash function to calculate the hash value of the document primary key and map the hash value to each shard of ElasticSearch.

[0087] Query optimization: When executing a query, a hash function is used to determine the shard where the document primary key is located, and then each related shard is queried in parallel to reduce the query load on a single shard and improve query efficiency.

[0088] Example:

[0089] const { createHash} = require('crypto');

[0090] / / Simulate hash sharding function

[0091] function getShard(documentId, numShards) {

[0092] const hash = createHash('sha256');

[0093] hash.update(documentId);

[0094] const hashValue = BigInt('0x' + hash.digest('hex')); / / Use BigInt to process large integers

[0095] return Number(hashValue % BigInt(numShards)); / / Calculate the shard number

[0096] }

[0097] / / Simulate query request construction

[0098] const documentIds = ["user_doc001", "user_doc002", "user_doc003"]; / / List of primary keys of documents queried by users

[0099] const numShards = 5; / / Total number of shards

[0100] const shardNum = getShard(documentIds[0], numShards); / / Get the shard number based on the first document primary key

[0101] The embodiment of the present invention further provides a system 100 for retaining query order based on ElasticSearch, comprising:

[0102] The receiving module 10 is used to receive a query request submitted by a user;

[0103] Acquisition module 20, module, used to obtain query requests and reconstruct query conditions according to user requirements;

[0104] The query module 30 uses ElasticSearch to perform a query and obtain the query results in the original order;

[0105] The output module 40 is used to serialize the results according to preset rules and return them to the user.

[0106] The query module uses ElasticSearch to query, and uses a hash algorithm to evenly distribute the document primary key in each shard of ElasticSearch to ensure that the data of each shard is aligned with the query data during the query. The specific steps are as follows:

[0107] Hash sharding: Use a hash function to calculate the hash value of the document primary key and map the hash value to each shard of ElasticSearch;

[0108] Query optimization: When executing a query, a hash function is used to determine the shard where the document primary key is located, and then each related shard is queried in parallel to reduce the query load on a single shard and improve query efficiency.

[0109] Through the above-mentioned technical means, the present invention realizes a method, device and electronic device that can ensure the sorting order of query results while allowing flexible rewriting of queries. This technology not only improves the customizability and flexibility of the system, but also significantly improves user experience and system performance.

[0110] This embodiment provides an electronic device, including: at least one processor and at least one memory;

[0111] The memory is used to store one or more program instructions;

[0112] The processor is used to run one or more program instructions to execute the intelligent patent search method as described above.

[0113] This embodiment provides a computer-readable storage medium, which contains one or more program instructions, and the one or more program instructions are used to execute the intelligent patent search method as described above.

[0114] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A method for retaining query order based on ElasticSearch, characterized in that: include: Step S1, receiving a query request submitted by a user; Step S2, obtaining a query request and reconstructing the query conditions according to user requirements; Step S3, use ElasticSearch to perform a query and obtain the original sorted query results; Step S4, serialize the results according to preset rules and return them to the user.

2. The method for preserving query order as claimed in claim 1, characterized in that: The query is performed using ElasticSearch, and the hash algorithm is used to evenly distribute the document primary key in each shard of ElasticSearch to ensure that the data of each shard is aligned with the query data during the query. The specific steps are as follows: Hash sharding: Use a hash function to calculate the hash value of the document primary key and map the hash value to each shard of ElasticSearch; Query optimization: When executing a query, a hash function is used to determine the shard where the document primary key is located, and then each related shard is queried in parallel to reduce the query load on a single shard and improve query efficiency.

3. The method for preserving query order as claimed in claim 1, characterized in that: The query using ElasticSearch also includes a query rewriting part, comparing the primary key part in the current text with the primary key of the user query part, and sorting the user query part according to the order-preserving flow. The specific steps are as follows: Intrusive code implementation: Use a script or plug-in to rewrite the scoring part of ElasticSearch; Primary key comparison: Compare the document primary key in the user query with the primary key of the actual document to ensure that the results are returned in the order of the primary keys in the user query.

4. The method for preserving query order as claimed in claim 1, characterized in that: The specific steps of serializing the results according to the preset rules include: Data structure design: Design a data structure to store the document primary key and corresponding content of each query result item; Serialization process: After the query results are returned, the primary key and content of each document are serialized into a predetermined object and organized into a data stream in an order-preserving manner.

5. The method for preserving query order as claimed in claim 4, characterized in that: The data structure is a JSON array, and the predetermined object is a JSON object.

6. A system for preserving query order based on ElasticSearch, characterized in that: include: A receiving module, used for receiving a query request submitted by a user; Acquisition module,The module is used to obtain query requests and reconstruct query conditions according to user needs; The query module uses ElasticSearch to query and obtain the original sorted query results; Output module,The module is used to serialize the results according to preset rules and return them to the user.

7. The system for preserving query order as claimed in claim 6, characterized in that: The query module uses ElasticSearch to perform queries and uses a hash algorithm to evenly distribute the document primary key in each shard of ElasticSearch to ensure that the data of each shard is aligned with the queried data during the query. The specific steps are as follows: Hash sharding: Use a hash function to calculate the hash value of the document primary key and map the hash value to each shard of ElasticSearch; Query optimization: When executing a query, a hash function is used to determine the shard where the document primary key is located, and then each related shard is queried in parallel to reduce the query load on a single shard and improve query efficiency.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor and at least one memory; The memory is used to store one or more program instructions; The processor is used to run one or more program instructions to execute the method according to any one of claims 1 to 4.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium includes one or more program instructions, and the one or more program instructions are used to execute the method according to any one of claims 1 to 4.

Citation Information

Patent Citations

  • Data query method and device, electronic equipment and readable storage medium

    CN111177476A

  • Method and device for querying state data result

    CN114924881A

  • Data query method and device, equipment and storage medium

    CN116962516A

  • Multi-condition query method based on ElasticSearch

    CN118606340A

  • Systems And Methods For Processing Structured Queries Over Clusters

    US20220050844A1