Query method and device based on search engine

Abstract

The invention provides a query method and device based on a search engine, the method is realized by a search engine module constructed based on a full-text search engine, and the method comprises the following steps: the search engine module receives a user query request containing a user query condition from a service system; the search engine module obtains a main data index corresponding to the query request based on a mapping relationship between a pre-stored query condition and the main data index and the query request, and returns the main data index to the service system, so that the service system performs query on the main data index based on the returned main data index; obtaining full data corresponding to the query request from a structured database in which a mapping relationship between the main data index and the full data is pre-stored; wherein the mapping relation between the query condition and the main data index is obtained based on the business operation record data of the business system. According to the method, efficient and real-time complex query can be realized in a general scene with large data volume and multiple conditions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data query, in particular to a query method and device based on a search engine. BACKGROUND

[0002] In a large business system, there are often business queries with multiple complex conditions. In this case, due to the performance of the structured database, it is difficult to achieve high query efficiency when a large amount of data is stored or associated with multiple tables or multiple fields in a single table are query conditions. Low query efficiency will affect the user experience and continuously occupy server resources.

[0003] For complex queries with large data volume and multiple conditions, existing business systems often use data warehouses (big data) or caching methods to improve query efficiency: ① The data warehouse method is to integrate the required data into a wide table through data synchronization, and then use the powerful performance support of Apache Doris to effectively solve the query efficiency problem under complex conditions. However, the learning cost and construction cost of the data warehouse are relatively high. Since the data warehouse requires an additional process of synchronizing data, it is difficult to meet the requirements of real-time business based on the data warehouse query method. ② The caching method refers to using cache databases such as Redis, MemoryCache, etc. to store query conditions as keys (key) and query results as values (value) in the database to improve query efficiency. However, this method is only suitable for some query scenarios (such as querying hot data or a query condition that can cover multiple scenarios), and has high limitations.

[0004] Therefore, there is an urgent need for an efficient, low-cost, and real-time query method to support complex queries in general scenarios with large data volume and multiple conditions. SUMMARY

[0005] In view of this, the embodiments of the present application provide a query method and device based on a search engine, which can overcome the defects of the query methods of data warehouses and caches, and provide efficient, low-cost, real-time, and complex queries suitable for general scenarios.

[0006] One aspect of the present application provides a query method based on a search engine, which is implemented by a data indexing system including a search engine module constructed based on a full-text search engine. The method comprises the following steps: receiving, by the search engine module, a user query request containing user query conditions from a business system; wherein the query request is a programming language generated based on a natural language query text input by a user into the business system; The search engine module obtains the main data index corresponding to the query request based on the mapping relationship between the pre-stored query condition and the main data index and the query request, and returns the main data index to the business system, so that the business system obtains the full data corresponding to the query request from the structured database in which the mapping relationship between the main data index and the full data is pre-stored based on the returned main data index. The mapping relationship between the query condition and the main data index is obtained by the following method: In the data falling into the database stage, the search engine module obtains the business operation record data from the business system; wherein the business operation record data includes the field information of the main data index and the corresponding full data; The search engine module extracts the query condition corresponding to each main data index from the full data corresponding to each main data index, and constructs a mapping relationship between the extracted query condition and the corresponding main data index.

[0007] In some embodiments of the present application, the data index system further comprises a query complexity evaluation module, which comprises a query parser and a classifier; If the search engine module has multiple different full-text search engines, before the search engine module receives the user query request, the method further comprises: The query parser in the query complexity evaluation module receives the user query request containing the user query condition, and extracts the data statistical feature and the data hierarchical structure feature corresponding to the user query condition; The extracted data statistical feature and data hierarchical structure feature are input into the classifier in the query complexity evaluation module, and the query complexity corresponding to the user query condition is output, and the full-text search engine used to determine the main data index corresponding to the user query request is selected based on the obtained query complexity.

[0008] In some embodiments of the present application, the method further comprises: under the set update condition, the search engine module obtains the historical query log data from the business system, and updates the parameters of the full-text search engine used to construct the search engine module based on the historical query log data by using the reinforcement learning algorithm.

[0009] In some embodiments of the present application, the historical query log data includes query context and query performance indicators; The parameters of the full-text search engine used to construct the search engine module are updated based on the historical query log data by using the reinforcement learning algorithm, which comprises: The parameters of the full-text search engine used to construct the search engine module are updated based on the query context by using the agent, so as to maximize the query performance indicators through iterative learning.

[0010] In some embodiments of the present application, for real-time services, the search engine module acquires the time of the business operation record data at the same time as the structured database acquires the business operation record data from the business system at the data falling into the database stage.

[0011] In some embodiments of the present application, the data indexing system further comprises a transaction coordination module. The search engine module acquires the time of the business operation record data at the same time as the structured database acquires the business operation record data from the business system is achieved by the following way: The search engine module acquires the time of the business operation record data at the same time as the structured database acquires the business operation record data from the business system is achieved by the following way:

[0012] In some embodiments of the present application, the method further comprises: Under the set trigger condition, after the business system sends the first data verification request to the structured database and obtains the full amount of data corresponding to the stored master data index from the structured database, the search engine module receives the second data verification request from the business system; wherein the second data verification request contains the master data index information obtained by the business system based on the first data verification request; The search engine module compares the master data index information in the second data verification request with the mapping relationship between the query condition and the master data index stored in advance; if there is a master data index in the search engine module that is not contained in the structured database, the search engine module sends the first data synchronization instruction containing the specific master data index to the business system, so that the business system sends the corresponding business operation record data to the structured database; if there is a master data index in the structured database that is not contained in the search engine module, the search engine module sends the second data synchronization instruction containing the specific master data index to the business system, and receives the corresponding business operation record data from the business system, thereby updating the mapping relationship between the query condition and the master data index.

[0013] In some embodiments of the present application, the data indexing system further comprises a cache module. The method further comprises: obtaining user historical query behavior data from the business system and inputting it into a pre-trained heat prediction model to predict the query hot spot information in the set future period, so that the business operation record data related to the query hot spot information is stored in the cache module in the set future period.

[0014] In some embodiments of the present application, the cache module comprises a Bloom filter, which is used to transmit the user query request to the search engine module in the case that the cache module does not have the full amount of data corresponding to the user query condition.

[0015] Another aspect of the present invention provides a search device based on a search engine, including a processor, a memory, and a computer program / instructions stored in the memory. The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the device implements the steps of the method described in any of the above embodiments.

[0016] The present invention proposes a search engine-based query method and apparatus. Based on a full-text search engine, a data indexing system is constructed between the business system and the structured database. This system encapsulates complex queries involving multiple table fields, as well as query conditions without table partitioning keys, within the search engine module. By matching query conditions, the corresponding full data can be retrieved from the structured database storing the actual data. The method proposed in this application supports complex queries in general scenarios involving large data volumes and multiple conditions, and has significant advantages such as high query efficiency, low cost, and high real-time performance.

[0017] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0018] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings: Figure 1 This is a timing diagram of data querying based on a data indexing system in one embodiment of the present invention.

[0020] Figure 2 This is a timing diagram of asynchronous data storage in one embodiment of the present invention.

[0021] Figure 3 This is a timing diagram of data verification in one embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0023] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0024] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0025] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0026] For complex queries involving large datasets and multiple conditions, current data warehouse query methods are costly and struggle to meet real-time business requirements. Furthermore, caching is only suitable for certain query scenarios. Due to these shortcomings, this application deploys an indexing system built on a full-text search engine between the business system and the structured database. This indexing system supports complex queries involving large datasets and multiple conditions. Using this indexing system, the business system can achieve high-efficiency, low-cost, real-time queries applicable to general scenarios.

[0027] The query method proposed in this application can be implemented by a data indexing system. The core of the data indexing system is a search engine module built on a full-text search engine. In addition, the data indexing system may also include a query complexity evaluation module, a transaction coordination module, and a caching module. The search engine module is mainly used to determine the corresponding master data index based on user query conditions from the business system, thereby quickly retrieving the full data corresponding to the user query conditions in the structured database. The query complexity evaluation module is used to determine the full-text search engine used in the data storage and data query stages when multiple different full-text search engines exist in the search engine module. The transaction coordination module is used to deploy the corresponding storage tasks in specific transactions during the data storage stage. The caching module is used to predict query hotspots and store business operation records related to query hotspots. Furthermore, the full-text search engine mentioned in this application can be a search engine that supports full-text indexing, such as ElasticSearch, Solr, or Algolia. These search engines are used to process large amounts of unstructured text data and support fast search and analysis functions.

[0028] like Figure 1 As shown, the main process of data querying based on the data indexing system during the data query phase is as follows: Step S01: The user initiates a query by inputting natural language query text into the business system.

[0029] Step S02: The business system receives the natural language query text input by the user, generates a corresponding programming language as the user query request (the user query request includes user query conditions), and transmits the user query request to the search engine module in the data indexing system. That is, the user query request (or the user query conditions in the user query request) is obtained based on the natural language query text input by the user into the business system. This application does not specifically limit its generation method; for example, a semantic understanding model (such as a large language model or BERT model) in the business system can be used to parse the natural language query text input by the user, and the user query request (or the user query conditions in the user query request) can be generated based on the parsing results. Both the user query request and the user query conditions are presented in the form of a programming language.

[0030] Step S03: The search engine module retrieves the master data index information corresponding to the user's query conditions based on the user's query conditions in the received user query request and the pre-stored mapping relationship between query conditions and master data index, and returns the retrieved master data index information to the business system.

[0031] In some embodiments of the present invention, the process of obtaining the main data index corresponding to the user query request based on the mapping relationship between the pre-stored query conditions and the main data index and the user query request needs to be completed in the search engine module. Since the search engine module mainly relies on the full-text search engine to find the main data index, the implementation process of step S03 is as follows: Step S11, the search engine module parses the user query conditions (including index sharding (dividing the user query conditions into multiple keywords), spell correction, and keyword expansion, etc.), and assigns corresponding weights to each field (field weights are used to measure the importance of different fields in the relevance of search results; different settings of field weights will directly affect the relevance and ranking of search results); Step S12, searches for the query conditions corresponding to each keyword in the inverted index (i.e., the mapping relationship between the pre-stored query conditions and the main data index); Step S13, sorts the found query conditions according to keyword relevance and webpage quality (such as the authority of the webpage and the number of links), thereby quickly finding the most relevant query conditions to the user query in massive amounts of data.

[0032] Step S04: The business system generates a full data request containing the master data index based on the master data index information returned by the data indexing system, and transmits the full data request to the structured database. Similar to user query requests, the full data request is also presented in the form of a programming language.

[0033] Step S05: The structured database matches the pre-stored mapping relationship (the mapping relationship between the master data index and the full data) with the master data index information in the full data request to retrieve the target full data (i.e., the full data corresponding to the user's query request) corresponding to the master data index information in the full data request, and returns it to the business system. Note that the mapping relationship stored in the search engine module (i.e., the mapping relationship between query conditions and the master data index) and the mapping relationship stored in the structured database (i.e., the mapping relationship between the master data index and the full data) mentioned in this application are formed during the data storage phase and can be updated through data verification mechanisms, etc.

[0034] Step S06: The business system presents the full target data returned by the structured database to the user through methods such as list rendering.

[0035] Regardless of whether the user's query conditions correspond to a multi-table or partitioned data structure, as long as the data indexing system has a corresponding mapping relationship for that query condition, the query efficiency for retrieving the full amount of data from the structured database can support business requirements. Furthermore, compared to a query solution that simply builds a search engine in front of the structured database, this application also has significant advantages in terms of query performance and variable costs. This is due to the existence of the same master data index in the mapping relationships stored in the search engine module and the structured database.

[0036] In this application, if Figure 1 The main reason the data query process shown is feasible is that, during the data storage phase, the same business operation record data is stored separately in the search engine module and the structured database. This ensures that the mapping relationships stored in the search engine module and the structured database contain the same master data index. Furthermore, these master data indexes are the original field information contained in the business operation record data within the business system. Therefore, when querying data after storage is complete, the target full data can be retrieved in the structured database through simple master data index field matching, based on the user's query conditions. Specifically, when users operate actual business processes through the business system, corresponding business operation record data is generated (e.g., business operation record data can be obtained from the business system's operation logs). This business operation record data includes master data indexes and corresponding full data information. Therefore, during the data storage phase before data querying using the data indexing system, the mapping relationships between query conditions and master data indexes, as well as between master data indexes and full data, can be obtained based on the business operation record data generated by the business system. These relationships are then stored separately in the search engine module and the structured database. Notably, the master data indexes corresponding to different full data sets are different; that is, the master data index field information in the business operation record data is a unique index.

[0037] It should be noted that during the data storage phase, this application design only stores all field information in the business operation record data in the structured database. However, when the data index system built on the search engine is used as the storage medium, it is difficult to meet transaction requirements and handle complex data relationships. Therefore, the data index system only stores the mapping relationship between the query conditions extracted from the business operation record data and the master data index, rather than storing all data fields in the business operation record data in the data index system.

[0038] The process of establishing the mapping relationship between the search engine module and the structured database during the data storage phase is as follows: The data indexing system (specifically, the search engine module) obtains business operation record data from the business system and, based on natural language processing, machine learning models, or data mining tools (such as Tableau, Power BI, etc.), extracts the query conditions corresponding to each master data index from the full data corresponding to each master data index. A mapping relationship is established between the extracted query conditions and the master data index, and this mapping relationship is stored in the search engine module. The structured database obtains business operation record data from the business system, and after data preprocessing (data filtering, pagination, etc.), a mapping relationship is established between the master data index and the full data, and this mapping relationship is stored in the structured database. The process of establishing the mapping relationship between query conditions and the master data index may not be implemented by the data indexing system (or the search engine module). For example, an additional system or module capable of data transmission with the business system and the search engine module can be designed. After obtaining the mapping relationship between the query conditions and the master data index using the business operation record data from the business system, this mapping relationship is transmitted to the search engine module for storage. This invention is not limited to this.

[0039] In some embodiments of the present invention, since the same business operation record data needs to be stored in a structured database and a data indexing system during the data storage phase, this application can propose synchronous and asynchronous data storage methods to meet the real-time requirements of the business. Specifically, for real-time business, a synchronous data storage method is adopted during the data storage phase, so that the time when the search engine module obtains the business operation record data is the same as the time when the structured database obtains the business operation record data from the business system (i.e., the start execution time of the data storage task of the data indexing system and the data storage task of the structured database is the same); for example Figure 2 As shown, for non-real-time business, an asynchronous data storage method can be adopted in the data storage stage, so that the time when the search engine module obtains the business operation record data is different from the time when the structured database obtains the business operation record data from the business system (e.g., the data storage task of the structured database is executed 10 minutes after the data storage task of the data indexing system is executed).

[0040] As an example, depending on the real-time requirements of the business, data can be synchronously or asynchronously persisted between the data indexing system and the structured database. This synchronization or asynchronous persistence can be achieved by deploying a transaction coordination module between the business system and the search engine module, specifically as follows: If the business has low real-time requirements, asynchronous persistence can be used. This involves using the transaction coordination module to deploy the data persistence tasks for the data indexing system and the structured database in different transactions (using distributed transactions for these two persistence tasks). If the business has high real-time requirements, synchronous persistence can be used. This involves using the transaction coordination module to deploy the data persistence tasks for the data indexing system and the structured database in the same transaction, and using a Try-Confirm-Cancel (TCC) mechanism to ensure bidirectional transaction synchronization between the search engine and the structured database. Furthermore, if asynchronous persistence is used, when generating distributed transactions using the transaction coordination module, the Raft protocol can be used to ensure data consistency (i.e., ensuring that the data persistence tasks for the data indexing system and the structured database use the same business operation record data for execution).

[0041] Furthermore, while the TCC mechanism can be used to ensure simultaneous success or failure of data storage operations in the data indexing system and structured database when high real-time business requirements are met, thus achieving data consistency, asynchronous data storage may result in one side failing while the other succeeds (e.g., the data storage task for the search engine succeeds, but the data storage task for the structured database fails). Therefore, to ensure data consistency, this application also designs a data verification mechanism between the data indexing system and the structured database as a data consistency guarantee strategy. Specifically, when the data verification mechanism is activated under set trigger conditions, it compares the master data indexes in the data indexing system and the structured database to determine whether the business operation record data has been successfully stored. If the master data indexes are inconsistent, a data compensation process is automatically triggered. The data verification process may specifically include: Step S21: The business system sends a first data verification request to the structured database; after receiving the first data verification request, the structured database returns the stored master data index and the corresponding full data to the business system. This application can be designed so that when the master data index in the mapping relationship stored by the search engine module and the mapping relationship stored by the structured database differs, the business system retrieves the full data corresponding to the specific master data index from the structured database. Alternatively, when the structured database returns the master data index based on the first data verification request, it can also return the full data corresponding to the returned master data index stored in the structured database to the business system. This application does not specifically limit the time when the structured database returns the full data to the business system during data verification; the design is up to the user. Furthermore, when the structured database returns the master data index (and the corresponding full data) to the business system, it can return it in batches or return all master data indexes stored in the structured database together.

[0042] Step S22: Compare the master data index information returned by the structured database in step S21 with the master data index in the mapping relationship stored by the search engine module. In this application, this step can be performed by the business system, at which point the business system also needs to obtain the master data index information in the mapping relationship stored by the search engine module; or it can be performed as follows: Figure 3 As shown, this step is performed by the search engine module. At this time, the business system must at least transmit the master data index information returned by the structured database to the search engine module. This application does not specifically limit the execution subject of step S22. The following description uses the search engine module as the execution subject.

[0043] More specifically, the business system generates a second data verification request based on the master data index information returned by the structured database (at this time, the second data verification request includes at least the master data index information obtained by the business system based on the first data verification request, and may also include the full data returned by the structured database); the search engine module receives the second data verification request from the business system and compares the master data index information in the second data verification request with the pre-stored query conditions and master data index mapping relationship.

[0044] If the data indexing system (or search engine module) contains a specific master data index that is not included in the structured database (i.e., the specific master data index is included in the mapping relationship stored in the search engine module but not in the mapping relationship stored in the structured database), the data indexing system sends a first data synchronization instruction containing the specific master data index information to the business system. After receiving the first data synchronization instruction, the business system extracts the business operation record data corresponding to the specific master data index and sends the data to the structured database (e.g., generates a data write instruction based on the data), so that the structured database re-writes the data based on the data. If the structured database contains a specific master data index that is not included in the data indexing system (or search engine module) (i.e., the specific master data index is included in the mapping relationship stored in the structured database but not in the mapping relationship stored in the search engine module), the search engine module sends a second data synchronization instruction containing the specific master data index to the business system. After receiving the second data synchronization instruction, the business system extracts the business operation record data corresponding to the specific master data index and sends the data to the search engine module, thereby causing the search engine module to update the mapping relationship between the stored query conditions and the master data index. Although both the first data synchronization instruction and the second data synchronization instruction are sent by the data indexing system to the business system, they limit the different entities from which the business operation record data is returned. Therefore, in addition to specific master data index information, the first data synchronization instruction and the second data synchronization instruction may also include system identification information for receiving business operation record data when it is re-stored in the database.

[0045] In addition, the triggering conditions can be set manually or to meet the set execution time of a scheduled task (such as setting a data verification operation to be performed every day at midnight). Furthermore, this application can be designed so that the data synchronization verification operation can be triggered by the business system.

[0046] In some embodiments of the present invention, data verification can be performed without comparing the master data index; instead, the mapping relationships stored in the search engine module can be updated directly. Specifically, the business system can query the stored master data index and the corresponding full data from the structured database in batches, and extract the corresponding field information from the retrieved full data as the redefined query conditions. The business system then transmits the master data index and the redefined query conditions to the data indexing system, which updates the stored mapping relationships batch by batch based on the unique master data index.

[0047] As an example, a search engine module may contain one or more full-text search engines. If the search engine module is built using only one full-text search engine, then that full-text search engine can be directly used for data storage and querying. If the search engine module is built using multiple different full-text search engines, then the appropriate full-text search engine can be selected for both the data storage and querying stages. For example, in the data querying stage, if the search engine module has multiple different full-text search engines, the priority of using the full-text search engines in the search engine module can be determined based on the query complexity of the user's query conditions, and then the appropriate full-text search engine can be selected to perform the data query. Similarly, in the data storage stage, the storage complexity can be calculated based on information such as the load of each full-text search engine and the size of the business operation record data. The priority of using the full-text search engines can be determined based on the storage complexity, and then the appropriate full-text search engine can be selected for data storage according to this priority.

[0048] In some embodiments of the present invention, when multiple different full-text search engines exist in the search engine module, a multi-engine collaborative architecture can be constructed using a query complexity assessment module during the data query stage. The query complexity assessment module mainly consists of a query parser and a classifier. The query parser extracts information related to query complexity from the user's query conditions. It can be constructed using neural networks such as GloVe (Global Vectors for Word Representation) or BERT (Bidirectional Encoder Representations from Transformers), or large language models. The classifier classifies the complexity levels based on the information extracted by the query parser, and then the query complexity assessment module determines the full-text search engine to execute the query task. The information related to query complexity mentioned in this application can include statistical features of the data extracted based on the user's query conditions (such as the number of aggregation functions in the user's query conditions) and hierarchical features of the data (such as the nesting depth in the user's query conditions). The information related to query complexity may also include the estimated amount of full data corresponding to the user's query conditions and the number of filtering conditions, etc., but the present invention is not limited to these.

[0049] For example, suppose the search engine module is a hybrid query cluster built on ElasticSearch and Solr. The user's query conditions (or user query requests containing user query conditions) are input into the query parser, which extracts the data statistical features and data hierarchical structure features corresponding to the user's query conditions. The extracted data statistical features and data hierarchical structure features are concatenated into an overall feature (which can be called the complexity evaluation feature corresponding to the user's query conditions) through weighting and other methods. The complexity evaluation feature is then input into a classifier (such as a decision tree model), and the output is a quantified query complexity (such as the classifier can quantify the query complexity into a score of 0 to 10).

[0050] This invention does not address the method of selecting a full-text search engine for data querying from multiple full-text search engines based on the obtained query complexity. For example, when the query complexity is quantified as 0 to 10 points, if the query complexity corresponding to the query condition is ≥7 points, Solr is used to execute the complex aggregation query; if the query complexity is ≤5 points, the query task is assigned to Elasticsearch; if the query complexity is between (5,7), either Elasticsearch or Solr is selected to execute the data query task.

[0051] Similar to the process of selecting a full-text search engine during data querying, this application will not elaborate on the selection process of a full-text search engine during data storage. To facilitate subsequent data verification and synchronization between the data indexing system and the structured database, this application designs that, during the data storage stage, multiple full-text search engines in the search engine module must apply a cross-engine data synchronization protocol to ensure the consistency of storage mapping relationships in each full-text search engine.

[0052] This application utilizes reinforcement learning algorithms to establish an index optimization mechanism to ensure the search engine module maintains optimal query efficiency. The specific process for online optimization of the search engine module based on this index optimization mechanism is as follows: Under set update conditions, the search engine module obtains historical query log data from the business system and uses a reinforcement learning algorithm to adaptively update the parameters of the full-text search engine used to construct the search engine module based on the historical query log data, thereby achieving search performance tuning. The set update conditions can include meeting a set update time (e.g., performing strategy evaluation and update every 5 minutes) or the search engine module's query efficiency being lower than a set query efficiency threshold, etc., and this invention is not limited to these. The historical query log data can include query context and query performance indicators. The query context can include information such as user query conditions, the size of the target full data corresponding to the user query conditions, query type, query keyword distribution characteristics, and query load. Query performance indicators can include performance indicators such as query latency, query response time, and query throughput. For example, based on the description of steps S11-S13, the parameters of the full-text search engine can include relevant parameters of the index sharding strategy and keyword weighting strategy.

[0053] In some embodiments of the present invention, reinforcement learning algorithms are used to update the parameters of the full-text search engine used to construct the search engine module based on historical query log data, including: using the parameters of the full-text search engine as actions, the query context as the state, and the query performance index as the reward; using an intelligent agent to update the parameters of the full-text search engine used to construct the search engine module based on the query context, so as to maximize the query performance index through iterative learning (maximizing the query performance index in this application refers to improving the query performance of the search engine module to the highest level).

[0054] As an example, after the reinforcement learning algorithm is updated, a canary release mechanism should be adopted to avoid index invalidation affecting user operations, such as validating the full-text search engine parameters with 10% traffic. In addition to updating the full-text search engine parameters, large language models can also be used to optimize the statements or code of query conditions in the search engine module.

[0055] In some embodiments of the present invention, to further improve query speed, this application may also construct a caching module in the data indexing system to cache business operation record data related to query hot information, and establish a caching architecture in conjunction with the caching function of the business system and the caching module. For example, it may be designed to store only the business operation record data related to query hot information in the caching module; or it may be designed with a three-level caching structure, using the local memory cache of the business system to store business operation record data related to ultra-hot query information (e.g., query information with a time-to-live (TTL) of 5 minutes is considered ultra-hot information), using the Redis cluster of the business system to store business operation record data related to medium-hot query information (e.g., query information with a TTL of 30 minutes is considered medium-hot information), and using the built-in cache of the search engine module to store business operation record data related to cold query information (e.g., information with a TTL of 2 hours is considered cold query information). This application does not specifically limit the design of the caching architecture and can be adjusted according to requirements.

[0056] Assuming that only business operation records related to query hotspot information are stored in the caching module, the caching process is as follows: The caching module retrieves historical user query behavior data from the business system and inputs this data (or popularity-related features extracted from historical user query behavior data) into a pre-trained popularity prediction model to predict query hotspot information within a set future time period. This allows the business operation records related to query hotspot information to be stored in the caching module within that future time period. The popularity prediction model can be built based on time series models or large language models, etc. Historical user query behavior data can include historical query frequency, time period distribution of query types, and click-through rate of query results, etc., but this invention is not limited to these. To ensure the accuracy of the query hotspot information in the caching module, the prediction results can be updated every 15 minutes using the popularity prediction model, and the top 20% of hot queries can be marked.

[0057] The caching module needs to be placed before the search engine module. If the cache module does not contain the full data corresponding to the user's query conditions, the search engine module can be used for further data retrieval. This application can use a cache penetration protection mechanism to set a Bloom filter in the cache module. This Bloom filter is used to transmit the user's query request to the search engine module if the cache module does not contain the full data corresponding to the user's query conditions.

[0058] As an example, the user-facing interface of the business system in this application can be designed as an idempotent API (Application Programming Interface) to avoid users repeatedly submitting tasks when operating the business system. Furthermore, this application can deploy a load balancing module in front of the business system and the data search system to monitor the system's health status in real time and automatically switch over faulty nodes.

[0059] This application supports queries in traditional databases handling large volumes of data by extracting and storing complex query conditions, ensuring data consistency across data source structures, employing a multi-engine collaborative architecture, implementing dynamic index optimization mechanisms, and building a caching module, while also achieving high query efficiency. Specifically, the method proposed in this application has the following significant advantages: ① A full-text search engine was used to build an index query system (i.e., a data index system) between the business system and the structured database, and all complex query conditions (such as complex queries involving multiple table fields and query conditions for non-table keys in the case of table partitioning) were pre-encapsulated into the data index system.

[0060] ② All field information in the business operation record data is centrally stored in the structured database, while only the mapping relationship between the master data index and the query conditions is stored in the data index system, thus avoiding a large resource pressure on the data index system.

[0061] ③ By using a dual-write method to store business operation records in the database, real-time queries can be achieved. Furthermore, the query scheme proposed in this application has a lower cost compared to data warehouse query methods, and its query efficiency is almost the same as that of cached query methods, and it is applicable to all complex query conditions.

[0062] ④ Since the business system stores the source data of business operation records and the structured database stores the full data after it has been stored, the data indexing system designed in this application will have a large margin of error tolerance and can adaptively design various index architectures based on requirements.

[0063] Corresponding to the above method, the present invention also provides a search engine-based query device (i.e., a data indexing system), which includes a computer device, the computer device including a processor and a memory, the memory storing computer programs / instructions, the processor executing the computer programs / instructions stored in the memory, and when the computer programs / instructions are executed by the processor, the device implements the steps of the method described above.

[0064] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0065] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0066] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0067] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A search method based on a search engine, characterized in that, This method is implemented by a data indexing system, which includes a search engine module built based on a full-text search engine. The method includes the following steps: The search engine module receives user query requests containing user query conditions from the business system; wherein, the query request is a programming language generated based on the natural language query text input by the user into the business system. The search engine module obtains the master data index corresponding to the query request based on the pre-stored mapping relationship between query conditions and master data index and the query request, and returns it to the business system, so that the business system can obtain the full data corresponding to the query request from the structured database that pre-stores the mapping relationship between master data index and full data based on the returned master data index; The mapping relationship between the query conditions and the master data index is obtained in the following way: During the data storage phase, the search engine module retrieves business operation record data from the business system; wherein, the business operation record data includes the master data index and the corresponding field information of the full data. The search engine module extracts the query conditions corresponding to each master data index from the full data corresponding to each master data index, and establishes a mapping relationship between the extracted query conditions and the corresponding master data index.

2. The method according to claim 1, characterized in that, The data indexing system also includes a query complexity evaluation module, which includes a query parser and a classifier; If the search engine module has multiple different full-text search engines, the method further includes the following steps before the search engine module receives the user query request: The query parser in the query complexity evaluation module receives the user query request containing the user query conditions and extracts the data statistical features and data hierarchical structure features corresponding to the user query conditions. The extracted statistical features and hierarchical features of the data are input into the classifier in the query complexity evaluation module, and the query complexity corresponding to the user query conditions is output. Based on the obtained query complexity, a full-text search engine is selected to determine the master data index corresponding to the user query request.

3. The method according to claim 1, characterized in that, The method further includes: under set update conditions, the search engine module obtains historical query log data from the business system, and uses a reinforcement learning algorithm to update the parameters of the full-text search engine used to construct the search engine module based on the historical query log data.

4. The method according to claim 3, characterized in that, The historical query log data includes query context and query performance metrics; The step of using reinforcement learning algorithms to update the parameters of the full-text search engine used to construct the search engine module based on the historical query log data includes: By leveraging an intelligent agent, the parameters of the full-text search engine used to build the search engine module are updated based on the query context, in order to maximize query performance metrics through iterative learning.

5. The method according to claim 1, characterized in that, For real-time business operations, during the data storage phase, the search engine module retrieves the business operation record data at the same time as the structured database retrieves the business operation record data from the business system.

6. The method according to claim 1, characterized in that, The data indexing system also includes a transaction coordination module; The timing at which the search engine module retrieves business operation record data coincides with the timing at which the structured database retrieves business operation record data from the business system is achieved through the following method: The transaction coordination module deploys the task of the search engine module to obtain business operation record data and the task of the structured database to obtain business operation record data from the business system in the same transaction, and uses an attempt-confirm-cancel mechanism to execute these two tasks simultaneously.

7. The method according to claim 1, characterized in that, The method further includes: Under the set trigger conditions, after the business system sends a first data verification request to the structured database and obtains the full data corresponding to the stored master data index from the structured database, the search engine module receives a second data verification request from the business system; wherein, the second data verification request includes the master data index information obtained by the business system based on the first data verification request; The search engine module compares the master data index information in the second data verification request with the pre-stored mapping relationship between the query conditions and the master data index. If the search engine module contains a master data index that is not included in the structured database, the search engine module sends a first data synchronization instruction containing the specific master data index to the business system, causing the business system to send the corresponding business operation record data to the structured database. If the structured database contains a master data index that is not included in the search engine module, the search engine module sends a second data synchronization instruction containing the specific master data index to the business system and receives the corresponding business operation record data from the business system, thereby updating the mapping relationship between the query conditions and the master data index.

8. The method according to claim 1, characterized in that, The data indexing system also includes a caching module; The method further includes: obtaining historical query behavior data of users from the business system and inputting it into a pre-trained popularity prediction model to predict query hotspot information within a set future time period, thereby storing the business operation record data related to the query hotspot information in the cache module within the set future time period.

9. The method according to claim 8, characterized in that, The caching module includes a Bloom filter, which is used to transmit the user query request to the search engine module when the caching module does not have full data corresponding to the user query conditions.

10. A search engine-based query device, comprising a processor, a memory, and a computer program / instructions stored in the memory, characterized in that, The processor is configured to execute the computer program / instructions, and when the computer program / instructions are executed, the device implements the steps of the method as described in any one of claims 1 to 9.

Images