Database retrieval processing method, device and equipment based on analyzer index

Abstract

The embodiment of the invention discloses a database retrieval processing method, device and equipment based on an analyzer index. A specific embodiment of the method comprises the following steps: carrying out query analysis on a retrieval task request to obtain query element information; performing query verification on the target analyzer name to obtain a verification result, and performing association query on the target analyzer name and the target column identifier to obtain a target inverted index identifier; generating query execution information according to the target inverted index identifier and the search word text; according to the query execution information, triggering physical query information, and according to the physical query information, determining a corresponding reverse index reader in a reverse index reader resource pool; generating a row identifier set according to the inverted index reader and the search word text; and obtaining a data record set according to the row identifier set, and sending the data record set to a client corresponding to the target user. According to the embodiment, the query delay and the resource load can be reduced.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular, to a database retrieval processing method, device and equipment based on an analyzer index. BACKGROUND

[0002] In the field of database full-text retrieval, the traditional method usually stores data redundantly (i.e., replicates multiple column data in a table to implement different search strategies) or uses a single index strategy (i.e., only establishes an inverted index of a fixed segmenter for the same column) to support multiple search requirements of the same text field.

[0003] However, when performing full-text retrieval on a database in the above manner, the following technical problems often exist: The single index strategy cannot be flexibly adapted to multiple search scenarios (such as precise matching requiring standard segmentation), so that users cannot select a segmentation strategy as needed, resulting in low retrieval efficiency and insufficient accuracy of retrieval results, and when burst traffic is concentrated on a single search strategy, the inverted index reader bound to the analyzer becomes a bottleneck resource, a large number of concurrent queries need to monopolize processing time, resulting in query delay and increased resource load.

[0004] The above information disclosed in this BACKGROUND section is only for the purpose of enhancing the understanding of the background of the present disclosure and, therefore, can include information that does not form the prior art that is already known in this country to those ordinary skilled in the art. SUMMARY

[0005] The summary of the present disclosure is intended to introduce the general concepts of the present disclosure in a simplified form, which will be described in detail in the specific embodiments section below. The summary of the present disclosure is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of the present disclosure propose a database retrieval processing method, device, electronic equipment and computer readable medium based on an analyzer index to solve one or more of the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of the present disclosure provide a database retrieval processing method based on an analyzer index, the method comprising: in response to receiving a retrieval task request of a target user on a database, performing query analysis on the retrieval task request to obtain query element information, wherein the query element information comprises a target column identifier, a search word text, and a target analyzer name; performing query verification on the target analyzer name to obtain a verification result, and in response to the verification result indicating validity, performing associated query on the target analyzer name and the target column identifier in an index mapping information set to obtain a target inverted index identifier; generating query execution information according to the target inverted index identifier and the search word text; triggering physical query information according to the query execution information, and determining a corresponding inverted index reader in an inverted index reader resource pool according to the physical query information; generating a row identifier set according to the inverted index reader and the search word text; obtaining a data record set in a target data storage system according to the row identifier set, and sending the data record set to a client corresponding to the target user.

[0008] In a second aspect, some embodiments of the present disclosure provide a database retrieval processing apparatus based on a multi-analyzer index, the apparatus comprising: a query analysis unit configured to, in response to receiving a retrieval task request of a target user on a database, perform query analysis on the retrieval task request to obtain query element information, wherein the query element information comprises a target column identifier, a search word text, and a target analyzer name; a query verification unit configured to perform query verification on the target analyzer name to obtain a verification result, and in response to the verification result indicating validity, perform associated query on the target analyzer name and the target column identifier in an index mapping information set to obtain a target inverted index identifier; a generation unit configured to generate query execution information according to the target inverted index identifier and the search word text; a triggering unit configured to trigger physical query information according to the query execution information, and determine a corresponding inverted index reader in an inverted index reader resource pool according to the physical query information; a second generation unit configured to generate a row identifier set according to the inverted index reader and the search word text; an obtaining unit configured to obtain a data record set in a target data storage system according to the row identifier set, and send the data record set to a client corresponding to the target user.

[0009] In a third aspect, some embodiments of the present disclosure provide an electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method described in any of the implementations of the first aspect.

[0010] In a fourth aspect, some embodiments of the present disclosure provide a computer readable medium having stored thereon a computer program, wherein the program, when executed by a processor, implements the method described in any implementation manner of the first aspect.

[0011] The above various embodiments of the present disclosure have the following beneficial effects: the database retrieval processing method based on analyzer index of some embodiments of the present disclosure can reduce query delay and reduce resource load. Specifically, the reason for causing query delay and increasing resource load is that a single index strategy cannot be flexibly adapted to multiple search scenarios (such as accurate matching requiring standard segmentation), so that the user cannot select the segmentation strategy as needed, resulting in low retrieval efficiency and insufficient accuracy of retrieval results, and when burst traffic is concentrated on a single search strategy, the analyzer-bound inverted index reader becomes a bottleneck resource, and a large number of concurrent queries need to monopolize processing time, resulting in query delay and increasing resource load. Based on this, the database retrieval processing method based on analyzer index of some embodiments of the present disclosure, first, in response to receiving a retrieval task request of a target user to a database, query analysis is performed on the above retrieval task request to obtain query element information, wherein the query element information includes a target column identifier, a search word text and a target analyzer name. Thus, the user can select the segmentation strategy as needed. Secondly, the target analyzer name is queried and verified to obtain a verification result, and in response to the verification result representing validity, the target analyzer name and the target column identifier are associated and queried in the index mapping information set to obtain a target inverted index identifier. Thus, the use of invalid analyzers or incorrect indexes can be avoided, and the problem of a single index being unable to adapt to multiple scenarios can be solved. Thirdly, query execution information is generated according to the target inverted index identifier and the search word text. Thus, the user query information can be converted into executable query instructions. Next, physical query information is triggered according to the query execution information, and a corresponding inverted index reader in the inverted index reader resource pool is determined according to the physical query information. Thus, the concentrated traffic can be diverted from the overloaded reader to the low-load reader. Then, a row identifier set is generated according to the inverted index reader and the search word text. Thus, the index query can be completed based on the analyzer specified by the user, and the retrieval logic can be ensured to be consistent with the user's needs. Finally, a data record set is obtained in the target data storage system according to the row identifier set, and the data record set is sent to the client corresponding to the target user. Thus, it can be ensured that the returned records are the matching results retrieved by the user-specified analyzer strategy, and invalid data queries caused by strategy mismatch can be avoided. Ultimately, query delay can be reduced and resource load can be reduced. BRIEF DESCRIPTION OF DRAWINGS

[0012] The above and other features, aspects and advantages of various embodiments of the present disclosure will become more apparent from the following detailed description, taken in conjunction with the accompanying drawings, in which like reference numerals refer to like elements throughout. The drawings are not necessarily to scale, the emphasis instead being placed upon illustrating the principles of the embodiments of the present disclosure.

[0013] Figure 1 is a flowchart of some embodiments of the database retrieval processing method based on analyzer index according to the present disclosure; Figure 2 is a structural schematic diagram of some embodiments of the database retrieval processing apparatus based on multi-analyzer index according to the present disclosure; Figure 3 is a structural schematic diagram of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0014] Embodiments of the present disclosure will be described in detail with reference to the drawings, wherein the same or similar components are denoted by the same reference numerals, and therefore repeated description is omitted. While the present disclosure is shown and described in connection with certain embodiments, it is not intended to be limited to the presented embodiments, but instead, the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and fully convey the scope of the present disclosure to those skilled in the art. It should be noted that the drawings are only for the purpose of illustrating preferred embodiments of the present disclosure and are not intended to limit the present disclosure.

[0015] It should also be noted that, in the drawings, only parts related to the present disclosure are shown. The embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0016] It should be noted that the terms "first", "second", and the like in the present disclosure are used only to distinguish different devices, modules or units, and do not imply the order or interdependence of the functions performed by these devices, modules or units.

[0017] It should be noted that the terms "one", "multiple" in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that "one" or "multiple" should be understood as "one or more" unless otherwise explicitly stated in the context.

[0018] The names of the messages or information exchanged between the devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0019] The present disclosure will be described in detail with reference to the drawings and in conjunction with embodiments.

[0020] Figure 1Flow 100 illustrates some embodiments of a database retrieval processing method based on an analyzer index according to the present disclosure. The database retrieval processing method based on an analyzer index includes the following steps: Step 101, in response to receiving a retrieval task request of a target user to a database, performing query parsing on the retrieval task request to obtain query element information.

[0021] In some embodiments, the execution subject (e.g., a server) of the database retrieval processing method based on an analyzer index can perform query parsing on the retrieval task request to obtain query element information in response to receiving a retrieval task request of a target user to a database. The retrieval task request can be a SQL query string. The analyzer index can be an inverted index constructed after the analyzer processes the text. The analyzer (Analyzer) can be a component for converting the search word text input by the user into a minimum semantic unit (term set) suitable for index matching to support different search scenarios, such as a MySQL query analyzer, which can include a standard tokenization analyzer, a prefix tokenization analyzer, and a keyword tokenization analyzer.

[0022] In some optional implementations of some embodiments, the execution subject can perform query parsing on the retrieval task request to obtain query element information by the following steps: Step one, performing lexical analysis on the retrieval task request to obtain a sequence of lexical units. In practice, the execution subject can identify the syntax elements in the retrieval task request by a predefined regular expression rule to obtain a sequence of lexical units. The predefined regular expression rule can be used to split the continuous request text into independent lexical units. As an example, the regular expression rule can be used to split “SELECT FROM users WHEREfirst_name MATCH ‘alice’ USING ANALYZER ‘my_ngram__analyzer’” into [SELECT, , FROM, users, WHERE, first_name, MATCH, ‘alice’, USING, ANALYZER, ‘my_ngram__analyzer].

[0023] Step two, performing syntax analysis on the above sequence of lexical units to construct an abstract syntax tree. In practice, the above execution subject can construct the above sequence of lexical units into a tree structure according to the structure of the SELECT statement to obtain an abstract syntax tree. The abstract syntax tree (AST) can include a root node and a child node. The root node can be a SELECT query node (used to represent that the query statement is a SELECT query statement). The child node can include a FROM clause node (used to store the table name, such as the table name "users"), a WHERE clause node (used to store the query condition). The WHERE clause can include a MATCH predicate node. The MATCH predicate node can be used to store the retrieval condition, including a column name node (storing the target column identifier, such as "first_name"), a query text node (used to store the search word text, such as "'alice'"), and an extension clause node (used to store additional parameters such as the analyzer name, such as USING ANALYZER'my_ngram__analyzer').

[0024] Step three, generating a target predicate node according to the abstract syntax tree. The target predicate node can be the retrieval condition stored in the MATCH predicate node in the abstract syntax tree.

[0025] Step four, extracting the target column identifier from the target predicate node to obtain the target column identifier. The target column identifier can be the target column identifier stored in the column name node in the abstract syntax tree.

[0026] Step five, extracting the search word text from the target predicate node to obtain the search word text. The search word text can be the search word text stored in the query text node in the abstract syntax tree.

[0027] Step six, extracting the analyzer name from the target predicate node to obtain the target analyzer name. In practice, the execution subject can query whether the abstract syntax tree contains an extension clause node. In response to the extension clause node being not empty, the analyzer name (such as my_ngram__analyzer) stored in the extension clause node is taken as the target analyzer name. In response to the extension clause node being empty, the default analyzer name is taken as the target analyzer name. The default analyzer name can be the name of the analyzer corresponding to the pre-set standard segmentation rule.

[0028] Step seven, determining the target column identifier, the search word text, and the target analyzer name as query element information.

[0029] In step 102, the target analyzer name is queried and verified, a verification result is obtained, and in response to the verification result indicating validity, the target analyzer name and the target column identifier are associated and queried in the index mapping information set, and a target inverted index identifier is obtained.

[0030] In some embodiments, the execution subject can query and verify the target analyzer name, obtain a verification result, and in response to the verification result indicating validity, associate and query the target analyzer name and the target column identifier in the index mapping information set, and obtain a target inverted index identifier. The index mapping information in the index mapping information set can be information for representing the correspondence relationship of the target column identifier, the target analyzer name, and the inverted index identifier.

[0031] In the process of adopting the technical solutions to solve the technical problems in the background art, for the scene to be applied: mixed execution of high-concurrency queries and index state maintenance during business peak period or high-concurrency transactions and malicious query scenarios (such as during e-commerce promotion, a large number of users place orders and pay, and real-time sales analysis tables need to be generated), the following technical problems often occur: during the online reconstruction of high-frequency inverted indexes during business peak periods (such as e-commerce promotion, social platform hot events, and financial transaction peaks), the index reconstruction process is in an intermediate state (such as the switching between new and old files is not completed), resulting in errors and incompleteness of the final query results. In view of the following demand characteristics required for this application scenario: multiple concurrent queries, query data consistency, and efficient retrieval of target indexes, the following solutions are adopted: In some optional implementations of some embodiments, the execution subject can query and verify the target analyzer name, obtain a verification result, and in response to the verification result indicating validity, associate and query the target analyzer name and the target column identifier in the index mapping information set, and obtain a target inverted index identifier by the following steps: First, an initial query snapshot version is obtained. In practice, the execution subject can sequentially combine a local physical clock (which can be a TSC timestamp counter, such as 1650000000) and a logical counter (initially 0, which is incremented by 1 when a new transaction is submitted, such as 3) to generate an initial query snapshot version (such as 1650000000, 3). The initial query snapshot version can be used to represent a logical timestamp (such as a logical clock HLC timestamp) for data consistency.

[0032] Secondly, the initial query snapshot version is clock calibrated to obtain a query snapshot version. In practice, the execution subject can adjust the initial query snapshot version according to a preset maximum allowed deviation (e.g., ±5 ms) to obtain the query snapshot version. As an example, in response to a deviation direction of "+2 ms", the clock value of the initial query snapshot version is reduced by 2 ms (e.g., 1650000000 is adjusted to 1649999998). The maximum allowed deviation can include a deviation direction and a global physical clock reference. The deviation direction is used to represent the fast or slow of the local physical clock relative to the global physical clock, which has positive and negative (e.g., "+2 ms" represents 2 ms faster than the local clock). The global physical clock reference is used to represent the latest global physical clock value maintained by the time service (e.g., 1650000002, which represents 2 ms behind the physical clock part of the initial query snapshot version).

[0033] Thirdly, the query snapshot version and the target analyzer name are used to perform snapshot reading on the multi-version analyzer metadata set to obtain a target analyzer metadata record. In practice, the execution subject can read the metadata record corresponding to the target analyzer name from the multi-version analyzer metadata set according to the query snapshot version to obtain the target analyzer metadata record. The multi-version analyzer metadata in the multi-version analyzer metadata set can represent the configuration information of the analyzer. The configuration information of the analyzer can include the analyzer name and the TokenFilters.

[0034] Fourthly, the target analyzer metadata record is subjected to multi-dimensional verification to obtain a verification result. The multi-dimensional verification includes integrity verification, validity verification, and compatibility verification. The integrity verification can be used to verify whether the target analyzer metadata record has null values. As an example, the integrity verification can be: in response to the target analyzer metadata record having no null values, the verification is successful. The validity verification is used to verify the state field of the target analyzer metadata record. As an example, the validity verification can be: in response to the state field being represented as "enabled", the verification is successful. The compatibility verification is used to verify whether the query snapshot version is within the range of the version field of the target analyzer metadata record. As an example, the compatibility verification can be: in response to the query snapshot version being within the range of the version field (e.g., "V2.0-V3.5") of the target analyzer metadata record, the verification is successful. The verification result can be used to represent validity or invalidity. As an example, in response to the integrity verification, the validity verification, and the compatibility verification all representing successful verification, a verification result representing validity is obtained.

[0035] In the fifth step, the target analyzer metadata record and the verification result are abnormally corrected to obtain a final verification result. In practice, in response to the integrity verification failure, the compatibility verification and the validity verification success, the execution subject can supplement the analyzer name and the token filter in the target analyzer metadata record with the standard tokenization rule analyzer and the standard tokenization rule to obtain the final verification result. The final verification result can be used to represent validity or invalidity.

[0036] In the sixth step, in response to the final verification result representing validity, a consistent hash query is performed in the index mapping information set according to the target column identifier, the target analyzer name and the query snapshot version to obtain a candidate index record table. In practice, in response to the final verification result representing validity, the execution subject can perform a consistent hash query in the index mapping information set according to the target column identifier, the target analyzer name and the query snapshot version to obtain a candidate index record table by the following steps: In the first step, an enhanced hash value is generated according to the target column identifier, the target analyzer name and the query snapshot version. In practice, the execution subject can determine the target column identifier (such as “user_name”), the target analyzer name (such as “my_ngram_analyzer”) and the query snapshot version (such as “V2.5”) as a hash key (such as “user_name:my_ngram_analyzer:V2.5”). Then, a dynamic random salt value is introduced for the hash key to obtain an enhanced hash key (such as “user_name:my_ngram_analyzer:V2.5:salt_1234”). As an example, the last four digits of the current timestamp can be determined as a dynamic random salt value (such as “salt_1234”). Finally, the hash value of the enhanced hash key is determined through a hash algorithm (such as MD5, SHA-256, etc.) to obtain an enhanced hash value.

[0037] In the second step, a shard corresponding to the enhanced hash value is queried in the index mapping information set to obtain a shard list. In practice, the execution subject can find a shard matching the enhanced hash value and the query snapshot version in the index mapping information set to obtain a shard list. The shards in the shard list can be used to store each index mapping information in the index mapping information set.

[0038] Step three, interval tree acceleration retrieval is performed on the above-mentioned shard list to obtain a candidate index record table. In practice, the above-mentioned execution subject can sort the shards in the above-mentioned shard list in ascending order according to the index creation version number corresponding to the shards to obtain a shard sequence. Then, through the interval tree algorithm, index records in the above-mentioned shard list that match the prefix of the above-mentioned hash key (target column identifier and target analyzer name) and cover the above-mentioned query snapshot version in the version range are retrieved to obtain a candidate index record table. Among them, the candidate index records in the above-mentioned candidate index record table can include index identifier (which can be the storage path of the physical file), index creation version number.

[0039] Step seven, according to the above-mentioned candidate index record table, a candidate index record sequence is generated. In practice, the above-mentioned execution subject can convert the above-mentioned index creation version number into a digital string (such as V2.3 converted into 203), and sort the candidate index records in the above-mentioned candidate index record table in ascending order to obtain a candidate index record sequence.

[0040] Step eight, based on the above-mentioned query snapshot version, binary search is performed on the above-mentioned candidate index record sequence to obtain an effective index record. In practice, the above-mentioned execution subject can find the candidate index record corresponding to the maximum version number that meets the condition of "index creation version number ≤ query snapshot version" in the above-mentioned candidate index record sequence through binary search, as the effective index record.

[0041] Step nine, the index file physical fingerprint and the data snapshot anchor point corresponding to the above-mentioned effective index record are extracted. Among them, the index file physical fingerprint can be the hash digest (such as SHA-256 value) of the index file pointed to by the above-mentioned effective index record, which is used to uniquely identify the file entity. The data snapshot anchor point can be the storage location identifier of the data snapshot pointed to by the above-mentioned effective index record in the index construction.

[0042] Step ten, according to the above-mentioned index file physical fingerprint and the above-mentioned data snapshot anchor point, the target inverted index identifier is generated. In practice, the above-mentioned execution subject can splice the above-mentioned index file physical fingerprint and the above-mentioned data snapshot anchor point to obtain the target inverted index identifier.

[0043] The first step to the tenth step and its related content as one of the invention points of the embodiment of the present disclosure solves the technical problem that the index reconstruction process is in an intermediate state (such as the new and old file switching is not completed) when the high-frequency inverted index is reconstructed online during the business peak (such as e-commerce promotion, social platform hot events, financial transaction peak), resulting in errors and incompleteness of the final query result. The reason for the final query error or incomplete data is that the index reconstruction process is in an intermediate state (such as the new and old file switching is not completed) when the high-frequency inverted index is reconstructed online during the business peak (such as e-commerce promotion, social platform hot events, financial transaction peak), resulting in errors and incompleteness of the final query result. If the above factors are solved, the problem of query data error or incomplete data can be solved. In order to achieve this effect, first, the initial query snapshot version is obtained. Thus, the target data state to be matched during retrieval can be determined. Second, the initial query snapshot version is clock calibrated to obtain a query snapshot version. Thus, the problem of unordered versions caused by clock deviation can be solved. Third, the multi-version analyzer metadata set is read according to the query snapshot version and the target analyzer name to obtain the target analyzer metadata record. Thus, the analyzer metadata corresponding to the time node can be read based on the query snapshot version. Fourth, the target analyzer metadata record is verified in multiple dimensions to obtain a verification result. Thus, the effectiveness of the analyzer can be verified to ensure that the analyzer is in a usable state. Fifth, the target analyzer metadata record and the verification result are abnormally corrected to obtain a final verification result. Thus, fault tolerance can be improved. Sixth, in response to the fact that the final verification result is valid, a consistent hash query is performed in the index mapping information set according to the target column identifier, the target analyzer name, and the query snapshot version to obtain a candidate index record table. Thus, the candidate index record corresponding to the query snapshot version can be screened out. Seventh, the candidate index record sequence is generated according to the candidate index record table. Thus, the efficiency of the binary search can be improved. Eighth, the binary search is performed on the candidate index record sequence based on the query snapshot version to obtain an effective index record. Thus, the index created under the query snapshot version can be accurately located and determined to be up-to-date. Ninth, the index file physical fingerprint and the data snapshot anchor point corresponding to the effective index record are extracted. Thus, the integrity of the index file can be verified and the data state during construction can be associated. Tenth, the target inverted index identifier is generated according to the index file physical fingerprint and the data snapshot anchor point. Thus, the index instance that is effective, complete, and matches the query snapshot version can be uniquely identified. Finally, the problem of query data error or incomplete data can be solved.

[0044] In step 103, query execution information is generated according to the target inverted index identifier and the search word text.

[0045] In some embodiments, the execution subject can generate query execution information according to the target inverted index identifier and the search term text.

[0046] In some optional implementations of some embodiments, the execution subject can generate query execution information according to the target inverted index identifier and the search term text by the following steps: Step one, generate a physical storage path set and a backend node identifier set according to the target inverted index identifier. In practice, the execution subject can query the physical storage path and the backend node identifier corresponding to the target inverted index identifier in the index metadata registry to obtain the physical storage path set and the backend node identifier set. The index metadata registry can be a system table for storing all inverted index configuration information. The inverted index configuration information can include index identifier, physical file path and backend node identifier. The backend node identifier can be the unique identifier of the backend node (physical server or storage node) storing the inverted index.

[0047] Step two, generate index operation description information according to the search term text. In practice, the execution subject can combine the analyzer rule corresponding to the target inverted index identifier and the search term text into index operation description information. The index operation description information can include search term text and analyzer rule. The analyzer rule can be a predefined query rule, which can include standard tokenization rule, prefix tokenization rule and keyword tokenization rule. The standard tokenization rule can be used in full-text retrieval scenarios to divide text into independent word items according to word boundaries. For example, spaces, punctuation and commonly used stop words can be used as word boundaries for division. The prefix tokenization rule can be used for automatic completion to divide text from the starting position into multiple word strings with increasing length. For example, "alice" can be divided into "a", "al", "ali", "alic" and "alice". The keyword tokenization rule can be used in precise matching scenarios without tokenization processing of text, and the entire text is taken as a word item.

[0048] Step three, generate query execution fragment information according to the index operation description information, the target inverted index identifier and the physical storage path set. In practice, the execution subject can combine the index operation description information, the target inverted index identifier and the physical storage path set into query execution fragment information.

[0049] Step four, generating query execution information according to the query execution fragment information and the set of backend node identifiers. In practice, the execution subject can group the query execution fragment information by backend node identifier to obtain a set of node fragment mappings. Then, the execution subject adds execution parameters to the set of node fragment mappings to obtain an updated set of node fragment mappings. After that, the execution subject combines the updated set of node fragment mappings, the set of backend node identifiers, and the target inverted index identifier into query execution information. The execution parameters can include execution priority (e.g., high, medium, and low) and timeout time (e.g., 500 ms).

[0050] Step 104, triggering physical query information according to the query execution information, and determining the corresponding inverted index reader in the inverted index reader resource pool according to the physical query information.

[0051] In some embodiments, the execution subject can trigger physical query information according to the query execution information, and determine the corresponding inverted index reader in the inverted index reader resource pool according to the physical query information. The inverted index reader resource pool can be a resource pool for managing inverted index readers. The inverted index reader can be associated with an analyzer and inherit the query rules and configuration parameters of the analyzer. The configuration parameters can include but are not limited to tokenization rules, stop word lists, and case conversion. As an example, the inverted index reader can be PostingsEnum in Lucene.

[0052] In the process of applying the technical solutions to solve the technical problems in the background art, for the scenario to be applied: sudden traffic is concentrated in a single search strategy (e.g., a large number of users use the same search mode and specify the same analyzer due to a hot event, such as searching for hot keywords through prefix association search), which is often accompanied by the following technical problems: a large number of search methods of the same type cause the inverted index reader bound to the hot analyzer to become a bottleneck resource, a large number of concurrent queries monopolize the processing time of the reader, causing a sharp increase in request delay, the readers corresponding to other analyzers are in an idle state, resources cannot be utilized evenly, and thus resource utilization is reduced. For the following demand characteristics required for this application scenario: dynamic perception of real-time load of the reader and balanced allocation, we decide to use the following solutions: In some optional implementations of some embodiments, the execution subject can trigger physical query information according to the query execution information, and determine the corresponding inverted index reader in the inverted index reader resource pool according to the physical query information by the following steps: In the first step, a real-time load dataset of the inverted index reader resource pool is obtained. The real-time load data in the real-time load dataset can include the number of queries to be processed, CPU usage, memory occupancy, a current analyzer name, and an inverted index identifier.

[0053] In the second step, a target load dataset is determined according to the real-time load dataset and the target inverted index identifier. In practice, the execution subject can select real-time load data matching the target inverted index identifier from the real-time load dataset as target load data to obtain the target load dataset.

[0054] In the third step, an inverted index reader subset is generated according to the target load dataset. In practice, the execution subject can determine a comprehensive load rate of each target load data in the target load dataset, and determine an average load rate according to the comprehensive load rate of each target load data. Then, an analyzer corresponding to a comprehensive load rate less than a load threshold is determined as a target inverted index reader to obtain the inverted index reader subset. The load threshold can be determined by the product of the average load rate and (1 minus the average load rate). The comprehensive load rate can be obtained by weighted summation of the number of queries to be processed, the CPU usage, and the memory occupancy. As an example, the following formula is used to determine the comprehensive load rate: Comprehensive load rate = number of queries to be processed × 0.4 + CPU usage × 0.3 + memory occupancy × 0.3.

[0055] In the fourth step, a temporary inverted index reader is created in response to the inverted index reader subset being empty. In practice, the execution subject can copy the configuration parameters of the target inverted index reader to an empty inverted index reader to obtain the temporary inverted index reader. The configuration parameters of the temporary inverted index reader are completely consistent with the configuration parameters of the target inverted index reader.

[0056] In the fifth step, an available inverted index reader set is generated according to the temporary inverted index reader and the inverted index reader subset. In practice, the execution subject can determine the temporary inverted index reader and each inverted index reader in the inverted index reader subset as an available inverted index reader to obtain the available inverted index reader set.

[0057] In the sixth step, a target inverted index reader is generated according to the available inverted index reader set. In practice, the execution subject can determine the weight of each available inverted index reader in the available inverted index reader set, and determine the available inverted index reader with the highest weight as the target inverted index reader. The following formula is used to determine the weight: Weight = 1 / Comprehensive Load Rate.

[0058] In the seventh step, the physical query information is generated according to the target inverted index reader and the target inverted index identifier. The physical query information can be an instruction package for the backend node to execute the query, including the reader instance ID, the index file path, and the query token set (e.g., "a", "al", "ali", "alic", "alice"). The reader instance ID can be a unique identifier of the inverted index reader.

[0059] In the eighth step, the corresponding inverted index reader in the inverted index reader resource pool is determined according to the physical query information. In practice, the execution subject can select the inverted index reader corresponding to the reader instance ID in the physical query information from the inverted index reader resource pool.

[0060] The first step to the eighth step and its related content as one of the invention points of the embodiment of the present disclosure solves the technical problem that a large number of search methods of the same type cause the inverted index reader bound to the hot spot analyzer to become a bottleneck resource, a large number of concurrent queries monopolize the processing time of the reader, causing the request delay to surge, the readers corresponding to other analyzers are in an idle state, the resources cannot be balanced, and the resource utilization rate is reduced. The reason for the low resource utilization rate is that a large number of search methods of the same type cause the inverted index reader bound to the hot spot analyzer to become a bottleneck resource, a large number of concurrent queries monopolize the processing time of the reader, causing the request delay to surge, the readers corresponding to other analyzers are in an idle state, the resources cannot be balanced, and the resource utilization rate is reduced. If the above factors are solved, the problem of long transmission time can be solved. In order to achieve this effect, first, according to the query execution information, the real-time load data set of the inverted index reader resource pool is obtained. In this way, the load state of the reader can be fully mastered. Second, according to the real-time load data set and the target inverted index identifier, the target load data set is determined. In this way, the overloaded analyzer can be optimized, and the invalid resource scheduling overhead can be reduced. Third, according to the target load data set, the inverted index reader subset is generated. In this way, the readers with light load can be screened from the existing resources, and the idle resources are preferentially used for query. Fourth, in response to the inverted index reader subset being empty, a temporary inverted index reader is created. In this way, dynamic expansion can be performed when the existing resources are exhausted, and query blocking caused by no available reader can be avoided. Fifth, according to the temporary inverted index reader and the inverted index reader subset, the available inverted index reader set is generated. In this way, the potential available resources can be integrated to provide reasonable resources for task allocation. Sixth, according to the available inverted index reader set, the target inverted index reader is generated. In this way, the query task can be allocated to the most idle reader to achieve load balancing. Seventh, according to the target inverted index reader and the target inverted index identifier, the physical query information is generated. In this way, the reader and the content executing the query task can be determined to ensure that the task is accurately issued to the low-load instance. Eighth, according to the physical query information, the corresponding inverted index reader in the inverted index reader resource pool is determined. In this way, the instance finally executing the query can be locked, and other queries continue to monopolize the bottleneck resource. Finally, the resource utilization rate can be improved.

[0061] In step 105, a row identifier set is generated according to the inverted index reader and the search word text.

[0062] In some embodiments, the execution subject can generate a row identifier set according to the inverted index reader and the search word text.

[0063] In some optional implementations of some embodiments, the execution subject can generate the set of row identifiers from the inverted index reader and the search term text by the following steps: Step one, performing word segmentation on the search term text according to the preset analyzer rule corresponding to the inverted index reader to obtain a set of terms.

[0064] Step two, generating a set of row identifier lists from the set of terms and the inverted index reader. In practice, the execution subject can input each term in the set of terms and the target inverted index identifier into the query interface of the inverted index reader to generate a row identifier list, thereby obtaining a set of row identifier lists.

[0065] Step three, merging each row identifier list in the set of row identifier lists to obtain an initial set of row identifiers.

[0066] Step four, removing duplicate values from the initial set of row identifiers to obtain the set of row identifiers. In practice, the execution subject can remove duplicate values from the initial set of row identifiers to obtain the set of row identifiers.

[0067] Step 106, obtaining a set of data records from the target data storage system according to the set of row identifiers, and sending the set of data records to the client corresponding to the target user.

[0068] In some embodiments, the execution subject can obtain a set of data records from the target data storage system according to the set of row identifiers, and send the set of data records to the client corresponding to the target user. The target data storage system can be a system for storing the set of data records.

[0069] In practice, the execution subject can use each row identifier in the set of row identifiers as a query key to obtain the data record corresponding to the query key from the target data storage system, thereby obtaining the set of data records. Then, the set of data records is encapsulated to obtain a data record file, and the data record file is transmitted to the client through a network transmission protocol. The network transmission protocol can be HTTP protocol. The data record file can be a JSON format file.

[0070] The above various embodiments of the present disclosure have the following beneficial effects: the database retrieval processing method based on the analyzer index of some embodiments of the present disclosure can reduce query delay and reduce resource load. Specifically, the reason for causing query delay and increasing resource load is that a single index strategy cannot be flexibly adapted to multiple search scenarios (such as accurate matching requiring standard segmentation), so that the user cannot select the segmentation strategy as needed, resulting in low retrieval efficiency and insufficient accuracy of retrieval results, and when burst traffic is concentrated on a single search strategy, the analyzer-bound inverted index reader becomes a bottleneck resource, and a large number of concurrent queries need to monopolize processing time, resulting in query delay and increasing resource load. Based on this, the database retrieval processing method based on the analyzer index of some embodiments of the present disclosure, first, in response to receiving a retrieval task request of a target user to a database, the retrieval task request is parsed to obtain query element information, wherein the query element information includes a target column identifier, a search word text and a target analyzer name. Thus, the user can select the segmentation strategy as needed. Secondly, the target analyzer name is queried and verified to obtain a verification result, and in response to the verification result representing validity, in the index mapping information set, the target analyzer name and the target column identifier are associated and queried to obtain a target inverted index identifier. Thus, the use of invalid analyzers or incorrect indexes can be avoided, and the problem of a single index being unable to adapt to multiple scenarios can be solved. Thirdly, according to the target inverted index identifier and the search word text, query execution information is generated. Thus, the user query information can be converted into executable query instructions. Next, according to the query execution information, physical query information is triggered, and according to the physical query information, a corresponding inverted index reader in the inverted index reader resource pool is determined. Thus, the concentrated traffic can be diverted from the overloaded reader to the low-load reader. Then, according to the inverted index reader and the search word text, a row identifier set is generated. Thus, the index query can be completed based on the analyzer specified by the user, ensuring that the retrieval logic is consistent with the user's needs. Finally, according to the row identifier set, a data record set is obtained in the target data storage system, and the data record set is sent to the client corresponding to the target user. Thus, it can be ensured that the returned records are the matching results retrieved by the user-specified analyzer strategy, avoiding invalid data queries caused by strategy mismatch. Ultimately, query delay and resource load can be reduced.

[0071] Further reference Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a database retrieval processing device based on multiple analyzer indexes, which device embodiments correspond to the method embodiments shown in Figure 2 The device can be applied to various electronic devices.

[0072] AsFigure 2 As shown in FIG. 17, the database retrieval processing apparatus 200 based on multi-analyzer index of some embodiments includes a query analysis unit 201, a query verification unit 202, a generation unit 203, a triggering unit 204, a second generation unit 205, and an acquisition unit 206. The query analysis unit 201 is configured to, in response to receiving a retrieval task request of a target user to a database, perform query analysis on the retrieval task request to obtain query element information, wherein the query element information includes a target column identifier, a search word text, and a target analyzer name. The query verification unit 202 is configured to perform query verification on the target analyzer name to obtain a verification result, and in response to the verification result representing validity, perform associated query on the target analyzer name and the target column identifier in an index mapping information set to obtain a target inverted index identifier. The generation unit 203 is configured to generate query execution information according to the target inverted index identifier and the search word text. The triggering unit 204 is configured to trigger physical query information according to the query execution information, and determine a corresponding inverted index reader in an inverted index reader resource pool according to the physical query information. The second generation unit 205 is configured to generate a row identifier set according to the inverted index reader and the search word text. The acquisition unit 206 is configured to acquire a data record set in a target data storage system according to the row identifier set, and send the data record set to a client corresponding to the target user.

[0073] It can be understood that the units described in the apparatus 200 correspond to the respective steps in the method described above. Therefore, the operations, features, and advantages described above for the method also apply to the apparatus 200 and the units included therein, which will not be described here again. Figure 2 It can be understood that the units described in the apparatus 200 correspond to the respective steps in the method described above. Therefore, the operations, features, and advantages described above for the method also apply to the apparatus 200 and the units included therein, which will not be described here again.

[0074] Reference is made below to Figure 3 which shows a structural schematic diagram of an electronic device (e.g., an electronic device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0075] As Figure 3As shown, the electronic device 300 can include a processing device (e.g., a central processor, a graphics processor, etc.) 301 that can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 302 or loaded into a random access memory (RAM) 303 from a storage device 308. Various programs and data required for the operation of the electronic device 300 are also stored in the RAM 303. The processing device 301, the ROM 302, and the RAM 303 are connected to each other through a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0076] Generally, the following devices can be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication devices 309 can allow the electronic device 300 to communicate wirelessly or wired with other devices to exchange data. Although Figure 3 The electronic device 300 is shown with various devices, but it should be understood that all of the illustrated devices are not required, and more or fewer devices can alternatively be implemented. Figure 3 Each block shown in the flowcharts can represent one device, or a plurality of devices, as needed.

[0077] In particular, processes described above with reference to the flowcharts can be implemented as a computer software program according to some embodiments of the present disclosure. For example, some embodiments of the present disclosure include a computer program product including a computer program carried on a computer readable medium, the computer program containing program codes for performing the methods shown in the flowcharts. In some such embodiments, the computer program can be downloaded and installed from a network through the communication devices 309, or installed from the storage devices 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-described functions defined in the methods of some embodiments of the present disclosure are performed.

[0078] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0079] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0080] The computer readable medium can be included in the electronic device; or can exist independently of the electronic device. The computer readable medium carries one or more programs, when executed by the electronic device, cause the electronic device to: in response to receiving a search task request of a target user for a database, perform query analysis on the search task request to obtain query element information, wherein the query element information includes a target column identifier, a search word text, and a target analyzer name; perform query verification on the target analyzer name to obtain a verification result, and in response to the verification result representing validity, perform associated query on the target analyzer name and the target column identifier in an index mapping information set to obtain a target inverted index identifier; generate query execution information according to the target inverted index identifier and the search word text; trigger physical query information according to the query execution information, and determine a corresponding inverted index reader in an inverted index reader resource pool according to the physical query information; generate a row identifier set according to the inverted index reader and the search word text; obtain a data record set in a target data storage system according to the row identifier set, and send the data record set to a client corresponding to the target user.

[0081] Computer program code for carrying out operations of some embodiments of the disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0082] The computer program product of the first aspect can include a computer readable storage medium. The computer readable storage medium can include instructions. The instructions can include one or both of: instructions for causing a computer to implement a method as described above; and instructions for causing a computer to operate based on a system as described above. The computer readable storage medium can include one or more of: a magnetic disk; a magnetic tape; a magneto-optical disk; a semiconductor memory (e.g., a RAM, a ROM, a flash memory, etc.); and an optical disk.

[0083] The units described in some embodiments of the present disclosure can be implemented by means of software, or by means of hardware. The described units can also be provided in a processor, for example, a processor can be described as including a query parsing unit, a query verification unit, a generating unit, a triggering unit, a second generating unit, and an obtaining unit. In some cases, the names of these units do not constitute a limitation on the units themselves, for example, the query parsing unit can also be described as "a unit that, in response to receiving a retrieval task request of a target user to a database, parses the retrieval task request to obtain query element information".

[0084] The functions described above can be performed at least in part by one or more hardware logic components. For example, and without limitation, example types of hardware logic components that can be used include: field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), etc.

[0085] The above description is merely illustrative of the exemplary embodiments of the present disclosure and the technical principles of the application. It should be understood by those skilled in the art that the inventive scope of the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by the combinations of the above technical features or equivalent features without departing from the above inventive concept. For example, the above technical features can be replaced with the technical features disclosed in the embodiments of the present disclosure (but not limited to) having similar functions to form technical solutions.

Claims

1. A database retrieval processing method based on an analyzer index, comprising: in response to receiving a retrieval task request of a target user to a database, performing query parsing on the retrieval task request to obtain query element information, wherein the query element information comprises a target column identifier, a search word text, and a target analyzer name; performing query verification on the target analyzer name to obtain a verification result, and in response to the verification result representing validity, performing associated query on the target analyzer name and the target column identifier in an index mapping information set to obtain a target inverted index identifier; generating query execution information according to the target inverted index identifier and the search word text; determining a corresponding inverted index reader in an inverted index reader resource pool according to the query execution information and physical query information, and according to the physical query information; generating a row identifier set according to the inverted index reader and the search word text; obtaining a data record set in a target data storage system according to the row identifier set, and sending the data record set to a client corresponding to the target user.

2. The method of claim 1, wherein, The generating of the query execution information according to the target inverted index identifier and the search word text comprises: generating a physical storage path set and a back-end node identifier set according to the target inverted index identifier; generating index operation description information according to the search word text; generating query execution fragment information according to the index operation description information, the target inverted index identifier, and the physical storage path set; generating query execution information according to the query execution fragment information and the back-end node identifier set.

3. The method of claim 1, wherein, The generating of the row identifier set according to the inverted index reader and the search word text comprises: performing word segmentation processing on the search word text according to a preset analyzer rule corresponding to the inverted index reader to obtain a term set; generating a row identifier list set according to the term set and the inverted index reader; merging each row identifier list in the row identifier list set to obtain an initial row identifier set; performing deduplication on the initial row identifier set to obtain a row identifier set.

4. The method of claim 1, wherein, The generating of the query execution information according to the target inverted index identifier and the search word text comprises: generating index configuration information according to the target inverted index identifier; generating a search condition descriptor according to a predicate type corresponding to the search word text; generating query execution information according to the index configuration information and the search condition descriptor.

5. The method of claim 1, wherein, The query parsing on the retrieval task request of the target user to obtain the query element information comprises: performing lexical analysis on the retrieval task request to obtain a lexical unit sequence; performing syntax analysis on the lexical unit sequence to construct an abstract syntax tree; generating a target predicate node according to the abstract syntax tree; extracting a target column identifier from the target predicate node; extracting a search word text from the target predicate node; performing analyzer name extraction on the target predicate node to obtain a target analyzer name; determining the target column identifier, the search word text, and the target analyzer name as query element information.

6. A database retrieval processing apparatus based on a multi-analyzer index, comprising: a query analysis unit configured to, in response to receiving a retrieval task request of a target user on a database, perform query analysis on the retrieval task request to obtain query element information, wherein the query element information comprises a target column identifier, a search word text, and a target analyzer name; a query verification unit configured to perform query verification on the target analyzer name to obtain a verification result, and in response to the verification result being valid, perform associated query on the target analyzer name and the target column identifier in an index mapping information set to obtain a target inverted index identifier; a generation unit configured to generate query execution information according to the target inverted index identifier and the search word text; a triggering unit configured to trigger physical query information according to the query execution information, and determine a corresponding inverted index reader in an inverted index reader resource pool according to the physical query information; a second generation unit configured to generate a row identifier set according to the inverted index reader and the search word text; an acquisition unit configured to acquire a data record set in a target data storage system according to the row identifier set, and send the data record set to a client corresponding to the target user.

7. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon; when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-5.

8. A computer readable medium having stored thereon a computer program, wherein, The program is executed by the processor to implement the method of any one of claims 1-5. The program is executed by the processor to implement the method of any one of claims 1-5.

Images