In-memory database query method and related devices

Abstract

Embodiments of the present application provide a memory database query method and related equipment. The memory database query method is applied to a memory database, and the method comprises: establishing an operator index according to a secondary index of the memory database, the operator index comprising a prefix index, a suffix index, a substring index, a regular index and a regular substring index; receiving a data query instruction, determining an optimal operator index in the operator index; and performing a query using the optimal operator index to obtain a query result. The technical solution of the embodiments of the present application introduces multiple indexes in the memory database, dynamically selects an optimal operator index according to a query instruction, optimizes the query process, enables the memory database to cope with different types of query instructions, improves the overall throughput of the system, and reduces unnecessary computing overhead.

Description

Technical Field

[0001] This application relates to the fields of computer and communication technology, and more specifically, to a method and related equipment for querying an in-memory database. Background Technology

[0002] In rule engines within distributed stream processing architectures, a type of in-memory database matching operation is involved. This type of operation has extremely high performance requirements; no single operator can have a performance bottleneck. Operators typically include the following: prefix matching (headOf), suffix matching (tailOf), substring matching (substrOf), regular expression matching (matchBy), and regular expression substring matching (submatchBy). However, currently available open-source in-memory databases can only accelerate some operators, not maximize the performance of all of them. Summary of the Invention

[0003] The embodiments of this application provide a memory database query method and related device, which can at least to some extent overcome the problem that the prior art cannot maximize the performance of all operators in a memory structure.

[0004] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0005] According to one aspect of the embodiments of this application, a memory database query method is provided, applied to a memory database. The memory database query method includes: establishing an operator index based on a secondary index of the memory database, the operator index including a prefix index, a suffix index, a substring index, a regular expression index, and a regular expression substring index; receiving a data query instruction; determining the optimal operator index among the operator indexes; and performing a query using the optimal operator index to obtain a query result.

[0006] In some embodiments of this application, the in-memory database includes at least the following secondary indexes: prefix and suffix indexes, AC automaton indexes, and Hyperscan optimized indexes.

[0007] In some embodiments of this application, receiving a data query instruction and determining the optimal operator index in the operator index specifically includes: receiving a data query instruction, determining the query conditions contained in the data query instruction, and determining the optimal operator index based on the query conditions.

[0008] In some embodiments of this application, receiving a data query instruction and determining the query conditions included in the data query instruction specifically includes: receiving a data query instruction, converting the format of the data query instruction to obtain a target data query instruction, and determining the corresponding query conditions based on the target data query instruction.

[0009] In some embodiments of this application, the step of using the optimal operator index to perform a query and obtain query results specifically includes: using the optimal operator index to quickly locate the target data record; sorting the target data record to obtain query results.

[0010] In some embodiments of this application, before receiving the data query instruction and determining the optimal operator index in the operator index, the in-memory database query method further includes: in response to the data insertion instruction, inserting the target insertion data into the corresponding position of the in-memory database and updating the operator index.

[0011] In some embodiments of this application, the step of inserting the target data into the corresponding position of the in-memory database in response to a data insertion instruction and updating the operator index specifically includes: parsing the data insertion instruction to determine whether it contains a keyword; if the data insertion instruction contains a keyword, then only the target data is inserted into the corresponding position of the in-memory database, and the regular expression index is not updated; if the data insertion instruction does not contain a keyword, then the target data is inserted into the corresponding position of the in-memory database, and the regular expression index is updated simultaneously.

[0012] In some embodiments of this application, receiving a data query instruction and determining the optimal operator index in the operator index specifically includes: receiving a data query instruction and determining whether the regular expression index has been updated; if the regular expression index has been updated, then determining the regular expression index as the optimal operator index.

[0013] According to one aspect of the embodiments of this application, a memory database query device is provided, applied to a memory database. The memory database query device includes: an operator index building module, used to build an operator index based on the secondary index of the memory database, the operator index including a prefix index, a suffix index, a substring index, a regular expression index, and a regular expression substring index; an optimal index determination module, used to receive a data query instruction and determine the optimal operator index from the operator indexes; and an operator index acceleration module, used to perform a query using the optimal operator index to obtain a query result.

[0014] In some embodiments of this application, the in-memory database includes at least the following secondary indexes: prefix and suffix indexes, AC automaton indexes, and Hyperscan optimized indexes.

[0015] In some embodiments of this application, the optimal index determination module specifically includes: a query condition determination submodule, used to receive a data query instruction and determine the query conditions contained in the data query instruction; and an operator index determination submodule, used to determine the optimal operator index based on the query conditions.

[0016] In some embodiments of this application, the query condition determination submodule specifically includes: a format conversion unit, used to receive a data query instruction, convert the format of the data query instruction, and obtain a target data query instruction; and a condition determination unit, used to determine the corresponding query conditions based on the target data query instruction.

[0017] In some embodiments of this application, the operator index acceleration module specifically includes: a data positioning unit, used to quickly locate the target data record using the optimal operator index; and a data sorting unit, used to sort the target data record to obtain query results.

[0018] In some embodiments of this application, the in-memory database query device further includes: a target data insertion module, used to insert target data into the corresponding position of the in-memory database in response to a data insertion instruction, and to update the operator index.

[0019] In some embodiments of this application, the target data insertion module specifically includes: an instruction parsing submodule, used to parse the data insertion instruction and determine whether it contains a keyword; a first update submodule, used to insert the target data into the corresponding position of the memory database without updating the regular expression index if the data insertion instruction contains a keyword; and a second update submodule, used to insert the target data into the corresponding position of the memory database while updating the regular expression index if the data insertion instruction does not contain a keyword.

[0020] In some embodiments of this application, the optimal index determination module further includes: an update determination submodule, used to receive a data query instruction and determine whether the regular expression index has been updated; and a regular expression index submodule, used to determine that the regular expression index is the optimal operator index if the regular expression index has been updated.

[0021] According to one aspect of the embodiments of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the memory database query method as described in the above embodiments.

[0022] According to one aspect of the embodiments of this application, an electronic device is provided, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the memory database query method as described in the above embodiments.

[0023] A computer program product includes one or more computer programs, characterized in that, when the one or more computer programs are executed by one or more processors, they implement the steps of the memory database query method as described in the above embodiments.

[0024] In some embodiments of this application, the technical solutions provide that by introducing multiple indexes into the in-memory database and dynamically selecting the optimal operator index based on the query instruction, the query process is optimized. This allows the in-memory database to handle different types of query instructions (prefix matching, suffix matching, regular expression matching, etc.), improving the overall throughput of the system and reducing unnecessary computational overhead. Simultaneously, the efficient indexing mechanism in these embodiments accelerates the execution of the feature library matching operator, significantly reducing query response time and effectively solving the performance bottleneck problem of the feature library matching operator in stream processing systems. Especially when processing large-scale data, different types of indexes can greatly reduce query time, avoid the traditional linear scanning process, and optimize data query efficiency.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0027] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of this application can be applied is shown.

[0028] Figure 2 A flowchart illustrating a memory database query method provided in an embodiment of this application is shown.

[0029] Figure 3 It shows that according to Figure 2 A flowchart illustrating a specific implementation of step S200 in the memory database query method shown in the corresponding embodiment.

[0030] Figure 4A flowchart illustrating another in-memory database query method provided in an embodiment of this application is shown.

[0031] Figure 5 A schematic diagram of the structure of an in-memory database query device provided in an embodiment of this application is shown.

[0032] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0033] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0034] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0035] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0036] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0037] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of this application can be applied is shown.

[0038] like Figure 1 As shown, the system architecture may include terminal devices (such as...) Figure 1The device shown includes one or more of a smartphone 101, tablet 102, and portable computer 103 (which could also be a desktop computer, etc.), a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal device and the server 105. The network 104 can include various connection types, such as wired communication links, wireless communication links, etc.

[0039] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, there can be any number of terminal devices, networks, and servers. For example, server 105 could be a server cluster composed of multiple servers.

[0040] Users can use terminal devices to interact with server 105 via network 104 to receive or send messages, etc. Server 105 can be a server that provides various services. For example, users can use terminal device 103 (or terminal device 101 or 102) to upload various database commands to server 105, such as data query commands, data insertion commands, and table creation commands, etc. Server 105 can build operator indexes based on the secondary indexes of the memory database. The operator indexes include prefix indexes, suffix indexes, substring indexes, regular expression indexes, and regular expression substring indexes. S200: Receive data query commands and determine the optimal operator index from the operator indexes. S300: Use the optimal operator index to perform a query and obtain the query results.

[0041] It should be noted that the memory database query method provided in this application embodiment is generally executed by server 105, and correspondingly, the memory database query device is generally set in server 105. However, in other embodiments of this application, the terminal device may also have similar functions to the server, thereby executing the memory database query scheme provided in this application embodiment.

[0042] The implementation details of the technical solutions in the embodiments of this application are described in detail below:

[0043] Figure 2 A flowchart of a memory database query method according to an embodiment of this application is shown. This memory database query method can be executed by a server, which may be... Figure 1 The server shown has an in-memory database that includes at least the following secondary indexes: prefix and suffix indexes, AC automaton indexes, and Hyperscan optimized indexes. (See reference...) Figure 2 As shown, this in-memory database query method includes at least the following:

[0044] S100, an operator index is established based on the secondary index of the memory database. The operator index includes a prefix index, a suffix index, a substring index, a regular expression index, and a regular expression substring index.

[0045] S200, Receive data query instruction, and determine the optimal operator index in the operator index.

[0046] S300 uses the optimal operator index to perform the query and obtain the query results.

[0047] In this embodiment, by introducing various indexes (prefix index, suffix index, AC automaton index, Hyperscan optimized index, etc.) into the in-memory database and dynamically selecting the optimal operator index based on the query command, the query process is optimized, making it more efficient. This allows the in-memory database to handle different types of query commands (prefix matching, suffix matching, regular expression matching, etc.). Appropriate index acceleration improves the overall system throughput, reduces unnecessary computational overhead and memory consumption, enhances system resource utilization, and increases response speed. Simultaneously, the efficient indexing mechanism in this embodiment accelerates the execution of the feature library matching operator, significantly reducing query response time and effectively solving the performance bottleneck problem of the feature library matching operator in stream processing systems. Especially when processing large-scale data, different types of indexes can greatly reduce query time, avoid the traditional linear scanning process, and optimize data query efficiency.

[0048] In S100, various indexes are created in the in-memory database to optimize query performance for different matching operators. Specifically, prefix indexes optimize queries with a given string as a prefix and are suitable for the prefix (headOf) operator; suffix indexes optimize queries with a given string as a suffix and are suitable for the suffix (tailOf) operator; substring indexes, by introducing the AC automaton, are suitable for queries containing a given substring and optimize the substring (substrOf) operator; regular expression indexes, by introducing AC automatons and Hyperscan techniques, optimize regular expressions and support complex regular expression matching queries; and regular expression substring indexes support regular expression substring matching and improve query efficiency by optimizing the regular expression engine.

[0049] Specifically, a prefix index is a hash table or dictionary where the keys are prefixes of strings and the values ​​are the corresponding document IDs or data locations. During creation, the database iterates through the string fields of each data record, generating multiple prefix substrings for each, and stores them in the prefix index.

[0050] The specific steps for creating a prefix index are as follows: First, traverse each string field in the data table (such as text, log data, etc.); then generate prefix substrings of length from 1 to N (N is the string length) for each string, and use these prefix substrings as index keys. The value corresponding to each key is the identifier or position of all records containing that prefix; finally, store the index in a specific storage structure, which can be a hash table, a Trie tree, etc., depending on the query efficiency requirements.

[0051] In the corresponding query process, taking headOf("prefix") as an example, the database will directly search for items in the prefix index with prefix as the key and quickly return all matching data records that start with that prefix.

[0052] Suffix indexes are similar to prefix indexes, except they focus on the end of the string. By generating and storing suffix substrings of a string, they provide fast access to queries that end with a specific suffix.

[0053] The specific steps for creating a suffix index are as follows: First, iterate through the string fields in all records; then, generate all possible suffix substrings for each string (from position i to the end of the string); finally, store each suffix substring and its corresponding record position or document ID. The suffix index can be stored using an inverted index structure for fast retrieval.

[0054] In the corresponding query process, taking tailOf("suffix") as an example, the database will directly access the suffix index to find all matching data records ending with "suffix".

[0055] Substring matching typically requires support for matching strings at arbitrary positions, which can be efficiently handled using the Aho-Corasick automaton. The Aho-Corasick automaton is a multi-pattern matching algorithm that can support various substring matching methods using a state machine.

[0056] The specific steps for creating a substring index are as follows: construct a state graph of an AC automaton, insert all substrings that need to be matched into the automaton, where each substring is converted into a state path, and each state of the automaton represents a matching progress in the string; using the AC algorithm, when matching a query, start from each position of the text, track the matching process through the state machine until a matching substring is found.

[0057] In the corresponding query process, taking `substrOf("substring")` as an example, the AC automaton will scan the entire text and track the position of each character to find if a matching substring exists. Because the AC automaton processes multiple substring matches in parallel, the query efficiency is greatly improved.

[0058] Regular expression matching is typically implemented using a regular expression engine. To improve query performance, optimization tools such as Hyperscan can be used for efficient regular expression matching. Hyperscan is a high-performance regular expression matching library that supports parallel matching of multiple regular expressions.

[0059] The specific steps for creating a regular expression index are as follows: Hyperscan compiles all the regular expressions that need to be matched into an efficient state machine. In the database, the regular expressions are compiled into a representation pattern structure, which is stored in a specific container, which can be a state machine or a lookup table.

[0060] It should be noted that Hyperscan can combine multiple regular expressions into a single matching engine, enabling fast matching through shared state.

[0061] During the corresponding query process, taking matchBy("regex") as an example, Hyperscan will scan the entire data and use the state machine mechanism of regular expressions to perform multiple regular expression matching in parallel in one operation, returning the results that meet the conditions.

[0062] Regular expression substring indexes are similar to regular expression indexes, but they focus on whether the query matches any substring within the regular expression. Hyperscan can still be used to optimize substring matching.

[0063] The specific steps for creating a regular expression substring index are as follows: First, all the required regular expressions are compiled into a state machine that Hyperscan can process. Then, Hyperscan merges multiple regular expressions into a matching pattern to achieve the requirement of quickly scanning any substring in the string and providing more efficient query processing.

[0064] During the corresponding query process, taking subMatchBy("regex") as an example, the Hyperscan engine will scan the entire input text and use an optimized regular expression engine to find matches in all substrings.

[0065] In S200, upon receiving a data query instruction, the system analyzes and selects the most suitable index based on the query conditions. By evaluating operator indexes, the system automatically selects the optimal index type to execute the query. Here, "optimal" refers to the index type that provides the shortest query response time and the least computational overhead among different index types. For example, if the query conditions involve prefix matching, a prefix index will be prioritized; if the query conditions involve regular expression matching, a regular expression index will be prioritized; and if the query conditions involve substring matching, a substring index will be prioritized. This automated selection mechanism significantly improves query efficiency and reduces unnecessary computation and memory consumption.

[0066] Specifically, in some embodiments, the specific implementation of step S200 can be found in [reference needed]. Figure 3 . Figure 3 It is based on Figure 2 According to the detailed description of step S200 in the memory database query method shown in the corresponding embodiment, step S200 in the memory database query method may include the following steps:

[0067] S210, Receive a data query instruction and determine the query conditions included in the data query instruction.

[0068] S220, Determine the optimal operator index based on the query conditions.

[0069] In the embodiments of this application, by determining the optimal operator index based on the query conditions obtained from parsing the data query instruction, the query conditions can be automatically identified, and the most suitable index can be selected according to the condition type and the selectivity of the query. This significantly improves query efficiency and avoids the complexity of manually designing specific indexes for each query, ensuring the efficiency and flexibility of database queries. Therefore, the selection of the optimal operator index can effectively improve the performance of the in-memory database when processing complex queries, solving the performance bottleneck problem caused by improper index selection in complex data query scenarios.

[0070] In S210, by identifying the query conditions in the query command, important information can be provided for subsequent index selection, ensuring that the query process can select the appropriate index based on the conditions. This process automatically supports multiple query methods, avoids manual intervention, and improves query efficiency.

[0071] The query request can be an operation such as filtering, sorting, or aggregation on certain tables in the database.

[0072] Query conditions may include: comparison operators (such as equal to, not equal to, greater than, less than, etc.), string matching (such as prefix matching, suffix matching, substring matching), and complex matching conditions such as regular expressions.

[0073] Specifically, in some embodiments, the specific implementation of step S210 can be found in the following embodiments. This embodiment is based on... Figure 3 According to the detailed description of step S210 in the memory database query method shown in the corresponding embodiment, step S210 in the memory database query method may include the following steps:

[0074] Receive a data query instruction, convert the data query instruction into a format, and obtain the target data query instruction.

[0075] The corresponding query conditions are determined based on the target data query instruction.

[0076] In this embodiment, converting query commands of different formats into a unified format (i.e., target data query command) can eliminate the differences between query commands from different sources. Simultaneously, by formatting the query commands, on the one hand, query conditions can be identified more efficiently and accurately, and the optimal operator index can be selected based on these conditions, thereby optimizing query performance; on the other hand, it can adapt to query commands from different sources and of different types, enabling the database to support the query needs of various front-ends or applications without needing to write different parsing logic for each format.

[0077] In this embodiment, format conversion typically includes converting graphical query expressions or natural language queries into standard query languages ​​(such as SQL or other internal query languages); and converting query instructions from a protocol or format into a format that the database system can understand (such as converting from JSON, XML, etc., to the system's internal data structure).

[0078] It should be noted that format conversion can also include converting all indexes to lowercase and then converting query commands to lowercase using a custom case-insensitive syntax.

[0079] The ultimate goal of format conversion is to ensure that the system can understand and convert the input query into a uniform format for subsequent processing, regardless of the type of query.

[0080] After format conversion, the target data query command is now in a standard format that the system can understand and process. Next, the database system will analyze this standard format command to derive the query conditions. The specific content of the query conditions includes: field matching conditions, logical conditions, pattern matching conditions, and set queries.

[0081] Among these, field matching conditions refer to whether a field is equal to a certain value, or whether a field meets a certain range condition. Logical conditions refer to logical operations such as AND and OR in query conditions. Pattern matching conditions refer to prefix matching, suffix matching, and substring matching of string fields. Set queries refer to operations such as IN on a certain field, or queries within certain ranges.

[0082] The database system uses these conditions to determine whether existing indexes can be used to speed up queries, or whether a full table scan is necessary. By extracting and analyzing the query conditions, the system provides foundational data for subsequent steps such as index selection and query optimization.

[0083] In S220, query efficiency can be significantly improved by automatically selecting the optimal operator index based on query conditions. After selecting the most suitable index, the query can quickly find matching data in the database, reducing unnecessary full table scans or inefficient index scans, thereby improving the overall performance of the database.

[0084] Specifically, the most suitable index type can be determined based on the type of query conditions (e.g., prefix, suffix, substring, regular expression, etc.). For example, if the query condition involves prefix matching (e.g., LIKE 'abc%'), then a prefix index is chosen; if the query involves matching at any position in the string (e.g., LIKE '%abc%'), then a suffix index or a substring index is chosen; for more complex matching, a regular expression index or other types of specialized indexes may be chosen.

[0085] In some embodiments, among multiple candidate indexes, the optimal index can be selected based on the following criteria: selectivity of query conditions, complexity of query conditions, and index lookup efficiency.

[0086] Specifically, regarding the selectivity of query conditions, if the selectivity is high (i.e., it can filter out fewer records), then the selected index should be able to quickly filter out most of the non-matching records. Regarding the complexity of query conditions, if the query conditions contain multiple fields or complex regular expressions, the system may combine multiple indexes or select a composite index that can effectively handle complex queries. Regarding the index lookup efficiency, the system will evaluate the lookup time and resource consumption of different indexes and select the optimal index to execute the query.

[0087] In S300, query operations are accelerated by using optimal operator indexes, allowing the database to efficiently locate data that meets the query criteria and return the results.

[0088] For example, for prefix matching queries, the database will directly utilize the prefix index to quickly find data that meets the conditions in memory; for regular expression matching queries, the database will utilize Hyperscan-optimized regular expression indexes to avoid recompiling the regular expression for each query, thereby improving query performance. This step ensures that the system can return results quickly and efficiently based on actual query needs, thus improving the overall throughput and response speed of the system.

[0089] Specifically, in some embodiments, the specific implementation of step S300 can be found in the following embodiments. This embodiment is based on... Figure 2 According to the detailed description of step S300 in the memory database query method shown in the corresponding embodiment, step S300 in the memory database query method may include the following steps:

[0090] Using the optimal operator index, the target data record can be quickly located.

[0091] The target data records are sorted to obtain the query results.

[0092] In this embodiment, the optimal operator index is used to quickly locate the target data record, avoiding a full table scan and reducing the computational load during the query process, thereby significantly reducing the query time. This is especially effective in large-scale data scenarios, improving performance. After the query, the target data records are also sorted to ensure the accuracy of the query results and meet the user's sorting requirements, while avoiding complex sorting operations in the later stages of the query, thus optimizing overall query efficiency.

[0093] In other embodiments of this application, such as Figure 4 As shown, prior to S200, the in-memory database query method further includes:

[0094] S400, in response to the data insertion instruction, insert the target data into the corresponding position in the memory database and update the operator index.

[0095] In this embodiment, the index is updated immediately upon data insertion, ensuring that the index used during queries reflects the latest database state and avoiding the use of outdated indexes during queries, thereby improving query speed. Especially in environments where data changes frequently, it can quickly adapt to data changes and optimize query efficiency.

[0096] In some feasible implementations, index update operations and data insertion operations can be performed synchronously to reduce the latency caused by delayed index updates during queries and improve system response speed.

[0097] Specifically, in some embodiments, the specific implementation of step S400 can be found in the following embodiments. This embodiment is based on... Figure 4 According to the detailed description of step S400 in the memory database query method shown in the corresponding embodiment, step S400 in the memory database query method may include the following steps:

[0098] The data insertion instruction is parsed to determine whether it contains keywords.

[0099] If the data insertion instruction contains a keyword, the target data will be inserted into the corresponding position in the memory database, and the regular expression index will not be updated.

[0100] If the data insertion instruction does not contain a keyword, the target data is inserted into the corresponding position in the in-memory database, and the regular expression index is updated.

[0101] In this embodiment, for batch regular expression indexes, a keyword can be introduced. When the data insertion instruction contains the keyword, only the data insertion operation is performed without updating the regular expression index. When the data insertion instruction does not contain the keyword (i.e., when the last data is inserted), the regular expression index is updated together. This avoids the operation of recompiling and updating the regular expression index of the column every time an insertion occurs, thereby reducing system resource consumption and operation overhead.

[0102] Corresponding to the above embodiments, in some embodiments, the specific implementation process of step S200 may include the following steps:

[0103] Receive a data query instruction and determine whether the regular expression index has been updated.

[0104] If the regular expression index update is complete, then the regular expression index is determined to be the optimal operator index.

[0105] In this embodiment, correspondingly, since the regular expression index is not updated in real time, before executing the data query command, in addition to identifying the query conditions and the columns to be selected, it is also necessary to confirm whether the regular expression index has been updated. If the regular expression index has been compiled according to the lazy compilation rule (i.e., compilation is completed when the last data is inserted), the regular expression index can be used to quickly locate the data record that meets the query conditions, improving query efficiency. If the regular expression index has not been compiled, the query may be performed in other ways (such as not using the regular expression index and performing a full table scan).

[0106] The following describes an apparatus embodiment of this application, which can be used to execute the memory database query method in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the memory database query method described above.

[0107] Figure 5 A block diagram of a memory database query apparatus according to an embodiment of this application is shown.

[0108] Reference Figure 5As shown, an in-memory database query apparatus 500 according to an embodiment of this application is applied to an in-memory database, which includes at least the following secondary indexes: prefix and suffix indexes, AC automaton indexes, and Hyperscan optimized indexes. The in-memory database query apparatus 500 includes: an operator index building module 510, an optimal index determination module 520, and an operator index acceleration module 530.

[0109] The operator index building module 510 is used to build operator indexes based on the secondary indexes of the memory database. The operator indexes include prefix indexes, suffix indexes, substring indexes, regular indexes, and regular substring indexes. The optimal index determination module 520 is used to receive data query instructions and determine the optimal operator index from the operator indexes. The operator index acceleration module 530 is used to perform queries using the optimal operator index and obtain query results.

[0110] In some embodiments of this application, the optimal index determination module specifically includes: a query condition determination submodule, used to receive a data query instruction and determine the query conditions contained in the data query instruction; and an operator index determination submodule, used to determine the optimal operator index based on the query conditions.

[0111] In some embodiments of this application, the query condition determination submodule specifically includes: a format conversion unit, used to receive a data query instruction, convert the format of the data query instruction, and obtain a target data query instruction; and a condition determination unit, used to determine the corresponding query conditions based on the target data query instruction.

[0112] In some embodiments of this application, the operator index acceleration module specifically includes: a data positioning unit, used to quickly locate the target data record using the optimal operator index; and a data sorting unit, used to sort the target data record to obtain query results.

[0113] In some embodiments of this application, the in-memory database query device further includes: a target data insertion module, used to insert target data into the corresponding position of the in-memory database in response to a data insertion instruction, and to update the operator index.

[0114] In some embodiments of this application, the target data insertion module specifically includes: an instruction parsing submodule, used to parse the data insertion instruction and determine whether it contains a keyword; a first update submodule, used to insert the target data into the corresponding position of the memory database without updating the regular expression index if the data insertion instruction contains a keyword; and a second update submodule, used to insert the target data into the corresponding position of the memory database while updating the regular expression index if the data insertion instruction does not contain a keyword.

[0115] In some embodiments of this application, the optimal index determination module further includes: an update determination submodule, used to receive a data query instruction and determine whether the regular expression index has been updated; and a regular expression index submodule, used to determine that the regular expression index is the optimal operator index if the regular expression index has been updated.

[0116] In this embodiment, by introducing various indexes (prefix index, suffix index, AC automaton index, Hyperscan optimized index, etc.) into the in-memory database and dynamically selecting the optimal operator index based on the query command, the query process is optimized, making it more efficient. This allows the in-memory database to handle different types of query commands (prefix matching, suffix matching, regular expression matching, etc.). Appropriate index acceleration improves the overall system throughput, reduces unnecessary computational overhead and memory consumption, enhances system resource utilization, and increases response speed. Simultaneously, the efficient indexing mechanism in this embodiment accelerates the execution of the feature library matching operator, significantly reducing query response time and effectively solving the performance bottleneck problem of the feature library matching operator in stream processing systems. Especially when processing large-scale data, different types of indexes can greatly reduce query time, avoid the traditional linear scanning process, and optimize data query efficiency.

[0117] Figure 6 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0118] It should be noted that, Figure 6 The computer system of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0119] like Figure 6 As shown, the computer system includes a Central Processing Unit (CPU) 1801, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 1802 or programs loaded from storage portion 1808 into Random Access Memory (RAM) 1803, such as performing the methods described in the above embodiments. The RAM 1803 also stores various programs and data required for system operation. The CPU 1801, ROM 1802, and RAM 1803 are interconnected via a bus 1804. An Input / Output (I / O) interface 1805 is also connected to the bus 1804.

[0120] The following components are connected to I / O interface 1805: an input section 1806 including a keyboard, mouse, etc.; an output section 1807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1808 including a hard disk, etc.; and a communication section 1809 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1809 performs communication processing via a network such as the Internet. A drive 1810 is also connected to I / O interface 1805 as needed. Removable media 1811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1810 as needed so that computer programs read from them can be installed into storage section 1808 as needed.

[0121] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1809, and / or installed from removable medium 1811. When the computer program is executed by central processing unit (CPU) 1801, it performs various functions defined in the system of this application.

[0122] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0123] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0124] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0125] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0126] This specification also provides a computer program product that stores at least one instruction, said at least one instruction being loaded and executed by the processor as described above. Figures 1-4 The method described in the illustrated embodiment can be found in the following document for a detailed execution process. Figures 1-4 The specific details of the illustrated embodiments will not be elaborated here.

[0127] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0128] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0129] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0130] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for querying an in-memory database, characterized in that, Applied to in-memory databases, the in-memory database query method includes: Operator indexes are built based on the secondary indexes of the memory database. The operator indexes include prefix indexes, suffix indexes, substring indexes, regular expression indexes, and regular expression substring indexes. Receive a data query instruction and determine the optimal operator index from the operator index; The query is performed using the optimal operator index to obtain the query results.

2. The in-memory database query method as described in claim 1, characterized in that, The in-memory database includes at least the following secondary indexes: prefix and suffix indexes, AC automaton indexes, and Hyperscan optimized indexes.

3. The in-memory database query method as described in claim 1, characterized in that, The process of receiving a data query instruction and determining the optimal operator index from the operator index specifically includes: Receive a data query instruction and determine the query conditions included in the data query instruction; Based on the query conditions, determine the optimal operator index.

4. The in-memory database query method as described in claim 3, characterized in that, The process of receiving a data query instruction and determining the query conditions included in the data query instruction specifically includes: Receive a data query instruction, convert the data query instruction into a format, and obtain the target data query instruction; The corresponding query conditions are determined based on the target data query instruction.

5. The in-memory database query method as described in claim 1, characterized in that, The query using the optimal operator index to obtain the query results specifically includes: Using the aforementioned optimal operator index, the target data record can be quickly located; The target data records are sorted to obtain the query results.

6. The in-memory database query method as described in claim 1, characterized in that, Before receiving the data query instruction and determining the optimal operator index in the operator index, the in-memory database query method further includes: In response to a data insertion instruction, the target data is inserted into the corresponding position in the in-memory database, and the operator index is updated.

7. A memory database query device, characterized in that, The memory database query device, applied to an in-memory database, includes: The operator index building module is used to build operator indexes based on the secondary indexes of the memory database. The operator indexes include prefix indexes, suffix indexes, substring indexes, regular expression indexes, and regular expression substring indexes. The optimal index determination module is used to receive data query instructions and determine the optimal operator index among the operator indexes; The operator index acceleration module is used to perform queries using the optimal operator index and obtain query results.

8. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the memory database query method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the memory database query method as described in any one of claims 1 to 6.

10. A computer program product comprising one or more computer programs, characterized in that, When the one or more computer programs are executed by one or more processors, they implement the steps of the memory database query method according to any one of claims 1 to 6.

Images