Data query method and apparatus, and computing device cluster

Abstract

The invention discloses a data query method and device and a computing device cluster, and belongs to the technical field of databases. After a data query request including a first word is obtained, a first index area corresponding to the first word in a database index structure is determined, k index fragments corresponding to the first index area are read to determine a first data block, and the first data block is queried to determine a query result. Wherein the database index structure comprises m BF index structures and n index fragments based on column storage, each BF index structure comprises n index units, each index fragment comprises m index units located in the same column, and the first index area comprises m first index segments corresponding to a first word in the m BF index structures, the first index area corresponds to k index fragments in the n index fragments, k is equal to 1 or 2, and k is smaller than n. The data query performance can be improved.

Description

[0001] This application claims priority to a Chinese patent application filed on October 28, 2023, with application number 202311426787.3 and application name “A data query method, device and computing device cluster”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of database technology, and in particular to a data query method and device, and a computing device cluster. Background Art

[0003] Managing data through a database is an effective means of data management. For example, recording (or storing) data blocks through a database, querying data blocks recorded in the database based on query services provided by the database, etc. all fall within the scope of managing data through a database.

[0004] Generally, a database is used not only to record data blocks, but also to record index information of words in the data blocks, so as to facilitate querying data blocks based on the index information of words in data query scenarios. For example, an index structure is created in the database, and the index structure is used to record the index information of words. For example, an index structure based on a bloom filter (BF) is a common index structure, also called a BF index structure.

[0005] However, in the traditional BF index structure, the index information of a word may be recorded at any position in the BF index structure, which means that when querying based on the traditional BF index structure, the entire content of the BF index structure needs to be read, which leads to poor query performance. Summary of the invention

[0006] The present application provides a data query method and device, and a computing device cluster. The technical solution of the present application is as follows.

[0007] In a first aspect, a data query method is provided, the method comprising: obtaining a first data query request, the first data query request comprising a first word; determining a first index area corresponding to the first word in a database index structure, wherein the database index structure comprises m BF index structures and n index fragments based on column storage, the m BF index structures correspond to m data blocks, each BF index structure is used to record existence index information of words in the corresponding data block, the existence index information of any word in any data block is in the index segment corresponding to the any word in the BF index structure corresponding to the any data block, each BF index structure comprises n index units, and each index fragment comprises the m BF index structures. m index units located in the same column in the BF index structure, the length of the index unit is equal to the length of the index segment, the length of the index segment is less than the length of the BF index structure, the first index area includes the m first index segments corresponding to the first word in the m BF index structures, and the first index area corresponds to k index segments in the n index segments, k is equal to 1 or 2, and k is less than n, and m and n are both positive integers; read the k index segments corresponding to the first index area to determine at least one first data block, the first index segment in the BF index structure corresponding to the first data block includes the existence index information of the first word; query the at least one first data block to determine the query result of the first data query request. Wherein, the index unit and the index segment are both a segment on the BF index structure, the length of the index unit is the number of bits included in the index unit, the length of the index segment is the number of bits included in the index segment, and the length of the BF index structure is the number of bits included in the BF index structure. Wherein, the data query method is executed by a data query device.

[0008] The technical solution provided by the present application is that the database index structure includes m BF index structures and n index fragments based on column storage, each BF index structure includes n index units, and each index fragment includes m index units located in the same column of the m BF index structures. Based on this database index structure, after the data query device obtains a first data query request including a first term, the data query device determines the first index area corresponding to the first term in the database index structure, and the data query device only needs to read the k index fragments corresponding to the first index area to determine at least one first data block, and then determine the query result of the first data query request by querying the at least one first data block. That is, the data query device can realize data query without reading the full content of the database index structure, so the input / output (I / O) cost of the data query is low, the memory overhead is small, and the query performance is good.

[0009] Optionally, each of the m BF index structures includes n×s bits, each index unit in each BF index structure includes s bits, and the bits in each BF index structure are used to record the existence index information of the words in the data block corresponding to each BF index structure, and s is a positive integer. For example, the length of each index unit is s bits.

[0010] Optionally, the database index structure further includes m×w unit index structures, each BF index structure in the m BF index structures corresponds to w unit index structures in the m×w unit index structures, and w is a positive integer;

[0011] Each unit index structure corresponding to each BF index structure in the m BF index structures corresponds to at least one index unit in each BF index structure, different unit index structures correspond to different index units, and each unit index structure corresponding to each BF index structure is used to record secondary index information of the index unit corresponding to each unit index structure;

[0012] The secondary index information recorded in each unit index structure includes one or more of the following information: the row number of the hit row of the index unit corresponding to each unit index structure, the number of hit rows of the index unit corresponding to each unit index structure, and row number identification information;

[0013] The hit row of the index unit corresponding to each unit index structure is the data row where the target word is located in the data block corresponding to each BF index structure, and the index segment corresponding to the target word hits the index unit corresponding to each unit index structure;

[0014] The row number identification information indicates whether the number of hit rows of the index unit corresponding to each unit index structure is greater than a specified number.

[0015] Optionally, the secondary index information recorded in each unit index structure includes the row number of the hit row of the index unit corresponding to each unit index structure, and the at least one first data block is queried to determine the query result of the first data query request, including: scanning the data row indicated by the row number of the first unit index structure record corresponding to each first data block in the at least one first data block to determine the query result of the first data query request, wherein the first unit index structure corresponding to each first data block in the at least one first data block is a unit index structure corresponding to the first index segment (i.e., the index segment corresponding to the first word) in the BF index structure of each first data block.

[0016] The technical solution provided by the present application is that since the secondary index information recorded in the unit index structure includes the row number of the hit row of the index unit corresponding to the unit index structure, when the data query device queries the first data block, it only needs to scan the data row indicated by the row number recorded in the first unit index structure corresponding to the first data block, and there is no need to scan the entire content of the first data block. Therefore, the query process can be simplified and the query efficiency can be improved.

[0017] Optionally, the secondary index information recorded in each unit index structure includes row number identification information and the row number of the hit row of the index unit corresponding to each unit index structure, and the at least one first data block is queried to determine the query result of the first data query request, including: for each first data block in the at least one first data block: when the row number identification information recorded in the first unit index structure corresponding to the first data block indicates that the number of hit rows of the index unit corresponding to the first unit index structure is less than or equal to a specified number, scanning the data row indicated by the row number recorded in the first unit index structure corresponding to the first data block to determine the query result of the first data query request; when the row number identification information recorded in the first unit index structure corresponding to the first data block indicates that the number of hit rows of the index unit corresponding to the first unit index structure is greater than a specified number, scanning each data row in the first data block to determine the query result of the first data query request.

[0018] In the technical solution provided by the present application, the secondary index information recorded in the unit index structure includes row number identification information and the row number of the hit row of the index unit corresponding to the unit index structure. Therefore, when the data query device queries the first data block, when the row number identification information recorded in the first unit index structure corresponding to the first data block indicates that the number of hit rows is less than or equal to the specified number, the data query device only needs to scan the data row indicated by the row number recorded in the first unit index structure corresponding to the first data block, and does not need to scan the entire content of the first data block, thereby improving the query efficiency. When the row number identification information recorded in the first unit index structure corresponding to the first data block indicates that the number of hit rows is greater than the specified number, it means that there may be more data rows in the first data block that include the first term. Therefore, the data query device scans each data row in the first data block to determine the query result of the first data query request, thereby improving the accuracy of the query result.

[0019] Optionally, the secondary index information recorded in each unit index structure includes the row number of the hit row of the index unit corresponding to each unit index structure, and the method also includes: obtaining a second data query request, the second data query request includes a first term and a second term; determining a first index area corresponding to the first term and a second index area corresponding to the second term in the database index structure, the second index area includes m second index segments corresponding to the second term in the m BF index structures, and the second index area corresponds to p index slices in the n index slices, p is equal to 1 or 2, and p is less than n; reading k index slices corresponding to the first index area and p index slices corresponding to the second index area to determine at least one second data block, wherein the first index segment and the second index segment in the BF index structure corresponding to the second data block respectively include the existence index information of the first term and the second term The target data block is determined from the at least one second data block according to the row number of the first unit index structure record corresponding to the at least one second data block and the row number of the second unit index structure record corresponding to the at least one second data block, wherein the first unit index structure corresponding to the second data block is a unit index structure corresponding to the first index segment in the BF index structure of the second data block, the second unit index structure corresponding to the second data block is a unit index structure corresponding to the second index segment in the BF index structure of the second data block, the target data block includes a target data row, and the target data row is a data row indicated by the same row number in the row number of the first unit index structure record corresponding to the target data block and the row number of the second unit index structure record corresponding to the target data block; the target data row in the target data block is scanned to determine the query result of the second data query request.

[0020] The technical solution provided by the present application is that after the data query device obtains the second data query request including the first word and the second word, the data query device determines the first index area corresponding to the first word in the database index structure and the second index area corresponding to the second word in the database index structure. The data query device only needs to read the k index fragments corresponding to the first index area and the p index fragments corresponding to the second index area to determine at least one second data block, and then determine the target data block from the at least one second data block, and determine the query result of the second data query request by scanning the target data row in the target data block. That is, the data query device can realize the joint query of the first word and the second word without reading the full content of the database index structure, and the query I / O cost is low, the memory overhead is small, and the query performance is good. In addition, the data query device determines the target data block from the at least one second data block according to the row number of the first unit index structure record corresponding to the at least one second data block and the row number of the second unit index structure record corresponding to the at least one second data block, that is, the at least one second data block is screened, which can avoid scanning the second data block that is misjudged.

[0021] Optionally, the secondary index information recorded in each unit index structure includes the number of hit rows of the index unit corresponding to each unit index structure, and the method also includes: obtaining a first aggregate query request, the first aggregate query request being used to query the number of data including a first term; determining a first index area corresponding to the first term in the database index structure; reading k index shards corresponding to the first index area to determine at least one first data block; determining a query result of the first aggregate query request based on the number of hit rows recorded in the first unit index structure corresponding to the at least one first data block, wherein the first unit index structure corresponding to each first data block in the at least one first data block is a unit index structure corresponding to the first index segment in the BF index structure of each first data block.

[0022] The technical solution provided by the present application is that after the data query device obtains the first aggregate query request, the data query device determines the first index area corresponding to the first word in the database index structure. The data query device only needs to read the k index fragments corresponding to the first index area to determine at least one first data block, and then determines the query result of the first aggregate query request according to the number of hit rows recorded in the first unit index structure corresponding to the at least one first data block. Among them, the hit rows recorded in the first unit index structure corresponding to the at least one first data block can be approximately considered as the number of data rows including the first word. Therefore, the data query device can realize the approximate aggregate query for the first word without reading the full content of the database index structure, and the query I / O cost is low, the memory overhead is small, and the query performance is good.

[0023] Optionally, the secondary index information recorded in each unit index structure includes the number of hit rows of the index unit corresponding to each unit index structure, and the method further includes: obtaining a second aggregate query request, the second aggregate query request is used to query the number of data including the first term and the second term; determining a first index area corresponding to the first term and a second index area corresponding to the second term in the database index structure, the second index area includes m second index segments corresponding to the second term in the m BF index structures, and the second index area corresponds to p index shards in the n index shards, p is equal to 1 or 2, and p is less than n; reading k index shards corresponding to the first index area and p index shards corresponding to the second index area to determine A second data block, wherein the first index segment and the second index segment in the BF index structure corresponding to the second data block respectively include existence index information of the first word and existence index information of the second word; according to the smaller number of the number of hit rows recorded by the first unit index structure corresponding to the second data block and the number of hit rows recorded by the second unit index structure corresponding to the second data block, determine the query result of the second aggregate query request, wherein the first unit index structure corresponding to the second data block is a unit index structure corresponding to the first index segment in the BF index structure of the second data block, and the second unit index structure corresponding to the second data block is a unit index structure corresponding to the second index segment in the BF index structure of the second data block.

[0024] The technical solution provided by the present application is that after the data query device obtains the second aggregate query request, the data query device determines the first index area corresponding to the first word in the database index structure and the second index area corresponding to the second word in the database index structure. The data query device only needs to read the k index fragments corresponding to the first index area and the p index fragments corresponding to the second index area to determine the second data block, and then determine the query result of the second aggregate query request according to the smaller number of the number of hit rows recorded in the first unit index structure corresponding to the second data block and the number of hit rows recorded in the second unit index structure corresponding to the second data block. The number of hit rows recorded in the first unit index structure corresponding to the second data block can be approximately considered as the number of data rows including the first word, and the number of hit rows recorded in the second unit index structure corresponding to the second data block can be approximately considered as the number of data rows including the second word. Therefore, the data query device can realize the approximate aggregate query for the first word and the second word without reading the full content of the database index structure, for example, realizing the joint and (and) approximate aggregate query of the first word and the second word, and the query has low I / O cost, low memory overhead, and good query performance.

[0025] Optionally, each of the m BF index structures includes a low-frequency index space and a high-frequency index space, the high-frequency index space is used to record the existence index information of high-frequency words in the data block corresponding to each BF index structure, and the low-frequency index space is used to record the existence index information of low-frequency words in the data block corresponding to each BF index structure.

[0026] The technical solution provided in the present application divides the BF index structure into a low-frequency index space and a high-frequency index space, and uses the high-frequency index space to record the existence index information of high-frequency words, and uses the low-frequency index space to record the existence index information of low-frequency words. This can prevent high-frequency words from polluting low-frequency words, thereby avoiding misjudgment during data query due to such pollution and ensuring query performance.

[0027] Optionally, determining a first index area corresponding to a first word in a database index structure includes: determining a starting position of the first index area according to the first word; and determining the first index area according to the starting position and a length of an index segment in the database index structure.

[0028] Optionally, before obtaining the first data query request, the method also includes: obtaining a data write request, the data write request being used to write a first data block in a database; creating a BF index structure for the first data block in response to the data write request; obtaining a first word in the first data block; determining the first index segment in the BF index structure of the first data block based on the first word; and recording existence index information of the first word in the first index segment.

[0029] The technical solution provided by the present application is that after a data query device obtains a data write request for writing a first data block in a database, the data query device creates a BF index structure of the first data block, determines a first index segment in the BF index structure of the first data block according to a first word in the first data block, and records the existence index information of the first word in the first index segment. That is, the data query device records the existence index information of the first word in the database index structure, so that it is convenient to query data for the first word according to the existence index information of the first word recorded in the BF index structure of the first data block.

[0030] Optionally, the first word is a complete word or a prefix of the complete word. The second word is a complete word or a prefix of the complete word. That is, the BF index structure of the present application supports data queries based on complete words and also supports fuzzy queries based on prefixes. Whether it is a data query based on a complete word or a fuzzy query based on a prefix, the query performance is good.

[0031] In a second aspect, a data query device is provided, comprising at least one functional module, the at least one functional module being used to execute the data query method provided in the first aspect or any optional manner of the first aspect. The at least one module can be implemented based on software, hardware, or a combination of software and hardware, and the at least one module can be arbitrarily combined or divided based on a specific implementation.

[0032] In a third aspect, a computing device is provided, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory so that the computing device executes the data query method provided in the first aspect or any optional method of the first aspect.

[0033] In a fourth aspect, a computing device cluster is provided, comprising at least one computing device, each of the at least one computing device comprising a processor and a memory, the processor of the at least one computing device being used to execute a computer program stored in the memory of the at least one computing device, so that the computing device cluster executes the data query method provided in the first aspect or any optional manner of the first aspect.

[0034] In a fifth aspect, a data query system is provided, comprising a client and a computing device cluster, wherein the client is communicatively connected to the computing device cluster, the computing device cluster is used to execute the data query method provided in the first aspect or any optional method of the first aspect, and the computing device cluster comprises the data query device provided in the second aspect.

[0035] In a sixth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed, the data query method provided in the first aspect or any optional manner of the first aspect is implemented. For example, when the computer program is executed by a computing device, a computing device cluster, a virtual machine, a container, a data query device, one or more processors, etc., the data query method provided in the first aspect or any optional manner of the first aspect is implemented.

[0036] In a seventh aspect, a computer program product including program instructions is provided, and when the program instructions are executed, a data query method as provided in the first aspect or any optional manner of the first aspect is implemented. For example, when the program instructions are executed by a computing device, a computing device cluster, a virtual machine, a container, a data query device, one or more processors, etc., a data query method as provided in the first aspect or any optional manner of the first aspect is implemented.

[0037] The technical effects of the second to seventh aspects mentioned above can refer to the technical effects of the first aspect and its optional implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0039] Figure 2 is a schematic diagram of a database index structure provided in an embodiment of the present application;

[0040] Figure 3 is a schematic diagram of another database index structure provided in an embodiment of the present application;

[0041] Figure 4 is a flow chart of a data query method provided by an embodiment of the present application;

[0042] Figure 5 yes Figure 2 A schematic diagram of a first index area in the database index structure shown;

[0043] Figure 6 yes Figure 3 A schematic diagram of a first index area in the database index structure shown;

[0044] Figure 7 yes Figure 2 A schematic diagram of another first index area in the database index structure shown;

[0045] Figure 8 yes Figure 3 A schematic diagram of another first index area in the database index structure shown;

[0046] Fig. 9 is a schematic diagram of multiple mapping positions in a first index area provided in an embodiment of the present application;

[0047] Fig.10 is a flow chart of another data query method provided by an embodiment of the present application;

[0048] Fig.11 yes Figure 3 A schematic diagram of a first index area and a second index area in the database index structure shown;

[0049] Fig.12 yes Figure 3 A schematic diagram of another first index area and a second index area in the database index structure shown;

[0050] Fig.13 is a flow chart of another data query method provided in an embodiment of the present application;

[0051] Fig.14 is a flow chart of another data query method provided in an embodiment of the present application;

[0052] Fig.15 is a flow chart of another data query method provided in an embodiment of the present application;

[0053] Fig.16 is a flow chart of another data query method provided in an embodiment of the present application;

[0054] Fig.17 is a flow chart of a data writing method provided by an embodiment of the present application;

[0055] Fig.18 is a schematic diagram of index segments of different words provided in an embodiment of the present application;

[0056] Fig.19 is a schematic diagram of another index segment of different words provided in an embodiment of the present application;

[0057] Fig. 20 is a schematic diagram of a data management method provided in an embodiment of the present application;

[0058] Fig.21 is a schematic diagram of a data query device provided in an embodiment of the present application;

[0059] Fig. 22 is a schematic diagram of a computing device provided in an embodiment of the present application;

[0060] Fig.23 is a schematic diagram of a computing device cluster provided in an embodiment of the present application;

[0061] Fig.24 It is a schematic diagram of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0063] Managing data through a database is an effective means of data management. For example, recording data blocks through a database, querying data blocks recorded in a database based on the query service provided by the database, etc. all fall into the category of managing data through a database. Currently, many customers are expected to generate hundreds of terabytes (TB) of data per day, and as their business expands, the amount of data generated by customers per day is still growing. In the scenario where the amount of data is constantly growing, managing data through a database is particularly important, and the database can improve data management efficiency to a certain extent.

[0064] Typically, a database is used not only to record data blocks, but also to record the index information of words in the data blocks, so as to query the data blocks based on the index information of the words in the query scenario. For example, when recording a data block in a database, that is, when a data block is stored in the database, an index structure is created in the database, and the index information of the words in the data block is recorded using the index structure. For example, common index structures include index structures based on bloom filters (BF), inverted index structures, etc., and the index structure based on BF is also called a BF index structure. For example, the database uses documents to record data blocks. For the inverted index structure, the index information of the word includes the identifier of the document where the word is located, the number of times the word appears in each document including the word, the position of the word in each document including the word, etc. For the BF index structure, the index information of the word includes indication information for indicating whether the word exists, also known as the existence index information of the word. Among them, the data blocks involved in the above description can be log data blocks or other data blocks. For example, a data block is recorded in multiple documents, each of the multiple documents is used to record a data row in the data block (or each of the multiple documents is a data row in the data block), each of the multiple documents includes multiple words, and when the data block is put into the database, the data block is segmented to obtain multiple words, and each word included in the data block and the index information of each word are recorded in the database.

[0065] Among them, the inverted index structure is an index structure used for full-text retrieval (also known as full-text query), which includes multiple inverted linked lists of multiple words, and the inverted linked list of each word is used to record the index information of the word. The index information of each word includes the identifier of the document where the word is located, the number of times the word appears in each document including the word, the position of the word in each document including the word, etc. For example, the Elasticsearch (ES) index structure is an inverted index structure implemented based on Lucene. Elasticsearch is a search server based on Lucene, which is a high-performance, scalable information retrieval tool library. Elasticsearch is used for distributed full-text retrieval.

[0066] Among them, the BF index structure is a probabilistic index structure with high space efficiency. A BF index structure includes a bit array and multiple hash functions. A bit array includes multiple bits, and each bit is used to store 0 or 1. The BF index structure can provide basic full-text query capabilities, which is used to query whether an element is in a set. For example, a BF index structure corresponds to a data block, and the BF index structure is used to record the existence index information of the words in the corresponding data block, so as to query whether a word is in the corresponding data block. The bit array in the BF index structure includes one or more bits corresponding to each word in the data block, and the 0 or 1 stored in these one or more bits is the existence index information of each word. For example, the index structure used by the index-free log engine represented by ClickHouse is the BF index structure. ClickHouse is a column storage database.

[0067] However, the indexing mechanism of the inverted index structure is heavy, which hinders the business from moving to the cloud and makes it difficult for customers to afford and use it. For example, the inverted index structure includes words and inverted linked lists of words. The more data blocks the database needs to store, the more words the inverted index structure needs to record, and the larger the inverted linked list of words, so the indexing mechanism of the inverted index structure is heavy. The heavy indexing mechanism leads to high storage costs of the inverted index structure and cache costs for queries based on the inverted index structure (the inverted index structure needs to be read into memory when querying based on the inverted index structure), which hinders the business from moving to the cloud. In the traditional BF index structure, the index information of a word may be recorded at any position in the BF index structure, which means that when querying data based on the traditional BF index structure, the entire content of the BF index structure needs to be read (the entire content of the BF index structure needs to be read into memory), which in turn leads to high input / output (I / O) costs for queries, large memory overhead, and poor query performance. In addition, the traditional BF index structure uses row storage, which means that when querying based on the traditional BF index structure, it is necessary to jump to read multiple bits in the BF index structure (that is, randomly read the index information), resulting in poor performance in reading the index information. In addition, the traditional BF index structure is relatively simple and cannot support prefix fuzzy query capabilities, that is, it cannot support querying data based on prefix words.

[0068] However, most of the words recorded in the database are low-frequency words, and there are massive data query scenarios in the database (for example, querying data blocks within a month). The database needs to support full-text retrieval capabilities, aggregate query capabilities, multi-word joint and query capabilities, multi-word joint or query capabilities, prefix fuzzy query capabilities, etc. It is also necessary to ensure that the database has low I / O costs and high query performance. The current inverted index structure and BF index structure are difficult to meet these requirements of the database.

[0069] The embodiment of the present application provides a data query method and device, and a computing device cluster. The data query method is executed by a data query device. In the embodiment of the present application, the database index structure includes m BF index structures and n index fragments based on column storage. The m BF index structures correspond to m data blocks, and each BF index structure is used to record the existence index information of words in the corresponding data block. The existence index information of any word in any data block is in the index segment corresponding to the any word in the BF index structure corresponding to the any data block, and the length of the index segment is less than the length of the BF index structure. Each of the m BF index structures includes n index units, and each index fragment includes m index units located in the same column of the m BF index structures, and the length of the index unit is equal to the length of the index segment. Based on this database index structure, after the data query device obtains a first data query request including a first word, the data query device determines the first index area corresponding to the first word in the database index structure. The first index area includes m first index segments corresponding to the first word in the m BF index structures, and the first index area corresponds to k index fragments in the n index fragments, k is equal to 1 or 2, and k is less than n. The data query device reads the k index fragments corresponding to the first index area to determine at least one first data block. The data query device queries the at least one first data block to determine the query result of the first data query request. That is, the data query device only needs to read the k index fragments corresponding to the first word in the database index structure to determine at least one first data block, and the data query device does not need to read the full content of the database index structure (that is, it does not need to read the entire database index structure) to realize data query, so the query I / O cost is low, the memory overhead is small, and the query performance is good. In addition, the embodiment of the present application does not need to randomly read the index information recorded in the BF index structure, so the query performance can be improved. The database index structure provided by the embodiment of the present application enables the database to support full-text retrieval capability, approximate aggregation query capability, multi-word joint and query capability, multi-word joint or query capability, prefix fuzzy query capability, etc.

[0070] The technical solution of the embodiment of the present application is introduced below. First, the application scenario of the embodiment of the present application is introduced.

[0071] The application scenario of the embodiment of the present application includes a computing device or a computing device cluster composed of multiple computing devices, and a data query device is deployed in the computing device or the computing device cluster, and the data query method provided by the embodiment of the present application is executed by the data query device. Optionally, the application scenario also includes a client, and the data query device performs data query by responding to a data query request from the client.

[0072] As an example, see Figure 1 , which shows a schematic diagram of an application scenario of an embodiment of the present application. The application scenario includes a computing device cluster 10 and a client 20. The client 20 is connected to the computing device cluster 10 in communication. For example, the client 20 is connected to the computing device cluster 10 in communication via a network. The network can be a local area network or the Internet, or other networks. Among them, a database is deployed in the computing device cluster 10, and the computing device cluster 10 can manage data through the database. Optionally, the application scenario also includes one or more data sources ( Figure 1 (not shown), the computing device cluster 10 manages the data generated by the one or more data sources through the database.

[0073] The user can interact with the computing device cluster 10 through the client 20. In one implementation, the user triggers the client 20 to send a data query request to the computing device cluster 10 by operating the client 20, and the computing device cluster 10 executes the data query method provided in the embodiment of the present application based on the data query request. Specifically, the data query request includes a word, and the computing device cluster 10 responds to the data query request and queries the index information of the word in the database. The computing device cluster 10 queries the data block related to the word in the database according to the index information of the word to determine the query result, and feeds back the query result to the client 20. Optionally, the data query request carries a query statement, and the query statement includes a word.

[0074] In an optional embodiment, the user triggers the client 20 to send a data write request to the computing device cluster 10 by operating the client 20, and the data write request is used to request to write a data block in the database; the computing device cluster 10 performs word segmentation on the data block requested to be written by the data write request to obtain at least one word, and the computing device cluster 10 records each word in the data block and the index information of each word in the database to implement the storage of the data block. In this way, the computing device cluster 10 can subsequently query the data block according to the query request including the word.

[0075] In an embodiment of the present application, the computing device cluster 10 can implement the data query method provided in the embodiment of the present application by running an executable program. For example, the executable program is presented in the form of an application installation package. After the computing device cluster 10 installs the application installation package, it can implement the data query method provided in the embodiment of the present application by running the executable program. In an optional embodiment, a database engine is deployed in the computing device cluster 10, and the database engine implements the data query method provided in the embodiment of the present application by running the executable program. For example, the database is used to manage log data, and the database engine deployed in the computing device cluster 10 is a log engine. Among them, all or part of the executable program and the database engine can be implemented as a data query device to execute the data query method provided in the embodiment of the present application.

[0076] In one implementation, the client 20 is deployed on a terminal device. The terminal device may be a desktop computer, a laptop computer, a tablet computer, a laptop computer, a mobile phone, a smart phone, a cloud host, a portable mobile terminal, a multimedia player, an e-book reader, a wearable device, a smart home appliance, an artificial intelligence device, an intelligent wearable device or a vehicle-mounted terminal device, etc. The computing device cluster 10 includes at least one computing device, and the computing device may be a server. Optionally, the computing device cluster 10 is a cluster composed of several computing devices, or a cloud computing service center. The cloud computing service center is deployed with a large number of basic resources such as computing resources, storage resources, and network resources owned by the cloud service provider. The cloud computing service center uses these basic resources to run the database and implement the data query method of the present application.

[0077] In one implementation, the computing device cluster 10 includes multiple computing devices, the database is deployed in the multiple computing devices, and the data blocks in the database are recorded in the multiple computing devices. This database is also called a distributed database (DDB). In another implementation, the computing device cluster 10 includes one computing device, the database is deployed in the one computing device, and the data blocks in the database are recorded in the one computing device. This database is also called a centralized database. Which database to use can be set according to business needs.

[0078] It should be understood that the above introduction to the application scenarios is only for illustrative purposes and is not intended to limit the technical solutions of the embodiments of the present application. A person skilled in the art may know that the application scenarios may be appropriately adjusted as business needs change, and the embodiments of the present application do not list them one by one.

[0079] The above is an introduction to the application scenarios of the embodiments of the present application. The following is an introduction to the embodiments of the data query method of the present application. In order to facilitate understanding of the data query method provided by the embodiments of the present application, the database index structure involved in the data query method is first introduced.

[0080] Please refer to Figure 2 , which shows a schematic diagram of a database index structure provided by an embodiment of the present application. Figure 2 As shown, the database index structure includes m BF index structures and n index fragments based on column storage. The m BF index structures are BF index structures 1 to m. The n index fragments are index fragments 1 to n. Both m and n are positive integers. For example, m = 1024, n = 4096.

[0081] Among them, the m BF index structures correspond to m data blocks, and each of the m BF index structures is used to record the existence index information of the words in the corresponding data block. The existence index information of any word in any data block is used to indicate the existence of the any word in the any data block, that is, to indicate whether the any word is included in the any data block. The existence index information of any word in any data block is in the index segment corresponding to the any word in the BF index structure corresponding to the any data block, and the length of the index segment is less than the length of the BF index structure. For example, BF index structures 1 to m correspond one-to-one to data blocks 1 to m. BF index structure 1 is used to record the existence index information of the words in data block 1, and the existence index information of any word in data block 1 is in the index segment corresponding to the any word in BF index structure 1. BF index structure 2 is used to record the existence index information of the words in data block 2, and the existence index information of any word in data block 2 is in the index segment corresponding to the any word in BF index structure 2. BF index structure 3 is used to record the existence index information of words in data block 3. The existence index information of any word in data block 3 is in the index segment corresponding to the word in BF index structure 3. And so on. BF index structure m is used to record the existence index information of words in data block m. The existence index information of any word in data block m is in the index segment corresponding to the word in BF index structure m.

[0082] Each of the m BF index structures includes n index units, and each of the n index fragments based on column storage includes m index units located in the same column of the m BF index structures, and the length of the index unit is equal to the length of the index segment. For example, BF index structure 1 includes index units 11 to 1n, BF index structure 2 includes index units 21 to 2n, BF index structure 3 includes index units 31 to 3n, and so on, and BF index structure m includes index units m1 to mn. Index units 11 to m1 are located in the same column, index units 12 to m2 are located in the same column, index units 13 to m3 are located in the same column, and so on, index units 1n to mn are located in the same column. Index fragment 1 includes index units 11 to m1, index fragment 2 includes index units 12 to m2, index fragment 3 includes index units 13 to m3, and so on, index fragment n includes index units 1n to mn. Among them, column storage is relative to row storage. In column storage, the data in a column exists in a continuous storage form in the storage medium. In row storage, the data in a row exists in a continuous storage form in the storage medium. For example, for the n index shards based on column storage, the index information in index units 11 to m1 exists in a continuous storage form in the storage medium, the index information in index units 12 to m2 exists in a continuous storage form in the storage medium, the index information in index units 13 to m3 exists in a continuous storage form in the storage medium, and so on, the index information in index units 1n to mn exists in a continuous storage form in the storage medium. That is, the index information in index shard 1 exists in a continuous storage form in the storage medium, the index information in index shard 2 exists in a continuous storage form in the storage medium, the index information in index shard 3 exists in a continuous storage form in the storage medium, and so on, the index information in index shard n exists in a continuous storage form in the storage medium. For example, the index information in index shard 2 and the index information in index shard 1 exist in a continuous storage form in the storage medium, the index information in index shard 3 and the index information in index shard 2 exist in a continuous storage form in the storage medium, and so on, the index information in index shard n and the index information in index shard n-1 exist in a continuous storage form in the storage medium. In the embodiment of the present application, the n index shards based on column storage are arranged, so that when performing data query, the index information in the BF index structure can be read continuously (the index information is read according to the granularity of the index shards, that is, after reading the index information in one index shard, the index information in another index shard is read, and the index information in each index shard exists in a continuous storage form, so the index information in the BF index structure can be continuously read), instead of jumping to read the index information in the BF index structure, which helps to ensure query performance.

[0083] In an optional embodiment, each of the m BF index structures includes n×s bits, each index unit in each BF index structure includes s bits, and the bits in each BF index structure are used to record the existence index information of words in the data block corresponding to each BF index structure, and s is a positive integer. By way of example, BF index structures 1 to m include n×s bits respectively, index units 11 to 1n, index units 21 to 2n, index units 31 to 3n...index units m1 to mn include s bits respectively, BF index structures 1 to m correspond one-to-one to data blocks 1 to m, the bits in BF index structure 1 are used to record the existence index information of words in data block 1, the bits in BF index structure 2 are used to record the existence index information of words in data block 2, the bits in BF index structure 3 are used to record the existence index information of words in data block 3, and so on, the bits in BF index structure m are used to record the existence index information of words in data block m. Figure 2 In the figure, index unit xy is any index unit among index units 11~1n, index units 21~2n, index units 31~3n...index units m1~mn, "x" is any value among 1~m, and "y" is any value among 1~n, which is not limited to the embodiments of the present application.

[0084] In an embodiment of the present application, an index segment is a segment in a BF index structure, and an index segment is also called a piece segment (segment) or an index segment. The length of an index segment is the number of bits included in the index segment. An index unit is a segment in a BF index structure, and the length of an index unit is the number of bits included in the index unit. The length of a BF index structure is the number of bits included in the BF index structure. For example, the length of an index segment and the length of an index unit are both s bits, and the length of a BF index structure is n×s bits, and the value of s and the value of n can be set according to the situation. In an example, s=8byte=8×8bit=64bit, n=4096, n×s=4096×64bit=262144bit=32KB.

[0085] For optional embodiments, please refer to Figure 3 , which shows a schematic diagram of another database index structure provided by an embodiment of the present application. Figure 2 Based on the database index structure shown, Figure 3The database index structure shown also includes m×w unit index structures, and each BF index structure in the m BF index structures corresponds to w unit index structures in the m×w unit index structures, where w is a positive integer. Each unit index structure corresponding to each BF index structure in the m BF index structures corresponds to at least one index unit in the BF index structure, and different unit index structures correspond to different index units. The embodiment of the present application takes the example that each unit index structure corresponding to each BF index structure corresponds to two index units in the BF index structure. For example, Figure 3 As shown, the m×w unit index structures include unit index structures 11~1w, unit index structures 21~2w, unit index structures 31~3w...unit index structures m1~mw. Unit index structures 11~1w correspond to BF index structure 1, and unit index structure 11 corresponds to index units 11~12, unit index structure 12 corresponds to index units 13~14...unit index structure 1w corresponds to index units 1(n-1)~1n. Unit index structures 21~2w correspond to BF index structure 2, and unit index structure 21 corresponds to index units 21~22, unit index structure 22 corresponds to index units 23~24...unit index structure 2w corresponds to index units 2(n-1)~2n. Unit index structures 31 to 3w correspond to BF index structure 3, and unit index structure 31 corresponds to index units 31 to 32, unit index structure 32 corresponds to index units 33 to 34, etc. Unit index structure 3w corresponds to index units 3(n-1) to 3n. Similarly, unit index structures m1 to mw correspond to BF index structure m, and unit index structure m1 corresponds to index units m1 to m2, unit index structure m2 corresponds to index units m3 to m4, etc. Unit index structure mw corresponds to index units m(n-1) to mn.

[0086] In an embodiment of the present application, each unit index structure corresponding to each BF index structure is used to record the secondary index information of the index unit corresponding to each unit index structure. The secondary index information recorded by each unit index structure includes one or more of the following information: the row number of the hit row of the index unit corresponding to each unit index structure, the number of hit rows of the index unit corresponding to each unit index structure, and the row number identification information. The hit row of the index unit corresponding to each unit index structure is the data row where the target word is located in the data block corresponding to the corresponding BF index structure, and the index segment corresponding to the target word hits the index unit corresponding to each unit index structure. For example, the starting position of the index segment corresponding to the target word hits the index unit corresponding to each unit index structure. In other words, the starting position of the index segment corresponding to the target word falls in the index unit corresponding to each unit index structure. The target word is in the data block corresponding to the BF index structure. The row number identification information indicates whether the number of hit rows of the index unit corresponding to each unit index structure is greater than the specified number. The row number identification information can be 0 or 1. "0" indicates that the number of hit rows of the index unit corresponding to each unit index structure is not greater than the specified number. "1" indicates that the number of hit rows of the index unit corresponding to each unit index structure is greater than the specified number. The specified number can be set according to actual conditions. Among them, when the row number identification information indicates that the number of hit rows of the index unit corresponding to each unit index structure is greater than the specified number, it can be considered that the index unit corresponding to each unit index structure is a high-frequency hit index unit. By way of example, the secondary index information recorded by each unit index structure includes: the row number of the hit row of the index unit corresponding to each unit index structure, the number of hit rows of the index unit corresponding to each unit index structure, and the row number identification information. Each unit index structure includes a hit row number field, a row number identification field, and at least one row number field. The hit row number field is used to record the number of hit rows of the index unit corresponding to each unit index structure, the row number identification field is used to record the row number identification information, and the at least one row number field is used to record the row number of the hit row of the index unit corresponding to each unit index structure. By way of example, in Figure 3 In the above example, the unit index structure xz is any unit index structure among the unit index structures 11 to 1w, 21 to 2w, 31 to 3w, etc., and "x" is any value among 1 to m, and "z" is any value among 1 to w. Figure 3As shown, each unit index structure in the unit index structure 11~1w, unit index structure 21~2w, unit index structure 31~3w... unit index structure m1~mw includes a hit row number field, a row number identification field and g row number fields, where g is a positive integer. For example, g=16. Among them, g=16 can be a value calculated by mathematical calculation. For example, mathematical calculations are performed on a data block including 3000 rows of data, and it is found that when g=16, it can be applicable to most scenarios, so the embodiment of the present application adopts g=16.

[0087] In one example, BF index structure 1 corresponds to data block 1, unit index structures 11-1w all correspond to BF index structure 1, and index units 11-12 correspond to unit index structure 11, index units 13-14 correspond to unit index structure 12, and so on, index units 1(n-1)-1n correspond to unit index structure 1w. Unit index structure 11 is used to record secondary index information of index units 11-12, and the secondary index information recorded by unit index structure 11 includes the row number of the hit row of index units 11-12, the number of hit rows of index units 11-12, and row number identification information 11, and the row number identification information 11 is used to indicate whether the number of hit rows of index units 11-12 is greater than a specified number, and the hit rows of index units 11-12 are the data rows where the words that hit index units 11-12 in data block 1 are located, or in other words, among the multiple words included in data block 1, the corresponding index segments hit the data rows where those words (i.e., target words) of index units 11-12 are located. The unit index structure 12 is used to record the secondary index information of the index units 13-14. The secondary index information recorded in the unit index structure 12 includes the row number of the hit row of the index units 13-14, the number of hit rows of the index units 13-14, and the row number identification information 12. The row number identification information 12 is used to indicate whether the number of hit rows of the index units 13-14 is greater than the specified number. The hit rows of the index units 13-14 are the data rows where the words that hit the index units 13-14 in the data block 1 are located, or in other words, among the multiple words included in the data block 1, the corresponding index segments hit the data rows where those words (that is, the target words) of the index units 13-14 are located. And so on. The unit index structure 1w is used to record the secondary index information of index units 1(n-1)~1n. The secondary index information recorded in the unit index structure 1w includes the row number of the hit row of index units 1(n-1)~1n, the number of hit rows of index units 1(n-1)~1n and the row number identification information 1w. The row number identification information 1w is used to indicate whether the number of hit rows of index units 1(n-1)~1n is greater than the specified number. The hit rows of index units 1(n-1)~1n are the data rows where the words that hit index units 1(n-1)~1n in data block 1 are located, or in other words, among the multiple words included in data block 1, the corresponding index segments hit the data rows where those words (i.e., target words) of index units 1(n-1)~1n are located. For example, data block 1 includes word 11, word 12, word 13, word 14, word 15 and word 16. The index segment corresponding to word 11 and the index segment corresponding to word 12 both hit index unit 11 (for example, the starting position of the index segment corresponding to word 11 and the starting position of the index segment corresponding to word 12 both hit index unit 11). The index segment corresponding to word 13 hits index unit 12 (for example, the starting position of the index segment corresponding to word 13 hits index unit 12).The index segment corresponding to word 14 hits index unit 13 (for example, the starting position of the index segment corresponding to word 14 hits index unit 13). The index segment corresponding to word 15 hits index unit 14 (for example, the starting position of the index segment corresponding to word 15 hits index unit 14). The index segment corresponding to word 16 hits index unit 1n (for example, the starting position of the index segment corresponding to word 16 hits index unit 1n). Then the hit rows of index units 11 to 12 include the data rows where each word in words 11 to 13 is located in data block 1. The hit rows of index units 13 to 14 include the data rows where each word in words 14 to 15 is located in data block 1. The hit rows of index units 1(n-1) to 1n include the data rows where word 16 is located in data block 1. The secondary index information recorded in the unit index structure 11 includes the row number of the data row where each word in the words 11 to 13 is located in the data block 1, the number of hit rows of index units 11 to 12 "3" and the row number identification information 11. The row numbers of the data rows where the words 11 to 13 are located in the data block 1 are respectively recorded in the three row number fields of the unit index structure 11, the number of hit rows of index units 11 to 12 "3" is recorded in the hit row number field of the unit index structure 11, and the row number identification information 11 is recorded in the row number identification field of the unit index structure 11. The secondary index information recorded in the unit index structure 12 includes the row number of the data row where each word in words 14 to 15 is located in the data block 1, the number of hit rows of index units 13 to 14 "2" and the row number identification information 12. The row numbers of the data rows where words 14 to 15 are located in the data block 1 are respectively recorded in the two row number fields of the unit index structure 12, the number of hit rows of index units 13 to 14 "2" is recorded in the hit row number field of the unit index structure 12, and the row number identification information 12 is recorded in the row number identification field of the unit index structure 12. The secondary index information recorded in the unit index structure 1w includes the row number of the data row where the word 16 is located in the data block 1, the number of hit rows of index units 1(n-1)~1n "1" and the row number identification information 1w. The row number of the data row where the word 16 is located in the data block 1 is recorded in a row number field of the unit index structure 1w, the number of hit rows of index units 1(n-1)~1n "1" is recorded in the hit row number field of the unit index structure 1w, and the row number identification information 1w is recorded in the row number identification field of the unit index structure 1w.For example, the specified number is 16, the number of hit rows "3" of index units 11 to 12, the number of hit rows "2" of index units 13 to 14, and the number of hit rows "1" of index units 1(n-1) to 1n are not greater than the specified number, therefore, row number identification information 11 is used to indicate that the number of hit rows "3" of index units 11 to 12 is not greater than the specified number, row number identification information 12 is used to indicate that the number of hit rows "2" of index units 13 to 14 is not greater than the specified number, and row number identification information 1w is used to indicate that the number of hit rows "1" of index units 1(n-1) to 1n is not greater than the specified number. For example, row number identification information 11, row number identification information 12, and row number identification information 1w are all "0".

[0088] In another example, BF index structure 2 corresponds to data block 2, unit index structures 21-2w all correspond to BF index structure 2, and index units 21-22 correspond to unit index structure 21, index units 23-24 correspond to unit index structure 22, and so on, index units 2(n-1)-2n correspond to unit index structure 2w. Unit index structure 21 is used to record secondary index information of index units 21-22, and the secondary index information recorded by unit index structure 21 includes the row number of the hit row of index units 21-22, the number of hit rows of index units 21-22, and row number identification information 21, which is used to indicate whether the number of hit rows of index units 21-22 is greater than a specified number, and the hit row of index units 21-22 is the data row where the words that hit index units 21-22 in data block 2 are located, or in other words, among the multiple words included in data block 2, the corresponding index segments hit index units 21-22 (that is, the target words) The data row where the words are located. The unit index structure 22 is used to record the secondary index information of the index units 23-24. The secondary index information recorded in the unit index structure 22 includes the row number of the hit row of the index units 23-24, the number of hit rows of the index units 23-24, and the row number identification information 22. The row number identification information 22 is used to indicate whether the number of hit rows of the index units 23-24 is greater than the specified number. The hit rows of the index units 23-24 are the data rows where the words that hit the index units 23-24 in the data block 2 are located, or in other words, among the multiple words included in the data block 2, the corresponding index segments hit the index units 23-24 (that is, the target words) The data rows are located. And so on. The unit index structure 2w is used to record the secondary index information of the index units 2(n-1)~2n. The secondary index information recorded in the unit index structure 2w includes the row number of the hit row of the index unit 2(n-1)~2n, the number of hit rows of the index unit 2(n-1)~2n and the row number identification information 2w. The row number identification information 2w is used to indicate whether the number of hit rows of the index unit 2(n-1)~2n is greater than the specified number. The hit row of the index unit 2(n-1)~2n is the data row where the words that hit the index unit 2(n-1)~2n in the data block 2 are located, or in other words, among the multiple words included in the data block 2, the corresponding index segments hit the data row where those words (that is, the target words) of the index unit 2(n-1)~2n are located. For example, the data block 2 includes words 21, 22, 23, 24, 25, 26, 27, 28 and 29.The index segment corresponding to word 21, the index segment corresponding to word 22, the index segment corresponding to word 23, and the index segment corresponding to word 24 all hit index unit 21 (for example, the starting position of the index segment corresponding to word 21, the starting position of the index segment corresponding to word 22, the starting position of the index segment corresponding to word 23, and the starting position of the index segment corresponding to word 24 all hit index unit 21). The index segment corresponding to word 25 and the index segment corresponding to word 26 both hit index unit 22 (for example, the starting position of the index segment corresponding to word 25 and the starting position of the index segment corresponding to word 26 both hit index unit 22). The index segment corresponding to word 27 hits index unit 23 (for example, the starting position of the index segment corresponding to word 27 hits index unit 23). The index segment corresponding to word 28 and the index segment corresponding to word 29 both hit index unit 24 (for example, the starting position of the index segment corresponding to word 28 and the starting position of the index segment corresponding to word 29 both hit index unit 24). Then the hit row of index units 21-22 includes the data row where each word in terms 21-26 is located in data block 2. The hit row of index units 23-24 includes the data row where each word in terms 27-29 is located in data block 2. The secondary index information recorded in the unit index structure 21 includes the row number of the data row where each word in terms 21-26 is located in data block 2, the number of hit rows of index units 21-22 "6" and row number identification information 21, the row numbers of the data rows where words 21-26 are located in data block 2 are respectively recorded in the six row number fields of the unit index structure 21, the number of hit rows of index units 21-22 "6" is recorded in the hit row number field of the unit index structure 21, and the row number identification information 21 is recorded in the row number identification field of the unit index structure 21. The secondary index information recorded in the unit index structure 22 includes the row number of the data row where each word in the words 27-29 is located in the data block 2, the number of hit rows of index units 23-24 "3" and the row number identification information 22, the row numbers of the data rows where the words 27-29 are located in the data block 2 are respectively recorded in the three row number fields of the unit index structure 22, the number of hit rows of index units 23-24 "2" is recorded in the hit row number field of the unit index structure 22, and the row number identification information 22 is recorded in the row number identification field of the unit index structure 22. For example, the specified number is 16, the number of hit rows of index units 21-22 "6" and the number of hit rows of index units 23-24 "3" are not greater than the specified number, therefore, the row number identification information 21 is used to indicate that the number of hit rows of index units 21-22 "6" is not greater than the specified number, and the row number identification information 22 is used to indicate that the number of hit rows of index units 23-24 "3" is not greater than the specified number. For example, the row number identification information 21 and the row number identification information 22 are both "0".

[0089] The above mainly introduces the secondary index information recorded by the unit index structure corresponding to BF index structure 1 and the secondary index information recorded by the unit index structure corresponding to BF index structure 2. The secondary index information recorded by the unit index structure corresponding to other BF index structures can refer to the above description and will not be repeated here. In addition, in the above introduction, the secondary index information recorded by each unit index structure includes the row number of the hit row of the index unit corresponding to each unit index structure, the number of hit rows of the index unit corresponding to each unit index structure, and the row number identification information as an example. In some embodiments, when the row number identification information indicates that the number of hit rows of the index unit corresponding to each unit index structure is greater than the specified number, it means that the index unit corresponding to each unit index structure is a high-frequency hit index unit. When performing subsequent data query, it is usually necessary to perform a full scan of the data block corresponding to the BF index structure corresponding to each unit index structure. Therefore, each unit index structure may not record the row number of the hit row of the index unit corresponding to each unit index structure, and the embodiment of the present application does not limit this.

[0090] In an optional embodiment, each of the m BF index structures includes a low-frequency index space and a high-frequency index space, wherein the high-frequency index space is used to record the existence index information of high-frequency words in the data block corresponding to each BF index structure, and the low-frequency index space is used to record the existence index information of low-frequency words in the data block corresponding to each BF index structure. In one embodiment, the lowest v bits in the BF index structure are divided into the high-frequency index space, and the part of the BF index structure other than the lowest v bits is determined as the low-frequency index space, and the value of v is set according to the situation. For example, the length of the BF index structure is 262144 bits (i.e., 32KB), the lowest 65536 bits (i.e., 8KB) in the BF index structure are divided into the high-frequency index space, and the remaining 196608 bits (i.e., 24KB) in the BF index structure are determined as the low-frequency index space. Among them, high-frequency words and low-frequency words can be determined by a word classification model, and the word classification model is a machine learning model pre-trained to determine high-frequency words and low-frequency words. For example, the word classification model mines the relationship between words in a large amount of sample data through an offline learning mapping method, and based on the model learned by such mining, the input parameter of the word classification model is a word, and the output parameter of the word classification model is the indication information of whether the word is a high-frequency word or a low-frequency word.

[0091] For example, BF index structure 1 corresponds to data block 1, BF index structure 1 includes low-frequency index space 11 and high-frequency index space 12, the length of BF index structure 1 is 32KB, high-frequency index space 12 is the lowest 8KB of BF index structure 1, low-frequency index space 11 is the remaining 24KB of BF index structure 1, high-frequency index space 12 is used to record the existence index information of high-frequency words in data block 1, and low-frequency index space 11 is used to record the existence index information of low-frequency words in data block 1. BF index structure 2 corresponds to data block 2, BF index structure 2 includes low-frequency index space 21 and high-frequency index space 22, the length of BF index structure 2 is 32KB, high-frequency index space 22 is the lowest 8KB of BF index structure 2, low-frequency index space 21 is the remaining 24KB of BF index structure 2, high-frequency index space 22 is used to record the existence index information of high-frequency words in data block 2, and low-frequency index space 21 is used to record the existence index information of low-frequency words in data block 2. And so on. The BF index structure m corresponds to the data block m. The BF index structure m includes a low-frequency index space m1 and a high-frequency index space m2. The high-frequency index space m2 is the lowest 8KB of the BF index structure m, and the low-frequency index space m1 is the remaining 24KB of the BF index structure m. The high-frequency index space m2 is used to record the existence index information of high-frequency words in the data block m, and the low-frequency index space m1 is used to record the existence index information of low-frequency words in the data block m.

[0092] The indexing performance of the BF index structure will be affected by the false positive rate, which in turn affects the query performance based on the BF index structure. The embodiment of the present application can reduce the false positive rate and ensure the query performance based on the BF index structure by dividing the BF index structure into a low-frequency index space and a high-frequency index space. Specifically, in a scenario where index information is recorded based on the BF index structure, both low-frequency words and high-frequency words are mapped to the BF index structure according to hash mapping. If the BF index structure is not divided into a low-frequency index space and a high-frequency index space, when the hash value of a low-frequency word is the same as the hash value of another high-frequency word, and the low-frequency word and the high-frequency word are located in different data blocks, the low-frequency word and the high-frequency word will be mapped to the same position of different BF index structures. For example, the low-frequency word is located in data block 1, and the high-frequency word is located in data block 2. The low-frequency word will be mapped to position 1 of the BF index structure 1 corresponding to data block 1 (for example, the 1st bit, the 10th bit, the 15th bit and the 20th bit in the BF index structure 1), and the high-frequency word will be mapped to position 1 of the BF index structure 2 corresponding to data block 2 (for example, the 1st bit, the 10th bit, the 15th bit and the 20th bit in the BF index structure 2). The existence index information of the low-frequency word is recorded in position 1 of the BF index structure 1, and the existence index information of the high-frequency word is recorded in position 1 of the BF index structure 2. The existence index information of the low-frequency word is the same as the existence index information of the high-frequency word. In this way, when querying the low-frequency word, the data blocks found according to the existence index information recorded in the BF index structure include data block 1 and data block 2, but data block 2 does not include the low-frequency word, which causes a misjudgment. This misjudgment is caused by the contamination of the low-frequency word by the high-frequency word, and this misjudgment will affect the query performance of the scan, the query performance of the aggregate (count) query, etc. The embodiment of the present application can avoid the low-frequency word and the high-frequency word being mapped to the same position of different BF index structures by dividing the BF index structure into a low-frequency index space and a high-frequency index space, thereby avoiding the high-frequency word from polluting the low-frequency word and causing a query misjudgment, and ensuring the query performance based on the BF index structure. It should be noted that mapping a certain word to the BF index structure refers to recording the existence index information of the certain word in the BF index structure, and the process of recording the existence index information of a word in the BF index structure is also called the process of mapping the word to the BF index structure. In addition, in a scan query scenario, usually after a target number of data including the term to be queried is queried, for example, after a target number of data rows including the term to be queried is queried, data is trimmed based on the time of the last queried data among the target number of data to obtain the query result.If the term to be queried is a low-frequency term or a non-existent term, it is difficult to query the target number of data including the term to be queried, and thus it is difficult to perform data clipping based on time, which results in poor performance in querying low-frequency terms in scan query scenarios. The embodiment of the present application divides the BF index structure into a low-frequency index space and a high-frequency index space, so that in the scan query scenario, it can be determined whether the term to be queried is a low-frequency term or a high-frequency term based on the position of the existence index information of the term to be queried in the BF index structure. In the case where the term to be queried is a high-frequency term, the data block found based on the existence index information of the term to be queried is scanned to query the data including the term to be queried. In the case where the term to be queried is a low-frequency term, the data block is not scanned to avoid performing an invalid scanning process (because when the term to be queried is a low-frequency term or a non-existent term, it is difficult to query the target number of data including the term to be queried by scanning the data block. In this case, scanning ultimately fails to obtain a query result, so it is considered an invalid scanning process).

[0093] The following is an introduction to the method embodiments of the present application.

[0094] Please refer to Figure 4 , which shows a flow chart of a data query method provided by an embodiment of the present application. The data query method is executed by a data query device. For example, the data query device is deployed in Figure 1 The computing device cluster 10 is shown.

[0095] like Figure 4 As shown, the data query method includes the following steps S401 to S404.

[0096] S401. Obtain a first data query request, where the first data query request includes a first word.

[0097] The first word is a complete word or a prefix of the complete word. For example, the complete word is "abcd", the prefix of the complete word can be "a", "ab" or "abc", and the first word can be the complete word "abcd" or any prefix of the complete word "abcd", for example, the first word is "a", "ab" or "abc".

[0098] In an optional embodiment, the first data query request is used to query data including the first word. The first data query request includes a query statement, and the query statement includes the first word. For example, the first data query request is a scan query request, and the query statement is a scan query statement.

[0099] In an optional embodiment, the data query device receives a first data query request sent by a client to obtain the first data query request, and the first data query request is triggered by a user operating the client. Specifically, the data query device is connected to the client in communication, for example, a computing device or a computing device cluster on which the data query device is deployed is connected to the client in communication, the user operates the client to trigger the first data query request, the client sends the first data query request to the data query device, and the data query device receives the first data query request.

[0100] S402. Determine a first index region corresponding to the first term in a database index structure, wherein the database index structure includes m BF index structures and n index shards based on column storage, each BF index structure includes n index units, each index shard includes m index units located in the same column of the m BF index structures, the first index region includes m first index segments corresponding to the first term in the m BF index structures, and the first index region corresponds to k index shards of the n index shards, k is equal to 1 or 2, and k is less than n.

[0101] Wherein, m and n are both positive integers, for example, m=1024, n=4096.

[0102] Among them, the m BF index structures correspond to m data blocks, that is, the m BF index structures correspond to the m data blocks one by one. Each BF index structure in the m BF index structures is used to record the existence index information of the words in the corresponding data block. The existence index information of any word in any data block is used to indicate the existence of the any word in the any data block. The existence index information of any word in any data block is in the index segment corresponding to the any word in the BF index structure corresponding to the any data block, the length of the index segment is less than the length of the BF index structure, the length of the index unit is equal to the length of the index segment, and thus the length of the index unit is also less than the length of the BF index structure. Among them, the BF index structure used to record the existence index information of the word can be understood as the BF index structure used to map the word, and recording the existence index information of the word in the BF index structure can be understood as mapping the word to the BF index structure.

[0103] In an optional embodiment, each of the m BF index structures includes n×s bits, each index unit in each BF index structure includes s bits, and each index segment also includes s bits. The bits in each BF index structure are used to record the existence index information of the words in the data block corresponding to each BF index structure, and s is a positive integer. For example, the length of each BF index structure in the m BF index structures is n×s bits, and the length of the index segment and the length of the index unit are both s bits, which is not limited in the embodiment of the present application.

[0104] It should be noted that the database index structure described in this embodiment can be Figure 2 or Figure 3 The database index structure shown in the figure is detailed in the following table. Figure 2 and Figure 3 And its related description, this embodiment mainly introduces the implementation process of determining the first index area corresponding to the first word in the database index structure, and the detailed introduction of the database index structure is not repeated here.

[0105] In an optional embodiment, the data query device obtains the first word included in the first data query request, and the data query device determines the starting position (that is, the starting boundary) of the first index area corresponding to the first word in the database index structure based on the first word, and the data query device determines the first index area based on the starting position and the length of the index segment in the database index structure. Among them, the distance between the starting position of the first index area and the ending position (that is, the ending boundary) of the first index area is equal to the length of the index segment. The first index area includes m first index segments corresponding to the first word in the m BF index structures, and the m first index segments are located one-to-one in the m BF index structures. The first index area corresponds to k index fragments among the n index fragments, and k is equal to 1 or 2.

[0106] In an optional embodiment, the data query device determines the starting position of the first index area corresponding to the first word in the database index structure according to the first word, including: the data query device obtains the hash value of the first word (for the sake of convenience of distinction, the hash value is called the target hash value); the data query device determines the starting position of the first index area in the database index structure according to the target hash value of the first word. In a specific embodiment, the data query device uses a target hash function to perform hash calculation on the first word to obtain the target hash value of the first word; the data query device determines the offset value of the first word according to the target hash value of the first word; the data query device determines the starting position of the first index area in the database index structure according to the offset value of the first word. For example, the data query device determines the position indicated by the offset value of the first word in the database index structure as the starting position of the first index area. In one embodiment, the data query device uses a part of the target hash value of the first word (for example, referred to as the first part) as the offset value of the first word, and the first part includes a plurality of consecutive bits. The starting position of the first part can be the highest bit or the lowest bit of the target hash value. This embodiment of the present application is not limited to this.

[0107] In an optional embodiment, the data query device determines the first index area according to the starting position of the first index area and the length of the index segment in the database index structure, including: the data query device determines the end position of the first index area in the database index structure according to the starting position of the first index area and the length of the index segment in the database index structure; the data query device determines the first index area according to the starting position of the first index area and the end position of the first index area. For example, the data query device determines the position after the starting position and with a distance of s bits from the starting position as the end position of the first index area.

[0108] In the embodiment of the present application, the starting position of the first index region may coincide with the starting boundary of a certain index slice among the n index slices (or the starting position of the first index region is the starting boundary of the certain index slice), and the starting position of the first index region may also be located in a certain index slice among the n index slices (that is, the starting position of the first index region does not coincide with the starting boundary of any index slice among the n index slices). Depending on the starting position of the first index region, the value of k is different.

[0109] In one embodiment, the starting position of the first index region coincides with the starting boundary of a certain index fragment among the n index fragments, k is equal to 1, the first index region corresponds to 1 index fragment, and the 1 index fragment is the certain index fragment. Since the distance between the starting position of the first index region and the ending position of the first index region is equal to the length of the index segment, and the length of the index fragment is equal to the length of the index segment (specifically, the length of the index unit in the index fragment is equal to the length of the index segment), therefore, when the starting position of the first index region coincides with the starting boundary of a certain index fragment, the ending position of the first index region coincides with the ending boundary of the certain index fragment.

[0110] For examples, please refer to Figure 5 and Figure 6 , which shows a schematic diagram of a first index area provided in an embodiment of the present application. Figure 5 It shows Figure 2 The first index area in the database index structure shown, Figure 6 It shows Figure 3 The first index area in the database index structure shown. Figure 5 and Figure 6 As shown, the starting position of the first index region coincides with the starting boundary of index fragment 1 (the left boundary of index fragment 1), and the ending position of the first index region coincides with the ending boundary of index fragment 1 (the right boundary of index fragment 1). The first index region corresponds to one index fragment, and the one index fragment is index fragment 1. The first index region includes m first index segments, and the m first index segments are index units 11 to m1 in the first index region. For example, the length of an index segment (such as the first index segment) is s bits, and the distance between the starting position of the first index region and the ending position of the first index region is s bits.

[0111] In another embodiment, the starting position of the first index region is located in a certain index slice among the n index slices, k is equal to 2, the first index region corresponds to 2 index slices, the 2 index slices are adjacent, and one of the 2 index slices is the certain index slice. Since the distance between the starting position of the first index region and the ending position of the first index region is equal to the length of the index segment, and the length of the index slice is equal to the length of the index segment (the length of the index unit in the index slice is equal to the length of the index segment), therefore, in the case where the starting position of the first index region is located in a certain index slice, the ending position of the first index region is located in another index slice adjacent to the certain index slice, and the other index slice of the 2 index slices is the index slice where the ending position of the first index region is located.

[0112] For examples, please refer to Figure 7 and Figure 8 , which shows a schematic diagram of another first index area provided in an embodiment of the present application. Figure 7 It is shown that Figure 2 The first index area in the database index structure shown, Figure 8 It is shown that Figure 3 The first index area in the database index structure shown. Figure 7 and Figure 8 As shown, the starting position of the first index region is located in index shard 1, the ending position of the first index region is located in index shard 2, and the first index region corresponds to two index shards, which are index shard 1 and index shard 2. The first index region includes m first index segments, which are located in the m BF index structures in a one-to-one correspondence, and each of the m first index segments is composed of a portion of the corresponding BF index structure located in the first index region. For example, the first index segment in BF index structure 1 is composed of the part of index unit 11 located in the first index area and the part of index unit 12 located in the first index area, the first index segment in BF index structure 2 is composed of the part of index unit 21 located in the first index area and the part of index unit 22 located in the first index area, the first index segment in BF index structure 3 is composed of the part of index unit 31 located in the first index area and the part of index unit 32 located in the first index area, and so on, the first index segment in BF index structure m is composed of the part of index unit m1 located in the first index area and the part of index unit m2 located in the first index area. For example, the length of the index segment is s bits, and the distance between the starting position of the first index area and the ending position of the first index area is s bits.

[0113] In an optional embodiment, each of the m BF index structures includes a low-frequency index space and a high-frequency index space, and the database index structure includes a low-frequency index area and a high-frequency index area, the low-frequency index area includes the low-frequency index space of the m BF index structures, and the high-frequency index area includes the high-frequency index space of the m BF index structures. In the case where the first word is a high-frequency word, the data query device determines a first index area corresponding to the first word in the high-frequency index area of ​​the database index structure, the first index area is located in the high-frequency index area, and the m first index segments included in the first index area are located in the high-frequency index space of the m BF index structures in a one-to-one correspondence. In the case where the first word is a low-frequency word, the data query device determines a first index area corresponding to the first word in the low-frequency index area of ​​the database index structure, the first index area is located in the low-frequency index area, and the m first index segments included in the first index area are located in the low-frequency index space of the m BF index structures in a one-to-one correspondence. For example, Figures 5 to 8 The first index region shown may be located in the high-frequency index region or in the low-frequency index region.

[0114] S403. Read the k index fragments corresponding to the first index area to determine at least one first data block, wherein the first index segment in the BF index structure corresponding to the first data block includes existence index information of the first word.

[0115] The m data blocks mentioned above include the at least one first data block.

[0116] In an optional embodiment, the data query device reads the k index fragments corresponding to the first index area into the memory. Afterwards, the data query device obtains multiple hash values ​​of the first term (for ease of distinction, the multiple hash values ​​are referred to as mapping hash values); the data query device determines multiple mapping positions of the first term in the first index area according to the multiple mapping hash values; the data query device determines at least one first data block in the m data blocks according to the values ​​of the multiple mapping positions. The multiple mapping positions correspond one-to-one to the multiple mapping hash values. By way of example, for each of the multiple mapping hash values, the data query device determines the position indicated by the mapping hash value in the first index area as a mapping position.

[0117] Among them, each of the multiple mapping positions is a bit, each of the multiple mapping positions is located in the m first index segments included in the first index area, each of the m first index segments includes the multiple mapping positions, the m first index segments are located one-to-one in m BF index structures, and the m BF index structures correspond one-to-one to the m data blocks. The data query device determines at least one first data block in the m data blocks according to the values ​​of the multiple mapping positions, including: for each first index segment in the m first index segments, the data query device reads the values ​​of the multiple mapping positions in the first index segment; the data query device determines whether the data block corresponding to the first index segment (that is, the data block corresponding to the BF index structure including the first index segment) may include the first word according to the values ​​of the multiple mapping positions in the first index segment; when the data query device determines that the data block corresponding to the first index segment may include the first word, the data query device determines the data block corresponding to the first index segment as the first data block; when the data query device determines that the data block corresponding to the first index segment cannot include the first word, the data query device does not determine the data block corresponding to the first index segment as the first data block. In this way, the data query device can determine at least one first data block in the m data blocks. In one embodiment, the data query device determines whether the data block corresponding to the first index segment may include the first word based on the values ​​of the multiple mapping positions in the first index segment, including: the data query device determines whether the values ​​of the multiple mapping positions in the first index segment are all 1; when the values ​​of the multiple mapping positions in the first index segment are all 1, the data query device determines that the data block corresponding to the first index segment may include the first word; when the value of at least one of the multiple mapping positions in the first index segment is not 1, the data query device determines that the data block corresponding to the first index segment cannot include the first word.

[0118] As an example, the m first index segments are first index segments 11-m1, and the first index segments 11-m1 are located in the BF index structures 1-m in a one-to-one correspondence. The BF index structures 1-m correspond to the data blocks 1-m in a one-to-one correspondence. For example, the length of the first index segments 11-m1 is 64 bits, and the multiple mapping positions are 4 mapping positions. Please refer to Fig. 9 , which shows 4 mapping locations in the first index area (for simplicity Fig. 9 A schematic diagram of the first index area is not shown in FIG. Fig. 9 The first index segment 11-m1 shown in can be Figure 5 or Figure 6 The m first index segments in the first index area shown may also be Figure 7 or Figure 8There are m first index segments in the first index area shown. Fig. 9 As shown, each of the first index segments 11 to m1 includes mapping positions 11 to 14. In each first index segment, mapping position 11 is the 8th bit in the first index segment, mapping position 12 is the 16th bit in the first index segment, mapping position 13 is the 40th bit in the first index segment, and mapping position 14 is the 60th bit in the first index segment. For example, the data query device determines whether the values ​​of mapping positions 11 to 14 in the first index segment 11 are all 1; when the values ​​of mapping positions 11 to 14 in the first index segment 11 are all 1, the data query device determines that data block 1 (that is, the data block corresponding to the BF index structure 1 including the first index segment 11) may include the first word, and the data query device determines data block 1 as the first data block; when the value of at least one of the mapping positions 11 to 14 in the first index segment 11 is not 1, for example, when the value of mapping position 11 in the first index segment 11 is 0, the data query device determines that data block 1 cannot include the first word, and the data query device does not determine data block 1 as the first data block. The data query device determines whether the values ​​of mapping positions 11 to 14 in the first index segment 21 are all 1; when the values ​​of mapping positions 11 to 14 in the first index segment 21 are all 1, the data query device determines that data block 2 (that is, the data block corresponding to the BF index structure 2 including the first index segment 21) may include the first word, and the data query device determines data block 2 as the first data block; when the value of at least one of the mapping positions 11 to 14 in the first index segment 21 is not 1, for example, when the value of mapping position 12 in the first index segment 21 is 0, the data query device determines that data block 2 cannot include the first word, and the data query device does not determine data block 2 as the first data block. And so on. The data query device determines whether the values ​​of mapping positions 11 to 14 in the first index segment m1 are all 1; when the values ​​of mapping positions 11 to 14 in the first index segment m1 are all 1, the data query device determines that the data block m (that is, the data block corresponding to the BF index structure m including the first index segment m1) may include the first word, and the data query device determines the data block m as the first data block; when the value of at least one of the mapping positions 11 to 14 in the first index segment m1 is not 1, the data query device determines that the data block m cannot include the first word, and the data query device does not determine the data block m as the first data block.

[0119] In an optional embodiment, the data query device obtains multiple mapping hash values ​​of the first word, including: the data query device uses each mapping hash function of multiple mapping hash functions to perform hash calculation on the first word to obtain a mapping hash value of the first word. The data query device can obtain multiple mapping hash values ​​of the first word, the multiple mapping hash values ​​correspond to the multiple mapping hash functions one by one, and each mapping hash value of the multiple mapping hash values ​​is obtained by performing hash calculation on the first word using the corresponding mapping hash function.

[0120] In another optional embodiment, the data query device obtains multiple mapping hash values ​​of the first word, including: the data query device uses each mapping hash function of a plurality of mapping hash functions to perform hash calculation on the target hash value of the first word to obtain a mapping hash value of the first word. The data query device can obtain multiple mapping hash values ​​of the first word, the multiple mapping hash values ​​correspond to the multiple mapping hash functions one by one, and each mapping hash value of the multiple mapping hash values ​​is obtained by using the corresponding mapping hash function to perform hash calculation on the target hash value of the first word. Among them, the target hash value of the first word is the hash value of the first word obtained by using the target hash function to perform hash calculation on the first word, that is, the target hash value of the first word calculated in S402.

[0121] In another optional embodiment, the data query device obtains multiple mapping hash values ​​of the first word, including: the data query device uses each mapping hash function of a plurality of mapping hash functions to perform hash calculation on the second part of the target hash value of the first word to obtain a mapping hash value of the first word. The data query device can obtain multiple mapping hash values ​​of the first word, and the multiple mapping hash values ​​correspond to the multiple mapping hash functions one by one. Each mapping hash value of the multiple mapping hash values ​​is obtained by performing hash calculation on the second part of the target hash value of the first word using the corresponding mapping hash function. Among them, the target hash value of the first word is the hash value of the first word obtained by performing hash calculation on the first word using the target hash function, that is, the target hash value of the first word calculated in S402. The second part of the target hash value of the first word can be the remaining part except the first part of the target hash value.

[0122] It should be noted that the above-mentioned multiple mapping hash functions are different, and the multiple mapping hash functions may be different from the target hash function described in S402. Each of the above-mentioned m BF index structures is associated with the multiple mapping hash functions. In other words, each of the above-mentioned m BF index structures includes the multiple mapping hash functions, and the multiple mapping hash functions are used to map the words in the data block corresponding to each of the m BF index structures to each BF index structure. In addition, the above-mentioned embodiment of the data query device obtaining the mapping hash value of the first word is only used as an example of the present application. The data query device can also obtain the mapping hash value of the first word in other ways. The embodiment of the present application does not limit the implementation method of the data query device obtaining the mapping hash value of the first word.

[0123] S404. Query the at least one first data block to determine a query result of the first data query request.

[0124] In an optional embodiment, the data query device scans each first data block in the at least one first data block, and the data query device determines the query result of the first data query request based on the scanning result of the at least one first data block. The query result of the first data query request is data including the first term. By way of example, the query result of the first data query request includes at least one of the following: a data block including the first term, the number of data blocks including the first term, an identifier of a data block including the first term, a data row including the first term, the number of data rows including the first term, a row number of a data row including the first term, etc.

[0125] In an optional embodiment, the database index structure further includes m×w unit index structures, and each of the m BF index structures corresponds to w unit index structures in the m×w unit index structures. Each unit index structure corresponding to each of the m BF index structures corresponds to at least one index unit in each BF index structure, and different unit index structures correspond to different index units. Each unit index structure corresponding to each BF index structure is used to record the secondary index information of the index unit corresponding to each unit index structure. The secondary index information recorded by each unit index structure includes one or more of the following information: the row number of the hit row of the index unit corresponding to each unit index structure, the number of hit rows of the index unit corresponding to each unit index structure, and the row number identification information. Among them, the hit row of the index unit corresponding to each unit index structure is the data row where the target word is located in the data block corresponding to each BF index structure, and the index segment corresponding to the target word hits the index unit corresponding to each unit index structure. The row number identification information indicates whether the number of hit rows of the index unit corresponding to each unit index structure is greater than the specified number. For a detailed explanation of the unit index structure and the secondary index information it records, please refer to Figure 3 The illustrated embodiment and its description are not repeated here. This embodiment mainly introduces the implementation process of the data query device querying the first data block when the database index structure also includes a unit index structure.

[0126] In one embodiment, the secondary index information recorded in each unit index structure includes the row number of the hit row of the index unit corresponding to each unit index structure. The data query device scans the data row indicated by the row number of the first unit index structure record corresponding to each first data block in the at least one first data block to determine the query result of the first data query request. That is, for each first data block in the at least one first data block, the data query device only scans the data row indicated by the row number of the first unit index structure record corresponding to the first data block, without scanning the entire content of the first data block. In this way, the process of the data query device scanning the first data block can be simplified, the efficiency of the data query device in querying the first data block can be improved, and the query performance can be guaranteed.

[0127] In another embodiment, the secondary index information recorded in each unit index structure includes row number identification information and the row number of the hit row of the index unit corresponding to each unit index structure. For each first data block in the at least one first data block mentioned above: when the row number identification information recorded in the first unit index structure corresponding to the first data block indicates that the number of hit rows of the index unit corresponding to the first unit index structure is less than or equal to the specified number, the data query device scans the data row indicated by the row number recorded in the first unit index structure corresponding to the first data block to determine the query result of the first data query request; when the row number identification information recorded in the first unit index structure corresponding to the first data block indicates that the number of hit rows of the index unit corresponding to the first unit index structure is greater than the specified number, the data query device scans each data row in the first data block to determine the query result of the first data query request. Among them, when the row number identification information of the first unit index structure record corresponding to the first data block indicates that the number of hit rows of the index unit corresponding to the first unit index structure is less than or equal to the specified number, it means that in the process of mapping the words in the first data block to the BF index structure of the first data block, the index unit corresponding to the first unit index structure is hit less times, and the index unit corresponding to the first unit index structure is a low-frequency hit index unit, which further indicates that the possibility of the existence of the first word in the first data block is low. Therefore, the data query device only scans the data row indicated by the row number of the first unit index structure record corresponding to the first data block, thereby improving the efficiency of querying the first data block. When the row number identification information recorded in the first unit index structure corresponding to the first data block indicates that the number of hit rows of the index unit corresponding to the first unit index structure is greater than the specified number, it means that in the process of mapping the words in the first data block to the BF index structure of the first data block, the index unit corresponding to the first unit index structure is hit more times, and the index unit corresponding to the first unit index structure is a high-frequency hit index unit, which further indicates that the possibility of the first word existing in the first data block is high. Therefore, the data query device scans the entire content of the first data block, thereby ensuring the accuracy of the query result of the query on the first data block.

[0128] In the embodiment of the present application, for each first data block in the at least one first data block, the first unit index structure corresponding to the first data block is a unit index structure corresponding to the first index segment in the BF index structure of the first data block. As mentioned above, each BF index structure corresponds to w unit index structures, and each unit index structure corresponding to each BF index structure corresponds to at least one index unit in each BF index structure. Moreover, it can be understood from the above description that the first index segment in each BF index structure corresponds to k index units in the BF index structure. For example, for Figure 6 In the case shown in FIG. 1 , the first index segment in each BF index structure corresponds to an index unit in the BF index structure. Figure 8 In the case shown, the first index segment in each BF index structure corresponds to two index units in the BF index structure. In the embodiment of the present application, the first unit index structure corresponding to each first data block refers to the unit index structure corresponding to the index unit corresponding to the first index segment in the BF index structure among the w unit index structures corresponding to the BF index structure of the first data block. For example, refer to Figure 6 , Figure 8 and Fig. 9 , BF index structures 1-m correspond to data blocks 1-m one by one, and first index segments 11-m1 correspond to each other in BF index structures 1-m one by one. It is assumed that the at least one first data block is data block 1, data block 2, and data block m. Figure 6 and Fig. 9 It can be seen that for Figure 6 The situation shown is: the first index segment 11 corresponds to index unit 11, and index unit 11 corresponds to unit index structure 11, so the first unit index structure corresponding to data block 1 is unit index structure 11 (the unit index structure corresponding to index unit 11 corresponding to the first index segment 11 in the BF index structure 1 of data block 1); the first index segment 21 corresponds to index unit 21, and index unit 21 corresponds to unit index structure 21, so the first unit index structure corresponding to data block 2 is unit index structure 21 (the unit index structure corresponding to index unit 21 corresponding to the first index segment 21 in the BF index structure 2 of data block 2); the first index segment m1 corresponds to index unit m1, and index unit m1 corresponds to unit index structure m1, so the first unit index structure corresponding to data block m is unit index structure m1 (the unit index structure corresponding to index unit m1 corresponding to the first index segment m1 in the BF index structure m of data block m). Figure 8 and Fig. 9 It can be seen that for Figure 8The situation shown is: the first index segment 11 corresponds to index units 11-12, and index units 11-12 correspond to unit index structure 11, so the first unit index structure corresponding to data block 1 is unit index structure 11 (the unit index structure corresponding to index units 11-12 corresponding to the first index segment 11 in the BF index structure 1 of data block 1); the first index segment 21 corresponds to index units 21-22, and index units 21-22 correspond to unit index structure 21, so the first unit index structure corresponding to data block 2 is unit index structure 21 (the unit index structure corresponding to index units 21-22 corresponding to the first index segment 21 in the BF index structure 2 of data block 2); the first index segment m1 corresponds to index units m1-m2, and index units m1-m2 correspond to unit index structure m1, so the first unit index structure corresponding to data block m is unit index structure m1 (the unit index structure corresponding to index units m1-m2 corresponding to the first index segment m1 in the BF index structure m of data block m).

[0129] In an optional embodiment, after the data query device determines the query result of the first data query request, the data query device sends the query result of the first data query request to the client. For example, the data query device generates a first data query response, the first data query response includes the query result of the first data query request, and the data query device sends the first data query response to the client.

[0130] In summary, the data query method provided by the embodiment of the present application, the database index structure includes m BF index structures and n index shards based on column storage, each BF index structure includes n index units, and each index shard includes m index units located in the same column of the m BF index structures. Based on this database index structure, after the data query device obtains a first data query request including a first term, the data query device determines the first index area corresponding to the first term in the database index structure, and the data query device only needs to read the k index shards corresponding to the first index area to determine at least one first data block, and then query the at least one first data block to determine the query result of the first data query request. The data query device can implement data query without reading the entire content of the database index structure, so the query I / O cost is low, the memory overhead is small, and the query performance is good. For example, m=1024, the database index structure includes 1024 BF index structures, and the size of each BF index structure is 32KB. For Figure 2The database index structure shown in the figure has a total content of 1024×32KB=32MB. The 1024 BF index structures correspond to 1024 data blocks, and the size of each data block is 1MB. The total size of the 1024 data blocks is 1024×1MB=1GB. The length of the index segment is 8 bytes, and the first index area corresponds to k index fragments; when k=1, the data volume of the first index area corresponding to k index fragments is 1024×8byte=8KB; when k=2, the data volume of the first index area corresponding to k index fragments is 1024×16byte=16KB. Therefore, for Figure 2 The database index structure shown in the figure only needs to read 8KB or 16KB of content in the database index structure for each scan of a 1GB data block, without having to read the entire content of the database index structure. This results in lower query I / O costs, less memory overhead, and better query performance.

[0131] Figure 4 The embodiment shown is described by implementing a scan query based on a word as an example. The embodiment of the present application can also implement a multi-word joint and scan query. The following takes the first word and the second word joint and scan query as an example to introduce the data query method provided by the embodiment of the present application.

[0132] Please refer to Fig.10 , which shows a flow chart of another data query method provided by an embodiment of the present application. The data query method is executed by a data query device. For example, the data query device is deployed in Figure 1 The computing device cluster 10 is shown.

[0133] like Fig.10 As shown, the data query method includes the following steps S1001 to S1005.

[0134] S1001. Obtain a second data query request, where the second data query request includes a first word and a second word.

[0135] The implementation method of the data query device acquiring the second data query request can refer to the implementation method of the data query device acquiring the first data query request in S401, which is not repeated here. In an optional embodiment, the second data query request is used to request to query data including the first word and the second word, and the second data query request can be a joint and scan query request of the first word and the second word.

[0136] For example, the second data query request includes a query statement, and the query statement includes a first word and a second word. For example, the second data query request is a scan query request, and the query statement is a scan query statement. The first word is a complete word or a prefix of the complete word. The second word is a complete word or a prefix of the complete word. The first word is different from the second word.

[0137] S1002. Determine a first index area corresponding to the first word and a second index area corresponding to the second word in a database index structure, wherein the database index structure includes m BF index structures, n index fragments based on column storage, and m×w unit index structures, each BF index structure includes n index units, each index fragment includes m index units located in the same column of the m BF index structures, each BF index structure corresponds to w unit index structures in the m×w unit index structures, each unit index structure corresponding to each BF index structure corresponds to at least one index unit in each BF index structure, and index units corresponding to different unit index structures are Differently, each unit index structure corresponding to each BF index structure is used to record the secondary index information of the index unit corresponding to each unit index structure, and the secondary index information recorded by each unit index structure includes the row number of the hit row of the index unit corresponding to each unit index structure, the first index area includes the m first index segments corresponding to the first word in the m BF index structures, and the first index area corresponds to k index slices out of the n index slices, the second index area includes the m second index segments corresponding to the second word in the m BF index structures, and the second index area corresponds to p index slices out of the n index slices, k and p are respectively equal to 1 or 2, and k and p are both less than n.

[0138] Wherein, m, n and w are all positive integers. For example, m = 1024, n = 4096, w = 2048. k and p may be equal or unequal.

[0139] The m BF index structures correspond to m data blocks. Each BF index structure is used to record the existence index information of words in the corresponding data block. The existence index information of any word in any data block is used to indicate the existence of the any word in the any data block. The existence index information of any word in any data block is in the index segment corresponding to the any word in the BF index structure corresponding to the any data block, the length of the index segment is less than the length of the BF index structure, and the length of the index unit is equal to the length of the index segment.

[0140] In an optional embodiment, each of the m BF index structures includes n×s bits, each index unit in each BF index structure includes s bits, and each index segment also includes s bits. The bits in each BF index structure are used to record the existence index information of the words in the data block corresponding to each BF index structure, and s is a positive integer. The secondary index information recorded by each unit index structure may also include one or more of the following information: the number of hit rows and row number identification information of the index unit corresponding to each unit index structure. Among them, the hit row of the index unit corresponding to each unit index structure is the data row where the target word is located in the data block corresponding to each BF index structure, and the index segment corresponding to the target word hits the index unit corresponding to each unit index structure. The row number identification information indicates whether the number of hit rows of the index unit corresponding to each unit index structure is greater than the specified number.

[0141] It should be noted that the database index structure in this embodiment can be Figure 3 The database index structure shown in the figure is detailed in the following table. Figure 3 The implementation process of the data query device determining the first index area corresponding to the first word in the database index structure, and the implementation process of the data query device determining the second index area corresponding to the second word in the database index structure, can refer to the description in S402, and will not be repeated here.

[0142] In an embodiment of the present application, the starting position of the first index area may coincide with the starting boundary of an index slice among the n index slices, and the starting position of the first index area may also be located in an index slice among the n index slices. Depending on the starting position of the first index area, the value of k is different. Similarly, the starting position of the second index area may coincide with the starting boundary of an index slice among the n index slices, and the starting position of the second index area may also be located in an index slice among the n index slices. Depending on the starting position of the second index area, the value of p is different. Moreover, the first index area and the second index area may be the same index area or different index areas. In the case where the first index area and the second index area are different index areas, there may or may not be overlapping areas between the first index area and the second index area, which is not limited in the embodiment of the present application. For the convenience of drawing, the following is an example in which the first index area and the second index area are different index areas and there is no overlapping area between the two.

[0143] In one embodiment, the starting position of the first index region coincides with the starting boundary of one of the n index slices, the starting position of the second index region coincides with the starting boundary of another index slice of the n index slices, k is equal to 1, p is equal to 1, and the first index region and the second index region correspond to one index slice respectively. For example, Fig.11 Shows Figure 3 A schematic diagram of a first index area and a second area in a database index structure shown in FIG. Fig.11 As shown, the starting position of the first index region coincides with the starting boundary of index shard 1 (the left boundary of index shard 1), the ending position of the first index region coincides with the ending boundary of index shard 1 (the right boundary of index shard 1), and the first index region corresponds to one index shard, which is index shard 1. The starting position of the second index region coincides with the starting boundary of index shard 3 (the left boundary of index shard 3), the ending position of the second index region coincides with the ending boundary of index shard 3 (the right boundary of index shard 3), and the second index region corresponds to one index shard, which is index shard 3. The first index region includes m first index segments, which are index units 11 to m1 in the first index region. The second index region includes m second index segments, which are index units 13 to m3 in the second index region. For example, the length of an index segment (e.g., a first index segment and a second index segment) is s bits, the distance between the starting position of the first index area and the ending position of the first index area is s bits, and the distance between the starting position of the second index area and the ending position of the second index area is s bits.

[0144] In another embodiment, the starting position of the first index area is located in an index slice among the n index slices, the starting position of the second index area is located in another index slice among the n index slices, k is equal to 2, p is equal to 2, the first index area and the second index area correspond to 2 index slices respectively, the 2 index slices corresponding to the first index area are adjacent, and the 2 index slices corresponding to the second index area are adjacent.

[0145] For example, Fig.12 Shows Figure 3 A schematic diagram of another first index area and a second area in a database index structure shown in FIG. Fig.12As shown, the starting position of the first index region is located in index shard 1, the ending position of the first index region is located in index shard 2, and the first index region corresponds to two index shards, which are index shard 1 and index shard 2. The starting position of the second index region is located in index shard 3, the ending position of the second index region is located in index shard 4, and the second index region corresponds to two index shards, which are index shard 3 and index shard 4. The first index region includes m first index segments, which are located in the m BF index structures in a one-to-one correspondence, and each of the m first index segments is composed of a portion of the corresponding BF index structure located in the first index region. For example, the first index segment in BF index structure 1 is composed of the portion of index unit 11 located in the first index area and the portion of index unit 12 located in the first index area, the first index segment in BF index structure 2 is composed of the portion of index unit 21 located in the first index area and the portion of index unit 22 located in the first index area, the first index segment in BF index structure 3 is composed of the portion of index unit 31 located in the first index area and the portion of index unit 32 located in the first index area, and so on, the first index segment in BF index structure m is composed of the portion of index unit m1 located in the first index area and the portion of index unit m2 located in the first index area. The second index area includes m second index segments, which are located in the m BF index structures in a one-to-one correspondence, and each of the m second index segments is composed of the portion of the corresponding BF index structure located in the second index area. For example, the second index segment in BF index structure 1 is composed of the part of index unit 13 located in the second index area and the part of index unit 14 located in the second index area, the second index segment in BF index structure 2 is composed of the part of index unit 23 located in the second index area and the part of index unit 24 located in the second index area, the second index segment in BF index structure 3 is composed of the part of index unit 33 located in the second index area and the part of index unit 34 located in the second index area, and so on, the second index segment in BF index structure m is composed of the part of index unit m3 located in the second index area and the part of index unit m4 located in the second index area. For example, the length of the index segment is s bits, the distance between the starting position of the first index area and the ending position of the first index area is s bits, and the distance between the starting position of the second index area and the ending position of the second index area is s bits.

[0146] It should be noted that the above two embodiments and Fig.11 and Fig.12Taking k and p as an example, in actual situations, k and p may not be equal. For example, the starting position of the first index region coincides with the starting boundary of one of the n index slices, and the starting position of the second index region is located in one of the n index slices. In this case, k is equal to 1, p is equal to 2, and the first index region can be as follows Fig.11 As shown, the second index area can be Fig.12 As another example, the starting position of the first index region is located in one of the n index slices, and the starting position of the second index region coincides with the starting boundary of one of the n index slices. In this case, k is equal to 2, p is equal to 1, and the first index region can be as follows: Fig.12 As shown, the second index area can be Fig.11 shown.

[0147] In an optional embodiment, each of the m BF index structures includes a low-frequency index space and a high-frequency index space, and the database index structure includes a low-frequency index area and a high-frequency index area, the low-frequency index area includes the low-frequency index space of the m BF index structures, and the high-frequency index area includes the high-frequency index space of the m BF index structures. In the case where the first word is a high-frequency word, the data query device determines a first index area corresponding to the first word in the high-frequency index area of ​​the database index structure, the first index area is located in the high-frequency index area, and the m first index segments included in the first index area are located in the high-frequency index space of the m BF index structures in a one-to-one correspondence. In the case where the first word is a low-frequency word, the data query device determines a first index area corresponding to the first word in the low-frequency index area of ​​the database index structure, the first index area is located in the low-frequency index area, and the m first index segments included in the first index area are located in the low-frequency index space of the m BF index structures in a one-to-one correspondence. In the case where the second word is a high-frequency word, the data query device determines a second index area corresponding to the second word in the high-frequency index area of ​​the database index structure, the second index area is located in the high-frequency index area, and the m second index segments included in the second index area are located in the high-frequency index space of the m BF index structures in a one-to-one correspondence. In the case where the second word is a low-frequency word, the data query device determines a second index area corresponding to the second word in the low-frequency index area of ​​the database index structure, the second index area is located in the low-frequency index area, and the m second index segments included in the second index area are located in the low-frequency index space of the m BF index structures in a one-to-one correspondence. For example, Figure 11 to Figure 12 The first index area and / or the second index area shown may be located in the high-frequency index area or in the low-frequency index area.

[0148] S1003. Read the k index fragments corresponding to the first index area and the p index fragments corresponding to the second index area to determine at least one second data block, wherein the first index segment and the second index segment in the BF index structure corresponding to the second data block respectively include the existence index information of the first word and the existence index information of the second word.

[0149] The m data blocks mentioned above include the at least one first data block.

[0150] The first index segment and the second index segment in the BF index structure corresponding to the second data block respectively include the existence index information of the first word and the existence index information of the second word, that is, the first index segment in the BF index structure corresponding to the second data block includes the existence index information of the first word, and the second index segment in the BF index structure corresponding to the second data block includes the existence index information of the second word.

[0151] In an optional embodiment, the data query device reads (or is referred to as scooping) the k index fragments corresponding to the first index area and the p index fragments corresponding to the second index area into the memory. Afterwards, the data query device obtains multiple mapping hash values ​​of the first term and multiple mapping hash values ​​of the second term. The data query device determines multiple mapping positions of the first term in the first index area according to the multiple mapping hash values ​​of the first term. The data query device determines multiple mapping positions of the second term in the second index area according to the multiple mapping hash values ​​of the second term. The data query device determines at least one second data block in the m data blocks according to the values ​​of the multiple mapping positions of the first term and the values ​​of the multiple mapping positions of the second term. For ease of description, the multiple mapping positions of the first term are referred to as multiple first mapping positions, and the multiple mapping positions of the second term are referred to as multiple second mapping positions. Among them, the multiple first mapping positions correspond one-to-one to the multiple mapping hash values ​​of the first term, and the multiple second mapping positions correspond one-to-one to the multiple mapping hash values ​​of the second term. For example, for each of the multiple mapping hash values ​​of the first term, the data query device determines the position indicated by the mapping hash value in the first index area as a first mapping position, and the data query device can obtain the multiple first mapping positions. For each of the multiple mapping hash values ​​of the second term, the data query device determines the position indicated by the mapping hash value in the second index area as the one second mapping position, and the data query device can obtain multiple second mapping positions. Wherein, each of the multiple first mapping positions and the multiple second mapping positions is a bit. Each of the multiple first mapping positions is located in the m first index segments included in the first index area, and each of the m first index segments includes the multiple first mapping positions. Each of the multiple second mapping positions is located in the m second index segments included in the second index area, and each of the m second index segments includes the multiple second mapping positions. The m first index segments are located in the m BF index structures in a one-to-one correspondence, the m second index segments are located in the m BF index structures in a one-to-one correspondence, and the m BF index structures are located in a one-to-one correspondence with the m data blocks.The data query device determines at least one second data block in the m data blocks according to the values ​​of the multiple mapping positions (i.e., multiple first mapping positions) of the first word and the values ​​of the multiple mapping positions (i.e., multiple second mapping positions) of the second word, including: for the first index segment and the second index segment in each of the m BF index structures, the data query device reads the values ​​of the multiple first mapping positions in the first index segment and the values ​​of the multiple second mapping positions in the second index segment; the data query device determines, according to the values ​​of the multiple first mapping positions in the first index segment, whether the data block corresponding to the BF index structure is The data query device determines whether the data block corresponding to the BF index structure may include the second word according to the values ​​of the multiple second mapping positions in the second index segment; when the data query device determines that the data block corresponding to the BF index structure may include the first word and the second word, the data query device determines the data block corresponding to the BF index structure as the second data block; when the data query device determines that the data block corresponding to the BF index structure may not include the first word and / or the second word, the data query device does not determine the data block corresponding to the BF index structure as the second data block. In this way, the data query device can determine at least one second data block among the m data blocks. In one embodiment, the data query device determines whether the data block corresponding to the BF index structure may include the first word based on the values ​​of the multiple first mapping positions in the first index segment, including: the data query device determines whether the values ​​of the multiple first mapping positions in the first index segment are all 1; when the values ​​of the multiple first mapping positions in the first index segment are all 1, the data query device determines that the data block corresponding to the BF index structure may include the first word; when the value of at least one of the multiple first mapping positions in the first index segment is not 1, the data query device determines that the data block corresponding to the BF index structure cannot include the first word. The data query device determines whether the data block corresponding to the BF index structure may include the second word based on the values ​​of the multiple second mapping positions in the second index segment, including: the data query device determines whether the values ​​of the multiple second mapping positions in the second index segment are all 1; when the values ​​of the multiple second mapping positions in the second index segment are all 1, the data query device determines that the data block corresponding to the BF index structure may include the second word; when the value of at least one of the multiple second mapping positions in the second index segment is not 1, the data query device determines that the data block corresponding to the BF index structure cannot include the second word.

[0152] It should be noted that the implementation process of the data query device obtaining the mapping hash value of the first word and the implementation process of the data query device obtaining the mapping hash value of the second word can both refer to the implementation process of the data query device obtaining the mapping hash value of the first word in S403, which will not be repeated here. In this embodiment, the implementation method of the data query device obtaining the mapping hash value of the first word and the implementation method of the data query device obtaining the mapping hash value of the second word may be the same or different. The mapping hash function used by the data query device to obtain the mapping hash value of the first word may be the same as the mapping hash function used by the data query device to obtain the mapping hash value of the second word.

[0153] S1004. Determine a target data block from the at least one second data block according to the row number of the first unit index structure record corresponding to the at least one second data block and the row number of the second unit index structure record corresponding to the at least one second data block.

[0154] Among them, the first unit index structure corresponding to the second data block is a unit index structure corresponding to the first index segment in the BF index structure of the second data block. The second unit index structure corresponding to the second data block is a unit index structure corresponding to the second index segment in the BF index structure of the second data block. The target data block includes a target data row, which is a data row indicated by the same row number in the row number recorded in the first unit index structure corresponding to the target data block and the row number recorded in the second unit index structure corresponding to the target data block. Among them, the explanation of the first unit index structure corresponding to the second data block, the explanation of the second unit index structure corresponding to the second data block, the explanation of the first unit index structure corresponding to the target data block, and the explanation of the second unit index structure corresponding to the target data block can all refer to the relevant description of the first unit index structure corresponding to the first data block in S404, which will not be repeated here.

[0155] In an optional embodiment, for each second data block in the at least one second data block, the data query device obtains the row number of the first unit index structure record corresponding to the second data block and the row number of the second unit index structure record corresponding to the second data block. The data query device determines whether the row number of the first unit index structure record corresponding to the second data block and the row number of the second unit index structure record corresponding to the second data block have the same row number. When the data query device determines that the row number of the first unit index structure record corresponding to the second data block and the row number of the second unit index structure record corresponding to the second data block have the same row number, the data query device determines the second data block as a target data block, and the data query device determines the same row number of the row number of the first unit index structure record corresponding to the second data block and the row number of the second unit index structure record corresponding to the second data block as the row number of the target data row in the target data block. When the data query device determines that the row number of the first unit index structure record corresponding to the second data block and the row number of the second unit index structure record corresponding to the second data block do not have the same row number, the data query device does not determine the second data block as a target data block. In this way, the data query device can determine at least one target data block from the at least one second data block, and the data query device can determine the row number of the target data row in each target data block in the at least one target data block.

[0156] S1005. Scan the target data row in the target data block to determine the query result of the second data query request.

[0157] Optionally, the number of target data blocks is at least one, and the data query device scans the target data row in at least one target data block, and determines the query result of the second data query request based on the scanning result of the at least one target data block.

[0158] S1005 is explained by taking the data query device scanning the target data rows in the target data block as an example. In an optional embodiment, for each target data block in the at least one target data block, when the row number identification information recorded in the first unit index structure corresponding to the target data block is used to indicate that the number of hit rows of the index unit corresponding to the first unit index structure is less than or equal to the specified number, and when the row number identification information recorded in the second unit index structure corresponding to the target data block is used to indicate that the number of hit rows of the index unit corresponding to the second unit index structure is less than or equal to the specified number, the data query device scans the target data rows in the target data block to determine the query result of the second data query request. When the row number identification information recorded in the first unit index structure corresponding to the target data block is used to indicate that the number of hit rows of the index unit corresponding to the first unit index structure is greater than the specified number, and / or, when the row number identification information recorded in the second unit index structure corresponding to the target data block is used to indicate that the number of hit rows of the index unit corresponding to the second unit index structure is greater than the specified number, the data query device scans each data row in the target data block to determine the query result of the second data query request.

[0159] The query result of the second data query request is data including the first word and the second word. For example, the query result of the second data query request includes at least one of the following: a data block including the first word and the second word, the number of data blocks including the first word and the second word, an identifier of a data block including the first word and the second word, a data row including the first word and the second word, the number of data rows including the first word and the second word, and the row number of data rows including the first word and the second word.

[0160] In an optional embodiment, after the data query device determines the query result of the second data query request, the data query device sends the query result of the second data query request to the client. For example, the data query device generates a second data query response, the second data query response includes the query result of the second data query request, and the data query device sends the second data query response to the client.

[0161] In summary, in the data query method provided by the embodiment of the present application, after the data query device obtains the second data query request including the first word and the second word, the data query device determines the first index area corresponding to the first word and the second index area corresponding to the second word in the database index structure. The data query device only needs to read the k index fragments corresponding to the first index area and the p index fragments corresponding to the second index area to determine at least one second data block, and then query the at least one second data block to determine the query result of the second data query request. The data query device can realize the joint query of the first word and the second word without reading the full content of the database index structure, so the query I / O cost is low, the memory overhead is small, and the query performance is good.

[0162] Furthermore, when the data query device queries the at least one second data block, the data query device determines the target data block from the at least one second data block according to the row number of the first unit index structure record corresponding to the at least one second data block and the row number of the second unit index structure record corresponding to the at least one second data block. The data query device determines the query result of the second data query request by scanning the target data block. Therefore, the data query method provided in the embodiment of the present application can realize the joint AND scanning query of the first word and the second word, and in the joint AND scanning query scenario of the first word and the second word, the query performance can be improved.

[0163] In addition, in the scenario of multi-word joint and scanning query, if different words in the multi-words in the same data block are located in different rows of the data block, for example, the first word is located in the first row of the data block, and the second word is located in the second row of the data block, the data block can still be hit (or determined) based on the BF index structure. However, the different words in the same data block are located in different rows of the data block, which will lead to misjudgment (the data block may not include the different words, but the BF index structure will determine that the data block includes the different words), affecting the query performance. The embodiment of the present application sets a unit index structure on the basis of the BF index structure, so that after determining at least one second data block according to the BF index structure, the at least one second data block is screened according to the row number recorded in the unit index structure corresponding to the at least one second data block to determine the target data block, and then the query result is confirmed by scanning the target data row in the target data block. The embodiment of the present application avoids scanning the data block where the misjudgment occurs through such screening, thereby ensuring the query performance.

[0164] Fig.10The embodiment shown takes the first word and the second word combined and scan query as an example. The data query method provided in the embodiment of the present application can also implement three or more words combined and scan query. The implementation method of three or more words combined and scan query can refer to Fig.10 In the illustrated embodiment, in a query scenario where three or more words are combined and scanned, query performance can also be improved.

[0165] also, Fig.10 The illustrated embodiment takes the case where the database index structure includes a unit index structure as an example. When the database index structure does not include a unit index structure, the embodiment of the present application can also implement a joint and scan query of the first term and the second term. For example, after the data query device determines at least one second data block (that is, after executing S1003), the data query device scans the at least one second data block to determine the query result of the second data query request, and the data query device does not need to execute the above S1004 and S1005. That is, the above S1004 and S1005 can be replaced by: scanning the at least one second data block to determine the query result of the second data query request.

[0166] Fig.10 The embodiment shown takes the first word and the second word combined and scan query as an example to illustrate, and the embodiment of the present application can also implement multi-word combined or scan query. The following takes the first word and the second word combined or scan query as an example to introduce the data query method provided by the embodiment of the present application.

[0167] Please refer to Fig.13 , which shows a flow chart of another data query method provided by an embodiment of the present application. The data query method is executed by a data query device. For example, the data query device is deployed in Figure 1 The computing device cluster 10 is shown.

[0168] like Fig.13 As shown, the data query method includes the following steps S1301 to S1304.

[0169] S1301. Obtain a second data query request, where the second data query request includes a first word and a second word.

[0170] The implementation method of the data query device obtaining the second data query request can refer to the implementation method of the data query device obtaining the first data query request in S401, which is not repeated here. In an optional embodiment, the second data query request is used to request to query data including the first word and / or the second word, and the second data query request is a joint or scan query request of the first word and the second word.

[0171] For example, the second data query request includes a query statement, and the query statement includes a first word and a second word. For example, the second data query request is a scan query request, and the query statement is a scan query statement. The first word is a complete word or a prefix of the complete word. The second word is a complete word or a prefix of the complete word. The first word is different from the second word.

[0172] S1302. Determine a first index area corresponding to the first word and a second index area corresponding to the second word in a database index structure, wherein the database index structure includes m BF index structures and n index shards based on column storage, each BF index structure includes n index units, each index shard includes m index units located in the same column of the m BF index structures, the first index area includes m first index segments corresponding to the first word in the m BF index structures, and the first index area corresponds to k index shards of the n index shards, the second index area includes m second index segments corresponding to the second word in the m BF index structures, and the second index area corresponds to p index shards of the n index shards, k and p are respectively equal to 1 or 2, and k and p are both less than n.

[0173] The implementation process of S1302 can refer to the implementation process of S1002, which will not be described in detail here.

[0174] S1303. Read the k index slices corresponding to the first index area and the p index slices corresponding to the second index area to determine at least one second data block, wherein the first index segment in the BF index structure corresponding to the second data block includes the existence index information of the first word and / or the second index segment in the BF index structure corresponding to the second data block includes the existence index information of the second word.

[0175] The m data blocks mentioned above include the at least two first data blocks.

[0176] In an optional embodiment, the data query device reads the k index fragments corresponding to the first index area and the p index fragments corresponding to the second index area into the memory. Afterwards, the data query device obtains multiple mapping hash values ​​of the first word and multiple mapping hash values ​​of the second word. The data query device determines multiple mapping positions of the first word in the first index area according to the multiple mapping hash values ​​of the first word. The data query device determines multiple mapping positions of the second word in the second index area according to the multiple mapping hash values ​​of the second word. The data query device determines at least one second data block in the m data blocks according to the values ​​of the multiple mapping positions of the first word and the values ​​of the multiple mapping positions of the second word. For ease of description, the multiple mapping positions of the first word are referred to as multiple first mapping positions, and the multiple mapping positions of the second word are referred to as multiple second mapping positions. Among them, each mapping position of the multiple first mapping positions and the multiple second mapping positions is a bit. Each first mapping position of the multiple first mapping positions is located in the m first index segments included in the first index area, and each first index segment of the m first index segments includes the multiple first mapping positions. Each of the plurality of second mapping positions is located in the m second index segments included in the second index area, and each of the m second index segments includes the plurality of second mapping positions. The m first index segments are located in the m BF index structures in a one-to-one correspondence, and the m second index segments are located in the m BF index structures in a one-to-one correspondence, and the m BF index structures correspond to the m data blocks in a one-to-one correspondence. The data query device determines at least one second data block in the m data blocks according to the values ​​of the multiple first mapping positions of the first word and the values ​​of the multiple second mapping positions of the second word, including: for the first index segment and the second index segment in each BF index structure in the m BF index structures, the data query device reads the values ​​of the multiple first mapping positions in the first index segment and the values ​​of the multiple second mapping positions in the second index segment; the data query device determines whether the data block corresponding to the BF index structure may include the first word according to the values ​​of the multiple first mapping positions in the first index segment; the data query device determines whether the data block corresponding to the BF index structure may include the second word according to the values ​​of the multiple second mapping positions in the second index segment; when the data query device determines that the data block corresponding to the BF index structure may include the first word and / or the second word, the data query device determines the data block corresponding to the BF index structure as the second data block; when the data query device determines that the data block corresponding to the BF index structure may not include the first word or the second word, the data query device does not determine the data block corresponding to the BF index structure as the second data block.That is, when the data query device determines that the data block corresponding to the BF index structure may include the first word, or may include the second word, or may include the first word and the second word, the data query device determines the data block corresponding to the BF index structure as the second data block. When the data query device determines that the data block corresponding to the BF index structure may not include the first word or the second word, the data query device does not determine the data block corresponding to the BF index structure as the second data block. In this way, the data query device can determine at least one second data block among the m data blocks. Among them, the implementation method of the data query device determining whether the data block corresponding to the BF index structure may include the first word according to the values ​​of the multiple first mapping positions in the first index segment, and the implementation method of the data query device determining whether the data block corresponding to the BF index structure may include the second word according to the values ​​of the multiple second mapping positions in the second index segment, can refer to the relevant description in S1003, which will not be repeated here.

[0177] S1304. Query the at least one second data block to determine a query result of the second data query request.

[0178] In an optional embodiment, the data query device scans each second data block in the at least one second data block, and the data query device determines the query result of the second data query request based on the scanning result of the at least one second data block. Wherein, the query result of the second data query request is data including the first word and / or the second word. By way of example, the query result of the second data query request includes at least one of the following: a data block including the first word and / or the second word, the number of data blocks including the first word and / or the second word, an identifier of a data block including the first word and / or the second word, a data row including the first word and / or the second word, the number of data rows including the first word and / or the second word, the row number of the data row including the first word and / or the second word, etc.

[0179] In an optional embodiment, the database index structure also includes m×w unit index structures, and each of the above-mentioned m BF index structures corresponds to w unit index structures in the m×w unit index structures. Each unit index structure corresponding to each BF index structure in the m BF index structures corresponds to at least one index unit in each BF index structure. Each unit index structure corresponding to each BF index structure is used to record the secondary index information of the index unit corresponding to each unit index structure. The secondary index information recorded by each unit index structure includes one or more of the following information: the row number of the hit row of the index unit corresponding to each unit index structure, the number of hit rows of the index unit corresponding to each unit index structure, and the row number identification information. For a detailed explanation of the unit index structure and the secondary index information recorded therein, please refer to Figure 3 The illustrated embodiment and its description are not repeated here. This embodiment mainly introduces the implementation process of the data query device querying the second data block when the database index structure also includes a unit index structure.

[0180] In one embodiment, the secondary index information recorded in each unit index structure includes the row number of the hit row of the index unit corresponding to each unit index structure. The data query device scans the data row indicated by the row number of the first unit index structure record corresponding to each second data block in the at least one second data block and the data row indicated by the row number of the second unit index structure record corresponding to each second data block to determine the query result of the second data query request. That is, for each second data block in the at least one second data block, the data query device only scans the data row indicated by the row number of the first unit index structure record corresponding to the second data block and the data row indicated by the row number of the second unit index structure record corresponding to the second data block, without scanning the entire content of the second data block. In this way, the process of the data query device scanning the second data block can be simplified, the query efficiency of querying the second data block can be improved, and the query performance can be guaranteed.

[0181] In another embodiment, the secondary index information recorded in each unit index structure includes row number identification information and the row number of the hit row of the index unit corresponding to each unit index structure. For each second data block in the at least one second data block mentioned above: when the row number identification information of the first unit index structure record corresponding to the second data block indicates that the number of hit rows of the index unit corresponding to the first unit index structure is less than or equal to the specified number, and the row number identification information of the second unit index structure record corresponding to the second data block indicates that the number of hit rows of the index unit corresponding to the second unit index structure is less than or equal to the specified number, the data query device scans the data rows indicated by the row number of the first unit index structure record corresponding to the second data block and the data rows indicated by the row number of the second unit index structure record corresponding to the second data block to determine the query result of the second data query request; when the row number identification information of the first unit index structure record corresponding to the second data block indicates that the number of hit rows of the index unit corresponding to the first unit index structure is greater than the specified number, and / or the row number identification information of the second unit index structure record corresponding to the second data block indicates that the number of hit rows of the index unit corresponding to the second unit index structure is greater than the specified number, the data query device scans each data row in the second data block to determine the query result of the second data query request. Therefore, on the one hand, the efficiency of querying the second data block can be improved, and on the other hand, the accuracy of the query result can be improved.

[0182] It should be noted that, in the present embodiment, the first unit index structure corresponding to the second data block is a unit index structure corresponding to the first index segment in the BF index structure of the second data block, and the first index segment in the BF index structure of the second data block is an index segment included in the first index region in the BF index structure. The second unit index structure corresponding to the second data block is a unit index structure corresponding to the second index segment in the BF index structure of the second data block, and the second index segment in the BF index structure of the second data block is an index segment included in the BF index structure of the second index region. The explanation of the first unit index structure corresponding to the second data block and the explanation of the second unit index structure corresponding to the second data block can refer to the relevant description of the first unit index structure corresponding to the first data block in S404, which will not be repeated here.

[0183] In summary, in the data query method provided by the embodiment of the present application, after the data query device obtains the second data query request including the first word and the second word, the data query device determines the first index area corresponding to the first word and the second index area corresponding to the second word in the database index structure. The data query device only needs to read the k index fragments corresponding to the first index area and the p index fragments corresponding to the second index area to determine at least one second data block, and then query the at least one second data block to determine the query result of the second data query request. The data query device can realize the joint or scan query of the first word and the second word without reading the full content of the database index structure, so the query I / O cost is low, the memory overhead is small, and the query performance is good.

[0184] Fig.13 The embodiment shown takes the first word and the second word as an example for joint or scan query. The data query method provided in the embodiment of the present application can also implement three or more words joint or scan query. The implementation method of three or more words joint or scan query can refer to Fig.13 In the illustrated embodiment, three or more words are combined in an OR scan query scenario, which can also improve the query performance.

[0185] Figure 4 , Fig.10 and Fig.13 The embodiment shown takes scan query as an example for explanation, and the embodiment of the present application can also implement aggregate query. The following takes aggregate query as an example to introduce the data query method provided by the embodiment of the present application.

[0186] Please refer to Fig.14 , which shows a flow chart of another data query method provided by an embodiment of the present application. The data query method is executed by a data query device. For example, the data query device is deployed in Figure 1 The computing device cluster 10 is shown.

[0187] like Fig.14 As shown, the data query method includes the following steps S1401 to S1404.

[0188] S1401. Obtain a first aggregate query request, where the first aggregate query request is used to query the quantity of data including a first term.

[0189] The implementation method of the data query device obtaining the first aggregate query request can refer to the implementation method of the data query device obtaining the first data query request in S401, which is not repeated here. In an optional embodiment, the first aggregate query request includes an aggregate (count) query statement, and the aggregate query statement includes a first word. The first word is a complete word or a prefix of the complete word.

[0190] The number of data including the first word may be the number of data rows including the first word.

[0191] S1402. Determine a first index region corresponding to a first term in a database index structure, wherein the database index structure includes m BF index structures, n index fragments based on column storage, and m×w unit index structures, each BF index structure includes n index units, each index fragment includes m index units located in the same column of the m BF index structures, each BF index structure corresponds to w unit index structures in the m×w unit index structures, each unit index structure corresponding to each BF index structure corresponds to at least one index unit in each BF index structure, different unit index structures correspond to different index units, each unit index structure corresponding to each BF index structure is used to record secondary index information of the index unit corresponding to each unit index structure, the secondary index information recorded by each unit index structure includes the number of hit rows of the index unit corresponding to each unit index structure, the first index region includes m first index segments corresponding to the first term in the m BF index structures, and the first index region corresponds to k index fragments in the n index fragments, k is equal to 1 or 2, and k is less than n.

[0192] Wherein, m, n and w are all positive integers. For example, m=1024, n=4096, w=2048.

[0193] The implementation process of S1402 may refer to the implementation process of S402, which will not be described in detail here.

[0194] S1403. Read the k index fragments corresponding to the first index area to determine at least one first data block, wherein the first index segment in the BF index structure corresponding to the first data block includes existence index information of the first word.

[0195] The m BF index structures correspond to m data blocks, and the m data blocks include the at least one first data block.

[0196] The implementation process of S1403 may refer to the implementation process of S403, which will not be described in detail here.

[0197] S1404. Determine the query result of the first aggregate query request according to the number of hit rows recorded in the first unit index structure corresponding to the at least one first data block.

[0198] The first unit index structure corresponding to each first data block is a unit index structure corresponding to the first index segment in the BF index structure of the first data block. Please refer to the relevant description of S404 for the first unit index structure corresponding to the first data block, which will not be repeated here.

[0199] In an optional embodiment, for each first data block in the at least one first data block, the data query device obtains the number of hit rows recorded in the first unit index structure corresponding to the first data block, and the data query device determines the query result of the first aggregate query request based on the number of hit rows recorded in the first unit index structure corresponding to the at least one first data block.

[0200] In one embodiment, the at least one first data block is a first data block, and the data query device determines the number of hit rows recorded by the first unit index structure corresponding to the first data block as the query result of the first aggregate query request. The number of hit rows recorded by the first unit index structure corresponding to the first data block can be approximately considered as the number of data rows including the first term.

[0201] In another embodiment, the at least one first data block is a plurality of first data blocks, and the data query device determines the sum of the number of hit rows recorded by the first unit index structure corresponding to the plurality of first data blocks as the query result of the first aggregate query request. The sum of the number of hit rows recorded by the first unit index structure corresponding to the plurality of first data blocks can be approximately considered as the number of data rows including the first term.

[0202] Based on the above, it can be seen that the embodiments of the present application can implement an approximate aggregation query on the number of data rows that include the first term.

[0203] In an optional embodiment, after the data query device determines the query result of the first aggregate query request, the data query device sends the query result of the first aggregate query request to the client. For example, the data query device generates a first aggregate query response, the first aggregate query response includes the query result of the first aggregate query request, and the data query device sends the first aggregate query response to the client.

[0204] In summary, in the data query method provided by the embodiment of the present application, after the data query device obtains the first aggregate query request, the data query device determines the first index area corresponding to the first word in the database index structure, and the data query device only needs to read the k index fragments corresponding to the first index area to determine at least one first data block, and then determine the query result of the first aggregate query request according to the number of hit rows recorded in the first unit index structure corresponding to the at least one first data block. The data query device can realize approximate aggregate query for the first word without reading the full content of the database index structure, and realizes fast aggregate query in massive data, with good query performance.

[0205] Fig.14 The illustrated embodiment takes the database index structure including the unit index structure as an example. When the database index structure does not include the unit index structure, the embodiment of the present application can also implement an aggregate query based on a term. In this case, after the data query device determines at least one first data block (that is, after executing S1403), the data query device can scan the at least one first data block to determine the query result of the second data query request, and the data query device does not need to execute the above S1404. That is, the above S1404 can be replaced by: scanning the at least one first data block to determine the query result of the first aggregate query request.

[0206] Fig.14 The embodiment shown is described by implementing an aggregate query based on one word. The embodiment of the present application can also implement a multi-word joint and aggregate query. The data query method provided by the embodiment of the present application is introduced below by taking the first word and the second word joint and aggregate query as an example.

[0207] Please refer to Fig.15 , which shows a flow chart of another data query method provided by an embodiment of the present application. The data query method is executed by a data query device. For example, the data query device is deployed in Figure 1 The computing device cluster 10 is shown.

[0208] like Fig.15 As shown, the data query method includes the following steps S1501 to S1504.

[0209] S1501. Obtain a second aggregate query request, where the second aggregate query request is used to query the number of data including the first term and the second term.

[0210] The implementation method for the data query device to obtain the second aggregate query request can refer to the implementation method for the data query device to obtain the first data query request in S401. Optionally, the second aggregate query request includes an aggregate (count) query statement, which includes the second word and the second word. The first word is a complete word or a prefix of the complete word, and the second word is a complete word or a prefix of the complete word. The number of data including the first word and the second word can be the number of data rows including the first word and the second word.

[0211] S1502. Determine a first index area corresponding to the first word and a second index area corresponding to the second word in a database index structure, wherein the database index structure includes m BF index structures, n index fragments based on column storage, and m×w unit index structures, each BF index structure includes n index units, each index fragment includes m index units located in the same column of the m BF index structures, each BF index structure corresponds to w unit index structures in the m×w unit index structures, each unit index structure corresponding to each BF index structure corresponds to at least one index unit in each BF index structure, and index units corresponding to different unit index structures Differently, each unit index structure corresponding to each BF index structure is used to record the secondary index information of the index unit corresponding to each unit index structure, and the secondary index information recorded by each unit index structure includes the number of hit rows of the index unit corresponding to each unit index structure, the first index area includes the m first index segments corresponding to the first word in the m BF index structures, and the first index area corresponds to k index slices out of the n index slices, the second index area includes the m second index segments corresponding to the second word in the m BF index structures, and the second index area corresponds to p index slices out of the n index slices, k and p are respectively equal to 1 or 2, and k and p are both less than n.

[0212] Wherein, m, n and w are all positive integers. For example, m = 1024, n = 4096, w = 2048. k and p may be equal or unequal.

[0213] The implementation process of S1502 can refer to the implementation process of S1002, which will not be repeated here.

[0214] S1503. Read the k index fragments corresponding to the first index area and the p index fragments corresponding to the second index area to determine the second data block, wherein the first index segment and the second index segment in the BF index structure corresponding to the second data block respectively include the existence index information of the first word and the existence index information of the second word.

[0215] The number of the second data block is at least one, the m BF index structures correspond to m data blocks, and the m data blocks include the at least one second data block. The implementation process of S1502 can refer to the implementation process of S1003, which will not be described in detail here.

[0216] S1504. Determine the query result of the second aggregate query request based on the smaller number of the number of hit rows recorded in the first unit index structure corresponding to the second data block and the number of hit rows recorded in the second unit index structure corresponding to the second data block.

[0217] Among them, the first unit index structure corresponding to the second data block is a unit index structure corresponding to the first index segment in the BF index structure of the second data block. The second unit index structure corresponding to the second data block is a unit index structure corresponding to the second index segment in the BF index structure of the second data block. The explanation of the first unit index structure corresponding to the second data block and the explanation of the second unit index structure corresponding to the second data block can refer to the relevant description of the first unit index structure corresponding to the first data block in S404, which will not be repeated here.

[0218] In an optional embodiment, for each second data block in the at least one second data block, the data query device obtains the number of hit rows of the first unit index structure record corresponding to the second data block and the number of hit rows of the second unit index structure record corresponding to the second data block; the data query device determines the smaller number of the number of hit rows of the first unit index structure record corresponding to the second data block and the number of hit rows of the second unit index structure record corresponding to the second data block; the data query device determines the query result of the second aggregate query request based on at least one smaller number determined based on the number of hit rows of the first unit index structure record corresponding to the at least one second data block and the number of hit rows of the second unit index structure record corresponding to the at least one second data block.

[0219] In one embodiment, the at least one second data block is a second data block, and the data query device determines the smaller number based on the number of hit rows recorded in the first unit index structure corresponding to the second data block and the number of hit rows recorded in the second unit index structure corresponding to the second data block as the query result of the second aggregate query request. For example, the at least one second data block is data block 1, the number of hit rows recorded in the first unit index structure corresponding to data block 1 is C11, and the number of hit rows recorded in the second unit index structure corresponding to data block 1 is C12. The data query device determines the smaller number between C11 and C12 as the query result of the second aggregate query request. For example, C11 is less than C12, and therefore, the data query device determines C11 as the query result of the second aggregate query request.

[0220] In another embodiment, the at least one second data block is a plurality of second data blocks, and the data query device determines a smaller number based on the number of hit rows recorded in the first unit index structure corresponding to each second data block in the plurality of second data blocks and the number of hit rows recorded in the second unit index structure corresponding to each second data block. The data query device can determine a plurality of smaller numbers corresponding to the plurality of second data blocks one by one, and the data query device determines the sum of the plurality of smaller numbers as the query result of the second aggregate query request. For example, the plurality of second data blocks are data block 1, data block 2, and data block 3. The number of hit rows recorded in the first unit index structure corresponding to data block 1 is C11, and the number of hit rows recorded in the second unit index structure corresponding to data block 1 is C12. The data query device determines the smaller number between C11 and C12. The number of hit rows recorded in the first unit index structure corresponding to data block 2 is C21, and the number of hit rows recorded in the second unit index structure corresponding to data block 2 is C22. The data query device determines the smaller number between C21 and C22. The number of hit rows recorded in the first unit index structure corresponding to the data block 3 is C31, and the number of hit rows recorded in the second unit index structure corresponding to the data block 3 is C32, and the data query device determines the smaller number of C31 and C32. The data query device determines the smaller number of C11 and C12, the smaller number of C21 and C22, and the smaller number of C31 and C32 as the query result of the second aggregate query request. For example, C11 is smaller than C12, C21 is smaller than C22, and C31 is smaller than C32, so the data query device determines the sum of C11, C21, and C31 as the query result of the second aggregate query request.

[0221] In an embodiment of the present application, the number of hit rows recorded in the first unit index structure corresponding to the second data block can be approximately considered as the number of data rows including the first term in the second data block, and the number of hit rows recorded in the second unit index structure corresponding to the second data block can be approximately considered as the number of data rows including the second term in the second data block. An embodiment of the present application determines the query result of the second aggregate query request based on the number of hit rows recorded in the first unit index structure corresponding to the second data block and the number of hit rows recorded in the second unit index structure corresponding to the second data block, and can implement an approximate aggregate query on the number of data rows including the first term and the second term.

[0222] In an optional embodiment, after the data query device determines the query result of the second aggregate query request, the data query device sends the query result of the second aggregate query request to the client. For example, the data query device generates a second aggregate query response, the second aggregate query response includes the query result of the second aggregate query request, and the data query device sends the second aggregate query response to the client.

[0223] In summary, in the data query method provided by the embodiment of the present application, after the data query device obtains the second aggregate query request, the data query device determines the first index area corresponding to the first word and the second index area corresponding to the second word in the database index structure. The aggregate query device only needs to read the k index fragments corresponding to the first index area and the p index fragments corresponding to the second index area to determine at least one second data block, and then determine the query result of the second aggregate query request according to the number of hit rows recorded in the first unit index structure corresponding to the at least one second data block and the number of hit rows recorded in the second unit index structure corresponding to the at least one second data block. The aggregate query device can realize the joint and aggregate query for the first word and the second word without reading the entire content of the database index structure. In the joint and aggregate query scenario of the first word and the second word, fast aggregate query is realized, which can improve query performance.

[0224] Fig.15 The embodiment shown takes the first word and the second word as an example to illustrate the combined and aggregated query. The data query method provided in the embodiment of the present application can also implement the combined and aggregated query of three or more words. The implementation method of the combined and aggregated query of three or more words can refer to Fig.15 In the illustrated embodiment, in a query scenario where three or more words are combined and aggregated, query performance can also be improved.

[0225] also, Fig.15 The illustrated embodiment takes the case where the database index structure includes a unit index structure as an example. When the database index structure does not include a unit index structure, the embodiment of the present application can also implement a joint and aggregate query of the first term and the second term. For example, in this case, after the data query device determines at least one second data block (that is, after executing S1503), the data query device scans the at least one second data block to determine the query result of the second aggregate query request, and the data query device does not need to execute the above S1504. That is, the above S1504 can be replaced by: scanning the at least one second data block to determine the query result of the second aggregate query request.

[0226] Fig.15The embodiment shown takes the first word and the second word combined and aggregated query as an example to illustrate, and the embodiment of the present application can also realize multi-word combined or aggregated query. The following takes the first word and the second word combined or aggregated query as an example to introduce the data query method provided by the embodiment of the present application.

[0227] Please refer to Fig.16 , which shows a flow chart of another data query method provided by an embodiment of the present application. The data query method is executed by a data query device. For example, the data query device is deployed in Figure 1 The computing device cluster 10 is shown.

[0228] like Fig.16 As shown, the data query method includes the following steps S1601 to S1604.

[0229] S1601. Obtain a second aggregate query request, where the second aggregate query request is used to query the number of data including the first term and / or the second term.

[0230] The implementation method for the data query device to obtain the second aggregate query request can refer to the implementation method for the data query device to obtain the first data query request in S401, which will not be repeated here. In an optional embodiment, the second aggregate query request is used to request the number of data including the first term and / or the second term, and the second aggregate query request is a joint or aggregate query request of the first term and the second term. For example, the second aggregate query request includes an aggregate (count) query statement, which includes the first term and the second term. The first term is a complete term or a prefix of the complete term. The second term is a complete term or a prefix of the complete term. The first term is different from the second term.

[0231] S1602. Determine a first index area corresponding to the first word and a second index area corresponding to the second word in a database index structure, wherein the database index structure includes m BF index structures, n index fragments based on column storage, and m×w unit index structures, each BF index structure includes n index units, each index fragment includes m index units located in the same column of the m BF index structures, each BF index structure corresponds to w unit index structures in the m×w unit index structures, each unit index structure corresponding to each BF index structure corresponds to at least one index unit in each BF index structure, and index units corresponding to different unit index structures Differently, each unit index structure corresponding to each BF index structure is used to record the secondary index information of the index unit corresponding to each unit index structure, and the secondary index information recorded by each unit index structure includes the number of hit rows of the index unit corresponding to each unit index structure, the first index area includes the m first index segments corresponding to the first word in the m BF index structures, and the first index area corresponds to k index slices out of the n index slices, the second index area includes the m second index segments corresponding to the second word in the m BF index structures, and the second index area corresponds to p index slices out of the n index slices, k and p are respectively equal to 1 or 2, and k and p are both less than n.

[0232] The implementation process of S1602 can refer to the implementation process of S1002, which will not be repeated here.

[0233] S1603. Read the k index slices corresponding to the first index area and the p index slices corresponding to the second index area to determine at least one second data block, wherein the first index segment in the BF index structure corresponding to the second data block includes the existence index information of the first word and / or the second index segment in the BF index structure corresponding to the second data block includes the existence index information of the second word.

[0234] The implementation process of S1603 can refer to the implementation process of S1303, which will not be described in detail here.

[0235] S1604. Determine the query result of the second aggregate query request based on the smaller number of the number of hit rows recorded in the first unit index structure corresponding to the second data block and the number of hit rows recorded in the second unit index structure corresponding to the second data block.

[0236] The implementation process of S1604 can refer to the implementation process of 1504, which will not be repeated here.

[0237] It should be noted that, in this embodiment, the number of hit rows recorded in the first unit index structure corresponding to the second data block can be approximately considered as the number of data rows including the first term in the second data block, and the number of hit rows recorded in the second unit index structure corresponding to the second data block can be approximately considered as the number of data rows including the second term in the second data block. The embodiment of the present application determines the query result of the second aggregate query request based on the number of hit rows recorded in the first unit index structure corresponding to the second data block and the number of hit rows recorded in the second unit index structure corresponding to the second data block, and can implement an approximate aggregate query on the number of data rows including the first term and / or the second term.

[0238] In summary, in the data query method provided by the embodiment of the present application, after the data query device obtains the second aggregate query request, the data query device determines the first index area corresponding to the first word and the second index area corresponding to the second word in the database index structure. The aggregate query device only needs to read the k index fragments corresponding to the first index area and the p index fragments corresponding to the second index area to determine at least one second data block, and then determine the query result of the second aggregate query request according to the number of hit rows recorded in the first unit index structure corresponding to the at least one second data block and the number of hit rows recorded in the second unit index structure corresponding to the at least one second data block. The aggregate query device can realize the joint or aggregate query for the first word and the second word without reading the entire content of the database index structure. In the joint or aggregate query scenario of the first word and the second word, fast aggregate query can be realized, which can improve query performance.

[0239] Fig.16 The embodiment shown takes the first word and the second word as an example for joint or aggregate query. The data query method provided in the embodiment of the present application can also implement three or more words joint or aggregate query. The implementation method of three or more words joint or aggregate query can refer to Fig.16 In the illustrated embodiment, query performance can also be improved in a query scenario where three or more words are combined or aggregated.

[0240] also, Fig.16The illustrated embodiment takes the case where the database index structure includes a unit index structure as an example. When the database index structure does not include a unit index structure, the embodiment of the present application can also implement a joint or aggregate query of the first term and the second term. For example, in this case, after the data query device determines at least one second data block (that is, after executing S1603), the data query device scans the at least one second data block to determine the query result of the second aggregate query request, and the data query device does not need to execute the above S1604. That is, the above S1604 can be replaced by: scanning the at least one second data block to determine the query result of the second aggregate query request.

[0241] In the embodiment of the present application, the m data blocks corresponding to the m BF index structures are pre-written into the database, and the database index structure is created in the process of writing the m data blocks into the database. The process of writing the m data blocks into the database is similar, so the following takes writing a first data block into the database as an example to introduce the implementation process of writing the m data blocks into the database.

[0242] Please refer to Fig.17 , which shows a flow chart of a data writing method provided by an embodiment of the present application. The data writing method is executed by a data writing device. For example, the data writing device is deployed in Figure 1 In the computing device cluster 10 shown in FIG. 1 , the data writing device and the aforementioned data query device may be the same device or different devices. Fig.17 As shown, the data writing method includes the following steps S1701 to S1705.

[0243] S1701. Obtain a data write request, where the data write request is used to write a first data block D1 in a database.

[0244] The implementation method of the data writing device obtaining the data writing request can refer to the implementation method of the data query device obtaining the first data query request in S401, which will not be described in detail here. Among them, the data writing request may include a first data block D1, the first data block D1 includes a first word, and the first data block D1 can be any first data block of the at least one first data block described in the above embodiments.

[0245] S1702. In response to the data write request, create a BF index structure of the first data block D1.

[0246] After the data writing device obtains the data writing request, the data writing device creates a BF index structure of the first data block D1. The BF index structure of the first data block D1 is used to record the existence index information of the words in the first data block D1. The existence index information of any word in the first data block D1 is used to indicate the existence of the any word in the first data block D1. The existence index information of any word in the first data block D1 is in the index segment corresponding to the any word in the BF index structure of the first data block D1. The existence index information of any word in the first data block D1 is the value of the mapping position corresponding to the any word in the BF index structure of the first data block D1. The mapping position of any word in the first data block D1 in the BF index structure of the first data block D1 is obtained by hash calculation. The length of the index segment is less than the length of the BF index structure of the first data block D1.

[0247] For example, the first data block D1 is the data block m described in the above embodiment, and the BF index structure of the first data block D1 is as follows: Figure 2 or Figure 3 The BF index structure m in the database index structure shown in FIG. The BF index structure includes index units m1 to mn, each of which has a length of s bits, and the length of the BF index structure m is n×s bits.

[0248] S1703. Obtain the first word in the first data block D1.

[0249] In an optional embodiment, the data writing device performs word division on the first data block D1 to obtain at least one word, and the at least one word includes a first word. The first word is a complete word or a prefix of the complete word. For example, the data writing device performs word division on the first data block D1 to obtain a complete word "abcd", and the prefix of the complete word "abcd" includes "a", "ab" or "abc", and the first word can be any word among "a", "ab", "abc" or "abcd".

[0250] S1704. Determine a first index segment in the BF index structure of the first data block D1 according to the first word.

[0251] In an optional embodiment, the data writing device determines the starting position (i.e., the starting boundary) of the first index segment in the BF index structure of the first data block D1 according to the first word, and the data writing device determines the first index segment in the BF index structure of the first data block D1 according to the starting position and the length of the first index segment. For example, the length of the first index segment is equal to the length of the index unit in the BF index structure of the first data block D1. The data writing device determines the first index segment in the BF index structure of the first data block D1 according to the starting position of the first index segment and the length of the index unit in the BF index structure of the first data block D1. Among them, the first index segment corresponds to the first word, and the first index segment can be called the index segment of the first word in the first data block D1. The first index segment corresponds to k index units in the BF index structure of the first data block D1, and k is equal to 1 or 2. In one example, the starting position of the first index segment coincides with the starting position of a certain index unit in the BF index structure of the first data block D1, and k is equal to 1. In another example, the starting position of the first index segment coincides with the starting position of a certain index unit in the BF index structure of the first data block D1, and k is equal to 2.

[0252] In an optional embodiment, the data writing device obtains the target hash value of the first word, and the data writing device determines the starting position of the first index segment in the BF index structure of the first data block D1 according to the target hash value of the first word. In a specific embodiment, the data writing device uses a target hash function to perform hash calculation on the first word to obtain the target hash value of the first word; the data writing device determines the offset value of the first word according to the target hash value of the first word; the data writing device determines the starting position of the first index segment in the BF index structure of the first data block D1 according to the offset value of the first word. For example, the data writing device determines the position indicated by the offset value of the first word in the BF index structure of the first data block D1 as the starting position of the first index segment. In one embodiment, the data writing device uses the first part of the target hash value of the first word as the offset value of the first word, and the first part includes a plurality of continuous bits. The starting position of the first part can be the highest bit or the lowest bit of the target hash value, which is not limited in the embodiment of the present application.

[0253] In an optional embodiment, the data writing device determines the end position of the first index segment in the BF index structure of the first data block D1 according to the start position of the first index segment and the length of the first index segment. The data writing device determines the first index segment as the portion between the start position of the first index segment and the end position of the first index segment. For example, the data writing device determines the end position of the first index segment as the position after the start position of the first index segment and the distance from the start position to the position of s bits.

[0254] For example, the first data block D1 is data block m, and the BF index structure of the first data block D1 is BF index structure m. Fig.18 and Fig.19 , Fig.18 and Fig.19 A schematic diagram showing index segments of different words in the first data block D1 (ie, data block m) is shown.

[0255] In an example, the first word is word a, and the index segment of word a is the index segment of the first word, that is, the first index segment. Fig.18 and Fig.19 It can be seen that the starting position of the first index segment coincides with the starting position of the index unit m1 (that is, the left boundary), and the ending position of the first index segment coincides with the ending position of the index unit m1 (that is, the right boundary). The first index segment corresponds to an index unit in the BF index structure of the first data block D1 (that is, the BF index structure m), and the index unit is the index unit m1.

[0256] In another example, the first word is word ab, and the index segment of word ab is the index segment of the first word, that is, the first index segment. Fig.18 and Fig.19 It can be seen that the starting position of the first index segment falls in index unit m2, and the ending position of the first index segment falls in index unit m3. The first index segment corresponds to two index units in the BF index structure (that is, BF index structure m) of the first data block D1, and the two index units are index unit m2 and index unit m3.

[0257] In another example, the first word is word abc, and the index segment of word abc is the index segment of the first word, that is, the first index segment. Fig.18 and Fig.19 It can be seen that the starting position of the first index segment falls in index unit m3, and the ending position of the first index segment falls in index unit m4. The first index segment corresponds to two index units in the BF index structure (that is, BF index structure m) of the first data block D1, and the two index units are index unit m3 and index unit m4.

[0258] In another example, the first word is word abcd, and the index segment of word abcd is the index segment of the first word, that is, the first index segment. Fig.18 and Fig.19 It can be seen that the starting position of the first index segment falls in index unit m4, and the ending position of the first index segment falls in index unit m5. The first index segment corresponds to two index units in the BF index structure (that is, BF index structure m) of the first data block D1, and the two index units are index unit m4 and index unit m5.

[0259] In an optional embodiment, the BF index structure of the first data block D1 includes a low-frequency index space and a high-frequency index space. When the first word is a high-frequency word, the data writing device determines the first index segment in the high-frequency index space of the BF index structure of the first data block D1. When the first word is a low-frequency word, the data writing device determines the first index segment in the low-frequency index space of the BF index structure of the first data block D1. Fig.18 and Fig.19 The index segment shown may be located in the low-frequency index space or the high-frequency index space of the BF index structure m1.

[0260] S1705. Record the existence index information of the first word in the first index segment.

[0261] In an optional embodiment, the data writing device obtains multiple mapping hash values ​​of the first word; the data writing device determines multiple mapping positions of the first word in the first index segment according to the multiple mapping hash values ​​of the first word; the data writing device records the existence index information of the first word in the multiple mapping positions. For example, the data writing device sets the values ​​of the multiple mapping positions to 1, and the value "1" of the multiple mapping positions is the existence index information of the first word. In other words, the value "1" of the multiple mapping positions constitutes the existence index information of the first word. Among them, the multiple mapping positions correspond one to one with the multiple mapping hash values. For example, for each mapping hash value in the multiple mapping hash values, the data writing device determines the position indicated by the mapping hash value in the first index segment as a mapping position of the first word. The implementation method of the data writing device obtaining multiple mapping hash values ​​of the first word can refer to the implementation method of the data query device obtaining multiple mapping hash values ​​of the first word in the above S403, which will not be repeated here.

[0262] In an example, Fig.18 and Fig.19As shown, the first word is word a, and the first index segment is the index segment of word a. The data writing device determines multiple mapping positions of word a in the index segment of word a according to multiple mapping hash values ​​of word a; the data writing device sets the values ​​of the multiple mapping positions to 1 to record the existence index information of word a in the index segment of word a.

[0263] In another example, Fig.18 and Fig.19 As shown, the first word is word ab, and the first index segment is the index segment of the word ab. The data writing device determines multiple mapping positions of the word ab in the index segment of the word ab according to multiple mapping hash values ​​of the word ab; the data writing device sets the values ​​of the multiple mapping positions to 1 to record the existence index information of the word ab in the index segment of the word ab.

[0264] In another example, Fig.18 and Fig.19 As shown, the first word is word abc, and the first index segment is the index segment of word abc. The data writing device determines multiple mapping positions of the word abc in the index segment of word abc according to multiple mapping hash values ​​of word abc; the data writing device sets the values ​​of the multiple mapping positions to 1 to record the existence index information of the word abc in the index segment of word abc.

[0265] In another example, Fig.18 and Fig.19 As shown, the first word is word abcd, and the first index segment is the index segment of word abcd. The data writing device determines multiple mapping positions of word abcd in the index segment of word abcd according to multiple mapping hash values ​​of word abcd; the data writing device sets the values ​​of the multiple mapping positions to 1 to record the existence index information of word abcd in the index segment of word abcd.

[0266] This embodiment is described by taking the example of a data writing device recording the existence index information of the first word in the first data block D1 in the BF index structure of the first data block D1. The data writing device can record the existence index information of each word in the first data block D1 in the BF index structure of the first data block D1. The implementation process of the data writing device recording the existence index information of each word in the first data block D1 in the BF index structure of the first data block D1 can refer to the above S1703 to S1705, and the embodiment of the present application will not be repeated here.

[0267] In an optional embodiment, after the data writing device creates the BF index structure of the first data block D1, the data writing device also creates w unit index structures corresponding to the BF index structure of the first data block D1, each unit index structure in the w unit index structures corresponds to at least one index unit in the BF index structure of the first data block D1, and different unit index structures in the w unit index structures correspond to different index units in the BF index structure of the first data block D1. The data writing device also records the secondary index information of the index unit corresponding to each unit index structure in each unit index structure in the w unit index structures. For example, the first data block D1 is data block m, and the BF index structure of the first data block D1 is as follows Figure 3 In the BF index structure m of the database index structure shown in FIG. 1 , the w unit index structures corresponding to the BF index structure of the first data block D1 are as follows: Figure 3 The unit index structures m1 to mw in the database index structure shown. The unit index structure m1 corresponds to the index units m1 to m2, the unit index structure m2 corresponds to the index units m3 to m4... The unit index structure mw corresponds to the index units m(n-1) to mn, the data writing device records the secondary index information of the index units m1 to m2 in the unit index structure m1, the data writing device records the secondary index information of the index units m3 to m4 in the unit index structure m2, and so on, the data writing device records the secondary index information of the index units m(n-1) to mn in the unit index structure mw.

[0268] In an optional embodiment, the data writing device records the secondary index information of the index unit corresponding to each unit index structure in each unit index structure according to the hit status of the index unit corresponding to each unit index structure in the w unit index structures by the words in the first data block D1. For example, the data writing device records one or more of the following information of the index unit corresponding to each unit index structure in each unit index structure in the w unit index structures: the row number of the hit row of the index unit corresponding to each unit index structure, the number of hit rows of the index unit corresponding to each unit index structure, and the row number identification information. Among them, the hit row of the index unit corresponding to each unit index structure is the data row where the target word is located in the data block corresponding to the corresponding BF index structure, and the index segment corresponding to the target word hits the index unit corresponding to each unit index structure. The row number identification information indicates whether the number of hit rows of the index unit corresponding to each unit index structure is greater than the specified number. Take the example of the data writing device recording the secondary index information of the index unit corresponding to the first unit index structure in the first unit index structure of the w unit index structures. The first unit index structure is the unit index structure corresponding to the index unit hit by the index segment (i.e., the first index segment) of the first word. For example, in the initial state, the row number of the hit row recorded by the first unit index structure is a null value, the number of hit rows recorded by the first unit index structure is 0, and the row number identification information recorded by the first unit index structure is the initial identification information, which is used to indicate that the number of hit rows of the index unit corresponding to the first unit index structure is not greater than the specified number. For example, the initial identification information is "0". After the data writing device determines that the index segment (i.e., the first index segment) of the first word hits the index unit corresponding to the first unit index structure, the data writing device adds 1 to the number of hit rows recorded in the first unit index structure, and the data writing device determines whether the number of hit rows recorded in the first unit index structure after adding 1 is greater than the specified number. If the number of hit rows recorded in the first unit index structure after adding 1 is not greater than (that is, less than or equal to) the specified number, the data writing device records the row number of the first word in the first data block D1 in the first unit index structure. If the number of hit rows recorded in the first unit index structure after adding 1 is greater than the specified number, the data writing device updates the row number identification information recorded in the first unit index structure, and the updated row number identification information indicates that the number of hit rows of the index unit corresponding to the first unit index structure is greater than the specified number. For example, the updated row number identification information is "1".

[0269] It should be noted that the index segment corresponding to a certain word hits the index unit corresponding to a certain unit index structure means that the index segment corresponding to the certain word (that is, the index segment of the certain word) hits the index unit corresponding to the certain unit index structure.

[0270] For examples, please refer to Fig.19 , the first data block D1 is data block m, the BF index structure of the first data block D1 is BF index structure m, and the w unit index structures corresponding to the BF index structure of the first data block D1 are unit index structures m1~mw. In the initial state, the row numbers of the hit rows recorded by the unit index structures m1~mw are all null values, the number of hit rows recorded by the unit index structures m1~mw is all 0, and the row number identification information recorded by the unit index structures m1~mw is all the initial identification information "0".

[0271] In an example, Fig.19 As shown, the first word is word a, and the index segment of word a hits index unit m1, and index unit m1 corresponds to unit index structure m1. After the data writing device determines that the index segment of word a hits index unit m1, the data writing device adds 1 to the number of hit rows recorded in unit index structure m1, and the data writing device determines whether the number of hit rows recorded in unit index structure m1 after adding 1 is greater than the specified number. If the data writing device determines that the number of hit rows recorded in unit index structure m1 after adding 1 is not greater than the specified number, the data writing device records the row number of word a in the first data block D1 (that is, data block m) in the unit index structure m1. If the data writing device determines that the number of hit rows recorded in unit index structure m1 after adding 1 is greater than the specified number, the data writing device updates the row number identification information recorded in unit index structure m1 to "1".

[0272] In another example, Fig.19 As shown, the first word is word ab, and the index segment of word ab (that is, the first index segment) hits index unit m2, and index unit m2 corresponds to unit index structure m1. After the data writing device determines that the index segment of word ab hits index unit m2, the data writing device adds 1 to the number of hit rows recorded in unit index structure m1, and the data writing device determines whether the number of hit rows recorded in unit index structure m1 after adding 1 is greater than the specified number. If the number of hit rows recorded in unit index structure m1 after adding 1 is not greater than the specified number, the data writing device records the row number of word ab in the first data block D1 (that is, data block m) in unit index structure m1. If the number of hit rows recorded in unit index structure m1 after adding 1 is greater than the specified number, the data writing device updates the row number identification information recorded in unit index structure m1 to "1".

[0273] In another example, Fig.19 As shown, the first word is the word abc, and the index segment of the word abc hits the index unit m3, and the index unit m3 corresponds to the unit index structure m2. After the data writing device determines that the index segment of the word abc hits the index unit m3, the data writing device adds 1 to the number of hit rows recorded in the unit index structure m2, and the data writing device determines whether the number of hit rows recorded in the unit index structure m2 after adding 1 is greater than the specified number. If the data writing device determines that the number of hit rows recorded in the unit index structure m2 after adding 1 is not greater than the specified number, the data writing device records the row number of the word abc in the first data block D1 (that is, data block m) in the unit index structure m2. If the data writing device determines that the number of hit rows recorded in the unit index structure m2 after adding 1 is greater than the specified number, the data writing device updates the row number identification information recorded in the unit index structure m2 to "1".

[0274] In another example, Fig.19 As shown, the first word is the word abcd, and the index segment of the word abcd hits the index unit m4, and the index unit m4 corresponds to the unit index structure m2. After the data writing device determines that the index segment of the word abcd hits the index unit m4, the data writing device adds 1 to the number of hit rows recorded in the unit index structure m2, and the data writing device determines whether the number of hit rows recorded in the unit index structure m2 after adding 1 is greater than the specified number. If the data writing device determines that the number of hit rows recorded in the unit index structure m2 after adding 1 is not greater than the specified number, the data writing device records the row number of the word abcd in the first data block D1 (that is, data block m) in the unit index structure m2. If the data writing device determines that the number of hit rows recorded in the unit index structure m2 after adding 1 is greater than the specified number, the data writing device updates the row number identification information recorded in the unit index structure m2 to "1".

[0275] The above embodiments are described by taking the data writing device recording secondary index information in the first unit index structure as an example. The data writing device can record the secondary index information of the index unit corresponding to the unit index structure in the unit index structure corresponding to the index unit hit by the index segment of each word in the first data block D1. For the specific recording process, please refer to the above embodiments, and the embodiments of this application will not be described in detail here. In an optional embodiment, after the data writing device records the index information of the words in the first data block D1 in the database index structure, the data writing device writes the first data block D1 into the database.

[0276] In summary, in the data writing method provided by the present application, after the data writing device obtains a data writing request for writing a first data block in a database, the data writing device creates a BF index structure of the first data block, and the data writing device determines a first index segment in the BF index structure of the first data block according to the first word in the first data block, and records the existence index information of the first word in the first index segment. That is, the data writing device records the existence index information of the first word in the database index structure, which can facilitate data query for the first word according to the existence index information of the first word recorded in the BF index structure of the first data block.

[0277] In order to facilitate understanding of the technical solution of the embodiment of the present application, the embodiment of the present application is introduced below with an example.

[0278] Please refer to Fig. 20 , which shows a schematic diagram of a data management method provided in an embodiment of the present application. Fig. 20 The database index structure in can be as follows Figure 2 or Figure 3 shown. Fig. 20 The learning module and the word mapping module are both located in a data management device, which can be the data query device described in the aforementioned embodiment, or the data writing device described in the aforementioned embodiment. As mentioned above, the data query device and the data writing device can be the same device. Optionally, the data management device, the data query device and the data writing device are the same database engine. For example, the database is used to manage log data, and the database engine is a log engine.

[0279] like Fig. 20As shown, the data source is used to provide data blocks to the data management device. The learning module in the data management device processes the data blocks provided by the data source (for example, segmenting the data blocks), the learning module performs word mining on the processed data blocks, and the learning module performs model training based on the mining results to generate a word classification model. The word mapping module in the data management device classifies the words in the data blocks provided by the data source based on the word classification model to determine high-frequency words and low-frequency words. For each data block provided by the data source: the word mapping module maps the high-frequency words in the data block to the high-frequency index area of ​​the BF index structure corresponding to the data block; the word mapping module maps the low-frequency words in the data block to the low-frequency index area of ​​the BF index structure corresponding to the data block. That is, the word mapping module records the existence index information of the high-frequency words in the data block in the high-frequency index area of ​​the BF index structure corresponding to the data block, and the word mapping module records the existence index information of the low-frequency words in the data block in the low-frequency index area of ​​the BF index structure corresponding to the data block. In addition, the data management device can also record secondary index information in the database index structure. After the data management device records the index information related to the words in the data blocks provided by the data source in the database index structure, the database index structure can provide index information to the data blocks, so that in the data query scenario, data query can be performed based on the index information related to the words. The database index structure provided by the embodiment of the present application has good writing performance, low storage cost, and good query performance.

[0280] The above is an introduction to the method embodiment of the present application. The following is an introduction to the device embodiment of the present application. The device of the present application can be used to execute the method of the present application. For details not disclosed in the device embodiment of the present application, please refer to the method embodiment.

[0281] Please refer to Fig.21 , which shows a schematic diagram of a data query device 2100 provided in an embodiment of the present application. The data query device 2100 is applied to Figure 1 The computing device cluster 10 shown. For example, the data query device 2100 is a computing device in the computing device cluster 10, or the data query device 2100 is deployed in at least one computing device in the computing device cluster 10. Fig.21 The data query device 2100 includes an acquisition module 2110 , a determination module 2120 , a reading module 2130 and a query module 2140 .

[0282] The acquisition module 2110 is configured to acquire a first data query request, where the first data query request includes a first term.

[0283] The determination module 2120 is used to determine a first index area corresponding to the first word in the database index structure, wherein the database index structure includes m BF index structures and n index fragments based on column storage, the m BF index structures correspond to m data blocks, each BF index structure is used to record the existence index information of the words in the corresponding data block, the existence index information of any word in any data block is in the index segment corresponding to the any word in the BF index structure corresponding to the any data block, each BF index structure includes n index units, each index fragment includes m index units located in the same column of the m BF index structures, the length of the index unit is equal to the length of the index segment, and the length of the index segment is less than the length of the BF index structure, the first index area includes m first index segments corresponding to the first word in the m BF index structures, and the first index area corresponds to k index fragments among the n index fragments, the k is equal to 1 or 2, and the k is less than the n, and the m and the n are both positive integers.

[0284] The reading module 2130 is used to read the k index fragments corresponding to the first index area to determine at least one first data block, wherein the first index segment in the BF index structure corresponding to the first data block includes the existence index information of the first word.

[0285] The query module 2140 is configured to query the at least one first data block to determine a query result of the first data query request.

[0286] Optionally, each of the m BF index structures includes n×s bits, each index unit in each BF index structure includes s bits, and the bits in each BF index structure are used to record the existence index information of words in the data block corresponding to each BF index structure, and s is a positive integer.

[0287] Optionally, the database index structure further includes m×w unit index structures, each BF index structure in the m BF index structures corresponds to w unit index structures in the m×w unit index structures, and w is a positive integer;

[0288] Each unit index structure corresponding to each BF index structure in the m BF index structures corresponds to at least one index unit in each BF index structure, different unit index structures correspond to different index units, and each unit index structure corresponding to each BF index structure is used to record the secondary index information of the index unit corresponding to each unit index structure;

[0289] The secondary index information recorded in each unit index structure includes one or more of the following information: the row number of the hit row of the index unit corresponding to each unit index structure, the number of hit rows of the index unit corresponding to each unit index structure, and row number identification information;

[0290] The hit row of the index unit corresponding to each unit index structure is the data row where the target word is located in the data block corresponding to each BF index structure, and the index segment corresponding to the target word hits the index unit corresponding to each unit index structure;

[0291] The row number identification information indicates whether the number of hit rows of the index unit corresponding to each unit index structure is greater than a specified number.

[0292] Optionally, the secondary index information recorded in each unit index structure includes the row number of the hit row of the index unit corresponding to each unit index structure, and the query module 2140 is used to: scan the data row indicated by the row number of the first unit index structure record corresponding to each first data block in the at least one first data block to determine the query result of the first data query request, and the first unit index structure corresponding to each first data block in the at least one first data block is a unit index structure corresponding to the first index segment in the BF index structure of each first data block.

[0293] Optionally, the secondary index information recorded in each unit index structure includes row number identification information and the row number of the hit row of the index unit corresponding to each unit index structure. The query module 2140 is used to: for each first data block in the at least one first data block: when the row number identification information recorded in the first unit index structure corresponding to the first data block indicates that the number of hit rows of the index unit corresponding to the first unit index structure is less than or equal to the specified number, scan the data row indicated by the row number recorded in the first unit index structure corresponding to the first data block to determine the query result of the first data query request; when the row number identification information recorded in the first unit index structure corresponding to the first data block indicates that the number of hit rows of the index unit corresponding to the first unit index structure is greater than the specified number, scan each data row in the first data block to determine the query result of the first data query request.

[0294] Optionally, the secondary index information recorded in each unit index structure includes the row number of the hit row of the index unit corresponding to each unit index structure. The acquisition module 2110 is further used to acquire a second data query request, the second data query request including the first term and the second term.

[0295] The determination module 2120 is also used to determine a first index area corresponding to the first word and a second index area corresponding to the second word in the database index structure, the second index area includes m second index segments corresponding to the second word in the m BF index structures, and the second index area corresponds to p index fragments among the n index fragments, p is equal to 1 or 2, and p is less than n.

[0296] The reading module 2130 is also used to read the k index fragments corresponding to the first index area and the p index fragments corresponding to the second index area to determine at least one second data block, wherein the first index segment and the second index segment in the BF index structure corresponding to the second data block respectively include the existence index information of the first word and the existence index information of the second word.

[0297] The determination module 2120 is further used to determine a target data block from the at least one second data block according to a row number of a first unit index structure record corresponding to the at least one second data block and a row number of a second unit index structure record corresponding to the at least one second data block, wherein the first unit index structure corresponding to the second data block is a unit index structure corresponding to a first index segment in a BF index structure of the second data block, the second unit index structure corresponding to the second data block is a unit index structure corresponding to a second index segment in the BF index structure of the second data block, and the target data block includes a target data row, which is a data row indicated by the same row number in the row number of the first unit index structure record corresponding to the target data block and the row number of the second unit index structure record corresponding to the target data block.

[0298] The query module 2140 is further configured to scan the target data row in the target data block to determine the query result of the second data query request.

[0299] Optionally, the secondary index information recorded in each unit index structure includes the number of hit rows of the index unit corresponding to each unit index structure. The acquisition module 2110 is also used to obtain a first aggregate query request, which is used to query the number of data including the first term.

[0300] The determination module 2120 is further configured to determine a first index region corresponding to the first word in the database index structure.

[0301] The reading module 2130 is further configured to read the k index fragments corresponding to the first index area to determine at least one first data block.

[0302] The query module 2140 is also used to determine the query result of the first aggregate query request based on the number of hit rows recorded in the first unit index structure corresponding to the at least one first data block, wherein the first unit index structure corresponding to each first data block in the at least one first data block is a unit index structure corresponding to the first index segment in the BF index structure of each first data block.

[0303] Optionally, the secondary index information recorded in each unit index structure includes the number of hit rows of the index unit corresponding to each unit index structure. The acquisition module 2110 is also used to obtain a second aggregate query request, which is used to query the number of data including the first term and the second term.

[0304] The determination module 2120 is also used to determine a first index area corresponding to the first word and a second index area corresponding to the second word in the database index structure, the second index area includes m second index segments corresponding to the second word in the m BF index structures, and the second index area corresponds to p index fragments among the n index fragments, p is equal to 1 or 2, and p is less than n.

[0305] The reading module 2130 is also used to read the k index fragments corresponding to the first index area and the p index fragments corresponding to the second index area to determine the second data block, wherein the first index segment and the second index segment in the BF index structure corresponding to the second data block respectively include the existence index information of the first word and the existence index information of the second word.

[0306] The query module 2140 is also used to determine the query result of the second aggregate query request based on the smaller number of the number of hit rows recorded by the first unit index structure corresponding to the second data block and the number of hit rows recorded by the second unit index structure corresponding to the second data block, wherein the first unit index structure corresponding to the second data block is a unit index structure corresponding to the first index segment in the BF index structure of the second data block, and the second unit index structure corresponding to the second data block is a unit index structure corresponding to the second index segment in the BF index structure of the second data block.

[0307] Optionally, each of the m BF index structures includes a low-frequency index space and a high-frequency index space, the high-frequency index space is used to record the existence index information of high-frequency words in the data block corresponding to each BF index structure, and the low-frequency index space is used to record the existence index information of low-frequency words in the data block corresponding to each BF index structure.

[0308] Optionally, the determination module 2120 is used to: determine a starting position of the first index area according to the first word; and determine the first index area according to the starting position and the length of the index segment in the database index structure.

[0309] Optionally, the acquisition module 2110 is further used to acquire a data write request, where the data write request is used to write the first data block in the database;

[0310] The data query device 2100 also includes: a write module 2150, which is used to: create a BF index structure of the first data block in response to the data write request; obtain a first word in the first data block; determine a first index segment in the BF index structure of the first data block based on the first word; and record the existence index information of the first word in the first index segment.

[0311] Optionally, the first word is a complete word or a prefix of the complete word.

[0312] In summary, in the technical solution provided by the embodiment of the present application, after the data query device obtains a data query request including a first word, the data query device determines the first index area corresponding to the first word in the database index structure, and the data query device only needs to read the k index fragments corresponding to the first index area to determine at least one first data block, and then query the at least one first data block to determine the query result of the data query request. The data query device does not need to read the entire content of the database index structure, so the query I / O cost is low, the memory overhead is small, and the query performance is good.

[0313] For the functional implementation of the acquisition module 2110, please refer to the relevant descriptions in the above steps S401, S1001, S1301, S1401, S1501, S1601 and S1701. For the functional implementation of the determination module 2120, please refer to the relevant descriptions in the above steps S402, S1002, S1004, S1302, S1402, S1502 and S1602. For the functional implementation of the reading module 2130, please refer to the relevant descriptions in the above steps S403, S1003, S1303, S1403, S1503 and S1603. For the functional implementation of the query module 2140, please refer to the relevant descriptions in the above steps S404, S1005, S1304, S1404, S1504 and S1604. For the functional implementation of the write module 2150, please refer to the relevant descriptions in the above steps S1702 to S1705.

[0314] In the embodiment of the present application, the acquisition module 2110, the determination module 2120, the reading module 2130, the query module 2140 and the writing module 2150 can all be implemented by software, or can all be implemented by hardware. For example, the following takes the determination module 2120 as an example to introduce the implementation method of the determination module 2120 implemented by software and the implementation method of the determination module 2120 implemented by hardware. Similarly, the implementation methods of the acquisition module 2110, the reading module 2130, the query module 2140 and the writing module 2150 can refer to the implementation method of the determination module 2120.

[0315] As an example of determining that module 2120 is implemented by software, determining module 2120 may be a software function module. Determining module 2120 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the computing instance running the code included in determining module 2120 may be one or more. For example, determining module 2120 may include code running on multiple hosts / virtual machines / containers. It should be noted that multiple hosts / virtual machines / containers for running the code may be distributed in the same region or in different regions. Further, multiple hosts / virtual machines / containers for running the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Among them, a region may generally include multiple AZs. Similarly, multiple hosts / virtual machines / containers for running the code may be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Among them, usually a VPC is set up in a region. A communication gateway can be set up in the VPC to achieve interconnection between different VPCs in the same region, as well as interconnection and cross-region communication between VPCs in different regions.

[0316] As an example of determining that the module 2120 is implemented by hardware, the module 2120 may be a hardware functional module. The module 2120 may include at least one computing device, such as at least one server. Alternatively, the module 2120 may be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. In the case where the module 2120 includes multiple computing devices, the multiple computing devices included in the module 2120 may be distributed in the same region or in different regions. The multiple computing devices included in the module 2120 may be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the module 2120 may be distributed in the same VPC or in multiple VPCs. The multiple computing devices may be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, or GALs.

[0317] It should be noted that, in the above embodiment, the acquisition module 2110 is used to execute steps S401, S1001, S1301, S1401, S1501, S1601 in the data query method and step S1701 in the data writing method. The determination module 2120 is used to execute steps S402, S1002, S1004, S1302, S1402, S1502 and S1602 in the data query method. The reading module 2130 is used to execute steps S403, S1003, S1303, S1403, S1503 and S1603 in the data query method. The query module 2140 is used to execute steps S404, S1005, S1304, S1404, S1504 and S1604 in the data query method. The writing module 2150 is used to execute steps S1702 to S1705 in the data writing method. However, this is only exemplary. In other embodiments, the acquisition module 2110 can be used to execute any step in the data query method and the data writing method, the determination module 2120 can be used to execute any step in the data query method and the data writing method, the reading module 2130 can be used to execute any step in the data query method and the data writing method, the query module 2140 can be used to execute any step in the data query method and the data writing method, and the writing module 2150 can be used to execute any step in the data query method and the data writing method. The steps that the acquisition module 2110, the determination module 2120, the reading module 2130, the query module 2140 and the writing module 2150 are responsible for implementing can be specified as needed, and the acquisition module 2110, the determination module 2120, the reading module 2130, the query module 2140 and the writing module 2150 respectively implement different steps in the data query method and the data writing method to realize the full functions of the data query device and the data writing device.

[0318] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the module is not elaborated in the above description. The specific working process of the module can refer to the corresponding content in the aforementioned method embodiment, which will not be repeated here.

[0319] The embodiment of the present application provides a computing device, including a memory and a processor. The memory is used to store a computer program. The processor is used to execute the computer program stored in the memory so that the computing device executes the data query method and data writing method provided by the embodiment of the present application. Wherein, the computer program includes an operating system and / or executable code (i.e., program instructions).

[0320] The computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0321] As an example, see Fig. 22 , which shows a schematic diagram of a computing device 100 provided in an embodiment of the present application. The computing device 100 includes a processor 101, a memory 102, a communication interface 103 and a bus 104. The processor 101, the memory 102 and the communication interface 103 are communicatively connected via the bus 104. It should be understood that in other embodiments, the processor 101, the memory 102 and the communication interface 103 may also be connected in other ways. The present application does not limit the number of processors and the number of memories in the computing device 100.

[0322] Among them, the processor 101 may include a general processor and / or a special processor. A general processor may be a processor that performs specific steps and / or operations by reading and executing a computer program stored in a memory (e.g., memory 102). The general processor may use data stored in the memory during the execution of the above steps and / or operations. A general processor may include a central processing unit (CPU), a microprocessor (MP), or a graphics processing unit (GPU). The CPU may be a single-core CPU (single-CPU) or a multi-core CPU (multi-CPU). A special processor is a processor specially designed to perform specific steps and / or operations. A special processor is a high-performance hardware module, and a special processor is also called a special hardware chip. A special processor may include a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a network processor (NP). Among them, the processor 101 includes at least one circuit (e.g., an integrated circuit) and has signal processing capabilities. During the implementation process, part or all of the functions of the data query method and the data writing method of the present application can be completed by an integrated circuit in the form of hardware or program instructions in the form of software in the processor 101.

[0323] Among them, the memory 102 can be various types of storage media. The memory 102 includes volatile memory and / or non-volatile memory. Volatile memory includes random access memory (RAM) or other types of static storage devices that can store static information and instructions. Non-volatile memory includes read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, optical storage (including read-only optical disk, optical disk storage, such as compact disc, laser disc, optical disk, digital versatile disc, Blu-ray disc, etc.), mechanical hard disk (HDD) or solid state drive (SSD), etc. The memory 102 can exist independently and be connected to the processor 101 through the bus 104, or it can be integrated with the processor 101. The memory 102 is used to store computer programs, and the computer programs stored in the memory 102 include an operating system and / or an executable code (i.e., a program instruction). The processor 101 implements the functions of the aforementioned acquisition module 2110, determination module 2120, reading module 2130, query module 2140, and write module 2150 by executing the executable code stored in the memory 102, thereby realizing the data query method and data writing method provided in the embodiment of the present application. That is, the program instructions for executing the data query method and the data writing method are stored in the memory 102. Optionally, the memory 102 also includes software modules and data required for other running processes such as an operating system, etc., which are not limited in the embodiment of the present application.

[0324] Among them, the communication interface 103 may include an input / output (I / O) interface, a physical interface, and a logical interface, etc., which are used to realize the interconnection of devices inside the computing device 100, and an interface for realizing the interconnection between the computing device 100 and other devices (such as computing devices, terminal devices, or network devices, etc.). The physical interface may be an Ethernet interface, which can be used to realize the interconnection between the computing device 100 and other devices. The logical interface is an interface inside the computing device 100, which can be used to realize the interconnection of devices inside the computing device 100. It is easy to understand that the communication interface 103 can be used for the computing device 100 to communicate with other devices, for example, the communication interface 103 is used for sending and receiving data between the computing device 100 and other devices. The communication interface 103 can also use a transceiver module such as a network interface card and a transceiver to realize the communication between the computing device 100 and other devices or a communication network. For example, the communication interface 103 can be a combination of one or more of the following devices: a network interface (such as an Ethernet interface), a wireless network card, and other devices with network access functions.

[0325] The bus 104 may be any type of communication bus, such as a system bus, for interconnecting devices (such as the processor 101, the memory 102, and the communication interface 103) within the computing device 100. The bus 104 may include a path for transmitting information between devices within the computing device 100. For example, the bus 104 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus 104 may be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Fig. 22 Although only one line is used to represent the bus 104, it does not mean that the bus 104 has only one bus or only one type of bus.

[0326] It should be noted that the above-mentioned multiple devices can be respectively arranged on independent chips, or at least partially or completely arranged on the same chip. Whether each device is independently arranged on different chips or integrated on one or more chips often depends on the needs of product design. The embodiments of the present application do not limit the specific implementation form of the above-mentioned devices.

[0327] Fig. 22The computing device 100 shown is only an example, and the computing device 100 may also include other components. The computing device 100 performs data query by executing all or part of the steps of the data query method provided in the above embodiment, and the computing device 100 performs data writing by executing all or part of the steps of the data writing method provided in the above embodiment.

[0328] The embodiment of the present application also provides a computing device cluster, which includes at least one computing device. Each computing device in the at least one computing device includes a processor and a memory. The memory of the at least one computing device is used to store computer programs. The processor of the at least one computing device is used to execute the computer program stored in the memory of the at least one computing device, so that the computing device cluster executes the data query method provided by the embodiment of the present application and the data writing method provided by the embodiment of the present application. Among them, the computer program stored in the memory of the at least one computing device may include an operating system and / or executable code (i.e., program instructions), and the memory of one or more computing devices in the at least one computing device may store the same program instructions for executing the data query method provided by the embodiment of the present application and the program instructions for executing the data writing method provided by the embodiment of the present application. The at least one computing device may include a server, for example, a central server, an edge server, or a local server in a local data center. In some embodiments, the at least one computing device may also include a terminal device such as a desktop computer, a laptop computer, or a smart phone. Among them, one or more computing devices in the at least one computing device are such as Fig. 22 Computing device 100 is shown.

[0329] As an example, see Fig.23 , which shows a schematic diagram of a computing device cluster provided in an embodiment of the present application. The computing device cluster includes multiple computing devices 100 ( Fig.23 Take four computing devices 100 as an example. The multiple computing devices 100 can be connected via a network ( Fig.23 The network may be a wide area network or a local area network. The structure of at least one computing device 100 in the computing device cluster may refer to Fig. 22 The structure of the computing device 100 shown. The memory 102 of one or more computing devices 100 in the computing device cluster may store the same program instructions for implementing the data query method provided in the embodiment of the present application and the same program instructions for executing the data writing method provided in the embodiment of the present application.

[0330] In some possible implementations, the memory 102 of at least one computing device 100 in the computing device cluster may also store some program instructions for executing the data query method. In other words, the combination of at least two computing devices 100 in the computing device cluster can jointly execute the program instructions for implementing the data query method provided in the embodiment of the present application. Similarly, the memory 102 of at least one computing device 100 in the computing device cluster may also store some program instructions for executing the data writing method. It should be noted that the memory 102 of different computing devices 100 in the computing device cluster may store different program instructions, and each computing device 100 in these different computing devices 100 is used to perform some functions of the data query device and / or some functions of the data writing device. That is, the program instructions stored in the memory 102 of each computing device 100 in these different computing devices 100 can realize the functions of one or more modules in the acquisition module 2110, the determination module 2120, the reading module 2230, the query module 2140 and the write module 2150.

[0331] As another example, see Fig.24 , which shows a schematic diagram of another computing device cluster provided in an embodiment of the present application. Fig.24 The computing device cluster includes computing device 100A and computing device 100B. The structure of computing device 100A and the structure of computing device 100B can be found in Fig. 22 The structure of the computing device 100 shown. The computing device 100A is connected to the computing device 100B via a network. Specifically, the communication interface 103 of the computing device 100A and the communication interface 103 of the computing device 100B are respectively connected to the network, so that the computing device 100A is connected to the computing device 100B via the network. In this example, the memory 102 of the computing device 100A stores program instructions for implementing the functions of the acquisition module 2110, the determination module 2120 and the reading module 2230, and the memory 102 of the computing device 100B stores program instructions for implementing the functions of the query module 2140 and the write module 2150. The computing device 100A and the computing device 100B jointly execute program instructions for implementing the data query method provided in the present application.

[0332] Fig.24 The connection mode between the computing device clusters shown may be that considering that the data query method provided by the present application requires a large amount of data and data index information to be stored, it is considered that the functions implemented by the acquisition module 2110, the determination module 2120 and the reading module 2230 are handed over to the computing device 100A for execution, and the functions implemented by the query module 2140 and the writing module 2150 are handed over to the computing device 100B for execution. It should be understood that Fig.24The functions of the computing device 100A shown in the figure may also be completed by multiple computing devices 100. Similarly, the functions of the computing device 100B may also be completed by multiple computing devices 100. This embodiment of the application does not limit this.

[0333] Based on the same inventive concept, an embodiment of the present application provides a data query system. The data query system includes a client and a computing device cluster. The client is connected to the computing device cluster for communication. The computing device cluster is used to execute Figure 4 , Fig.10 , Figures 13 to 17 The method provided by the embodiment shown. The computing device cluster includes Fig.21 The data query device provided by the illustrated embodiment.

[0334] For example, the data query system is as follows Figure 1 shown.

[0335] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium. A computer program is stored in the computer-readable storage medium, and when the computer program is executed, all or part of the steps of the data query method and / or data writing method provided in the embodiment of the present application are implemented. For example, when the computer program is executed by a computing device, a computing device cluster, a virtual machine, a container, a data query device, one or more processors, etc., all or part of the steps of the data query method provided in the embodiment of the present application are implemented. When the computer program is executed by a computing device, a computing device cluster, a virtual machine, a container, a data writing device, one or more processors, etc., all or part of the steps of the data query method provided in the embodiment of the present application are implemented.

[0336] The computer-readable storage medium may be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk).

[0337] Based on the same inventive concept, the embodiment of the present application provides a computer program product comprising program instructions, which implement all or part of the steps of the data query method and / or data writing method provided by the embodiment of the present application when the program instructions are executed. For example, when the program instructions are executed by a computing device, a computing device cluster, a virtual machine, a container, a data query device, one or more processors, etc., all or part of the steps of the data query method provided by the embodiment of the present application are implemented. When the program instructions are executed by a computing device, a computing device cluster, a virtual machine, a container, a data writing device, one or more processors, etc., all or part of the steps of the data query method provided by the embodiment of the present application are implemented. Wherein, the computer program product can be a software or program product that can be run on a computing device or stored in any available medium, including program instructions.

[0338] Based on the same inventive concept, an embodiment of the present application provides a chip, which includes a programmable logic circuit and / or program instructions. When the chip is running, it is used to implement all or part of the steps of the data query method and / or data writing method provided in the above method embodiment.

[0339] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product, and the computer program product includes one or more computer program instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a computer network, or other programmable device. The computer program instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that contains one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium, or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0340] It should be understood that the term "at least one" in this application refers to one or more, and "plurality" refers to two or more. In this application, unless otherwise specified, the symbol " / " generally means or, for example, A / B can represent A or B. The term "and / or" in this application is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, for the convenience of clear description, this application uses words such as "first", "second", and "third" to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art can understand that words such as "first", "second", and "third" do not limit the quantity and execution order.

[0341] Different types of embodiments such as method embodiments and device embodiments provided in the embodiments of the present application can refer to each other, the order of operations of the method embodiments can be appropriately adjusted, and the operations can be increased or decreased in response to the circumstances. Any technician familiar with the technical field can easily think of changes within the technical scope disclosed in the present application, and all of them should be covered within the scope of protection of the present application, so they will not be repeated here.

[0342] In the corresponding embodiments provided in the present application, it should be understood that the disclosed devices and the like can be implemented by other configuration methods. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical or other forms. The modules described as separate components may or may not be physically separated, and the components described as modules may or may not be physical modules, which may be located in one place or distributed on multiple network nodes. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0343] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A data query method, characterized in that: The method comprises: Obtaining a first data query request, wherein the first data query request includes a first term; Determine a first index area corresponding to the first word in a database index structure, wherein the database index structure includes m Bloom filter BF index structures and n index shards based on column storage, the m BF index structures correspond to m data blocks, each BF index structure is used to record existence index information of words in the corresponding data block, the existence index information of any word in any data block is in the index segment corresponding to the any word in the BF index structure corresponding to the any data block, each BF index structure includes n index units, each index segment includes m index units located in the same column in the m BF index structures, the length of the index unit is equal to the length of the index segment, the length of the index segment is less than the length of the BF index structure, the first index area includes m first index segments corresponding to the first word in the m BF index structures, and the first index area corresponds to k index shards in the n index shards, k is equal to 1 or 2, and k is less than n, and m and n are both positive integers; Read the k index fragments corresponding to the first index area to determine at least one first data block, wherein a first index segment in the BF index structure corresponding to the first data block includes existence index information of the first word; The at least one first data block is queried to determine a query result of the first data query request.

2. The method according to claim 1, characterized in that Each of the m BF index structures includes n×s bits, each index unit in each BF index structure includes s bits, and the bits in each BF index structure are used to record the existence index information of words in the data block corresponding to each BF index structure, and s is a positive integer.

3. The method according to claim 1 or 2, characterized in that: The database index structure further includes m×w unit index structures, each BF index structure in the m BF index structures corresponds to w unit index structures in the m×w unit index structures, and w is a positive integer; Each unit index structure corresponding to each BF index structure in the m BF index structures corresponds to at least one index unit in each BF index structure, different unit index structures correspond to different index units, and each unit index structure corresponding to each BF index structure is used to record secondary index information of the index unit corresponding to each unit index structure; The secondary index information recorded in each unit index structure includes one or more of the following information: the row number of the hit row of the index unit corresponding to each unit index structure, the number of hit rows of the index unit corresponding to each unit index structure, and row number identification information; The hit row of the index unit corresponding to each unit index structure is the data row where the target word is located in the data block corresponding to each BF index structure, and the index segment corresponding to the target word hits the index unit corresponding to each unit index structure; The row number identification information indicates whether the number of hit rows of the index unit corresponding to each unit index structure is greater than a specified number.

4. The method according to claim 3, characterized in that The secondary index information recorded in each unit index structure includes the row number of the hit row of the index unit corresponding to each unit index structure, The querying the at least one first data block to determine a query result of the first data query request includes: Scan the data row indicated by the row number of the first unit index structure record corresponding to each first data block in the at least one first data block to determine the query result of the first data query request, wherein the first unit index structure corresponding to each first data block in the at least one first data block is a unit index structure corresponding to the first index segment in the BF index structure of each first data block.

5. The method according to claim 3, characterized in that: The secondary index information recorded in each unit index structure includes the row number identification information and the row number of the hit row of the index unit corresponding to each unit index structure, The querying the at least one first data block to determine a query result of the first data query request includes: For each first data block of the at least one first data block: In a case where the row number identification information of the first unit index structure record corresponding to the first data block indicates that the number of hit rows of the index unit corresponding to the first unit index structure is less than or equal to the specified number, scanning the data row indicated by the row number of the first unit index structure record corresponding to the first data block to determine the query result of the first data query request; When the row number identification information recorded in the first unit index structure corresponding to the first data block indicates that the number of hit rows of the index unit corresponding to the first unit index structure is greater than the specified number, each data row in the first data block is scanned to determine the query result of the first data query request.

6. The method according to claim 3, characterized in that The secondary index information recorded in each unit index structure includes the row number of the hit row of the index unit corresponding to each unit index structure, and the method further includes: Obtaining a second data query request, where the second data query request includes the first term and a second term; Determine the first index region corresponding to the first word and the second index region corresponding to the second word in the database index structure, wherein the second index region includes m second index segments corresponding to the second word in the m BF index structures, and the second index region corresponds to p index fragments among the n index fragments, where p is equal to 1 or 2, and p is less than n; Reading the k index fragments corresponding to the first index area and the p index fragments corresponding to the second index area to determine at least one second data block, wherein a first index segment and a second index segment in the BF index structure corresponding to the second data block respectively include existence index information of the first word and existence index information of the second word; Determine a target data block from the at least one second data block according to a row number of a first unit index structure record corresponding to the at least one second data block and a row number of a second unit index structure record corresponding to the at least one second data block, wherein the first unit index structure corresponding to the second data block is a unit index structure corresponding to a first index segment in a BF index structure of the second data block, the second unit index structure corresponding to the second data block is a unit index structure corresponding to a second index segment in the BF index structure of the second data block, and the target data block includes a target data row, and the target data row is a data row indicated by the same row number in the row number of the first unit index structure record corresponding to the target data block and the row number of the second unit index structure record corresponding to the target data block; The target data row in the target data block is scanned to determine a query result of the second data query request.

7. The method according to claim 3, characterized in that The secondary index information recorded in each unit index structure includes the number of hit rows of the index unit corresponding to each unit index structure, and the method further includes: Obtaining a first aggregate query request, where the first aggregate query request is used to query the quantity of data including the first term; Determining the first index area corresponding to the first word in the database index structure; Reading the k index slices corresponding to the first index area to determine the at least one first data block; Determine the query result of the first aggregate query request based on the number of hit rows recorded in the first unit index structure corresponding to the at least one first data block, wherein the first unit index structure corresponding to each first data block in the at least one first data block is a unit index structure corresponding to the first index segment in the BF index structure of each first data block.

8. The method according to claim 3, characterized in that The secondary index information recorded in each unit index structure includes the number of hit rows of the index unit corresponding to each unit index structure, and the method further includes: Obtaining a second aggregate query request, where the second aggregate query request is used to query the quantity of data including the first term and the second term; Determine the first index region corresponding to the first word and the second index region corresponding to the second word in the database index structure, wherein the second index region includes m second index segments corresponding to the second word in the m BF index structures, and the second index region corresponds to p index fragments among the n index fragments, where p is equal to 1 or 2, and p is less than n; Reading the k index fragments corresponding to the first index area and the p index fragments corresponding to the second index area to determine a second data block, wherein a first index segment and a second index segment in a BF index structure corresponding to the second data block respectively include existence index information of the first word and existence index information of the second word; Determine the query result of the second aggregate query request based on the smaller number of hit rows recorded by the first unit index structure corresponding to the second data block and the number of hit rows recorded by the second unit index structure corresponding to the second data block, wherein the first unit index structure corresponding to the second data block is a unit index structure corresponding to the first index segment in the BF index structure of the second data block, and the second unit index structure corresponding to the second data block is a unit index structure corresponding to the second index segment in the BF index structure of the second data block.

9. The method according to any one of claims 1 to 8, characterized in that Each of the m BF index structures includes a low-frequency index space and a high-frequency index space, the high-frequency index space is used to record the existence index information of high-frequency words in the data block corresponding to each BF index structure, and the low-frequency index space is used to record the existence index information of low-frequency words in the data block corresponding to each BF index structure.

10. The method according to any one of claims 1 to 9, characterized in that The determining a first index area corresponding to the first word in the database index structure includes: Determine a starting position of the first index area according to the first word; The first index area is determined according to the starting position and the length of the index segment in the database index structure.

11. The method according to any one of claims 1 to 10, characterized in that Before obtaining the first data query request, the method further includes: Obtaining a data write request, where the data write request is used to write the first data block into a database; In response to the data write request, create a BF index structure of the first data block; Obtaining the first word in the first data block; Determine the first index segment in the BF index structure of the first data block according to the first word; The existence index information of the first word is recorded in the first index segment.

12. The method according to any one of claims 1 to 11, characterized in that The first word is a complete word or a prefix of the complete word.

13. A data query device, characterized in that: The device comprises: An acquisition module, configured to acquire a first data query request, wherein the first data query request includes a first term; a determination module, configured to determine a first index region corresponding to the first term in a database index structure, wherein the database index structure includes m Bloom filter BF index structures and n index fragments based on column storage, the m BF index structures correspond to m data blocks, each BF index structure is used to record existence index information of terms in a corresponding data block, the existence index information of any term in any data block is in an index segment corresponding to the any term in the BF index structure corresponding to the any data block, each BF index structure includes n index units, each index fragment includes m index units located in the same column in the m BF index structures, the length of the index unit is equal to the length of the index segment, the length of the index segment is less than the length of the BF index structure, the first index region includes m first index segments corresponding to the first term in the m BF index structures, and the first index region corresponds to k index fragments in the n index fragments, k is equal to 1 or 2, and k is less than n, and both m and n are positive integers; a reading module, configured to read the k index fragments corresponding to the first index region to determine at least one first data block, wherein a first index segment in the BF index structure corresponding to the first data block includes existence index information of the first word; The query module is used to query the at least one first data block to determine the query result of the first data query request.

14. The device according to claim 13, characterized in that Each of the m BF index structures includes n×s bits, each index unit in each BF index structure includes s bits, and the bits in each BF index structure are used to record the existence index information of words in the data block corresponding to each BF index structure, and s is a positive integer.

15. The device according to claim 13 or 14, characterized in that The database index structure further includes m×w unit index structures, each BF index structure in the m BF index structures corresponds to w unit index structures in the m×w unit index structures, and w is a positive integer; Each unit index structure corresponding to each BF index structure in the m BF index structures corresponds to at least one index unit in each BF index structure, different unit index structures correspond to different index units, and each unit index structure corresponding to each BF index structure is used to record secondary index information of the index unit corresponding to each unit index structure; The secondary index information recorded in each unit index structure includes one or more of the following information: the row number of the hit row of the index unit corresponding to each unit index structure, the number of hit rows of the index unit corresponding to each unit index structure, and row number identification information; The hit row of the index unit corresponding to each unit index structure is the data row where the target word is located in the data block corresponding to each BF index structure, and the index segment corresponding to the target word hits the index unit corresponding to each unit index structure; The row number identification information indicates whether the number of hit rows of the index unit corresponding to each unit index structure is greater than a specified number.

16. The device according to claim 15, characterized in that The secondary index information recorded in each unit index structure includes the row number of the hit row of the index unit corresponding to each unit index structure, and the query module is used to: Scan the data row indicated by the row number of the first unit index structure record corresponding to each first data block in the at least one first data block to determine the query result of the first data query request, wherein the first unit index structure corresponding to each first data block in the at least one first data block is a unit index structure corresponding to the first index segment in the BF index structure of each first data block.

17. The device according to claim 15, characterized in that The secondary index information recorded in each unit index structure includes the row number identification information and the row number of the hit row of the index unit corresponding to each unit index structure, and the query module is used to: For each first data block of the at least one first data block: In a case where the row number identification information of the first unit index structure record corresponding to the first data block indicates that the number of hit rows of the index unit corresponding to the first unit index structure is less than or equal to the specified number, scanning the data row indicated by the row number of the first unit index structure record corresponding to the first data block to determine the query result of the first data query request; When the row number identification information recorded in the first unit index structure corresponding to the first data block indicates that the number of hit rows of the index unit corresponding to the first unit index structure is greater than the specified number, each data row in the first data block is scanned to determine the query result of the first data query request.

18. The device according to claim 15, characterized in that The secondary index information recorded in each unit index structure includes the row number of the hit row of the index unit corresponding to each unit index structure, The acquisition module is further used to acquire a second data query request, where the second data query request includes the first term and the second term; The determining module is further used to determine the first index area corresponding to the first word and the second index area corresponding to the second word in the database index structure, the second index area includes m second index segments corresponding to the second word in the m BF index structures, and the second index area corresponds to p index fragments among the n index fragments, where p is equal to 1 or 2, and p is less than n; The reading module is further used to read the k index fragments corresponding to the first index area and the p index fragments corresponding to the second index area to determine at least one second data block, wherein the first index segment and the second index segment in the BF index structure corresponding to the second data block respectively include the existence index information of the first word and the existence index information of the second word; The determination module is further configured to determine a target data block from the at least one second data block according to a row number of a first unit index structure record corresponding to the at least one second data block and a row number of a second unit index structure record corresponding to the at least one second data block, wherein the first unit index structure corresponding to the second data block is a unit index structure corresponding to a first index segment in a BF index structure of the second data block, the second unit index structure corresponding to the second data block is a unit index structure corresponding to a second index segment in the BF index structure of the second data block, and the target data block includes a target data row, and the target data row is a data row indicated by a same row number in the row number of the first unit index structure record corresponding to the target data block and the row number of the second unit index structure record corresponding to the target data block; The query module is further configured to scan the target data row in the target data block to determine a query result of the second data query request.

19. The device according to claim 15, characterized in that The secondary index information recorded in each unit index structure includes the number of hit rows of the index unit corresponding to each unit index structure. The acquisition module is further used to acquire a first aggregate query request, where the first aggregate query request is used to query the amount of data including the first term; The determination module is further configured to determine the first index region corresponding to the first word in the database index structure; The reading module is further configured to read the k index fragments corresponding to the first index area to determine the at least one first data block; The query module is also used to determine the query result of the first aggregate query request based on the number of hit rows recorded in the first unit index structure corresponding to the at least one first data block, wherein the first unit index structure corresponding to each first data block in the at least one first data block is a unit index structure corresponding to the first index segment in the BF index structure of each first data block.

20. The device according to claim 15, characterized in that The secondary index information recorded in each unit index structure includes the number of hit rows of the index unit corresponding to each unit index structure. The acquisition module is further used to acquire a second aggregate query request, where the second aggregate query request is used to query the amount of data including the first term and the second term; The determining module is further used to determine the first index area corresponding to the first word and the second index area corresponding to the second word in the database index structure, the second index area includes m second index segments corresponding to the second word in the m BF index structures, and the second index area corresponds to p index fragments among the n index fragments, where p is equal to 1 or 2, and p is less than n; The reading module is further used to read the k index fragments corresponding to the first index area and the p index fragments corresponding to the second index area to determine a second data block, wherein a first index segment and a second index segment in the BF index structure corresponding to the second data block respectively include the existence index information of the first word and the existence index information of the second word; The query module is also used to determine the query result of the second aggregate query request based on the smaller number of hit rows recorded by the first unit index structure corresponding to the second data block and the number of hit rows recorded by the second unit index structure corresponding to the second data block, wherein the first unit index structure corresponding to the second data block is a unit index structure corresponding to the first index segment in the BF index structure of the second data block, and the second unit index structure corresponding to the second data block is a unit index structure corresponding to the second index segment in the BF index structure of the second data block.

21. The device according to any one of claims 13 to 20, characterized in that Each of the m BF index structures includes a low-frequency index space and a high-frequency index space, the high-frequency index space is used to record the existence index information of high-frequency words in the data block corresponding to each BF index structure, and the low-frequency index space is used to record the existence index information of low-frequency words in the data block corresponding to each BF index structure.

22. The device according to any one of claims 13 to 21, characterized in that The determining module is used to: Determine a starting position of the first index area according to the first word; The first index area is determined according to the starting position and the length of the index segment in the database index structure.

23. The device according to any one of claims 13 to 22, characterized in that The acquisition module is further used to acquire a data write request, where the data write request is used to write the first data block into the database; The device further comprises: a writing module, configured to: In response to the data write request, create a BF index structure of the first data block; Obtaining the first word in the first data block; Determine the first index segment in the BF index structure of the first data block according to the first word; The existence index information of the first word is recorded in the first index segment.

24. The device according to any one of claims 13 to 23, characterized in that The first word is a complete word or a prefix of the complete word.

25. A computing device cluster, characterized in that: The method comprises at least one computing device, each of which comprises a processor and a memory, and the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method as described in any one of claims 1 to 12.

26. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 12 is implemented.

27. A computer program product comprising program instructions, characterized in that When the program instructions are executed, the method according to any one of claims 1 to 12 is implemented.

Images