Data storage device, data query method, device, server and storage medium

Abstract

The embodiments of the present application disclose a data storage device, a data query method, an apparatus, a server and a storage medium, which belong to the field of databases. The data query method includes: receiving a data query request; responding to the data query request for querying a first data item, determining a target first-level data block that meets the query condition based on an offset index or a column value index; determining a target second-level data block that meets the query condition from the target first-level data block; determining a target column storage unit that meets the query condition from the target second-level data block; determining a target query content from the target column storage unit, the target query content being the row number to which the target column value belongs, or the column value belonging to the target row number. When the value range corresponding to the query condition is wide, the embodiments of the present application reduce the number of scans and improve the query efficiency, so that the server can locate the data block step by step according to the query condition and the index, and determine the data that meets the query condition from the data block, further improving the data query efficiency.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of databases, and in particular to a data storage device, a data query method, an apparatus, a server, and a storage medium. Background Art

[0002] A database can be considered as a place for storing electronic files, where users can add, query, update, delete, and perform other operations on the data in the files.

[0003] In related technologies, to improve the efficiency of random data reads, data is typically stored using a combination of column storage and inverted indexing, and column data is compressed using methods such as arithmetic compression coding. When querying data, the computer caches the complete data column corresponding to the column value or row identifier included in the query criteria, and then filters the data column to obtain the query results.

[0004] However, if the data storage method in the related technology is adopted, the corresponding data columns need to be cached when querying data. When there are many data columns hit by the column values ​​or row identifiers in the query conditions, it takes a long time for the device to cache and decompress the data, the query efficiency is low, and the data processing burden of the device is heavy. Summary of the Invention

[0005] The present invention provides a data storage device, a data query method, an apparatus, a server, and a storage medium that can reduce the number of scans when querying data, improve query efficiency, and reduce the data storage resources occupied by indexes. The technical solution is as follows:

[0006] In one aspect, an embodiment of the present application provides a data storage device, wherein the data storage device stores a first column corresponding to a first data item in a data table, and row data in the data table is sorted according to the value of the first data item:

[0007] The column storage structure of the first column storage includes: a column storage unit, primary data block information, secondary data block information, an offset index and a column value index;

[0008] The column storage unit includes a column value and a starting row number corresponding to the column value;

[0009] The primary data block information includes a correspondence between a primary data block identifier, a starting row number of the primary data block, and a first starting column value index, wherein the first starting column value index is an index of a starting column value corresponding to the primary data block in the column storage unit, and the primary data block is composed of at least one secondary data block;

[0010] The secondary data block information includes a correspondence between a secondary data block identifier, a primary data block identifier of the primary data block to which it belongs, a starting row number of the secondary data block, and a second starting column value index, wherein the second starting column value index is an index of a starting column value corresponding to the secondary data block in the column storage unit, and the secondary data block is composed of at least one of the column storage units;

[0011] The offset index includes a correspondence between the primary data block identifier and the starting row number of the primary data block;

[0012] The column value index includes the correspondence between the primary data block identifier and the primary data block starting column value.

[0013] On the other hand, an embodiment of the present application provides a data query method for querying data from the data storage device described in the above aspect, the method comprising:

[0014] Receive a data query request, wherein the data query request includes a query condition, and the query condition is a target column value or a target row number;

[0015] In response to the data query request for querying the first data item, determining a target primary data block that meets the query condition based on the offset index or the column value index;

[0016] Determining a target secondary data block that meets the query condition from the target primary data block;

[0017] Determining a target column storage unit that meets the query condition from the target secondary data block;

[0018] A target query content is determined from the target column storage unit, where the target query content is the row number to which the target column value belongs, or the column value belonging to the target row number.

[0019] On the other hand, an embodiment of the present application provides a data query device for querying data from the data storage device described in the above aspect, the device comprising:

[0020] A receiving module, configured to receive a data query request, wherein the data query request includes a query condition, and the query condition is a target column value or a target row number;

[0021] a first determining module, configured to, in response to the data query request for querying the first data item, determine a target primary data block that meets the query condition based on the offset index or the column value index;

[0022] A second determining module is configured to determine a target secondary data block that meets the query condition from the target primary data block;

[0023] A third determining module is configured to determine a target column storage unit that meets the query condition from the target secondary data block;

[0024] The fourth determining module is configured to determine a target query content from the target column storage unit, where the target query content is the row number to which the target column value belongs, or the column value belonging to the target row number.

[0025] On the other hand, an embodiment of the present application provides a server, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the data query method as described in the second aspect above.

[0026] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set. The at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the data query method as described in the second aspect above.

[0027] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a server reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the server to perform the data query method provided in various optional implementations of the above aspects.

[0028] The beneficial effects of the technical solutions provided by the embodiments of the present application include at least:

[0029] In the embodiment of the present application, since the column storage units are stored in sorted order, similar column values ​​are stored together. When the value range corresponding to the query condition is wide during a random query, the number of scans is reduced and the query efficiency is improved. In addition, two levels of data blocks are divided based on the column storage units. Each level of data block corresponds to data block information, which can indicate the starting position, starting value and inclusion relationship of the data block with other levels of data blocks, and set an index for the first-level data block from the perspectives of row number and column value, so that the server can locate the data block level by level according to the query condition and index, and finally determine the data that meets the query condition from the data block, thereby further improving the data query efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the data storage structure and data query process provided by related technologies;

[0031] Figure 2is a schematic diagram of a data storage structure and a data query process provided according to an exemplary embodiment of the present application;

[0032] Figure 3 is a schematic diagram of a data storage structure of a first data item provided according to an exemplary embodiment of the present application;

[0033] Figure 4 is a schematic diagram of a first data item writing process provided according to an exemplary embodiment of the present application;

[0034] Figure 5 is a schematic diagram of a first data item writing process provided according to another exemplary embodiment of the present application;

[0035] Figure 6 is a schematic diagram of a data storage structure of a second data item provided according to an exemplary embodiment of the present application;

[0036] Figure 7 is a flow chart of a data query method provided according to an exemplary embodiment of the present application;

[0037] Figure 8 is a schematic diagram of a data query process provided according to an exemplary embodiment of the present application;

[0038] Figure 9 is a schematic diagram of a data query process provided according to another exemplary embodiment of the present application;

[0039] Figure 10 is a flow chart of a data query method provided according to another exemplary embodiment of the present application;

[0040] Figure 11 This is a data query performance comparison chart of the data storage structure provided by the embodiment of the present application and the data storage structure of the related art;

[0041] Figure 12 This is a comparison chart of CPU usage between the data storage method provided by the embodiment of the present application and the data storage method of the related art;

[0042] Figure 13 This is a comparison chart of network I / O volume between the data storage method provided by the embodiment of the present application and the data storage method of the related art;

[0043] Figure 14 is a histogram of data query memory allocation provided according to an exemplary embodiment of the present application;

[0044] Figure 15 It is a histogram of data query memory allocation provided by a data query method in the related art;

[0045] Figure 16is a schematic diagram of a data compression result of a second data item provided according to an exemplary embodiment of the present application;

[0046] Figure 17 3. This is a schematic diagram of the data read cache ratio of the second data item provided according to an exemplary embodiment of the present application;

[0047] Figure 18 is a schematic diagram of query time consumption for a second data item provided according to an exemplary embodiment of the present application;

[0048] Figure 19 is a structural block diagram of a data query device provided according to an exemplary embodiment of the present application;

[0049] Figure 20 It is a structural block diagram of a server provided according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0051] In this document, "plurality" refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0052] In the related art, the data storage device uses a combination of inverted index and column storage to store primary key data items. The data storage device stores the values ​​of the primary key data items and the row identifiers of each value in the column storage. During the data query process, the data storage device determines the hit column storage and the column storage of the target data corresponding to the query result through inverted retrieval based on the values ​​of the primary key data items contained in the query conditions and the values ​​of other data items, and caches each column storage into the memory. According to the values ​​of each data item in the query conditions, a corresponding row identifier list is obtained. Since the column storage usually uses fixed-length encoding to store data, the position of the target data item in the target column storage can be determined based on the row identifier list, thereby extracting the target data item.

[0053] like Figure 1As shown, the query condition is to find rows that meet the primary key data items (expid) of 71000 and 79289, the first indicator data item (sitesetid) of 25, and the second indicator data item (producttypeid) of 12. The target query content is the column value of the target data item (bucketid) corresponding to the rows that meet the query condition. Based on the query condition, the server queries the data storage device for the expid column store 101, the sitesetid column store 102, the producttypeid column store 103, and the bucketid column store 105. Because the expid and bucketid of the query results need to be sent to the upper-level computing engine, the server caches the expid column store 101 and the bucketid column store 105 in memory and takes the intersection of the row identifiers corresponding to the expid, sitesetid, and producttypeid in the query condition to obtain a row identifier list 104. Based on the row identifier list 104, the server extracts the expid from the in-memory column store 101 and the bucketid from the column store 105.

[0054] However, if the data storage structure for primary key data items in related technologies is adopted, the data storage device must cache the complete expid column and bucketid column in memory before retrieving the data items. Furthermore, when the query conditions involve a large number of data items, the rows where the data items are located are spread across a wide range and are scattered, requiring multiple scans to find the corresponding row identifiers. This results in a large amount of random input / output (I / O), resulting in low data query efficiency. Furthermore, this encoding method has a low compression rate and requires a large amount of storage space.

[0055] In order to solve the above technical problems, the present application provides a data storage device and a data query method. Figure 2As shown, in the data storage device, expids are arranged in sequence and stored in a first column store 201. Each column store cell (valuemeta) in the first column store contains a corresponding column value (value) and a start row identifier (startrowid). The data storage device divides the first column store 101 into first-level data blocks (rounds), and further divides each first-level data block into second-level data blocks (blocks), with each first-level data block corresponding to at least one second-level data block. The data storage device stores second-level data block information and first-level data block information, as well as a column value index (ValueIndex) and an offset index (OffsetIndex) containing the correspondence between first-level data blocks and column store cells, facilitating the server to quickly locate the column store cell corresponding to the query condition when querying data. Based on the query condition, the server determines the first column store 201, the column store 202 for sitesetid, the column store 203 for producttypeid, and the column store 205 for bucketid. The server then takes the intersection of the row identifiers corresponding to the expid, sitesetid, and producttypeid in the query condition to obtain a row identifier list 204. During this process, the server only needs to determine the target primary data block (target round) in the first column store based on the column value and column value index in the query condition, and determine the target secondary data block (target block) based on the primary data block information and the secondary data block information. The target secondary data block is cached in memory, and the row identifier (rowrange) of the target column value is expanded. The column value of bucketid is then determined based on the obtained row identifier. In the embodiment of the present application, the server only needs to determine a small number of target secondary data blocks to obtain the query result. The server does not need to scan the first column store multiple times, which improves data query efficiency. In addition, the server only needs to cache the target secondary data block and does not need to cache the entire first column store, which reduces the memory occupied by data queries.

[0056] The following describes the data storage structure in the data storage device according to an embodiment of the present application.

[0057] The data storage device stores the first column (expid) corresponding to the first data item in the data table, and the row data in the data table is sorted according to the value of the first data item. Figure 3 As shown, the data storage device stores a first column 301 .

[0058] In one possible implementation, the values ​​of the first data item in the data storage device are stored in a certain order, such as from smallest to largest. Since other data items in the data table have a logical correspondence with the first data item, the row data in the data table is stored in a corresponding order according to the value of the first data item.

[0059] For illustration, please refer to Figure 4, which shows a process of writing a first data item. The original data of the first data item in the data table is 0,7,1,1,0,8,8,7,7,8,7,8,8,8,7. The data storage device first sorts the values ​​of the first data item from small to large, obtaining the sorted first data item [0,0,1,1,7,7,7,7,7,7,8,8,8,8,8,8].

[0060] like Figure 3 As shown, the column storage structure of the first column storage 301 includes: a column storage unit, primary data block information 302 , secondary data block information 303 , an offset index (OffsetIndex) 304 and a column value index (ValueIndex) 305 .

[0061] The column storage unit includes the column value and the starting row number corresponding to the column value. The data storage device divides the first column storage into primary data blocks, and divides the primary data blocks into at least one secondary data block. The offset index 304 and the column value index 305 are used to index the primary data blocks.

[0062] The column storage unit is a basic data unit for storing the first data item. Since the first data items are sorted and result in a concentration of data items of equal size, the data storage device uses the column storage unit to represent the first data items of equal size and their corresponding storage locations in order to reduce the storage space required for the first data items.

[0063] Optionally, the column storage unit includes a column value and a starting row number corresponding to the column value, and the starting row number is used to indicate the row where the first first data item corresponding to the column value is located.

[0064] Indicative, such as Figure 4 As shown, the sorted first data items are written into a new storage in the form of column storage units, and a column storage structure 401 of the first column storage is generated. The column storage units are [0, 0], [1, 2], [7, 4], [8, 10], and different column storage units store first data items with different values ​​and are arranged according to the values ​​of the first data items.

[0065] The first-level data block information includes the correspondence between the first-level data block identifier (roundid), the first-level data block starting row number (baserowid) and the first starting column value index (startvalueindex). The first starting column value index is the index of the starting column value corresponding to the first-level data block in the column storage unit. The first-level data block is composed of at least one second-level data block.

[0066] In a possible implementation, in order to quickly search for a column storage unit, the data storage device divides the column storage unit into two levels of data blocks and uses data block information for indexing.

[0067] Optionally, the data storage device divides a first column into a first-level data block, further divides the first-level data block into second-level data blocks, and simultaneously generates first-level data block information and second-level data block information. For two adjacent first-level data blocks, the last row of the previous first-level data block is adjacent to the last row of the next first-level data block. The first-level data block information includes the first-level data block starting row number and the first starting column value index, facilitating the data storage device to query data based on row number or column value.

[0068] Indicative, such as Figure 4 As shown, for the first data item [0,0,1,1,7,7,7,7,7,7,8,8,8,8,8,8], it is divided into a first-level data block, and the corresponding first-level data block information is [0,0,0], that is, the 0th first-level data block, starting at the 0th row, and the first column value belongs to the 0th column value unit.

[0069] The secondary data block information includes the correspondence between the primary data block identifier, the secondary data block identifier (blockid), the secondary data block starting row number (blockrowid) and the second starting column value index (startvalueindex) of the primary data block to which it belongs. The second starting column value index is the index of the corresponding starting column value of the secondary data block in the column storage unit. The secondary data block is composed of at least one column storage unit.

[0070] In a possible implementation, since the primary data block contains a large number of column storage units and the column storage units cannot be efficiently queried, the data storage device divides the primary data block into a predetermined number of column storage units to obtain at least one secondary data block.

[0071] Optionally, the data storage device divides the primary data block into 16k column storage units per secondary data block; or the division method is dynamically adjusted based on the number of column storage units in the primary data block, which is not limited in the embodiments of the present application. To enable the data storage device to efficiently query data, the secondary data information it generates includes the primary data block identifier of the primary data block to which it belongs, thereby establishing an index between the primary data block and the secondary data block. In addition, the secondary data block information also includes the starting row number and starting column value index of the secondary data block, which facilitates the data storage device to query data based on the row number or column value.

[0072] Indicative, such as Figure 4As shown, the data storage device divides the primary data blocks into three column storage units per secondary data block. Therefore, the 0th secondary data block contains the first three column storage units, and the secondary data block information is [0, 0, 0, 0]. The 1st secondary data block contains the last column storage unit, and the secondary data block information is [0, 1, 10, 3]. That is, the primary data block identifier of the corresponding primary data block is 0, the starting row number of the first column value of the first secondary data block in the corresponding primary data block is 10, and the first column value belongs to the 3rd column value unit.

[0073] The offset index contains the correspondence between the first-level data block identifier and the starting row number of the first-level data block. The column value index contains the correspondence between the first-level data block identifier and the starting column value of the first-level data block.

[0074] In one possible embodiment, in order to facilitate the rapid positioning of the first-level data blocks and avoid the situation where scanning the first-level data blocks results in a long query time, the data storage device also stores an index for the first-level data blocks, including an offset index that indexes the first-level data blocks from the row number, and a numerical index that indexes the first-level data blocks from the column value.

[0075] Indicative, such as Figure 4 As shown, the offset index corresponding to the 0th level-1 data block is [0,0], that is, the 0th level-1 data block, the starting row number of the level-1 data block is 0; the numerical index corresponding to the 0th level-1 data block is [0,0], that is, the 0th level-1 data block, the starting numerical value of the level-1 data block is 0.

[0076] To sum up, in the embodiment of the present application, since the column storage units are stored in sorted order, similar column values ​​are stored together. When the value range corresponding to the query condition is wide during the random query process, the number of scans is reduced and the query efficiency is improved; and two levels of data blocks are divided based on the column storage units, and each level of data block corresponds to data block information, which can indicate the starting position, starting value and inclusion relationship of the data block with other levels of data blocks, and set indexes for the first-level data blocks from the perspectives of row number and column value, so that the server can locate the data blocks level by level according to the query conditions and indexes, and finally determine the data that meets the query conditions from the data blocks, thereby further improving the data query efficiency.

[0077] In another possible implementation, a data table contains multiple first data items, and the data storage device creates an index for each first data item and sorts the first data items according to their sorting priorities. For example, first data item a with a high ranking is sorted first, and then first data item b with a lower ranking is sorted based on the sorted first data item a, and the position of first data item a is adjusted accordingly.

[0078] For illustration, please refer to Figure 5, which shows a process of writing two first data items into a data table. The data storage device first sorts the first data item a with a higher ranking, transforming the original data of the first data item a [0,7,1,1,0,8,8,7,7,8,7,8,8,7] into the sorted first data item a [0,0,1,1,7,7,7,7,7,7,8,8,8,8,8]. Then, the first data items b corresponding to the same values ​​in the first data item a are sorted in ascending order, transforming the original data of the first data item b [1,2,2,2,2,1,1,1,2,1,1,4,1,1,4,3] into the sorted first data item b [1,2,2,2,1,1,1,2,3,4,1,1,1,1,4,4].

[0079] Optionally, during the process of querying data, the data storage device usually queries the first data item of lower level based on the query result of the first data item of higher level. Therefore, when storing data, the data storage device first indexes the first data item of higher level, and then divides and indexes the first data item of lower level based on the data blocks of the first data item of higher level.

[0080] Indicative, such as Figure 5 As shown, after obtaining the column storage structure 401 for the first data item a, the data storage device establishes a column storage structure 501 for the first data item b based on the column storage structure 401. The data storage structure establishes the column storage units for the first data item b based on the column storage units of the first data item a. That is, first data items b corresponding to different column values ​​of the first data item a are located in different column storage units, and first data items b with different values ​​are also located in different column storage units. For example, first data items b corresponding to first data item a column values ​​of 0, 1, 7, and 8 are stored in different column storage units, while first data items b corresponding to first data item a column values ​​of 0, 1, and 2 are stored in different column storage units.

[0081] Optionally, the data storage device divides the primary data blocks according to the column value of the first data item a corresponding to the first data item b. Figure 5 As shown in FIG, since the first data item a contains 4 column values, the first data item b corresponds to 4 primary data blocks. Similar to the first data item a, the data storage device divides the primary data blocks according to the predetermined number of column storage units to obtain secondary data blocks. For example, Figure 5 As shown, the data storage device divides the primary data blocks into three column storage units per secondary data block. For the second primary data block [2,4,4], since it contains four column storage units, it is divided into two secondary data blocks. Similarly, the data storage device establishes an offset index and a value index for the first column storage b of the first data item b.

[0082] In a possible implementation, the data storage device further stores a second column corresponding to the second data item in the data table.

[0083] In some embodiments, the first data item is a primary key data item in a data table, and the second data item is a data item other than the primary key data item in the data table, such as a dimension item or an indicator item. Compared to the first data item, the second data item has a larger cardinality, and the data storage device classifies and compresses the second data item based on its characteristics.

[0084] The column storage structure of the second column storage includes: metadata (metadata) and coded data (blockdata), the metadata is composed of at least one metadata block (metablock), and the coded data is composed of at least one coded data block (datablock), and the metadata block corresponds to the coded data block one by one.

[0085] The coded data in the coded data block is obtained by compressing the original data according to the compression type.

[0086] In one possible implementation, a data storage device divides the second data item, sorted according to the values ​​of the first data item, into at least one data block according to a predetermined data volume. The data block is then compressed using a corresponding compression code based on characteristics of the data in each data block to obtain an encoded data block. To facilitate querying the second data item, the data storage device stores metadata corresponding to the encoded data.

[0087] The metadata block contains the encoding block meta information of the corresponding encoding data block, and the encoding block meta information includes the encoding block offset (offset), compressed size (compressedsize) and compression type (compressedtype).

[0088] The coding block offset is used to indicate the starting position of the coding data block in the second column memory, and the compressed size is the size of the coding data block.

[0089] Optionally, the encoding block meta information further includes the data type of the encoding data, which is used to indicate the data type of the original data in the encoding data block, such as integer type (int), long integer type (long) and short integer type (short).

[0090] In one possible implementation, since the data storage device divides the second column into storage units according to a predetermined data size, for example, each 16k rows is a coded data block, the server needs to determine the target coded data block based on the target row number and the block size of the coded data block in the coded data. For example, if the target row number is 6 and the data storage device divides each 3 rows into a coded data block, i.e., the block size of the coded data block is 3 rows, the target coded data block is determined to be the second coded data block.

[0091] Optionally, to facilitate rapid location and decompression of the original data during subsequent data queries, the data storage device indicates the starting position of the encoded data block using the encoded block offset in the encoded block meta-information and indicates the data size of the encoded data block using the compressed size. This allows the server to determine the starting and ending positions of the encoded data block in the second column store based on the encoded block offset and compressed size, and decompress the encoded data block based on the compression type in the encoded block meta-information. Optionally, the encoded block meta-information also includes the data type of the original data, facilitating computational processing of the original data based on the data type after the server obtains the original data.

[0092] For illustration, please refer to Figure 6 , which shows a column memory structure of a second column memory. The column memory structure includes coded data 601 and metadata 602, and the coded data blocks of the coded data 601 correspond to the metadata blocks of the metadata 602. The metadata 602 includes coded block meta information 603 corresponding to the coded data blocks. The coded block meta information 603 includes a coded block offset, a compressed size, a compression type, and a data type (datatype).

[0093] In an embodiment of the present application, by setting metadata corresponding to the encoded data of the second data item, the encoded block metadata of the encoded data block is stored, indicating the position and compression type of each encoded data block in the second column memory, so that the data storage device can quickly query the storage location of the second data item, and the second data item is stored in blocks, so that the data storage device only needs to cache and decompress the corresponding encoded data block to obtain the second data item without having to decompress the entire second column memory, and the compression method of each encoded data block can be known based on the metadata, thereby improving the efficiency of data decompression.

[0094] Please refer to Figure 7 , which shows a flow chart of a data query method provided by an exemplary embodiment of the present application. This embodiment is described using the method for a server with a data query function as an example. The method includes the following steps:

[0095] Step 701: Receive a data query request, where the data query request includes a query condition, which is a target column value or a target row number.

[0096] In one possible implementation, a server receives a data query request, which includes a query condition and a target query content. The query condition is a target column value or a target row number, and the target query content is a column value or a row number corresponding to a desired data item. Optionally, the data query request also includes a target data item, which is the data item to which the target query content belongs.

[0097] Optionally, the server queries the row number corresponding to the target column value and the row number or column value of other corresponding data items based on the target column value in the query condition; or queries the column value corresponding to the target row number and the row number or column value of other corresponding data items based on the target row number in the query condition.

[0098] Optionally, the server is the data storage device in the above embodiment, or a data query server connected to the data storage device.

[0099] Step 702 : In response to a data query request for querying a first data item, a target primary data block that meets a query condition is determined based on an offset index or a column value index.

[0100] In a possible implementation, when the query content in the data query request is the first data item, the query is performed using the offset index or column value index of the first data item and the data block information of each level of the first data item.

[0101] Optionally, the server selects a corresponding index according to a query condition in the data query request to determine a target first-level data block.

[0102] In one possible implementation, step 702 includes the following steps:

[0103] Step 1: In response to the query condition being a target column value, a target primary data block that meets the query condition is determined based on a column value index.

[0104] In one possible implementation, the data query request is to query the row number to which the target column value belongs based on the target column value of the first data item. Because the first data item is stored in blocks in the data storage device and a column value index containing a correspondence between a first-level data block identifier and a first-level data block starting column value is stored, the data storage device first traverses the column value index based on the target column value to determine the target first-level data block to which the target column value belongs.

[0105] Step 2: In response to the query condition being a target row number, a target primary data block that meets the query condition is determined based on the offset index.

[0106] In one possible implementation, the data query request is to query the column value corresponding to the target row number of the first data item based on the target row number. Because the first data item is stored in blocks in the data storage device and an offset index is stored for indexing the first-level data blocks based on the row number, the data storage device first traverses the offset index based on the target row number to determine the target first-level data block to which the target row number belongs.

[0107] Step 703: Determine a target secondary data block that meets the query condition from the target primary data blocks.

[0108] In a possible implementation, in order to further determine the position of the target column value or the column value corresponding to the target row number, the data storage device filters out the target secondary data blocks that meet the query conditions from the target primary data blocks.

[0109] Optionally, the data storage device determines at least one secondary data block included in the target primary data block based on the secondary data block information. After determining the target secondary data block, the data storage device caches the target secondary data block in memory so that the corresponding column storage unit can be subsequently expanded to obtain the target query content.

[0110] Step 704: Determine a target column storage unit that meets the query condition from the target secondary data block.

[0111] In one possible implementation, when the query condition is a target column value, the data storage device determines the target column storage unit contained in the target column value from the target secondary data block based on the target column value and the column values ​​contained in each column storage unit; when the query condition is a target row number, the data storage device determines the target column storage unit corresponding to the target row number from the target secondary data block based on the target row number and the starting row number contained in each column storage unit.

[0112] Step 705: Determine the target query content from the target column storage unit. The target query content is the row number to which the target column value belongs, or the column value belonging to the target row number.

[0113] In a possible implementation, when the data storage device determines a target column storage unit that meets the query condition, the target column storage unit is expanded from a cached target secondary data block to obtain the target query content.

[0114] Optionally, when the query condition is the target column value, the data storage device obtains the row number to which the target column value belongs based on the starting row number contained in the target column storage unit; when the query condition is the target row number, the data storage device obtains the column value of the target row number based on the column value corresponding to the target column storage unit.

[0115] In an embodiment of the present application, the server determines the target primary data block that meets the query conditions based on the query conditions and the offset index or column value index, and further determines the target secondary data block through binary search. Since the first data items are stored in sorted order, the column storage units corresponding to the query conditions are relatively concentrated, and only a few target secondary data blocks need to be determined to obtain the query results. The server does not need to scan the first column storage multiple times, thereby improving data query efficiency; and the server only needs to cache the target secondary data block, and does not need to cache the complete first column storage, thereby reducing the memory occupied by data queries.

[0116] In a possible implementation, when the query condition is the target column value of the first data item and the target query content is the row number to which the target column value belongs, steps 702 to 705 of the above data query method include the following steps:

[0117] In one possible implementation, step 702 includes step 702a:

[0118] Step 702a, determine the target first-level data block based on the first-level data block starting column value and the target column value in the column value index, wherein the first-level data block starting column value of the target first-level data block is less than or equal to the target column value, and the first-level data block starting column value of the next first-level data block corresponding to the target first-level data block is greater than or equal to the target column value.

[0119] In a possible implementation, when the query condition is the target column value of the first data item, the server obtains the column value index of the first-level data block.

[0120] Optionally, since the column value index includes the primary data block identifier and the primary data block starting column value, the server can traverse the column value index according to the target column value to determine the target primary data block to which the target column value belongs.

[0121] exist Figure 4 The first column is shown for reference only. Figure 8 , which shows a schematic diagram of querying the row number of a target column value based on the target column value 1. In the query condition, the target column value is 1. The server traverses the column value index. Since the starting column value of level 1 data block 0 is 0 and the starting column value of level 1 data block 1 is 9, level 1 data block 0 is determined to be the target level 1 data block.

[0122] In a possible implementation, step 703 includes step 703a and step 703b:

[0123] Step 703a: Obtain the second starting column value index of each secondary data block under the target primary data block from the secondary data block information.

[0124] In a possible implementation, since the secondary data block information includes the primary data block identifier of the primary data block to which each secondary data block belongs, the server filters out the secondary data blocks under the target primary data block from the secondary data block information according to the primary data block identifier of the target primary data block. Figure 8 As shown, according to the first-level data block identifier 0 of the target first-level data block, it is determined that the second-level data blocks under the target first-level data block include the second-level data block 0 and the second-level data block 1.

[0125] Optionally, the server obtains the second starting column value index of each secondary data block from the secondary data block information. Figure 8As shown, the second starting column value index of the secondary data block 0 is 0, and the second starting column value index of the secondary data block 1 is 3.

[0126] Step 703b, determine the target secondary data block by binary search based on the column value corresponding to the second starting column value index, wherein the secondary data block starting column value of the target secondary data block is less than or equal to the target column value, and the secondary data block starting column value of the next secondary data block corresponding to the target secondary data block is greater than or equal to the target column value.

[0127] In one possible implementation, the server uses a binary search method to determine the target secondary data block. Specifically, the server first determines the middle secondary data block, then compares the column value corresponding to the starting column value index of the secondary data block with the target column value to determine whether the target column value belongs to the secondary data block corresponding to the first half of the primary data blocks, or the secondary data block corresponding to the second half of the primary data blocks, and so on, until the target secondary data block is determined.

[0128] like Figure 8 As shown, since the column value corresponding to the starting column value index of secondary data block 0 is 0, which is less than the target column value, and the column value corresponding to the starting column value index of secondary data block 1 is 8, which is greater than the target column value, secondary data block 0 is determined to be the target secondary data block.

[0129] In one possible implementation, step 704 includes step 704a:

[0130] Step 704a: Determine the target column storage unit by binary search based on the column values ​​corresponding to the column storage units in the target secondary data block. The column value corresponding to the target column storage unit is the target column value.

[0131] In a possible implementation, the server also adopts a binary method to determine the target column storage unit. Figure 8 As shown, since the column value corresponding to the column storage unit 1 is 1, which is equal to the target column value, the column storage unit 1 is determined to be the target column storage unit.

[0132] In a possible implementation, step 705 includes step 705a and step 705b:

[0133] Step 705a: Obtain a first starting row number corresponding to the column value in the target column storage unit, and a second starting row number corresponding to the column value in the next column storage unit after the target column storage unit.

[0134] In one possible implementation, since the column storage unit only contains the column value and the starting row number, but does not contain the ending row number of the column storage unit, the server needs to determine the range of the target column value, that is, the row number to which the target column value belongs, based on the first starting row number of the target column storage unit and the second starting row number corresponding to the column value in the next column storage unit after the target column storage unit.

[0135] like Figure 8 As shown, when the server determines that column storage unit 1 is the target column storage unit, it obtains the starting row number 2 of column storage unit 1 and the starting row number 4 of column storage unit 2.

[0136] Step 705b: Determine the row number to which the target column value belongs according to the first starting row number and the second starting row number.

[0137] In a possible implementation, the server determines the starting row number of the target column value based on the first starting row number, and determines the ending row number of the target column value based on the second starting row number, that is, the ending row number of the target column value is the row number corresponding to the row before the second starting row number. Figure 8 As shown, the first starting row number is 2 and the second starting row number is 4, so the row numbers to which the target column value belongs are determined to be 2 and 3.

[0138] In another possible implementation, when the query condition is the target row number of the first data item, steps 702 to 705 of the above data query method include the following steps:

[0139] In one possible implementation, the server queries the corresponding column value based on the target row number of the first data item in the query condition. In the actual data query process, the server usually queries the row number of the first primary key data based on the target column value of the first data item in the query condition, and then queries the corresponding column value of the second primary key data based on the row number. Figure 9 In the embodiment of the present application, querying the column value of the second primary key data according to the row number of the first primary key data is used as an example for explanation.

[0140] In one possible implementation, step 702 includes step 702b:

[0141] Step 702b: Determine the target first-level data block according to the starting row number of the first-level data block and the target row number in the offset index, wherein the target row number belongs to the row number range of the target first-level data block.

[0142] In one possible implementation, after the server queries the target row number corresponding to the target column value in the first primary key data, it traverses the offset index stored in the first column corresponding to the second primary key data according to the row number to determine the target primary data block to which the target row number belongs.

[0143] Indicative, such as Figure 9 As shown, since the target row numbers are 2 and 3, the starting row number of the first-level data block 1 is 2, and the starting row number of the first-level data block 2 is 4, the first-level data block 1 is determined to be the target first-level data block.

[0144] In a possible implementation, step 703 includes step 703c and step 703d:

[0145] Step 703c: Obtain the secondary data block starting row number of each secondary data block under the target primary data block from the secondary data block information.

[0146] In a possible implementation, since the secondary data block information includes the primary data block identifier of the primary data block to which each secondary data block belongs, the server filters out the secondary data blocks under the target primary data block from the secondary data block information according to the primary data block identifier of the target primary data block. Figure 9 As shown, according to the first-level data block identifier 1 of the target first-level data block, it is determined that the second-level data blocks under the target first-level data block include the second-level data block 1.

[0147] Optionally, the server obtains the secondary data block starting row number of each secondary data block from the secondary data block information. Figure 9 As shown, the secondary data block starting row number of the secondary data block 1 is 2.

[0148] Step 703d: Determine the target secondary data block according to the starting row number of the secondary data block, wherein the target row number belongs to the row number range of the target secondary data block.

[0149] In one possible implementation, the server uses a binary search method to determine the target secondary data block. Specifically, the server first determines the middle secondary data block, then compares the starting row number of the secondary data block with the target row number to determine whether the target row number belongs to the secondary data block corresponding to the first half of the primary data blocks, or the secondary data block corresponding to the second half of the primary data blocks, and so on, until the target secondary data block is determined.

[0150] like Figure 9 As shown, since the target primary data block only includes one secondary data block, the secondary data block 1 is directly determined as the target secondary data block.

[0151] In one possible implementation, step 704 includes step 704b:

[0152] Step 704b: Determine the target column storage unit according to the starting row number of the column value corresponding to each column storage unit in the target secondary data block, and the target row number belongs to the row number range of the target column storage unit.

[0153] In a possible implementation, the server also adopts a binary method to determine the target column storage unit.

[0154] like Figure 9 As shown, since the starting row number of column memory unit 2 is 2 and the starting row number of column memory unit 3 is 4, which are greater than the target row number, column memory unit 2 is determined to be the target column memory unit.

[0155] In one possible implementation, step 705 includes step 705c:

[0156] Step 705c: extract the column value corresponding to the target row number from the target column storage unit.

[0157] In a possible implementation, since a large number of target row numbers may hit multiple target column storage units, the server determines the target second data block and caches the target second database block, expands the target column storage, and extracts the corresponding column value according to the target row number. Figure 9 As shown, since the row number range included in the target column storage unit is 2 and 3, the column value 2 in the target column storage unit is determined as the column value corresponding to the target row number.

[0158] Since the compression storage method of the second data item is different from that of the first data item, when a data query request is used to query the second data item, the server queries the corresponding encoded data according to the data query request, and decompresses the encoded data according to the metadata to obtain the original data. Figure 7 Based on the Figure 10 After step 701, the data query method further includes the following steps:

[0159] Step 706 : In response to the data query request for querying the second data item, cache a second column corresponding to the second data item.

[0160] In one possible implementation, the data storage device stores the second data item in columns, that is, a column of second data items is compressed and stored centrally. Therefore, when the server queries the second data item, it needs to cache the second column corresponding to the second data item in the memory.

[0161] Optionally, because some second data items are frequently queried, to avoid needing to cache the second column memory for every query, the server does not immediately clear the second column memory in memory after the query completes. Instead, it temporarily stores the second column memory in memory. When the time since the last query for the second data item in the second column memory reaches a predetermined time, the server deletes the second column memory and re-caches it for the next query.

[0162] Step 707: decompress the encoded data according to the metadata stored in the second column to obtain the original data.

[0163] In a possible implementation, since the data storage device stores the second column in blocks and compression methods of the coded data blocks may be different, the server decompresses the coded data according to the coded block meta information included in the metadata.

[0164] In one possible implementation, step 707 includes the following steps:

[0165] Step 1: Determine the target coded data block from the coded data and the target metadata block corresponding to the target coded data block in the metadata based on the target row number and the data block size of the coded data block in the coded data, where the data block size is the number of rows of the original data corresponding to the coded data block.

[0166] In one possible implementation, since the data storage device divides the second column into storage units according to a predetermined data size, for example, each 16k rows is a coded data block, the server needs to determine the target coded data block based on the target row number and the block size of the coded data block in the coded data. For example, if the target row number is 6 and the data storage device divides each 3 rows into a coded data block, i.e., the block size of the coded data block is 3 rows, the target coded data block is determined to be the second coded data block.

[0167] Optionally, the coding block meta information in the metadata includes a coding block offset, and the server obtains a corresponding target metadata block according to the target coding data block.

[0168] Step 2: extract the target coded data block from the coded data according to the coded block offset and compression size in the target metadata block.

[0169] In one possible implementation, the server determines the starting position of the target coded data block in the second column store based on the coded block offset in the target metadata block, and then determines the ending position of the target coded data block in the second column store based on the compressed size in the target metadata block, thereby extracting the target coded data block.

[0170] Step three: decompress the target coded data block according to the compression type in the target metadata block to obtain the original data.

[0171] In one possible implementation, since the compression types of different encoded data blocks may be different, the corresponding compression methods are also different. Therefore, the server needs to use the corresponding decompression method to decompress the target encoded data block according to the compression type in the target metadata block to obtain the original data.

[0172] Optionally, according to data characteristics of the encoded data block, the compression type of the second data item in the data storage device includes a first compression type and a second compression type, and the compression rate of the first compression type is higher than the compression rate of the second compression type.

[0173] Among them, if the proportion of target original data in the original data reaches a first threshold, or the proportion of cardinality in the original data is less than a second threshold, the original data is compressed using the first compression type, and the target original data is data with at least two leading bits being 0.

[0174] The cardinality refers to all the unique values ​​in the original data. For example, if the original data is [0, 1, 2, 2, 2, 4], then the cardinality of the original data is [0, 1, 2, 4].

[0175] Optionally, the first compression type is a high compression type, and the corresponding compression encoding method is a combination of a lossless compression algorithm (A. Lempel J. Ziv 4, LZ4) and a bit shuffle compression algorithm.

[0176] Illustratively, if the proportion of data in which at least two leading bits in the original data are 0 reaches 80%, or the cardinality proportion in the original data is less than 20%, the original data is compressed using the first compression type.

[0177] Optionally, when the number of cardinalities in the original data is 1, i.e., the original data in the data block is all identical, the data storage device compresses the original data using a third compression type, i.e., directly records the cardinalities and records the corresponding rows in the corresponding metadata. When the original data is all empty, the data storage device directly generates the corresponding metadata and marks the data segment as empty.

[0178] Optionally, if the proportion of cardinality in the original data is greater than the second threshold and less than the third threshold, the original data is compressed using the second compression type.

[0179] Optionally, the second compression type is normal compression, and the corresponding compression encoding method is to use only the LZ4 compression algorithm.

[0180] Illustratively, if the proportion of cardinality in the original data is greater than 20% and less than 30%, and the proportion of data with at least two leading bits being 0 is less than 80%, the second compression type is used for compression.

[0181] Optionally, the compression type of the second data item in the data storage device further includes a fourth compression type. When the original data does not meet the above data characteristics, the data storage device compresses the original data using a bit packing method.

[0182] Step 708: Search the original data for data that meets the query conditions.

[0183] In a possible implementation, the server filters the decompressed original data according to the query condition to obtain data that meets the query condition.

[0184] In an embodiment of the present application, the data storage device improves the data compression efficiency of the second data item by dividing the second data item and determining the corresponding compression method according to the data characteristics of the original data; and the data storage device stores metadata that records the metadata of the coding block, so that when the server queries the second data item, it determines the corresponding target coding data block according to the query conditions and the metadata, and only needs to decompress the target coding data block to obtain the original data, without having to decompress the complete second column storage, thereby reducing the data processing pressure of the server and improving the data query efficiency.

[0185] For illustration, please refer to Figures 11 to 18 , which shows the query performance comparison results of the data query method in the embodiment of the present application and the data query method in the related art. Figure 11 As shown, the query time of the new storage (i.e., the data storage structure in the embodiment of the present application) is much shorter than the query time of the original storage (i.e., the data storage structure in the related art). Moreover, when the amount of data increases, the query time of the new storage increases slightly, while the query time of the original storage increases significantly and the increase is unstable. Therefore, within the same data storage device, the data storage structure used in the embodiment of the present application can support a larger data scale.

[0186] like Figure 12 As shown in the figure, it shows the CPU usage of the data storage device using two data storage structures when querying data. Among them, 23:03 to 23:11 is the execution time of the query of the newly stored data, and the CPU usage is the highest at 11%; 23:11 to 23:38 is the execution time of the query of the original stored data, and the CPU usage is the highest at 66%. Figure 13 Figure 2 shows the network I / O usage of data storage devices using two different data storage structures when querying data. The query execution time for the new storage data was from 23:03 to 23:11, with a network I / O volume of 30MB. The query execution time for the original storage data was from 23:11 to 23:38, with an inbound flow of 736MB and an outbound flow of 2.9GB. This shows that compared to the original storage, the new storage has significantly reduced CPU usage and network I / O.

[0187] like Figure 14 and Figure 15 Figure 2 shows the memory allocation for the first data item query in the new and original storage. The new storage completed 100 queries in 7 minutes, with a stable memory allocation of around 10GB. The original storage, on the other hand, executed one query in 4 minutes, which was not completed. The memory allocation reached 22GB, and there was a long garbage collection pause, requiring a period of waiting before the next round of iterative computation could continue.

[0188] like Figure 16 As shown in FIG, which shows the data compression rate test results of the second data item in the embodiment of the present application, compared with the original data, the data size of the compressed second data item is reduced by at least 30% and at most 65%. Figure 17 and Figure 18 As shown, it shows the data query cache ratio and the first query time of the second data item in the embodiment of the present application. It can be seen that compared with the fourth compression type, the data of the first compression type and the second compression type have a smaller increase in cache ratio and a smaller increase in query time as the amount of data read by the server increases.

[0189] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0190] Please refer to Figure 19 , which shows a structural block diagram of a data query device provided by an embodiment of the present application. The device may include: a receiving module 1901, a first determining module 1902, a second determining module 1903, a third determining module 1904, and a fourth determining module 1905.

[0191] Receiving module 1901, configured to receive a data query request, wherein the data query request includes a query condition, and the query condition is a target column value or a target row number;

[0192] A first determining module 1902 is configured to, in response to the data query request for querying the first data item, determine a target primary data block that meets the query condition based on the offset index or the column value index;

[0193] A second determining module 1903 is configured to determine a target secondary data block that meets the query condition from the target primary data block;

[0194] A third determining module 1904 is configured to determine a target column storage unit that meets the query condition from the target secondary data block;

[0195] The fourth determining module 1905 is configured to determine a target query content from the target column storage unit, where the target query content is the row number to which the target column value belongs, or the column value belonging to the target row number.

[0196] Optionally, the first determining module 1902 is further configured to:

[0197] In response to the query condition being the target column value, determining the target primary data block that meets the query condition based on the column value index;

[0198] In response to the query condition being the target row number, the target primary data block that meets the query condition is determined based on the offset index.

[0199] Optionally, the first determining module 1902 is further configured to:

[0200] Determine the target first-level data block according to the first-level data block starting column value and the target column value in the column value index, wherein the first-level data block starting column value of the target first-level data block is less than or equal to the target column value, and the first-level data block starting column value of the next first-level data block corresponding to the target first-level data block is greater than or equal to the target column value;

[0201] The second determining module 1903 is further configured to:

[0202] Acquire a second starting column value index of each secondary data block under the target primary data block from the secondary data block information;

[0203] Determine the target secondary data block by binary search based on the column value corresponding to the second starting column value index, wherein the secondary data block starting column value of the target secondary data block is less than or equal to the target column value, and the secondary data block starting column value of the next secondary data block corresponding to the target secondary data block is greater than or equal to the target column value;

[0204] The third determining module 1904 is further configured to:

[0205] According to the column values ​​corresponding to the column storage units in the target secondary data block, the target column storage unit is determined by a binary search method, and the column value corresponding to the target column storage unit is the target column value;

[0206] The fourth determining module 1905 is further configured to:

[0207] Obtaining a first starting row number corresponding to a column value in the target column storage unit and a second starting row number corresponding to a column value in a column storage unit next to the target column storage unit;

[0208] The row number to which the target column value belongs is determined according to the first starting row number and the second starting row number.

[0209] Optionally, the first determining module 1902 is further configured to:

[0210] Determine the target first-level data block according to the first-level data block starting row number and the target row number in the offset index, wherein the target row number belongs to the row number range of the target first-level data block;

[0211] The second determining module 1903 is further configured to:

[0212] Obtaining the secondary data block starting row number of each secondary data block under the target primary data block from the secondary data block information;

[0213] Determining the target secondary data block according to the starting row number of the secondary data block, wherein the target row number belongs to the row number range of the target secondary data block;

[0214] The third determining module 1904 is further configured to:

[0215] Determine the target column storage unit according to the starting row number of the column value corresponding to each column storage unit in the target secondary data block, wherein the target row number belongs to the row number range of the target column storage unit;

[0216] The fourth determining module 1905 is further configured to:

[0217] The column value corresponding to the target row number is extracted from the target column storage unit.

[0218] Optionally, the device further includes:

[0219] a cache module, configured to cache a second column memory corresponding to the second data item in response to the data query request for querying the second data item;

[0220] a decompression module, configured to decompress the encoded data according to the metadata stored in the second column to obtain original data;

[0221] The query module is used to query the original data for data that meets the query condition.

[0222] Optionally, the decompression module is further configured to:

[0223] determining a target coded data block from the coded data and a target metadata block corresponding to the target coded data block in the metadata according to the target row number and a data block size of the coded data block in the coded data, wherein the data block size is the number of rows of original data corresponding to the coded data block;

[0224] extracting the target coded data block from the coded data according to the coded block offset and the compressed size in the target metadata block;

[0225] The target coded data block is decompressed according to the compression type in the target metadata block to obtain the original data.

[0226] Optionally, the compression type includes a first compression type and a second compression type, and the compression rate of the first compression type is higher than the compression rate of the second compression type;

[0227] If the proportion of target original data in the original data reaches a first threshold, or the proportion of cardinality in the original data is less than a second threshold, the original data is compressed using the first compression type, and the target original data is data with at least two leading bits being 0;

[0228] If the proportion of cardinality in the original data is greater than the second threshold and less than a third threshold, the original data is compressed using the second compression type.

[0229] Please refer to Figure 20 , which shows a schematic diagram of the structure of a server provided by an embodiment of the present application. Specifically:

[0230] The server 2000 includes a CPU 2001, a system memory 2004 including a random access memory (RAM) 2002 and a read-only memory (ROM) 2003, and a system bus 2005 connecting the system memory 2004 and the central processing unit 2001. The server 2000 also includes a basic I / O system 2006 that facilitates information transmission between various components within the computer, and a mass storage device 2007 for storing an operating system 2013, application programs 2014, and other program modules 2012.

[0231] The basic input / output system 2006 includes a display 2008 for displaying information and an input device 2009 such as a mouse and a keyboard for user input. The display 2008 and the input device 2009 are both connected to the central processing unit 2001 via an input / output controller 2010 connected to the system bus 2005. The basic input / output system 2006 may also include an input / output controller 2010 for receiving and processing input from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 2010 also provides output to a display screen, a printer, or other types of output devices.

[0232] The mass storage device 2007 is connected to the central processing unit 2001 via a mass storage controller (not shown) connected to the system bus 2005. The mass storage device 2007 and its associated computer-readable medium provide non-volatile storage for the server 2000. In other words, the mass storage device 2007 may include a computer-readable medium (not shown) such as a hard disk or a Compact Disc Read-Only Memory (CD-ROM) drive.

[0233] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), flash memory or other solid-state storage technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage media are not limited to the above-mentioned ones. The above-mentioned system memory 2004 and mass storage device 2007 can be collectively referred to as memory.

[0234] According to various embodiments of the present application, the server 2000 may also be connected to a remote computer on a network such as the Internet for operation. That is, the server 2000 may be connected to the network 2012 via the network interface unit 2011 connected to the system bus 2005, or the network interface unit 2011 may be used to connect to other types of networks or remote computer systems (not shown).

[0235] The memory also includes at least one instruction, at least one program, code set or instruction set, which is stored in the memory and configured to be executed by one or more processors to implement the above-mentioned data query method.

[0236] Optionally, the above-mentioned server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to these. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0237] An embodiment of the present application further provides a computer-readable storage medium, which stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the data query method described in the above embodiments.

[0238] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a server reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the server to perform the data query method provided in various optional implementations of the above aspects.

[0239] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable storage medium or transmitted as one or more instructions or codes on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0240] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A data storage device, characterized in that The data storage device stores a first column corresponding to a first data item in a data table, and the row data in the data table is sorted according to the value of the first data item: The column storage structure of the first column storage includes: a column storage unit, primary data block information, secondary data block information, an offset index and a column value index; The column storage unit includes a column value and a starting row number corresponding to the column value; The primary data block information includes a correspondence between a primary data block identifier, a starting row number of the primary data block, and a first starting column value index, wherein the first starting column value index is an index of a starting column value corresponding to the primary data block in the column storage unit, and the primary data block is composed of at least one secondary data block; The secondary data block information includes a correspondence between a secondary data block identifier, a primary data block identifier of the primary data block to which it belongs, a starting row number of the secondary data block, and a second starting column value index, wherein the second starting column value index is an index of a starting column value corresponding to the secondary data block in the column storage unit, and the secondary data block is composed of at least one of the column storage units; The offset index includes a correspondence between the primary data block identifier and the starting row number of the primary data block; The column value index includes the correspondence between the primary data block identifier and the primary data block starting column value.

2. The data storage device according to claim 1, wherein The data storage device further stores a second column corresponding to the second data item in the data table; The column storage structure of the second column storage includes metadata and coded data, wherein the metadata is composed of at least one metadata block, and the coded data is composed of at least one coded data block, and the metadata block corresponds to the coded data block in a one-to-one manner; The metadata block contains coding block meta information corresponding to the coding data block, and the coding block meta information includes coding block offset, compression size and compression type; The coded data in the coded data block is obtained by compressing the original data according to the compression type.

3. The data storage device according to claim 2, wherein: The first data item is a primary key data item in the data table, and the second data item is a data item in the data table other than the primary key data item.

4. A data query method, characterized in that: For querying data from the data storage device according to any one of claims 1 to 3, the method comprising: Receive a data query request, wherein the data query request includes a query condition, and the query condition is a target column value or a target row number; In response to the data query request for querying the first data item, determining a target primary data block that meets the query condition based on the offset index or the column value index; Determining a target secondary data block that meets the query condition from the target primary data block; Determining a target column storage unit that meets the query condition from the target secondary data block; A target query content is determined from the target column storage unit, where the target query content is the row number to which the target column value belongs, or the column value belonging to the target row number.

5. The method according to claim 4, characterized in that The determining, based on the offset index or the column value index, a target primary data block that meets the query condition includes: In response to the query condition being the target column value, determining the target primary data block that meets the query condition based on the column value index; In response to the query condition being the target row number, the target primary data block that meets the query condition is determined based on the offset index.

6. The method according to claim 5, characterized in that The determining the target primary data block that meets the query condition based on the column value index includes: Determine the target first-level data block according to the first-level data block starting column value and the target column value in the column value index, wherein the first-level data block starting column value of the target first-level data block is less than or equal to the target column value, and the first-level data block starting column value of the next first-level data block corresponding to the target first-level data block is greater than or equal to the target column value; The determining of a target secondary data block that meets the query condition from the target primary data block includes: Acquire a second starting column value index of each secondary data block under the target primary data block from the secondary data block information; Determine the target secondary data block by binary search based on the column value corresponding to the second starting column value index, wherein the secondary data block starting column value of the target secondary data block is less than or equal to the target column value, and the secondary data block starting column value of the next secondary data block corresponding to the target secondary data block is greater than or equal to the target column value; The determining of a target column storage unit that meets the query condition from the target secondary data block includes: According to the column values ​​corresponding to the column storage units in the target secondary data block, the target column storage unit is determined by a binary search method, and the column value corresponding to the target column storage unit is the target column value; The determining target query content from the target column storage unit includes: Obtaining a first starting row number corresponding to a column value in the target column storage unit and a second starting row number corresponding to a column value in a column storage unit next to the target column storage unit; The row number to which the target column value belongs is determined according to the first starting row number and the second starting row number.

7. The method according to claim 5, characterized in that The determining the target primary data block that meets the query condition based on the offset index includes: Determine the target first-level data block according to the first-level data block starting row number and the target row number in the offset index, wherein the target row number belongs to the row number range of the target first-level data block; The determining of a target secondary data block that meets the query condition from the target primary data block includes: Obtaining the secondary data block starting row number of each secondary data block under the target primary data block from the secondary data block information; Determining the target secondary data block according to the starting row number of the secondary data block, wherein the target row number belongs to the row number range of the target secondary data block; The determining of a target column storage unit that meets the query condition from the target secondary data block includes: Determine the target column storage unit according to the starting row number of the column value corresponding to each column storage unit in the target secondary data block, wherein the target row number belongs to the row number range of the target column storage unit; The determining target query content from the target column storage unit includes: The column value corresponding to the target row number is extracted from the target column storage unit.

8. The method according to claim 4, characterized in that After receiving the data query request, the method further includes: In response to the data query request for querying a second data item, caching a second column corresponding to the second data item; decompressing the encoded data according to the metadata stored in the second column to obtain original data; Query the original data for data that meets the query condition.

9. The method according to claim 8, characterized in that The query condition is the target row number, and the decompressing the encoded data according to the metadata in the second column store to obtain the original data includes: determining a target coded data block from the coded data and a target metadata block corresponding to the target coded data block in the metadata according to the target row number and a data block size of the coded data block in the coded data, wherein the data block size is the number of rows of original data corresponding to the coded data block; extracting the target coded data block from the coded data according to the coded block offset and the compressed size in the target metadata block; The target coded data block is decompressed according to the compression type in the target metadata block to obtain the original data.

10. The method according to claim 9, characterized in that The compression type includes a first compression type and a second compression type, the compression rate of the first compression type being higher than the compression rate of the second compression type; If the proportion of target original data in the original data reaches a first threshold, or the proportion of cardinality in the original data is less than a second threshold, the original data is compressed using the first compression type, and the target original data is data with at least two leading bits being 0; If the proportion of cardinality in the original data is greater than the second threshold and less than a third threshold, the original data is compressed using the second compression type.

11. A data query device, characterized in that: For querying data from the data storage device according to any one of claims 1 to 3, the apparatus comprises: A receiving module, configured to receive a data query request, wherein the data query request includes a query condition, and the query condition is a target column value or a target row number; a first determining module, configured to, in response to the data query request for querying the first data item, determine a target primary data block that meets the query condition based on the offset index or the column value index; A second determining module is configured to determine a target secondary data block that meets the query condition from the target primary data block; A third determining module is configured to determine a target column storage unit that meets the query condition from the target secondary data block; The fourth determining module is configured to determine a target query content from the target column storage unit, where the target query content is the row number to which the target column value belongs, or the column value belonging to the target row number.

12. A server, characterized in that: The server includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the data query method as described in any one of claims 4 to 10.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the data query method as described in any one of claims 4 to 10.

Images