Data indexing method, data reading method, device and electronic equipment

Abstract

The embodiment of the present application provides a data indexing method, a data reading method, a device and an electronic device. The method includes: dividing each column of data in the target data table into at least one shard according to the row number; for each shard, respectively reducing the query condition of each data in the shard to obtain the first code of each data; generating the first index of the shard according to the first code of each data in the shard; determining the first digit corresponding to the shard; determining the second code of each data in the shard according to the first digit corresponding to the shard as the second index of each data; for each data, saving the second index of the data corresponding to the internal row number of the data, and saving the first index of the shard to which the data belongs corresponding to the second index of the data. It can be achieved to reduce the amount of data constructed by the data index, thereby reducing the memory resources required for the data index.

Description

Technical Field

[0001] The present application relates to the technical field of analytical database and big data computing, and in particular to a data indexing method, a data reading method, a device and an electronic device. Background Art

[0002] In analytical databases, the following storage method is generally used for data storage: each column of data in the data table is stored in a different page. Corresponding to the above column storage method, data reading is based on pages, that is, when reading data, it is only possible to determine which page the data to be read is in, but it is not possible to specifically determine the location of the data to be read in the page, that is, it is impossible to directly determine the storage address of the data to be read in the database, so that all the data in the page needs to be read when reading data.

[0003] In the related art, a data index is usually constructed so that when reading data, the storage address of the data to be read in the database can be determined according to the data index. In the related art, the data itself and the storage address of the data are usually used as data indexes. Since there is a lot of data in the data table, the above data index construction method will make the amount of data constructed to obtain a large data index, so that the memory resources required to construct the data index are large. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a data indexing method, a data reading method, a device and an electronic device to reduce the amount of data in the constructed data index, thereby reducing the memory resources required for the data index. The specific technical solution is as follows:

[0005] The present application provides a data indexing method, the method comprising:

[0006] Divide each column of data in the target data table into at least one shard according to the row number;

[0007] For each of the shards, respectively perform dimension reduction processing on the query conditions of each data in the shard to obtain a first code for each data;

[0008] For each of the slices, generating a first index of the slice according to the first encoding of each data in the slice;

[0009] For each of the slices, determine a first digit corresponding to the slice, wherein the first digit is capable of making a conflict rate between second codes corresponding to the data less than a preset conflict rate threshold, the second code is a subcode of the first digit selected from the first code of the data in a preset order, and the first digit corresponding to at least one slice is less than the second digit corresponding to the slice, and the second digit is the number of digits of the first code of each data in the slice;

[0010] For each of the slices, determining the second codes of the data in the slices according to the first bit number corresponding to the slices as the second indexes of the data;

[0011] For each data, the second index of the data is saved corresponding to the internal row number of the data, and the first index of the shard to which the data belongs is saved corresponding to the second index of the data, wherein the internal row number is used to indicate the order of the data in the shard.

[0012] In a possible embodiment, determining the first digit corresponding to the slice includes:

[0013] Selecting a subcode of a third digit from the first code of each data of the slice in a preset order, wherein the third digit is initially a preset fourth digit, and the preset fourth digit is smaller than the second digit corresponding to the slice;

[0014] Determining a conflict rate between the selected sub-codes;

[0015] If the conflict rate is less than a preset conflict rate threshold, the third digit is determined as the first digit corresponding to the fragment;

[0016] If the conflict rate is not less than the preset conflict rate threshold, the third digit is increased, and the process returns to the step of selecting the sub-code of the third digit from the first code of each data of the fragment in a preset order.

[0017] In a possible embodiment, the preset fourth digit is 1, and the increasing the third digit includes:

[0018] The third digit is incremented by one.

[0019] In a possible embodiment, the respectively performing dimension reduction processing on the query conditions of each data in the slice to obtain the first encoding of each data includes:

[0020] Determine a fifth digit according to the amount of data in the shard, wherein the fifth digit is positively correlated with the amount;

[0021] The query conditions of each data in the slice are respectively reduced to the fifth digit to obtain the first code of each data.

[0022] In a possible embodiment, the method further includes:

[0023] For each preset column combination, data of each column in the preset column combination is stored in a page corresponding to the preset column combination.

[0024] In a possible embodiment, generating a first index of the slice according to the first encoding of each data in the slice includes:

[0025] Extracting a subcode of a preset order in the first code of each data in the slice as the subcode corresponding to each data in the slice;

[0026] Generate a first index of the slice according to the subcode corresponding to each data in the slice.

[0027] The present application also provides a data reading method, the method comprising:

[0028] Acquire a query condition indicated by a target query instruction for a target data table, wherein an index of the target data table is established in advance according to any of the aforementioned data indexing methods;

[0029] Performing dimensionality reduction processing on the query condition to obtain a third encoding of the query condition;

[0030] Determine, in the pre-saved first indexes, an index matching the third encoding as the target first index;

[0031] In the second index corresponding to the target first index, determine an index matching the third encoding as the target second index;

[0032] Determine an internal row number corresponding to the target second index as the target internal row number;

[0033] Read data according to the target internal line number.

[0034] In a possible embodiment, reading data according to the target internal row number includes:

[0035] Determine the column corresponding to each of the target first indexes; wherein the column corresponding to the target first index is the column to which the target shard belongs, and the target shard is the shard to which the target first index belongs;

[0036] If the determined column hits the preset column combination, data is read from the page corresponding to the preset column combination according to the target internal row number.

[0037] The present application also provides a data indexing device, the device comprising:

[0038] A sharding module is used to divide each column of data in the target data table into at least one shard according to the row number;

[0039] A dimension reduction processing module is used for performing dimension reduction processing on the query conditions of each data in each shard to obtain a first code of each data;

[0040] A first index generating module, configured to generate, for each of the slices, a first index of the slice according to the first encoding of each data in the slice;

[0041] a first digit determination module, configured to determine, for each of the slices, a first digit corresponding to the slice, wherein the first digit is capable of making a conflict rate between second codes corresponding to the data less than a preset conflict rate threshold, the second code is a subcode of the first digit selected from the first code of the data in a preset order, and the first digit corresponding to at least one slice is less than a second digit corresponding to the slice, and the second digit is the number of digits of the first code of each data in the slice;

[0042] A second index determination module is used to determine, for each of the slices, the second code of each data in the slice according to the first bit number corresponding to the slice, as the second index of each data;

[0043] An index saving module is used to save, for each data, a second index of the data corresponding to an internal row number of the data, and to save the first index of the shard to which the data belongs corresponding to the second index of the data, wherein the internal row number is used to indicate the sequence of the data in the shard.

[0044] The present application also provides a data reading device, the device comprising:

[0045] A query condition acquisition module, used to acquire the query condition indicated by the target query instruction for the target data table, wherein the index of the target data table is established in advance according to any of the above-mentioned data indexing methods;

[0046] A coding module, used for performing dimensionality reduction processing on the query condition to obtain a third code of the query condition;

[0047] A target first index determination module, used to determine an index matching the third code in the pre-saved first index as the target first index;

[0048] A target second index determination module, configured to determine, in the second index corresponding to the target first index, an index matching the third code as the target second index;

[0049] A target internal row number determination module, used to determine the internal row number corresponding to the target second index as the target internal row number;

[0050] A data reading module is used to read data according to the target internal row number.

[0051] The present application also provides an electronic device, including:

[0052] Memory, used to store computer programs;

[0053] The processor is used to implement any of the above-mentioned data indexing methods or data reading methods when executing the program stored in the memory.

[0054] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned data indexing methods or data reading methods is implemented.

[0055] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned data indexing methods or data reading methods.

[0056] Beneficial effects of the embodiments of the present application:

[0057] A data indexing method, a data reading method, a device and an electronic device provided by the embodiment of the present application can divide each column of the target data table into at least one shard according to the row number; for each shard, by respectively reducing the dimension of the query conditions of each data in the shard, the first encoding method of each data is obtained, and the data volume of each data is reduced, so that the data volume of the first encoding of each data is less than the data volume of each data itself. For each shard, according to the first encoding of each data in the shard, the first index of the shard is generated; for each shard, the first digit corresponding to the shard is determined. Since the first digit can make the conflict rate between the second encodings corresponding to each data less than the preset conflict rate threshold, the second encoding is the sub-encoding of the first digit selected from the first encoding of the data in a preset order. Therefore, in the same shard, the second encoding of each data is determined according to the first digit corresponding to the shard, and the determined second encoding is used as the second index of each data in the shard, so that the repetition rate between the second indexes of each data in the same shard is low, so that different data in the shard can be distinguished according to the second index. Moreover, since the first digit corresponding to at least one shard is smaller than the second digit corresponding to the shard, and the second digit is the digit of the first code of each data in the shard, the data amount of the second index of all data in the target data table is smaller than the data amount of the first code of all data. And as described above, the data amount of the first code of each data is smaller than the data amount of each data itself, therefore, the data amount of the second index of all data in the target data table is smaller than the data amount of all data itself. For each data, the second index of the data is saved corresponding to the internal row number of the data, and the first index of the shard to which the data belongs is saved corresponding to the second index of the data, wherein the internal row number is used to indicate the order of the data in the shard, and according to the saved corresponding relationship, the order of the data to be read in the shard can be determined, that is, the position of the data to be read in the shard can be determined. And as described above, the data amount of the second index of all data in the target data table is smaller than the data amount of all data itself, therefore, the embodiment of the present application can reduce the data amount of the constructed data index, thereby reducing the memory resources required for the data index.

[0058] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0060] Figure 1A schematic diagram of a flow chart of a data indexing method provided in an embodiment of the present application;

[0061] Figure 2 A schematic diagram of a storage index format provided in an embodiment of the present application;

[0062] Figure 3 A flowchart of a method for reducing the dimension of a query condition of data provided in an embodiment of the present application;

[0063] Figure 4 A schematic diagram of a flow chart of a method for generating a first index of a shard provided in an embodiment of the present application;

[0064] Figure 5a A schematic diagram of a process for determining the first digit by binary division provided in an embodiment of the present application;

[0065] Figure 5b A flowchart of a method for determining the first digit corresponding to a slice provided in an embodiment of the present application;

[0066] Figure 6 Another flowchart of a method for determining the first digit corresponding to a slice provided in an embodiment of the present application;

[0067] Figure 7 Another schematic diagram of a flow chart of a data indexing method provided in an embodiment of the present application;

[0068] Figure 8 A schematic diagram of another flow chart of the data indexing method provided in the embodiment of the present application;

[0069] Fig. 9 A schematic diagram of a flow chart of a data reading method provided in an embodiment of the present application;

[0070] Fig.10 Another schematic diagram of a data reading method according to an embodiment of the present application;

[0071] Fig.11 A schematic diagram of the structure of a data indexing device provided in an embodiment of the present application;

[0072] Fig.12 A schematic diagram of the structure of a data reading device provided in an embodiment of the present application;

[0073] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0075] In order to more clearly explain a data indexing method, a data reading method, a device and an electronic device provided by the present application, some terms appearing in this document are explained below:

[0076] Hash: Hash is also called hashing. Inputting an input of any length into a hash algorithm (i.e., a hash encoding algorithm) can get an output of a fixed length. In other words, the hash algorithm can transform an input of any length into an output of a fixed length. The fixed-length output is the hash value. The hash value space is usually much smaller than the input space, that is, the amount of hash value data is usually much smaller than the amount of input data.

[0077] Bitmap: A bitmap is a bit array, in which a bit can have two values: 0 or 1. Each bit corresponds to a row in a data table, and the value of the bit can be determined based on the value of the data in the row. Exemplarily, in this application, each slice corresponds to a bitmap, and each bit in the bitmap corresponds to a row in a slice. If a row in a slice is empty, that is, the row does not contain data, the value of the bit corresponding to the row is 0. If a row in a slice is not empty, that is, the row contains data, the value of the bit corresponding to the row is 1.

[0078] In order to more clearly illustrate the data indexing method provided by the present application, the possible application scenarios of the data indexing method provided by the present application will be exemplified below. It can be understood that the following examples are only possible application scenarios of the data indexing method provided by the present application. In other possible embodiments, the data indexing method provided by the present application can be applied to other possible application scenarios, and the following examples do not impose any limitations on this.

[0079] In analytical databases, the following storage method is generally used for data storage: each column of data in the data table is stored in a different page. In addition, in order to reduce random reads, each column of data can be divided into multiple pages for data storage. Corresponding to the above-mentioned column storage method, data reading is based on pages, that is, when reading data, it is only possible to determine which page the data to be read is in, but it is not possible to specifically determine the location of the data to be read in the page, that is, it is impossible to directly determine the storage address of the data to be read in the database, so that all the data in the page needs to be read when reading data.

[0080] In the related art, a data index is usually constructed so that when reading data, the storage address of the data to be read in the database can be determined according to the data index. In the related art, the data itself and the storage address of the data are usually used as data indexes. Since there is a lot of data in the data table, when a single data itself is also relatively large, the above data index construction method will make the amount of data constructed to obtain a large data index, and the amount of calculation required to construct the index is also large, so that the memory resources required to construct the data index are large, and the time required to construct the index is long.

[0081] Based on this, in order to reduce the amount of data in the constructed data index, thereby reducing the memory resources required for the data index and reducing the time required to construct the data index, the present application provides a data indexing method, such as Figure 1 As shown, the method includes:

[0082] S101, divide each column of data in the target data table into at least one fragment according to the row number.

[0083] S102, for each shard, respectively perform dimension reduction processing on the query conditions of each data in the shard to obtain a first code for each data.

[0084] S103: For each slice, generate a first index of the slice according to the first encoding of each data in the slice.

[0085] S104: For each fragment, determine the first digit corresponding to the fragment.

[0086] Among them, the first digit can make the conflict rate between the second codes corresponding to each data less than a preset conflict rate threshold, the second code is a sub-code of the first digit selected from the first code of the data according to a preset order, and the first digit corresponding to at least one slice is less than the second digit corresponding to the slice, and the second digit is the number of digits of the first code of each data in the slice.

[0087] S105 , for each slice, determine the second code of each data in the slice according to the first digit corresponding to the slice, as the second index of each data.

[0088] S106: For each data, save the second index of the data corresponding to the internal row number of the data, and save the first index of the shard to which the data belongs corresponding to the second index of the data.

[0089] The internal row number is used to indicate the order of data in the shard.

[0090] By using the embodiment of the present application, on the one hand, each column of the target data table can be divided into at least one shard according to the row number; for each shard, by respectively reducing the dimension of the query conditions of each data in the shard, the first encoding of each data is obtained, and the data volume of each data is reduced, so that the data volume of the first encoding of each data is less than the data volume of each data itself. For each shard, according to the first encoding of each data in the shard, the first index of the shard is generated; for each shard, the first digit corresponding to the shard is determined. Since the first digit can make the conflict rate between the second encodings corresponding to each data less than the preset conflict rate threshold, the second encoding is the sub-encoding of the first digit selected from the first encoding of the data in a preset order. Therefore, in the same shard, the second encoding of each data is determined according to the first digit corresponding to the shard, and the determined second encoding is used as the second index of each data in the shard, so that the repetition rate between the second indexes of each data in the same shard is low, so that different data in the shard can be distinguished according to the second index. Moreover, since the first digit corresponding to at least one shard is smaller than the second digit corresponding to the shard, and the second digit is the digit of the first code of each data in the shard, the data amount of the second index of all data in the target data table is smaller than the data amount of the first code of all data. And as described above, the data amount of the first code of each data is smaller than the data amount of each data itself, therefore, the data amount of the second index of all data in the target data table is smaller than the data amount of all data itself. For each data, the second index of the data is saved corresponding to the internal row number of the data, and the first index of the shard to which the data belongs is saved corresponding to the second index of the data, wherein the internal row number is used to indicate the order of the data in the shard, and according to the saved corresponding relationship, the order of the data to be read in the shard can be determined, that is, the position of the data to be read in the shard can be determined. And as described above, the data amount of the second index of all data in the target data table is smaller than the data amount of all data itself, therefore, the embodiment of the present application can reduce the data amount of the constructed data index, thereby reducing the memory resources required for the data index.

[0091] On the other hand, since the data volume of the first code of each data is smaller than the data volume of each data itself, in the subsequent process of constructing the data index, the embodiment of the present application only needs to process the first code with a smaller data volume to obtain the data index, thereby reducing the amount of calculation required in the process of constructing the data index and reducing the time required to construct the data index.

[0092] The above S101-S106 will be described exemplarily below:

[0093] In S101, different ways of dividing the column data of the target data table according to the row numbers may make the number of rows of data in each shard the same or different.

[0094] In a possible embodiment, the method of dividing the column data of the target data table according to the row number can be: determine a preset number of rows of data in the column data and divide them into one slice, and can divide the continuous preset number of rows of data in the column data into one slice, and can also divide the discontinuous preset number of rows of data in the column data into one slice, which is hereinafter referred to as method 1. The preset number of rows can be set according to actual needs, for example, the preset number of rows can be set to 10 rows, 20 rows, 30 rows, etc.

[0095] Exemplarily, assuming that the preset number of rows is 10 rows, and the target data table includes 100 rows of data, then for the Mth column of data in the target data table, the 1st to 10th rows of data in the column of data can be used as slice 1, and the 11th to 20th rows of data in the column of data can be used as slice 2, ..., and so on. Alternatively, the 10 rows of data in the 1st, 3rd, 5th, 7th, 9th, 11th, 13th, 15th, 17th, and 19th rows of data in the column of data can be used as slice 1, and the 10 rows of data in the 2nd, 4th, 6th, 8th, 10th, 12th, 14th, 16th, 18th, and 20th rows of data in the column of data can be used as slice 2, ..., and so on. Alternatively, the 1st to 10th rows of data in the column data may be used as fragment 1, and the 11th, 13th, 15th, 17th, 19th, 21st, 23rd, 25th, 27th, and 29th rows of data in the column data may be used as fragment 2, ..., and so on. In the above example, the number of rows of data in each fragment is the same.

[0096] In the above example, if the number of remaining rows is less than the preset number of rows during the sharding process of the column data, the remaining number of rows can be added to any shard obtained by sharding the column data, or the remaining number of rows can be used as new shards. For example, assuming that the preset number of rows is 10 rows, and the target data table includes 104 rows of data, then for the Mth column data in the target data table, the 1st to 100th rows of data in the column data are respectively divided into 10 shards, then the 101st to 104th rows of data can be added to any of the above 10 shards, or the 101st to 104th rows of data can be used as shard 11. In this example, the number of rows of data in each shard is different.

[0097] In another possible embodiment, the method of dividing the column data of the target data table according to the row number can also be: dividing a column data of the target data table into at least one shard, and the number of rows of data in each shard conforms to a preset row number rule, which is hereinafter referred to as method 2. The preset row number rule can be set according to actual needs, for example, the preset row number rule can be set to an arithmetic progression, a geometric progression, etc.

[0098] Exemplarily, assuming that the target data table includes 100 rows of data, for the Mth column of data in the target data table, the 1st to 10th rows of data in the column of data can be used as slice 1, the 11th to 30th rows of data in the column of data can be used as slice 2, the 31st to 60th rows of data in the column of data can be used as slice 3, and the 61st to 100th rows of data in the column of data can be used as slice 4. In this example, the sequence composed of the number of rows of data in each slice is an arithmetic progression. In the above example, the row data in the slice is continuous row data, and in other possible examples, the row data in the slice can also be discontinuous row data. In the above example, the number of rows of data in each slice is different.

[0099] In the above example, if the number of remaining rows is less than the preset number of rows during the sharding process of the column data, the remaining number of rows can be added to any shard obtained by sharding the column data, or the remaining number of rows can be used as new shards. For example, assuming that the preset number of rows is 10 rows, and the target data table includes 104 rows of data, then for the Mth column data in the target data table, the 1st to 100th rows of data in the column data are respectively divided into 4 shards, then the 101st to 104th rows of data can be added to any of the above 4 shards, or the 101st to 104th rows of data can be used as shard 5. In this example, the number of rows of data in each shard is different.

[0100] It can be understood that the above-mentioned method 1 and method 2 are only two possible methods of dividing the column data of the target data table according to the row number. The column data of the target data table can also be divided by other possible methods of dividing according to the row number. It is only necessary to ensure that a column of data in the target data table can be divided into at least one fragment. The present application does not impose any restrictions on the method of dividing the column data of the target data table.

[0101] Each column of data in the target data table can be divided into at least one shard in the same division method, or each column of data in the target data table can be divided into at least one shard in different division methods. Exemplarily, each column of data in the target data table can be divided into at least one shard in the aforementioned method 1. Assuming that the target data table includes M columns of data, the data in columns 1 to A of the target data table can be divided into at least one shard in the aforementioned method 1, and the data in columns A+1 to M of the target data table can be divided into at least one shard in the aforementioned method 2, where 1<A<M.

[0102] The above S101 has been described as an example. Figure 1 The embodiment shown, Figure 1 S102 - S106 in the illustrated embodiment are described for exemplary purposes.

[0103] In S102, for each shard, the query conditions for each data in the shard are set by the user in advance for the target data table. The query condition for a certain data refers to: the keyword used by the user when searching for the data, which can be one or more keywords. For example, assuming that the target data table is the statistical elements contained in each substance and the corresponding content of the elements, the corresponding header of the target data table can be as follows:

[0104] | Substance name | Element 1 | Element 1 content | Element 2 | Element 2 content | … | Element n content |.

[0105] When searching for data, users may use the substance name as a query condition to determine the elements contained in a substance and the corresponding content of the elements based on the substance name. The query condition for each data can be pre-set as the substance name corresponding to the row to which the data belongs. For example, assuming that a certain data is the data in the 5th row of the target data table, that is, the row to which the data belongs is the 5th row, and the data stored in the substance name column in the target data table is "water", that is, the substance name corresponding to the row to which the data belongs is "water", then "water" can be used as the query condition for the data.

[0106] When searching for data, users may also use elements as query conditions to determine which substances contain a certain element. The query conditions for each data can be pre-set as a set of all elements in the row to which the data belongs. For example, assuming that a certain data is the data in the 5th row of the target data table, that is, the row to which the data belongs is the 5th row, and the data stored in the 5th row of the target data table include: the substance name is a, element 1 is b, the content of element 1 is c, element 2 is d, and the content of element 2 is e, then b and d can be used as the query conditions for the data.

[0107] The query conditions of the data can be converted into fixed-length data as the first encoding of the data by inputting the query conditions of the data into a hash encoding algorithm. The query conditions of each data in the shard can also be reduced in dimension by prefix, bitmap and other dimensionality reduction processing methods. By reducing the dimension of the query conditions of the data, the amount of data of the query conditions of the data can be greatly reduced to reduce the memory resources required for the query conditions of the data. The following will be an exemplary description of how to reduce the dimension of the query conditions of each data in the shard, which will not be repeated here.

[0108] In S103, in a possible embodiment, for each slice, the first codes of the data in the slice may be concatenated to obtain a first index of the slice.

[0109] In another possible embodiment, for each slice, assuming that the first code of each data includes x bytes, a code of one byte can be arbitrarily selected from the first code of each data as the concatenated code corresponding to each data, and the concatenated codes corresponding to each data are concatenated to obtain the first index of the slice. The following will exemplarily explain how to generate the first index of the slice, which will not be repeated here.

[0110] In S104, the preset conflict rate threshold can be set according to actual needs or past experience. For example, the preset conflict rate threshold can be set to 0.1, 0.15, 0.2, etc. It can be understood that the larger the preset conflict rate threshold, the smaller the first digit determined, so that the amount of data of the second code selected based on the first digit is smaller, that is, the amount of data of the second index of the data is smaller, so that the memory resources required for the data index can be smaller. Since the first digit can make the conflict rate between the second codes corresponding to each data less than the preset conflict rate threshold, the smaller the preset conflict rate threshold, the smaller the conflict rate between the second codes selected based on the first digit, that is, the smaller the conflict rate between the second indexes of the data, the more different data in the shard can be distinguished according to the second index, and the more accurately the data to be read can be determined according to the second index when the data is read, reducing the situation of data reading errors that may occur during reading.

[0111] Based on this, if the user needs to fully reduce the memory resources required to build the index, the preset conflict rate threshold can be set larger; if the user needs to minimize the possibility of reading data errors during reading, the preset conflict rate threshold can be set smaller.

[0112] The number of bits in this application refers to the number of bytes. The preset order may refer to the order of arrangement between any bytes. For example, assuming that the first encoding includes 9 bytes, and the bytes in the first encoding are recorded as byte 1-byte 9 in order from front to back, the preset order may refer to the order of bytes from front to back, or the order of bytes from back to front, or the order of byte 1, byte 5, byte 3, byte 4, byte 2, byte 6, byte 8, byte 9, byte 7, and so on. The preset order set for different slices may be different or the same.

[0113] The order of bytes in the subcode selected according to the preset order can be the same as the preset order or the opposite of the preset order. Based on the above example, if the first digit is 3, the preset order refers to the order of bytes from front to back, then the order of bytes in the second code can be: byte 1, byte 2, byte 3, or: byte 3, byte 2, byte 1.

[0114] For each data in the slice, subcodes with different digits are selected in sequence according to a preset order. If the conflict rate between subcodes with the same digit corresponding to each data is less than a preset conflict rate threshold, the digit of the subcode is used as a candidate first digit. If multiple candidate first digits are determined for a certain slice, any candidate first digit corresponding to the slice can be used as the first digit corresponding to the slice, provided that the first digit corresponding to at least one slice is less than the second digit corresponding to the slice. The following is an exemplary description of how to determine the first digit corresponding to the slice, which will not be repeated here.

[0115] In S105, for each slice, the subcode of the first digit corresponding to the slice is selected from the first code of the data in a preset order to obtain the second code of each data in the slice as the second index of each data.

[0116] In S106, in a possible embodiment, multiple first index sets may be created first, each shard corresponds to a first index set, and the first index set is named as the first index of the shard corresponding to the first index set. The second index of each data in a shard and the internal row number of each data are stored in the first index set corresponding to the shard.

[0117] In this embodiment, the second index of each data in the slice and the internal row number of each data may be stored in the form of a table or an array.

[0118] The second index of each data in the shard can also be sorted and stored according to the order of the data in the shard, that is, the second index of each data in the shard can be sorted and stored according to the internal row number of the data. In the above method, the internal row number of each data can be reflected by the arrangement order between the second indexes of each data in the shard, so that the internal row number of each data does not need to be stored, further reducing the memory resources required to store the data index. In this embodiment, the stored index format can be as follows Figure 2 shown.

[0119] exist Figure 2 In the data structure for recording the target data table (hereinafter referred to as the data structure of the target data table), m columns of Data are included, namely, column a, column b, ..., column m, which can be respectively recorded as: Column a, Column b, ..., Column m; the index of the data can include indexes such as dimensionality reduction index and other indexes, wherein the dimensionality reduction index refers to the first index of the aforementioned shard and the second index of the data, the dimensionality reduction index can be recorded as reduce index, and the other index can be recorded as other index; in addition, the data structure of the target data table also includes footer information, and the footer information includes three types of information: filerfooter (file end information), checksum, and magic code.

[0120] See also Figure 2 , taking column a as an example, the first index of the shard in each column and the second index of the data are explained exemplarily. The shard index of each shard in column a is stored in column a index. Column a is divided into n shards according to the row number: shard 1, shard 2, ..., shard n, and the above shards can be recorded as: shard 1, shard 2, ..., shard n. The column a index also stores the following types of meta information, including: total num (total number of records) and shard size (shard size).

[0121] Taking shard n as an example, the shard index of each shard is exemplarily described. The second index of each data in shard n is stored in the shard index of shard n, such as Figure 2As shown in the figure, the second indexes of the z data in shard n are: k1, k2, ..., kz. In addition, the shard index also stores the following types of information, including: null_bitmap (bitmap mask of empty data), first index, shard entry num (the number of data in the shard), where the bitmap mask is used to record the sequence of empty data in shard n, and the shard index is the first index of shard n.

[0122] In another possible embodiment, a correspondence between the second index of the data, the internal row number of the data, and the first index of the shard to which the data belongs may be established for each data in the form of a table. In this embodiment, the header of the correspondence table may be as follows:

[0123] | The second index of the data | The internal row number of the data | The first index of the shard to which the data belongs |.

[0124] The above S101-S106 have been described as examples. Figure 1 In the embodiment shown, the data indexing method provided by the present application relies on obtaining the first encoding of each data after the query condition of each data in the shard is reduced in dimension when generating the first index of the shard and the second index of each data. Therefore, the following will illustrate how to reduce the dimension of the query condition of each data in the shard in the aforementioned S102. Figure 3 , the aforementioned S102 includes:

[0125] S1021, determine the fifth digit according to the amount of data in the shard.

[0126] Among them, the fifth digit is positively correlated with quantity.

[0127] The fifth digit is positively correlated with the quantity, which means that, without considering other factors except the quantity, the fifth digit increases as the quantity of data in the shard increases, and decreases as the quantity of data in the shard decreases. Increasing can refer to monotonically increasing or non-monotonic increasing; decreasing can refer to monotonically decreasing or non-monotonic decreasing.

[0128] S1022, reduce the query conditions of each data in the shard to the fifth digit respectively, and obtain the first code of each data.

[0129] exist Figure 3In the illustrated embodiment, specifically, a hash coding algorithm can be used to reduce the dimension of the query conditions of each data in the shard, and one of a 16-bit hash coding algorithm, a 32-bit hash coding algorithm, a 64-bit hash coding algorithm, and a 128-bit hash coding algorithm is adaptively selected according to the amount of data in the shard. Four quantity intervals can be set in advance: [0, y1), [y1, y2), [y2, y3), [y3, +∞). If the number of data in the shard is between [0, y1), the fifth digit is set to 2, that is, a 16-bit hash coding algorithm is selected to reduce the dimension of the query conditions of each data in the shard; if the number of data in the shard is between [y1, y2), the fifth digit is set to 4, that is, a 32-bit hash coding algorithm is selected to reduce the dimension of the query conditions of each data in the shard; if the number of data in the shard is between [y2, y3), the fifth digit is set to 8, that is, a 64-bit hash coding algorithm is selected to reduce the dimension of the query conditions of each data in the shard; if the number of data in the shard is between [y3, +∞), the fifth digit is set to 16, that is, a 128-bit hash coding algorithm is selected to reduce the dimension of the query conditions of each data in the shard.

[0130] When this embodiment is selected, if the amount of data in the shard is small, it can be considered that the probability of conflict when determining the second index of each data in the shard is small, then the amount of data in the first code of each data can be reduced in the following manner: determine the fifth digit according to the amount of data in the shard, and the fifth digit is positively correlated with the amount; reduce the query conditions of each data in the shard to the fifth digit respectively, and obtain the first code of each data. If the amount of data in the first code is small, the amount of data in the first index of the shard obtained based on the first code and the amount of data in the second index of each data will be small, thereby further reducing the amount of data in the constructed data index and reducing the memory resources required for the data index.

[0131] In the above, an exemplary description is given of how to perform dimension reduction processing on the query conditions of each data in the shard in the aforementioned S102. In the following, an exemplary description is given of how to generate the first index of the shard in the aforementioned S103. Figure 4 , the aforementioned S103 includes:

[0132] S1031, extracting a subcode of a preset order in the first code of each data in the slice as the subcode corresponding to each data in the slice.

[0133] A preset order may refer to: an order preset when the bytes in the first encoding of the data are sorted from front to back; or may refer to: an order preset when the bytes in the first encoding of the data are sorted from back to front.

[0134] Exemplarily, assume that the first code includes 9 bytes, and the bytes in the first code are recorded from front to back as: byte 1-byte 9. If the preset order is the 8th bit from front to back, byte 8 in the first code of each data in the slice is extracted as the sub-code corresponding to each data in the slice.

[0135] S1032: Generate a first index of the slice according to the sub-code corresponding to each data in the slice.

[0136] The subcodes corresponding to the data in the slice may be concatenated to obtain the first index of the slice.

[0137] By selecting this embodiment, only a sub-code of a preset order in the first code of each data in the slice can be extracted as the sub-code corresponding to each data in the slice, and the first index of the slice can be generated according to the sub-code corresponding to each data in the slice, so that there is no need to generate the first index of the slice according to all the data in the first code of each data in the slice, thereby reducing the amount of data in the obtained first index.

[0138] The above has been exemplified by how to generate the first index of the shard in the aforementioned S103. In the example of using a hash coding algorithm to reduce the dimension of the query conditions of each data in the shard, it can be understood that in the process of determining the second index of each data, the second index of each data in the same shard may conflict. If the conflict rate between the second indexes of each data in the same shard is high, it is impossible to distinguish the data in the same shard through the second index of each data. In the related art, the method of calculating hash can be optimized in the data storage address query scenario to solve the hash conflict problem, but the above method is not used in the scenario of constructing a data index, and the change of the hash calculation method may make the constructed data index only applicable to devices that can implement the above hash calculation method, making the applicability of the data indexing method low.

[0139] Based on this, in the embodiment of the present application, the conflict rate between the second codes of each data can be ensured to be less than the preset conflict rate threshold according to the first digit determined in the aforementioned S104, so that the second code of each data in the slice can be determined respectively according to the first digit corresponding to the slice, as the second index of each data, to ensure that the conflict rate between the second indexes of each data is less than the preset conflict rate threshold, thereby reducing the conflict rate of the second indexes of each data in the same slice. The following is an exemplary description of how to determine the first digit corresponding to the slice in the aforementioned S104.

[0140] In a possible embodiment, the first digit can be determined by binary search. Figure 5aAs shown, assuming that the first code includes 10 bytes, and the bytes in the first code are recorded as byte 1-byte 10 in order from front to back, the preset order may refer to the order of the bytes from front to back, first select byte 1-byte 5 in the first code of each data in the slice as the sub-code 1 corresponding to each data, if the conflict rate between the sub-codes 1 of each data (i.e., conflict rate 1) is less than the preset conflict rate threshold, then select byte 1-byte 3 in the first code of each data in the slice as the sub-code 2 corresponding to each data; if the conflict rate between the sub-codes 1 of each data is not less than the preset conflict rate threshold, then select byte 1-byte 8 in the first code of each data in the slice as the sub-code 3 corresponding to each data. For subcode 2, if the conflict rate between subcodes 2 of each data (i.e., conflict rate 2) is less than the preset conflict rate threshold, then byte 1 to byte 2 in the first code of each data in the slice is selected as the subcode 4 corresponding to each data; if the conflict rate between subcodes 2 of each data is not less than the preset conflict rate threshold, then byte 1 to byte 4 in the first code of each data in the slice is selected as the subcode 5 corresponding to each data. For subcode 3, if the conflict rate between subcodes 3 of each data (i.e., conflict rate 3) is less than the preset conflict rate threshold, then byte 1 to byte 7 in the first code of each data in the slice is selected as the subcode 6 corresponding to each data; if the conflict rate between subcodes 3 of each data is not less than the preset conflict rate threshold, then byte 1 to byte 9 in the first code of each data in the slice is selected as the subcode 7 corresponding to each data. The above judgment process is performed for subcode 4, subcode 5, subcode 6, and subcode 7 respectively to determine all the bits that can make the conflict rate between the second codes corresponding to each data less than the preset conflict rate threshold. While ensuring that the first bit corresponding to at least one fragment is less than the second bit corresponding to the fragment, any one of the determined bits is used as the first bit.

[0141] In another possible embodiment, see Figure 5b , the aforementioned S104 includes:

[0142] S1041, selecting a subcode of a third digit from the first code of each data of the slice according to a preset order.

[0143] The third digit is initially a preset fourth digit, and the preset fourth digit is smaller than the second digit corresponding to the shard.

[0144] If the number of bits of the first code of each data in the slice is 8, that is, the second number is 8, the preset fourth number can be set to 1, 3, 5, etc., which is less than 8.

[0145] S1042: Determine the conflict rate between the selected sub-codes.

[0146] The number of repeated subcodes in the selected subcodes is determined, and the ratio between the number of repeated subcodes and the number of selected subcodes is used as the conflict rate between the selected subcodes.

[0147] For example, if the number of selected subcodes is 20, and subcode 1=subcode 5=subcode 6, subcode 11=subcode 12, subcode 10=subcode 20, then the number of repeated subcodes is 7, and the conflict rate between the selected subcodes is 7 / 20=0.35.

[0148] S1043: If the conflict rate is less than a preset conflict rate threshold, the third digit is determined as the first digit corresponding to the fragment.

[0149] S1044, if the conflict rate is not less than the preset conflict rate threshold, the third digit is increased, and the process returns to the step of selecting the sub-code of the third digit from the first code of each data of the fragment in a preset order.

[0150] When adding the third digit, the third digit can be increased by any value. However, since the number of digits of the first code of each data in the slice is the second digit, if the increased third digit is greater than the second digit, the subcode of the increased third digit cannot be selected from the first code of each data in the slice according to the preset order. Therefore, when adding the third digit, it is necessary to make the increased third digit smaller than the second digit corresponding to the slice.

[0151] By selecting this embodiment, the first digit can be determined in the following manner: select the subcode of the third digit from the first code of each data of the slice in a preset order; the third digit is initially the preset fourth digit, and the preset fourth digit is less than the second digit corresponding to the slice; determine the conflict rate between the selected subcodes; if the conflict rate is less than the preset conflict rate threshold, determine the third digit as the first digit corresponding to the slice; if the conflict rate is not less than the preset conflict rate threshold, increase the third digit, and return to execute the step of selecting the subcode of the third digit from the first code of each data of the slice in a preset order. By determining the first digit, it is ensured that the conflict rate between the second codes of each data is less than the preset conflict rate threshold, so that the conflict rate between the second indexes of each data can be ensured to be less than the preset conflict rate threshold, thereby reducing the conflict rate of the second indexes of each data in the same slice.

[0152] In a possible embodiment, the aforementioned preset fourth digit is 1. In this embodiment, the aforementioned Figure 5b The flowchart shown can be Figure 6 As shown, see Figure 6 , the aforementioned S104 includes:

[0153] S1041, selecting a subcode of a third digit from the first code of each data of the slice according to a preset order.

[0154] S1042: Determine the conflict rate between the selected sub-codes.

[0155] S1043: If the conflict rate is less than a preset conflict rate threshold, the third digit is determined as the first digit corresponding to the fragment.

[0156] The foregoing Figure 5b S1041-S1043 have been exemplarily described in the illustrated embodiment, and reference may be made to the aforementioned related descriptions of S1041-S1043, which will not be repeated here.

[0157] S1044a, if the conflict rate is not less than the preset conflict rate threshold, the third digit is incremented by one, and the process returns to the step of selecting a sub-code of the third digit from the first codes of each data of the fragment in a preset order.

[0158] Initially, the third digit is 1. After executing S1041-S1042 for the first time, if the conflict rate is not less than the preset conflict rate threshold, the third digit is incremented by one. At this time, the third digit is 2, and the aforementioned S1041-S1042 is executed for the second time. If the conflict rate is still not less than the preset conflict rate threshold after executing S1041-S1042 for the second time, the third digit is incremented by one again. At this time, the third digit is 3, and the aforementioned S1041-S1042 is executed for the third time. If the conflict rate is still not less than the preset conflict rate threshold after executing S1041-S1042 for the second time, the third digit is incremented by one again, and the above process is repeated until the conflict rate is less than the preset conflict rate threshold and the first digit is determined. If the conflict rate is less than the preset conflict rate threshold after executing S1041-S1042 for the second time, the third digit is used as the first digit corresponding to the fragment, that is, the first digit is 2.

[0159] By selecting this embodiment, the preset fourth digit can be set to 1, and the third digit can be incremented by 1, so that the change of the third digit is continuous from small to large, and no possible value of the third digit will be skipped. That is, the possible values ​​of the third digit can be traversed in order from small to large, so that the minimum number of digits that can make the conflict rate between the second codes corresponding to each data less than the preset conflict rate threshold can be determined more accurately, and the minimum number of digits can be used as the first digit, thereby improving the accuracy of the determined first digit.

[0160] The above has been explained in an exemplary manner how to determine the first digit corresponding to the shard in the aforementioned S104. It can be understood that for fixed-scenario businesses, when reading data, users usually only pay attention to part of the data read. Compared with row storage, column storage can flexibly select the data columns to be read. However, if the data that the user needs to read is located in the same row of multiple columns (that is, the user performs high-concurrency data query), the column storage will require more columns to be read in the data reading. Then, the data needs to be read from the disk multiple times, and each read is in units of page. Each time a key to be read in a certain page is read, more useless data will be read, resulting in reduced data query efficiency under high-concurrency data query, especially when reading data from a mechanical disk. The reading time is linearly related to the number of columns of data to be read. The more columns of data to be read, the longer the reading time. If column storage is not used, but row storage is used, the advantages of column storage cannot be utilized, and the above problems will also occur when reading data.

[0161] Based on this, in order to read data from different columns in the same page and reduce the number of times required to read, see Figure 7 , the data indexing method provided in the embodiment of the present application includes:

[0162] S101, divide each column of data in the target data table into at least one fragment according to the row number.

[0163] S102, for each shard, respectively perform dimension reduction processing on the query conditions of each data in the shard to obtain a first code for each data.

[0164] S103: For each slice, generate a first index of the slice according to the first encoding of each data in the slice.

[0165] S104: For each fragment, determine the first digit corresponding to the fragment.

[0166] Among them, the first digit can make the conflict rate between the second codes corresponding to each data less than a preset conflict rate threshold, the second code is a sub-code of the first digit selected from the first code of the data according to a preset order, and the first digit corresponding to at least one slice is less than the second digit corresponding to the slice, and the second digit is the number of digits of the first code of each data in the slice.

[0167] S105 , for each slice, determine the second code of each data in the slice according to the first digit corresponding to the slice, as the second index of each data.

[0168] S106: For each data, save the second index of the data corresponding to the internal row number of the data, and save the first index of the shard to which the data belongs corresponding to the second index of the data. The internal row number is used to indicate the order of the data in the shard.

[0169] The foregoing Figure 1 S101 - S106 have been exemplarily described in the illustrated embodiment, and reference may be made to the aforementioned related descriptions of S101 - S106 , which will not be repeated here.

[0170] S107 , for each preset column combination, saving the data of each column in the preset column combination in a page corresponding to the preset column combination.

[0171] The preset column combination can be set according to the data content in the target data table according to actual needs or past experience. For example, referring to the example in S102 above, assuming that the target data table is the elements contained in each substance obtained by statistics and the corresponding content of the elements, then the header of the target data table can be as follows:

[0172] | Substance name | Element 1 | Element 1 content | Element 2 | Element 2 content | … | Element n content |.

[0173] If the correlation between an element contained in a substance and the corresponding content of the element is high, the user is likely to want to determine the content of the element contained in the substance while obtaining the element contained in the substance. Therefore, in this example, the preset column combination can be set as: preset column combination 1: element 1 and element 1 content, preset column combination 2: element 2 and element 2 content, ..., preset column combination n: element n and element n content.

[0174] Each preset column combination may correspond to a page, and the page corresponding to preset column combination 1 is recorded as page 1, the page corresponding to preset column combination 2 is recorded as page 2, ..., and the page corresponding to preset column combination n is recorded as page n. Then the data of each column belonging to preset column combination 1 is stored in page 1, that is, the data of the two columns of element 1 and element 1 content are stored in page 1; the data of each column belonging to preset column combination 2 is stored in page 2, that is, the data of the two columns of element 2 and element 2 content are stored in page 2; ...; and so on.

[0175] Since each column of the target data table is divided into at least one fragment according to the row number in the aforementioned S101, each preset column combination can correspond to multiple pages, and the fragments containing data with the same row number and different columns in the target data table are saved in the same page. Exemplarily, assuming that the target data table includes 30 rows of data, the 1st to 10th rows of data in the element 1 column are the data included in fragment 1; the 11th to 20th rows of data in the element 1 column are the data included in fragment 2; the 21st to 30th rows of data in the element 1 column are the data included in fragment 3; the 1st to 10th rows of data in the element 1 content column are the data included in fragment 4; the 11th to 20th rows of data in the element 1 content column are the data included in fragment 5; the 21st to 30th rows of data in the element 1 content column are the data included in fragment 6. The preset column combination 1 corresponds to page 1, page 2, and page 3, so shard 1 and shard 4 can be saved in page 1, shard 2 and shard 5 can be saved in page 2, and shard 3 and shard 6 can be saved in page 3.

[0176] In this embodiment, see Figure 2 As shown, Data includes not only the m columns of data, namely, column a, column b, ..., column m, but also merged column (preset column combination) information.

[0177] By selecting this embodiment, by setting preset column combinations, for each preset column combination, the data of each column in the preset column combination can be saved in the page corresponding to the preset column combination, so that the same page can include data of different columns with high correlation, so that when reading data, data of different columns can be read simultaneously in one page, reducing the different pages that need to be read and the number of times required for data reading, thereby reducing the time required for data reading and improving the efficiency of data reading.

[0178] The process of the data indexing method provided in this application can be as follows: Figure 8 As shown, the method includes:

[0179] S801, input data set.

[0180] The data set is the target data table mentioned above.

[0181] S802, row number fragmentation.

[0182] S802 corresponds to the aforementioned S101, and the relevant description of the aforementioned S101 can be referred to, which will not be repeated here.

[0183] S803, query condition dimension reduction coding.

[0184] S803 corresponds to the aforementioned S102, and the relevant description of the aforementioned S102 can be referred to, which will not be repeated here.

[0185] S804: Build a shard-level index.

[0186] S804 corresponds to the aforementioned S103, and the relevant description of the aforementioned S103 can be referred to, which will not be repeated here.

[0187] S805, get the coded bytes.

[0188] The coded byte is the sub-code in S1041 mentioned above. S805 corresponds to the Figure 6 S1041 in the illustrated embodiment, i.e., presetting the fourth digit to 1, can be referred to the related description of the aforementioned S1041, which will not be described again here.

[0189] S806, detecting a conflict.

[0190] S806 is to determine whether the conflict rate between the selected sub-codes is less than the preset conflict rate threshold. If the condition is met, that is, if the conflict rate between the selected sub-codes is less than the preset conflict rate threshold, the first digit is determined, that is, the second index of each data is determined, and then the following S807 is executed; if the condition is not met, that is, if the conflict rate between the selected sub-codes is not less than the preset conflict rate threshold, the third digit is incremented by one, and the process returns to execute the aforementioned S805.

[0191] S807, build internal line number mapping.

[0192] S807 corresponds to the aforementioned S106, and the relevant description of the aforementioned S106 can be referred to, which will not be repeated here. In S807, a bitmap is set for each slice to record the sequence of the empty data in a slice in the slice, that is, to record the row where the empty data in a slice is located in the slice.

[0193] S808, persistent index.

[0194] Since the data reading and writing speed on the disk is much slower than the memory reading speed, when the second index of each data is persistently stored on the disk, the second index of each data in the shard can be sorted and stored according to the order of the data in the shard, that is, the second index of each data in the shard can be sorted and stored according to the internal row number of the data, so that the internal row number of each data can be reflected by the arrangement order between the second indexes of each data in the shard, and the internal row number of each data is not stored, which further reduces the memory resources required for storing data indexes, that is, reduces the amount of data indexes stored on the disk.

[0195] S809, the result column merging code can be configured.

[0196] S809 corresponds to the aforementioned S107. Please refer to the relevant description of the aforementioned S107 and will not be repeated here.

[0197] Corresponding to the aforementioned columnar storage method, in the related technology, when reading data, it is necessary to enter the query conditions, determine the page that matches the query conditions in the analytical database through filtering methods such as bloomfilter or zonemap, read all the data in the page, and then determine the data that meets the query conditions from the read data. When the string of the data to be read is large, reading all the data in the entire page will consume a lot of resources. At the same time, if it is necessary to read data in multiple columns, it is necessary to load the data through multiple disk seeks. The more columns need to be read, the more disk seeks are needed, and the lower the efficiency of data query, that is, data reading, and the worse the performance.

[0198] In the related technology, data reading can also be performed in the following manner: the hash bucket value of each row is calculated through the row data of the row number of each column data in the column database, and the row number set with the same hash bucket value is determined according to the hash bucket value of each row, and then the column data corresponding to each row number in the row number set with the same hash bucket value is sent to the hash bucket corresponding to the row number set. However, the above method is applied to the shuffle (data repartitioning and reorganization) process of big data computing, and SIMD (Single Instruction Multiple Data, data vector) technology is applied to accelerate hash calculations, and cannot be applied to the scenario of data reading in analytical databases in this application, and cannot improve the efficiency of data reading.

[0199] Based on this, in order to reduce the time required for data reading and improve the efficiency and performance of data reading, corresponding to the aforementioned data indexing method, the embodiment of the present application also provides a data reading method, see Fig. 9 , methods include:

[0200] S901, obtaining a query condition indicated by a target query instruction for a target data table.

[0201] Wherein, the index of the target data table is established in advance according to any of the aforementioned data indexing methods.

[0202] S902: Perform dimensionality reduction processing on the query condition to obtain a third encoding of the query condition.

[0203] S903: Determine, from among the pre-saved first indexes, an index that matches the third encoding as a target first index.

[0204] S904: Determine, in the second index corresponding to the target first index, an index matching the third code as the target second index.

[0205] S905: Determine the internal row number corresponding to the target second index as the target internal row number.

[0206] S906, reading data according to the target internal row number.

[0207] By selecting this embodiment, the query condition indicated by the target query instruction for the target data table can be obtained; the index of the target data table is established in advance according to any of the aforementioned data indexing methods; the query condition is subjected to dimensionality reduction processing to obtain the third encoding of the query condition; in the pre-saved first index, the index matching the third encoding is determined as the target first index; in the second index corresponding to the target first index, the index matching the third encoding is determined as the target second index. After the target second index is determined, the specific internal row number of the data corresponding to the query condition in the shard can be determined. Even if multiple target second indexes are determined, the conflict rate of the second indexes of each data in the same shard is low. Therefore, compared with bloomfilter and the like, which can only locate whether there is data corresponding to the query condition in the page, and need to read the entire page data before making a specific judgment, the data reading method provided by the present application will only read the data corresponding to the query condition when reading the data, or may also read a small part of the data that does not correspond to the query condition, thereby greatly reducing the time required for data reading and improving the efficiency and performance of data reading.

[0208] Furthermore, as described above, the amount of data of the index generated according to the above-mentioned data indexing method is relatively small, and compared with the query condition itself, the amount of data of the third code of the query condition is also relatively small. Therefore, in the process of filtering data according to the third code of the query condition to obtain the target second index, the time and memory resources required for data filtering can be reduced, so that the third code can be directly cached in the memory of the disk, and the utilization rate of the CPU (Central Processing Unit) of the device to which the disk belongs can be greatly reduced.

[0209] The aforementioned S901 - S906 will be exemplarily described below.

[0210] In S901, the query condition indicated by the target query instruction may be a query condition corresponding to any data that the user needs to read. The method of establishing the index of the target data table can refer to the above description of the data index method, which will not be repeated here.

[0211] In S902, the method of performing dimensionality reduction processing on the query condition indicated by the target query instruction should be the same as the method of performing dimensionality reduction processing on the query condition of the data in the aforementioned S102. The method of performing dimensionality reduction processing on the query condition indicated by the target query instruction can refer to the aforementioned relevant description of S102, which will not be repeated here.

[0212] In S903, if a first index contains bytes that are repeated with the third code, the first index may be determined as an index that matches the third code, and the first index may be used as a target first index.

[0213] In S904, the second index corresponding to the target first index is determined based on the second index of the data saved in S106, the internal row number of the data, and the first index of the shard to which the data belongs. In the second index corresponding to the target first index, if a second index contains bytes that are repeated with the third code, the second index can be determined as an index that matches the third code, and the second index is used as the target second index. The second indexes corresponding to the target first index can be sorted in order from large to small, and then the index that matches the third code is determined in a binary way as the target second index.

[0214] In S905, based on the second index of the data saved in S106, the internal row number of the data, and the first index of the shard to which the data belongs, the internal row number corresponding to the target second index is determined as the target internal row number.

[0215] As described in S106 above, the second index of each data in the shard can be sorted and stored according to the order of the data in the shard, that is, the second index of each data in the shard can be sorted and stored according to the internal row number of the data. In this embodiment, the order of the target second index in the second index corresponding to the target first index can be determined, and the order of the target second index is the internal row number corresponding to the target second index, which is used as the target internal row number.

[0216] In S906, the location of the data corresponding to the query condition in the shard can be determined according to the target internal row number, so that the data corresponding to the query condition can be directly read.

[0217] Corresponding to the above Figure 7 The embodiment shown in the present application provides another data reading method, see Fig.10 , S906 in the aforementioned data reading method includes:

[0218] S9061, determine the column corresponding to the first index of each target.

[0219] Among them, the column corresponding to the target first index is the column to which the target shard belongs, and the target shard is the shard to which the target first index belongs.

[0220] S9062: If the determined column hits the preset column combination, data is read from the page corresponding to the preset column combination according to the target internal row number.

[0221] The hit in S9062 means that the preset column combination is a true subset of the set composed of the determined columns.

[0222] For example, corresponding to the aforementioned Figure 7 In the example of the embodiment, assuming that the target data table is the statistically obtained elements contained in each substance and the corresponding content of the elements, then the header of the target data table can be as follows:

[0223] | Substance name | Element 1 | Element 1 content | Element 2 | Element 2 content | … | Element n content |.

[0224] The preset column combinations are: preset column combination 1: element 1 and element 1 content, preset column combination 2: element 2 and element 2 content,…, preset column combination n: element n and element n content, and the page corresponding to preset column combination 1 is recorded as page 1, the page corresponding to preset column combination 2 is recorded as page 2,…, and the page corresponding to preset column combination n is recorded as page n.

[0225] If the columns corresponding to the determined first indexes of each target are: substance name, element 1, and element 1 content, respectively, and the determined columns hit the preset column combination 1, data can be read from page 1 corresponding to the preset column combination 1 according to the target internal row number. For the column substance name corresponding to the target first index, data is read from the shard to which the target first index belongs, that is, the target shard, according to the target internal row number.

[0226] If the columns corresponding to the determined first indexes of each target are: substance name, element 1, element 2, and the determined columns do not hit the preset column combination 1, the data can be read directly in the shard to which the first index of the target belongs, that is, the target shard, according to the target internal row number.

[0227] By selecting this embodiment, by setting a preset column combination, when the column corresponding to the first target index hits the preset column combination, data can be read from the page corresponding to the preset column combination according to the target internal row number. In other words, in the process of data reading, data of different columns can be read simultaneously in the same page, reducing the number of different pages to be read and the number of times required for data reading, thereby reducing the time required for data reading and improving the efficiency of data reading, so that the data reading method provided by the present application can support the application scenario of high-concurrency data query in analytical databases.

[0228] Corresponding to the aforementioned data indexing method, the present application embodiment provides a data indexing device, see Fig.11 , the device comprises:

[0229] A sharding module 1101 is used to divide each column of data in the target data table into at least one shard according to the row number;

[0230] A dimension reduction processing module 1102 is used to perform dimension reduction processing on the query conditions of each data in each shard, respectively, to obtain a first code of each data;

[0231] A first index generating module 1103 is used to generate a first index of each shard according to a first code of each data in the shard for each shard;

[0232] A first digit determination module 1104 is used to determine, for each slice, a first digit corresponding to the slice, wherein the first digit can make the conflict rate between the second codes corresponding to the data less than a preset conflict rate threshold, the second code is a subcode of the first digit selected from the first code of the data in a preset order, and the first digit corresponding to at least one slice is less than the second digit corresponding to the slice, and the second digit is the number of digits of the first code of each data in the slice;

[0233] The second index determination module 1105 is used to determine, for each slice, the second code of each data in the slice according to the first digit corresponding to the slice, as the second index of each data;

[0234] The index saving module 1106 is used to save the second index of the data corresponding to the internal row number of the data for each data, and save the first index of the shard to which the data belongs corresponding to the second index of the data, wherein the internal row number is used to indicate the order of the data in the shard.

[0235] In a possible embodiment, determining the first digit corresponding to the slice includes:

[0236] Selecting a subcode of a third digit from the first code of each data of the slice in a preset order, wherein the third digit is initially a preset fourth digit, and the preset fourth digit is smaller than the second digit corresponding to the slice;

[0237] Determining a conflict rate between the selected sub-codes;

[0238] If the conflict rate is less than a preset conflict rate threshold, the third digit is determined as the first digit corresponding to the fragment;

[0239] If the conflict rate is not less than the preset conflict rate threshold, the third digit is increased, and the process returns to the step of selecting the sub-code of the third digit from the first code of each data of the fragment in a preset order.

[0240] In a possible embodiment, the preset fourth digit is 1, and the increasing the third digit includes:

[0241] The third digit is incremented by one.

[0242] In a possible embodiment, the respectively performing dimension reduction processing on the query conditions of each data in the slice to obtain the first encoding of each data includes:

[0243] Determine a fifth digit according to the amount of data in the shard, wherein the fifth digit is positively correlated with the amount;

[0244] The query conditions of each data in the slice are respectively reduced to the fifth digit to obtain the first code of each data.

[0245] In a possible embodiment, the device further includes:

[0246] The preset column combination saving module is used for saving, for each preset column combination, the data of each column in the preset column combination in a page corresponding to the preset column combination.

[0247] In a possible embodiment, generating a first index of the slice according to the first encoding of each data in the slice includes:

[0248] Extracting a subcode of a preset order in the first code of each data in the slice as the subcode corresponding to each data in the slice;

[0249] Generate a first index of the slice according to the subcode corresponding to each data in the slice.

[0250] Corresponding to the aforementioned data acquisition method, the present application embodiment provides a data reading device, see Fig.12 , the device comprises:

[0251] The query condition acquisition module 1201 is used to acquire the query condition indicated by the target query instruction for the target data table, wherein the index of the target data table is established in advance according to any of the above data indexing methods;

[0252] The encoding module 1202 is used to perform dimensionality reduction processing on the query condition to obtain a third encoding of the query condition;

[0253] A target first index determination module 1203 is used to determine an index matching the third code in the pre-saved first index as the target first index;

[0254] A target second index determination module 1204 is used to determine an index matching the third code in the second index corresponding to the target first index as the target second index;

[0255] A target internal row number determination module 1205 is used to determine the internal row number corresponding to the target second index as the target internal row number;

[0256] The data reading module 1206 is used to read data according to the target internal row number.

[0257] In a possible embodiment, reading data according to the target internal row number includes:

[0258] Determine the column corresponding to each of the target first indexes; wherein the column corresponding to the target first index is the column to which the target shard belongs, and the target shard is the shard to which the target first index belongs;

[0259] If the determined column hits the preset column combination, data is read from the page corresponding to the preset column combination according to the target internal row number.

[0260] The present application also provides an electronic device, such as Fig.13 As shown, including:

[0261] Memory 1301, used for storing computer programs;

[0262] The processor 1302 is used to implement the following steps when executing the program stored in the memory 1301:

[0263] Divide each column of data in the target data table into at least one shard according to the row number;

[0264] For each of the shards, respectively perform dimension reduction processing on the query conditions of each data in the shard to obtain a first code for each data;

[0265] For each of the slices, generating a first index of the slice according to the first encoding of each data in the slice;

[0266] For each of the slices, determine a first digit corresponding to the slice, wherein the first digit is capable of making a conflict rate between second codes corresponding to the data less than a preset conflict rate threshold, the second code is a subcode of the first digit selected from the first code of the data in a preset order, and the first digit corresponding to at least one slice is less than the second digit corresponding to the slice, and the second digit is the number of digits of the first code of each data in the slice;

[0267] For each of the slices, determining the second codes of the data in the slices according to the first bit number corresponding to the slices as the second indexes of the data;

[0268] For each data, the second index of the data, the internal row number of the data, and the first index of the shard to which the data belongs are saved accordingly, wherein the internal row number is used to indicate the sequence of the data in the shard.

[0269] or,

[0270] Acquire a query condition indicated by a target query instruction for a target data table, wherein an index of the target data table is established in advance according to any of the aforementioned data indexing methods;

[0271] Performing dimensionality reduction processing on the query condition to obtain a third encoding of the query condition;

[0272] Determine, in the pre-saved first indexes, an index matching the third encoding as the target first index;

[0273] In the second index corresponding to the target first index, determine an index matching the third encoding as the target second index;

[0274] Determine an internal row number corresponding to the target second index as the target internal row number;

[0275] Read data according to the target internal line number.

[0276] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 1302, the communication interface, and the memory 1301 communicate with each other via the communication bus.

[0277] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0278] The communication interface is used for communication between the above electronic device and other devices.

[0279] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0280] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0281] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data indexing methods or data reading methods are implemented.

[0282] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any one of the data indexing methods or data reading methods in the above embodiments.

[0283] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0284] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0285] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the device, electronic device, computer-readable storage medium, and computer program product containing instructions, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0286] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A data indexing method, characterized in that: The method comprises: Divide each column of data in the target data table into at least one shard according to the row number; For each of the shards, respectively perform dimension reduction processing on the query conditions of each data in the shard to obtain a first code for each data; For each of the slices, generating a first index of the slice according to the first encoding of each data in the slice; For each of the slices, determine a first digit corresponding to the slice, wherein the first digit is capable of making a conflict rate between second codes corresponding to the data less than a preset conflict rate threshold, the second code is a subcode of the first digit selected from the first code of the data in a preset order, and the first digit corresponding to at least one slice is less than the second digit corresponding to the slice, and the second digit is the number of digits of the first code of each data in the slice; For each of the slices, determining the second codes of the data in the slices according to the first bit number corresponding to the slices as the second indexes of the data; For each data, the second index of the data is saved corresponding to the internal row number of the data, and the first index of the shard to which the data belongs is saved corresponding to the second index of the data, wherein the internal row number is used to indicate the order of the data in the shard.

2. The method according to claim 1, characterized in that The determining the first digit corresponding to the fragment includes: Selecting a subcode of a third digit from the first code of each data of the slice in a preset order, wherein the third digit is initially a preset fourth digit, and the preset fourth digit is smaller than the second digit corresponding to the slice; Determining a conflict rate between the selected sub-codes; If the conflict rate is less than a preset conflict rate threshold, the third digit is determined as the first digit corresponding to the fragment; If the conflict rate is not less than the preset conflict rate threshold, the third digit is increased, and the process returns to the step of selecting the sub-code of the third digit from the first code of each data of the fragment in a preset order.

3. The method according to claim 2, characterized in that The preset fourth digit is 1, and the increasing the third digit includes: The third digit is incremented by one.

4. The method according to claim 1, characterized in that The step of respectively reducing the query conditions of each data in the slice to obtain a first code of each data includes: Determine a fifth digit according to the amount of data in the shard, wherein the fifth digit is positively correlated with the amount; The query conditions of each data in the slice are respectively reduced to the fifth digit to obtain the first code of each data.

5. The method according to claim 1, characterized in that The method further comprises: For each preset column combination, data of each column in the preset column combination is stored in a page corresponding to the preset column combination.

6. The method according to claim 1, characterized in that Generating a first index of the slice according to the first encoding of each data in the slice includes: Extracting a subcode of a preset order in the first code of each data in the slice as the subcode corresponding to each data in the slice; Generate a first index of the slice according to the subcode corresponding to each data in the slice.

7. A data reading method, characterized in that: The method comprises: Acquire a query condition indicated by a target query instruction for a target data table, wherein an index of the target data table is established in advance according to any method of claims 1 to 6; Performing dimensionality reduction processing on the query condition to obtain a third encoding of the query condition; Determine, in the pre-saved first indexes, an index matching the third encoding as the target first index; In the second index corresponding to the target first index, determine an index matching the third encoding as the target second index; Determine an internal row number corresponding to the target second index as the target internal row number; Read data according to the target internal line number.

8. The method according to claim 7, characterized in that The step of reading data according to the target internal row number includes: Determine the column corresponding to each of the target first indexes; wherein the column corresponding to the target first index is the column to which the target shard belongs, and the target shard is the shard to which the target first index belongs; If the determined column hits the preset column combination, data is read from the page corresponding to the preset column combination according to the target internal row number.

9. A data indexing device, characterized in that: The device comprises: A sharding module is used to divide each column of data in the target data table into at least one shard according to the row number; A dimension reduction processing module is used for performing dimension reduction processing on the query conditions of each data in each shard to obtain a first code of each data; A first index generating module, configured to generate, for each of the slices, a first index of the slice according to the first encoding of each data in the slice; a first digit determination module, configured to determine, for each of the slices, a first digit corresponding to the slice, wherein the first digit is capable of making a conflict rate between second codes corresponding to the data less than a preset conflict rate threshold, the second code is a subcode of the first digit selected from the first code of the data in a preset order, and the first digit corresponding to at least one slice is less than a second digit corresponding to the slice, and the second digit is the number of digits of the first code of each data in the slice; A second index determination module is used to determine, for each of the slices, the second code of each data in the slice according to the first bit number corresponding to the slice, as the second index of each data; An index saving module is used to save, for each data, a second index of the data corresponding to an internal row number of the data, and to save the first index of the shard to which the data belongs corresponding to the second index of the data, wherein the internal row number is used to indicate the sequence of the data in the shard.

10. A data reading device, characterized in that: The device comprises: A query condition acquisition module, used to acquire the query condition indicated by the target query instruction for the target data table, wherein the index of the target data table is established in advance according to any method described in claims 1-6; A coding module, used for performing dimensionality reduction processing on the query condition to obtain a third code of the query condition; A target first index determination module, used to determine an index matching the third code in the pre-saved first index as the target first index; A target second index determination module, configured to determine, in the second index corresponding to the target first index, an index matching the third code as the target second index; A target internal row number determination module, used to determine the internal row number corresponding to the target second index as the target internal row number; A data reading module is used to read data according to the target internal row number.

11. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing the method described in any one of claims 1-6 or 7-8 when executing a program stored in a memory.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1-6 or 7-8 is implemented.

Images