Data processing method, device and equipment based on column storage database

Abstract

The invention discloses a data processing method, device and equipment based on a column storage database. The data processing method, device and equipment are used for improving data writing performance under the scene that data missing exists in the database. The method comprises the steps that for a target column in a data table, a bitmap corresponding to the target column is adopted to store a row missing identifier of the target column, whether data missing exists in each row, corresponding to the target column, of each bit of the bitmap or not is judged, and the target column is any column in the data table; storing actual data included in the target column; receiving a data reading request, wherein the data reading request comprises a to-be-read column; according to the bitmap corresponding to the to-be-read column and actual data included in the to-be-read column, a memory structure of the to-be-read column is established in a memory, the memory structure comprises a missing identification column and a data column, the row number of the missing identification column is consistent with that of the data column, each row of the missing identification column corresponds to each bit of the bitmap of the corresponding column, and each bit of the to-be-read column corresponds to the corresponding bit of the data column; the data columns have actual data included in the corresponding columns in the rows with the corresponding missing identification columns not having data missing.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a data processing method, device and equipment based on a column storage database. Background Art

[0002] Based on column storage databases, such as LSM Tree (Log Structured Merge Tree), when writing new data to a data table in the database, you can directly add a row of data to the data table. When you need to modify the data in the data table, you also add a row of modified data to the data table. When reading data, for data with the same row number, the latest row of data is returned to achieve the effect that the data has been modified.

[0003] In some scenarios, a complete row of data cannot be written, resulting in missing data in some rows of the data table. Currently, the usual solution is to read the latest value of the missing column according to the row number when writing data if there is a missing column in the written data to complete the currently written row. However, this solution requires a query completion process for each row of data written, which seriously affects the write performance of the database. Summary of the invention

[0004] In view of this, embodiments of the present application provide a data processing method, apparatus, and device based on a column storage database to improve the performance of data writing in a scenario where data is missing from the database.

[0005] To solve the above problems, the technical solutions provided in the embodiments of the present application are as follows:

[0006] In a first aspect, an embodiment of the present application provides a data processing method based on a column storage database, including:

[0007] For a target column in a data table, a bitmap corresponding to the target column is used to store a row missing flag of the target column, each bit of the bitmap corresponds to whether there is data missing in each row of the target column, and the target column is any column in the data table;

[0008] Storing actual data included in the target column;

[0009] receiving a data read request, wherein the data read request includes a column to be read;

[0010] According to the bitmap corresponding to the column to be read and the actual data included in the column to be read, a memory structure of the column to be read is established in the memory, the memory structure includes a missing identification column and a data column, the missing identification column is consistent with the number of rows of the data column, each row of the missing identification column corresponds to each bit of the bitmap of the corresponding column, and the data column has the actual data included in the corresponding column in the row where the corresponding missing identification column does not have data missing.

[0011] In a second aspect, an embodiment of the present application provides a data processing device based on a column storage database, including:

[0012] A storage unit is used for storing a row missing flag of a target column in a data table by using a bitmap corresponding to the target column, wherein each bit of the bitmap corresponds to whether there is data missing in each row of the target column, and the target column is any column in the data table;

[0013] The storage unit is further used to store the actual data included in the target column;

[0014] A receiving unit, configured to receive a data read request, wherein the data read request includes a column to be read;

[0015] An establishing unit is used to establish a memory structure of the column to be read in a memory according to the bitmap corresponding to the column to be read and the actual data included in the column to be read, wherein the memory structure includes a missing identification column and a data column, the missing identification column has the same number of rows as the data column, each row of the missing identification column corresponds to each bit of the bitmap of the corresponding column, and the data column has the actual data included in the corresponding column in the row where the corresponding missing identification column does not have data missing.

[0016] In a third aspect, an embodiment of the present application provides a data processing device based on a column storage database, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the data processing method based on the column storage database as described above is implemented.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes the data processing method based on the column storage database as described above.

[0018] It can be seen that the embodiments of the present application have the following beneficial effects:

[0019] The embodiment of the present application creates a bitmap for each column in the data table to store the row missing identifier of each column, and stores the actual data of each column at the same time, so as to store the data table when there is data missing. When reading data, the data is converted from the storage structure to the memory structure. In the memory structure, the data columns corresponding to each column of the data table are aligned with the missing identifier column, so as to facilitate reading the data of the corresponding row according to the row number, and realize data reading in the case of data missing. Since the embodiment of the present application supports data writing and reading in the scenario of data missing, there is no need to query the completion process when writing data, which improves the performance of data writing. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of the process of data reading optimization in the predicate push-down scenario in an embodiment of the present application;

[0021] Figure 2 A flowchart of a data processing method based on a column storage database provided in an embodiment of the present application;

[0022] Figure 3 A schematic diagram of a data storage structure in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of the memory structure of data in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of the execution of a predicate expression in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of the execution result of a predicate expression in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of a data processing device based on a column storage database provided in an embodiment of the present application;

[0027] Figure 8 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0029] In order to facilitate understanding and explanation of the technical solution provided by the embodiments of the present application, the background technology of the embodiments of the present application will be described below.

[0030] Based on column storage databases, such as LSM Tree (Log Structured Merge Tree), when writing new data to a data table in the database, you can directly add a row of data to the data table. When you need to modify the data in the data table, you also add a row of modified data to the data table. When reading data, data with the same row number (key) is merged, usually using the latest row of data to overwrite the old row of data. This strategy is called Merge on Read (MOR).

[0031] In actual applications, businesses often have scenarios where some columns need to be written, and a complete row of data cannot be written. For example, a data table consists of key columns: k1, k2 and value columns: v1, v2, v3, v4. The data in this data table comes from different sources. One source only writes: k1, k2, v1, v2 columns, and another source only writes: k1, k2, v3, v4 columns. In other words, different value columns are responsible for different sources, and only part of the value columns are written each time. This will cause some rows in the data table to have missing data.

[0032] In order to ensure row alignment between multiple columns, it is necessary to write corresponding row data for each column. At present, the common solution is to read the latest value of the missing column according to the row number to complete the currently written row when the data is written. The subsequent reading process is no different from the normal written data, and the storage engine does not need to support row data with missing columns. However, this solution requires a query completion process for each row of data written, which seriously affects the write performance of the database.

[0033] In addition, when reading data, in order to improve reading efficiency, it is also necessary to push the filter predicate (such as conditional expression) down to the storage engine, which will perform reading optimization. Common reading optimization methods used by storage engines include:

[0034] 1. With the help of the index, apply the predicate to the index to filter the data. For example, if the predicate is v1>100 (i.e. the value of the v1 column is greater than 100), and the index of rows 1-100 in the data table is the v1 column with a minimum value of 1 and a maximum value of 80, then applying v1>100 to the index will result in rows 1-100 in the data table not satisfying the filter predicate and can be skipped without reading the data of these 100 rows. For another example, if the index of rows 100-150 in the data table is the v1 column with a minimum value of 150 and a maximum value of 200, then the data corresponding to the index satisfies the predicate and rows 100-150 in the data table can be read.

[0035] 2. The predicate expression is executed in advance, and based on the execution result, the target row information is obtained, and then other target column data is read. For example, if the predicate is v1>100, and the value of column v2 is read, the rows in the data table where column v1 is greater than 100 can be determined, and then the value of column v2 can be read from these rows.

[0036] See also Figure 1 As shown, a schematic diagram of the process of storage engine performing read optimization in a scenario with predicate pushdown is shown. For example, according to the data read request, the columns to be read are k1, k2, v2, and v3 columns, and the predicate is k1=x, v1>y, that is, the object columns corresponding to the predicate are k1 and v1 columns. First, the predicate is applied to the index to obtain the candidate row number that satisfies the predicate, for example, there are 100 candidate rows. Then, according to the candidate row number, the object column corresponding to the predicate (k1, v1 columns) is read, and according to the column data, the predicate expression is executed to generate the final row number that satisfies the predicate, for example, there are 10 final target rows. Read the other columns to be read (k2, v2, and v3 columns) of the target row, and aggregate them with the read k1 column to generate the read result. The read result can be handed over to the computing engine for data analysis and processing.

[0037] In the case of missing data, to complete the above data reading process, the following problems need to be solved: How to handle data storage and reading for columns with missing rows in the data table; What changes need to be made to the index to ensure that when the index is used to filter data, the filtering result will not lose data; When the predicate expression is executed, how to handle the missing row information of the column to ensure that the execution result will not lose data.

[0038] Based on this, the embodiments of the present application provide a data processing method, device and equipment based on a columnar storage database. In order to solve the technical problem of reduced write performance caused by the query completion solution when writing data, it is designed from the perspective of the storage engine so that the storage engine supports row data with missing columns, and supports predicate pushdown and read optimization under the premise of missing data.

[0039] To facilitate understanding of the embodiments of the present application, a data processing method based on a column storage database provided in an embodiment of the present application is described below with reference to the accompanying drawings.

[0040] See also Figure 2 As shown in FIG. 1 , this figure is a flow chart of a data processing method based on a column storage database provided in an embodiment of the present application, such as Figure 2 As shown, the method may include S201-S204:

[0041] S201: For a target column in a data table, a bitmap corresponding to the target column is used to store a row missing flag of the target column, where each bit of the bitmap corresponds to whether there is data missing in each row of the target column, and the target column is any column in the data table.

[0042] The embodiments of the present application can be applied to a storage engine, which can implement operations such as writing and reading data in a database. When a data table is written to a storage engine, it is first necessary to establish a storage structure for the data table. For any column in a data table, the column can be called a target column, and whether there is data missing in a row in the column is a 0 / 1 status bit. Therefore, a bitmap structure can be used to store the row missing flag of each column.

[0043] In the embodiment of the present application, each column corresponds to a bitmap, each bit in the bitmap corresponds to a row of the column, and the number of bits in the bitmap is the total number of rows of the column. Each bit in the bitmap is a row missing flag, indicating whether there is data missing in the row. For example, a row missing flag of 1 indicates that there is data missing, and a row missing flag of 0 indicates that there is no data missing.

[0044] In order to improve storage efficiency, the bitmap in the storage structure can use encoding methods such as RLE (Run-length encoding) for lossless data compression storage. The embodiment of the present application does not limit the implementation form of the bitmap.

[0045] S202: storing actual data included in the target column.

[0046] When storing data in each column, for missing rows, it is no longer necessary to store the values ​​of the rows, so only the actual data included in each column in the data table is stored. For example, in a column, the first row includes a, the second row is missing, and the third row includes b, then the actual data a and b included in the column are stored. The actual data can be various types of data such as numerical values ​​and character strings.

[0047] See also Figure 3 As shown, a schematic diagram of the data storage structure is shown. For example, a column includes 6 rows, among which the 2nd and 4th rows have missing data. In the storage structure of the column, the bitmap corresponding to the column has 6 bits, the 2nd and 4th bits are row missing flags 1, and the other bits are row missing flags 0. The data column stores the actual data a, b, a, c included in the other 4 rows of the column. That is, the 1st row corresponds to data a, the 3rd row corresponds to data b, the 5th row corresponds to data a, and the 6th row corresponds to data c.

[0048] S203: Receive a data read request, where the data read request includes a column to be read.

[0049] When data reading is required, a data reading request may be received, the data reading request including columns to be read, that is, columns to be read, for example, columns k1, k2, v2, and v3 are columns to be read.

[0050] The data read request may also include a predicate, which is a conditional expression for a certain object column, indicating that the read data needs to satisfy the predicate. For example, if the predicate is k1=x, v1>y, the object columns corresponding to the predicate are k1 and v1 columns. It is necessary to read the data in the rows that simultaneously satisfy k1=x, v1>y in the k1, k2, v2, and v3 columns. In the case of missing data, how to execute the predicate will be explained in subsequent embodiments and will not be described here.

[0051] S204: According to the bitmap corresponding to the column to be read and the actual data included in the column to be read, a memory structure of the column to be read is established in the memory. The memory structure includes a missing identification column and a data column, the missing identification column has the same number of rows as the data column, each row of the missing identification column corresponds to each bit of the bitmap of the corresponding column, and the data column has the actual data included in the corresponding column in the row where the corresponding missing identification column does not have data missing.

[0052] When querying and reading a column, the data must be converted from the storage structure to the memory column structure, and the memory structure of the column to be read must be established in the memory according to the bitmap corresponding to the column to be read and the actual data included in the column to be read.

[0053] In actual applications, the corresponding memory structure is a nested column consisting of an outer missing flag column (unset column) and an inner data column. In this nested column, the number of rows in the data column is aligned with the number of rows in the outer missing flag column. Each row of the missing flag column corresponds to each bit of the corresponding bitmap, that is, the content of the missing flag column corresponds to the corresponding bitmap. Figure 1 The data column has corresponding actual data in the rows where the corresponding missing indicator column indicates that there is no missing data, and other rows are null values ​​or specified values ​​indicating that the data is null.

[0054] See also Figure 4 As shown in Figure 1, a schematic diagram of the memory structure of data is shown. In this example, Figure 3 The storage structure shown is converted into a memory structure. The memory structure includes a missing identification column and a data column. The missing identification column is consistent with the content of the bitmap. The missing identification column is 1 in the 2nd and 4th rows, indicating that there is data missing. The missing identification column in other rows is 0, indicating that there is no data missing. The number of rows of the data column and the missing identification column is 6. The data column includes actual data in the rows where there is no data missing, that is, the 1st row includes data a, the 3rd row includes data b, the 5th row includes data a, and the 6th row includes data c, while the 2nd and 4th rows are null values ​​or specified values.

[0055] In this way, through the memory structure, it is convenient to directly read the data of the corresponding row according to the row number, thereby improving the efficiency of data analysis and calculation.

[0056] The embodiment of the present application creates a bitmap for each column in the data table to store the row missing identifier of each column, and stores the actual data of each column at the same time, so as to store the data table when there is data missing. When reading data, the data is converted from the storage structure to the memory structure. In the memory structure, the data columns corresponding to each column of the data table are aligned with the missing identifier column, so as to facilitate reading the data of the corresponding row according to the row number, and realize data reading in the case of data missing. Since the embodiment of the present application supports data writing and reading in the scenario of data missing, there is no need to query the completion process when writing data, which improves the performance of data writing.

[0057] For a column in a data table, if there is missing row data, the predicate judgment cannot be performed, so the result of the predicate execution should be true, that is, the predicate condition is satisfied. For the index on the column in the data table, when the predicate is applied to the index, this logic also needs to be followed. In an embodiment of the present application, a method for processing the index in the data table when there is missing data in the data table is provided.

[0058] With respect to the maximum and minimum value indexes, in a possible implementation manner, the data processing method based on the column storage database provided in the embodiment of the present application may further include:

[0059] A1: Reads the maximum and minimum value indexes of the first object column in the data table.

[0060] If a column in the data table has a maximum and minimum index, then the column is the first object column and the maximum and minimum index is read. The maximum and minimum index records the maximum and minimum values ​​within a range of rows in the first object column. For example, through the maximum and minimum index, you can read that the maximum value of the v1 column in the range of rows 10 to 100 is 100 and the minimum value is 10.

[0061] A2: Read the first row number range in the bitmap corresponding to the first object column. The first row number range is the row number range corresponding to the maximum and minimum value indexes.

[0062] The row number range corresponding to the maximum and minimum value indexes is the first row number range. The content in the first row number range is read from the bitmap corresponding to the first object column to determine whether there is data missing in each row in the first row number range.

[0063] A3: If there is missing data in the first row number range, mark the maximum and minimum indexes as invalid so that the first row number range in the data table is not filtered when the maximum and minimum indexes are applied.

[0064] If there is missing data in the first row number range, it is actually impossible to determine the maximum and minimum values ​​of the first object column in the first row number range, so the maximum and minimum indexes need to be marked as invalid. When applying any predicate to the maximum and minimum indexes, the execution result needs to be returned as true (true), that is, the index cannot judge the predicate as false (false), and the first row number range in the data table is not filtered. The row numbers in the first row number range need to be retained as candidate row numbers.

[0065] Based on this, in a possible implementation manner, the data processing method based on the column storage database provided in the embodiment of the present application may also include:

[0066] If the data read request also includes a predicate, and the predicate is a conditional expression of the first object column, when the predicate is applied to the maximum and minimum value indexes, it is determined that the maximum and minimum value indexes satisfy the predicate, and all row numbers in the first row number range are output as candidate row numbers.

[0067] That is, if the data read request also includes a predicate, the predicate is a conditional expression of the first object column, for example, the predicate is v1>120. When the predicate is applied to the maximum and minimum indexes, since the maximum and minimum indexes have been marked as invalid, it is determined that the maximum and minimum indexes satisfy the predicate, and all row numbers in the first row number range are output as candidate row numbers. For example, the maximum and minimum indexes are v1 columns in the range of rows 10 to 100, and the maximum value is 100, and the minimum value is 10. Since there is missing data in the v1 column, rows 10 to 100 cannot be skipped by the predicate v1>120, and the maximum and minimum indexes are marked as invalid, and rows 10 to 100 are output as candidate row numbers.

[0068] In the embodiment of the present application, when there is data missing in the data table, the maximum and minimum value indexes are adjusted accordingly, and the adjusted maximum and minimum value indexes can be correctly applied when reading data.

[0069] With respect to the Bloom filter index, in a possible implementation manner, the data processing method based on the column storage database provided in the embodiment of the present application may further include:

[0070] B1: Read the Bloom filter index of the second object column in the data table.

[0071] If a column in the data table has a Bloom filter index, then the column is the second object column, and the Bloom filter index is read. The Bloom filter index builds a Bloom filter for the values ​​that appear in a range of rows in the second object column. For values ​​that have not appeared, when querying the Bloom filter, there is a high probability that the execution result returned is false (false), while for values ​​that have appeared, when querying the Bloom filter, the result returned is definitely true (true).

[0072] B2: Read the second row number range in the bitmap corresponding to the second object column, where the second row number range is the row number range corresponding to the Bloom filter index.

[0073] The row number range corresponding to the Bloom filter index is the second row number range, and the content in the second row number range is read from the bitmap corresponding to the second object column to determine whether there is data missing in each row in the second row number range.

[0074] B3: If there is missing data in the second row number range, mark the Bloom filter index as invalid so that the second row number range in the data table is not filtered when the Bloom filter index is applied.

[0075] Similar to the maximum and minimum indexes, if there is missing data in the second row number range, the Bloom filter index needs to be marked as invalid. When applying any predicate to the Bloom filter, the execution result needs to be returned as true, that is, the index cannot judge the predicate as false, and the second row number range in the data table is not filtered. The row numbers in the second row number range need to be retained as candidate row numbers.

[0076] Based on this, in a possible implementation, the data processing method based on the column storage database provided in the embodiment of the present application may also include:

[0077] If the data read request further includes a predicate, and the predicate is a conditional expression of the second object column, when the predicate is applied to the Bloom filter index, it is determined that the Bloom filter index satisfies the predicate, and all row numbers in the second row number range are output as candidate row numbers.

[0078] That is, if the data read request further includes a predicate, the predicate is a conditional expression of the second object column. When the predicate is applied to the Bloom filter index, since the Bloom filter index has been marked as invalid, it is determined that the Bloom filter index satisfies the predicate, and all row numbers in the second row number range are output as candidate row numbers.

[0079] In the embodiment of the present application, when there is data missing in the data table, the Bloom filter index is adjusted accordingly, and the adjusted Bloom filter index can be correctly applied when reading data.

[0080] Regarding the inverted index, in a possible implementation, the data processing method based on the column storage database provided in the embodiment of the present application may also include:

[0081] C1: Read the inverted index of the third object column in the data table.

[0082] If a column in the data table has an inverted index, then the column is the third object column and the inverted index is read. The inverted index establishes an inverted mapping between the values ​​that appear in a certain range in the third object column and the row numbers where the values ​​appear. For example, Figure 3-Figure 4 In the example, there are three values ​​a, b, and c. The inverted index needs to maintain the mapping between these three values ​​and the row numbers and establish an inverted chain: a->1, 5, b->3, c->6.

[0083] C2: Read the third row number range in the bitmap corresponding to the third object column, where the third row number range is the row number range corresponding to the inverted index.

[0084] The row number range corresponding to the inverted index is the third row number range, and the content in the third row number range is read from the bitmap corresponding to the third object column to determine whether there is data missing in each row in the third row number range.

[0085] C3: Create an inverted chain of missing values ​​based on the row numbers with missing data in the third row number range, and add the inverted chain of missing values ​​to the inverted index.

[0086] If there is missing data in the third row number range, you need to create a missing value inverted chain for the row number with missing data and add the missing value inverted chain to the inverted index. For example, based on the above example, you also need to add a missing value inverted chain to the inverted index: missing value->2, 4.

[0087] Based on this, in a possible implementation, the data processing method based on the column storage database provided in the embodiment of the present application may also include:

[0088] If the data read request further includes a predicate, and the predicate is a conditional expression of the third object column, when the predicate is applied to the inverted index, the row numbers in the inverted chain that satisfy the predicate and the row numbers in the missing value inverted chain are determined as candidate row numbers.

[0089] When there is no missing data, the predicate is applied to the index. For example, when the third object column = a, the postings chain a->1, 5 that satisfies the predicate can be read to obtain row numbers 1 and 5 as candidate row numbers. However, when there is missing data, this cannot be done directly. The row numbers of the rows with missing data need to be included in the candidate row number range. Therefore, it is also necessary to read the missing value postings chain. For example, when the predicate third object column = a is applied, the two postings chains a->1, 5 and missing value->2, 4 need to be merged to obtain 1, 2, 4, 5 as candidate row numbers.

[0090] In the embodiment of the present application, when there is data missing in the data table, the inverted index is adjusted accordingly, and the adjusted inverted index can be correctly applied when reading data.

[0091] In addition, the embodiment of the present application also provides an execution process of a predicate expression on a memory structure in a data missing scenario.

[0092] The first solution for executing predicate expressions that support missing rows, in a possible implementation, the data processing method based on a column storage database provided in the embodiment of the present application may also include:

[0093] D1: If the data read request further includes a predicate, and the predicate is a conditional expression of the fourth object column, a memory structure of the fourth object column is established according to a bitmap corresponding to the fourth object column and actual data included in the fourth object column.

[0094] If the data read request also includes a predicate, the predicate is a conditional expression of the fourth object column. The fourth object column may be a to-be-read column, and the memory structure of the fourth object column may be obtained at this time. When the fourth object column is not a to-be-read column, the storage structure of the fourth object column may be converted into a memory structure. Similarly, the memory structure of the fourth object column includes a missing identification column and a data column.

[0095] D2: Determine the target row number where the actual data included in the target data column satisfies the predicate, and determine the row number where the target missing identification column has missing data as the target row number that satisfies the predicate; the target data column is the data column in the memory structure of the fourth object column, and the target missing identification column is the missing identification column in the memory structure of the fourth object column.

[0096] By executing the predicate on the data column in the memory structure of the fourth object column, the row number in the data column (i.e., the target data column) that satisfies the predicate can be obtained as the target row number. If the fourth object column has an index, the fourth object column is the first object column, the second object column, or the third object column, and the row number that satisfies the predicate can be determined from the candidate row numbers. If the fourth object column does not have an index, the row number that satisfies the predicate is determined from all the row numbers. At the same time, the row number where the target missing identification column has data missing is determined as the target row number that satisfies the predicate, and the two together constitute the output target row number.

[0097] That is, each predicate conditional expression execution operator supports the memory structure of nested columns. When the operator is executed, if it finds that the missing flag value of a row is 1, the execution result of the operator on that row is set to true. )=is_unset(unset<>)OR F(A).

[0098] D3: Obtain the data of the column to be read within the target row number from the memory structure of the column to be read.

[0099] Finally, the actual data in the target row number is read from the memory structure of the column to be read.

[0100] However, this solution is quite destructive to the execution engine, and requires that each predicate conditional expression execution operator supports this solution, and the missing identification value needs to be checked during the calculation process of each predicate, which will affect the execution efficiency. Therefore, the embodiment of the present application also provides a second solution for supporting the execution of predicate expressions with missing rows.

[0101] The second solution for executing predicate expressions that support missing rows, in a possible implementation, the data processing method based on a column storage database provided in the embodiment of the present application may also include:

[0102] E1: If the data read request further includes a predicate, and the predicate is a conditional expression of the fourth object column, a memory structure of the fourth object column is established according to a bitmap corresponding to the fourth object column and actual data included in the fourth object column.

[0103] If the data read request also includes a predicate, the predicate is a conditional expression of the fourth object column. The fourth object column may be a to-be-read column, and the memory structure of the fourth object column may be obtained at this time. When the fourth object column is not a to-be-read column, the storage structure of the fourth object column may be converted into a memory structure. Similarly, the memory structure of the fourth object column includes a missing identification column and a data column.

[0104] E2: For rows where data is missing in the target missing identification column, a default value is written into the target data column; the target data column is the data column in the memory structure of the fourth object column, and the target missing identification column is the missing identification column in the memory structure of the fourth object column.

[0105] For the fourth object column, the missing identification column in the memory structure is the target missing identification column, and the data column in the memory structure is the target data column. First, the target missing identification column needs to be stripped off, and only the target data column is retained. That is, for the rows where the target missing identification column has missing data, a default value, such as 0, is written in the corresponding row of the target data column. The default value can be set according to the actual situation, and this embodiment of the application does not limit this.

[0106] E3: Determine the target row number where the actual data included in the target data column satisfies the predicate, and determine the row number where the target data column is a default value as the target row number that satisfies the predicate.

[0107] By executing the predicate on the updated data column, the row number in the target data column that satisfies the predicate can be obtained as the target row number. If the fourth object column has an index, the fourth object column is the first object column, the second object column, or the third object column, and the row number that satisfies the predicate can be determined from the candidate row numbers. If the fourth object column does not have an index, the row number that satisfies the predicate is determined from all the row numbers. At the same time, the row number of the target data column with the default value is determined as the target row number that satisfies the predicate, and the two together constitute the output target row number.

[0108] E4: Change the default value in the target data column to missing data.

[0109] If the fourth object column is a to-be-read column, the default value in the target data column also needs to be modified to data missing, such as a null value or a specified value representing that the data is null, and restored to the original memory structure.

[0110] E5: Obtain the data of the column to be read within the target row number from the memory structure of the column to be read.

[0111] Finally, the actual data in the target row number is read from the memory structure of the column to be read.

[0112] See also Figure 5 As shown in FIG. 1 , a schematic diagram of the execution of the predicate expression in this embodiment is shown. For example, the predicate is A>0 and B>0, that is, column A>0 and column B>0, column A and column B are the fourth object columns, and the column to be read is column A. The data of column A and column B are as follows Figure 5 As shown on the left, unset is a specified value that indicates that the data in the data column is empty. After removing the missing data column, as shown Figure 5As shown on the right, the default value 0 is written into the specified value in the data column. The result of executing the predicate expression for each row is (false, true, false). Since the default value appears in the first row (in column A) and the third row (in column B), the execution result is rewritten to (true, true, true), and the output target row numbers are 1, 2, and 3.

[0113] See also Figure 6 As shown, a schematic diagram of the execution result of the predicate expression in this embodiment is shown. Based on the above example, column A is used as a column to be read, and the unset information needs to be filled in, that is, the default value in the data column of column A is changed to unset, and the output result of the final execution engine is shown in the figure.

[0114] In this embodiment, it is only necessary to process the data column before the execution of the predicate expression, so that the data is stored in a column that can be recognized by the normal expression. After execution, the execution result is rewritten according to the unset information. The predicate expression execution engine does not need to recognize the unset information during the entire process. In addition, this simplified processing method may result in missed data filtering and multiple recalls. For example, based on the above example, the predicate expression is modified to A>0 and B>10. Although the first and third rows have unset information, they do not meet the predicate conditions. According to this solution, these two rows are still considered to be likely to meet the predicate and are returned as the execution result. This simplified processing process and the way of returning multiple data are usually allowed in the scenario of predicate push-down of the actual data column, and will not affect the correctness of the subsequent processing results.

[0115] Based on the data processing method based on a column storage database provided in the above method embodiment, the embodiment of the present application also provides a data processing device based on a column storage database, which will be described below in conjunction with the accompanying drawings.

[0116] See also Figure 7 As shown in FIG. 1 , this figure is a schematic diagram of the structure of a data processing device based on a column storage database provided in an embodiment of the present application. Figure 7 As shown, the data processing device based on the column storage database includes:

[0117] A storage unit 701 is used for storing a row missing flag of a target column in a data table using a bitmap corresponding to the target column, wherein each bit of the bitmap corresponds to whether there is data missing in each row of the target column, and the target column is any column in the data table;

[0118] The storage unit 701 is further used to store the actual data included in the target column;

[0119] A receiving unit 702 is configured to receive a data read request, wherein the data read request includes a column to be read;

[0120] The first establishing unit 703 is used to establish a memory structure of the column to be read in the memory according to the bitmap corresponding to the column to be read and the actual data included in the column to be read, the memory structure including a missing identification column and a data column, the missing identification column is consistent with the number of rows of the data column, each row of the missing identification column corresponds to each bit of the bitmap of the corresponding column, and the data column has the actual data included in the corresponding column in the row where the corresponding missing identification column does not have data missing.

[0121] In a possible implementation manner, the device further includes:

[0122] A first reading unit, configured to read the maximum and minimum value indexes of the first object column in the data table;

[0123] A second reading unit, configured to read a first row number range in the bitmap corresponding to the first object column, wherein the first row number range is a row number range corresponding to the maximum and minimum value indexes;

[0124] The first marking unit is used to mark the maximum and minimum value index as invalid if there is missing data in the first row number range, so that the first row number range in the data table is not filtered when the maximum and minimum value index is applied.

[0125] In a possible implementation manner, the device further includes:

[0126] A third reading unit, configured to read a Bloom filter index of a second object column in the data table;

[0127] A fourth reading unit, configured to read a second row number range in the bitmap corresponding to the second object column, where the second row number range is a row number range corresponding to the Bloom filter index;

[0128] The second marking unit is configured to mark the Bloom filter index as invalid if there is missing data in the second row number range, so that the second row number range in the data table is not filtered when the Bloom filter index is applied.

[0129] In a possible implementation manner, the device further includes:

[0130] A fifth reading unit, configured to read an inverted index of a third object column in the data table;

[0131] a sixth reading unit, configured to read a third row number range in the bitmap corresponding to the third object column, wherein the third row number range is a row number range corresponding to the inverted index;

[0132] An adding unit is used to establish a missing value inverted chain according to the row numbers with missing data in the third row number range, and add the missing value inverted chain to the inverted index.

[0133] In a possible implementation manner, the device further includes:

[0134] A first determining unit is configured to, if the data read request further includes a predicate, and the predicate is a conditional expression of the first object column, determine when the predicate is applied to the maximum and minimum value indexes, that the maximum and minimum value indexes satisfy the predicate, and output all row numbers in the first row number range as candidate row numbers.

[0135] In a possible implementation manner, the device further includes:

[0136] A second determining unit is configured to, if the data read request further includes a predicate, and the predicate is a conditional expression of the second object column, determine when the predicate is applied to the Bloom filter index that the Bloom filter index satisfies the predicate, and output all row numbers in the second row number range as candidate row numbers.

[0137] In a possible implementation manner, the device further includes:

[0138] a third determining unit, configured to determine, if the data read request further includes a predicate, and the predicate is a conditional expression of the third object column, when applying the predicate to the inverted index, the row numbers in the inverted chain that satisfy the predicate and the row numbers in the missing value inverted chain as candidate row numbers.

[0139] In a possible implementation manner, the device further includes:

[0140] a second establishing unit, configured to establish a memory structure of the fourth object column according to a bitmap corresponding to the fourth object column and actual data included in the fourth object column if the data read request further includes a predicate and the predicate is a conditional expression of the fourth object column;

[0141] a fourth determining unit, configured to determine a target row number in which actual data included in a target data column satisfies the predicate, and to determine a row number in which data in a target missing identification column is missing as the target row number in which the predicate is satisfied; the target data column is a data column in the memory structure of the fourth object column, and the target missing identification column is a missing identification column in the memory structure of the fourth object column;

[0142] The first acquisition unit is used to acquire the data of the data column of the to-be-read column within the target row number from the memory structure of the to-be-read column.

[0143] In a possible implementation manner, the device further includes:

[0144] a third establishing unit, configured to establish a memory structure of the fourth object column according to a bitmap corresponding to the fourth object column and actual data included in the fourth object column if the data read request further includes a predicate and the predicate is a conditional expression of the fourth object column;

[0145] A writing unit, used to write a default value in a target data column for a row where data in the target missing identification column is missing; the target data column is a data column in the memory structure of the fourth object column, and the target missing identification column is a missing identification column in the memory structure of the fourth object column;

[0146] A fifth determining unit, configured to determine a target row number at which actual data included in the target data column satisfies the predicate, and determine a row number at which the target data column is a default value as the target row number at which the predicate is satisfied;

[0147] A modification unit, used to modify the default value in the target data column to data missing;

[0148] The second acquisition unit is used to acquire the data of the data column of the to-be-read column within the target row number from the memory structure of the to-be-read column.

[0149] In addition, an embodiment of the present application further provides a computer program product, including computer program instructions, which, when executed on a computer, enable the computer to execute any of the data processing methods based on a column storage database as described above.

[0150] Based on the data processing method based on a column storage database provided by the above method embodiment, the present application also provides an electronic device, including: one or more processors; a storage device, on which one or more programs are stored, when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method based on a column storage database described in any of the above embodiments.

[0151] Reference below Figure 8, which shows a schematic diagram of the structure of an electronic device 1300 suitable for implementing the embodiment of the present application. The terminal device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (portable android devices), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs (televisions), desktop computers, etc. Figure 8 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0152] like Figure 8 As shown, the electronic device 1300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage device 1306 into a random access memory (RAM) 1303. In the RAM 1303, various programs and data required for the operation of the electronic device 1300 are also stored. The processing device 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0153] Typically, the following devices may be connected to the I / O interface 1305: an input device 1306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1306 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1309. The communication device 1309 may allow the electronic device 1300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 The electronic device 1300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0154] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 1309, or installed from the storage device 1306, or installed from the ROM 1302. When the computer program is executed by the processing device 1301, the above-mentioned functions defined in the method of the embodiment of the present application are executed.

[0155] The electronic device provided in the embodiment of the present application and the data processing method based on a column storage database provided in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0156] Based on the data processing method based on a column storage database provided in the above method embodiment, an embodiment of the present application provides a computer-readable medium on which a computer program is stored, wherein when the program is executed by a processor, the data processing method based on a column storage database as described in any of the above embodiments is implemented.

[0157] It should be noted that the computer-readable medium in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In an embodiment of the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In an embodiment of the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0158] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0159] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0160] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the data processing method based on the column storage database.

[0161] The computer program code for performing the operation of the embodiment of the present application can be written in one or more programming languages ​​or a combination thereof, and the above-mentioned programming languages ​​include but are not limited to object-oriented programming languages-such as Java, Smalltalk, C++, and also include conventional procedural programming languages-such as "C" language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network-including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0162] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0163] The units involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the unit / module does not, in some cases, constitute a limitation on the unit itself.

[0164] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0165] In the context of the present application embodiment, machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include but is not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0166] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0167] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0168] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0169] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0170] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method based on a column storage database, characterized in that: include: For a target column in a data table, a bitmap corresponding to the target column is used to store a row missing flag of the target column, each bit of the bitmap corresponds to whether there is data missing in each row of the target column, and the target column is any column in the data table; Storing actual data included in the target column; receiving a data read request, wherein the data read request includes a column to be read; According to the bitmap corresponding to the column to be read and the actual data included in the column to be read, a memory structure of the column to be read is established in the memory, the memory structure includes a missing identification column and a data column, the missing identification column is consistent with the number of rows of the data column, each row of the missing identification column corresponds to each bit of the bitmap of the corresponding column, and the data column has the actual data included in the corresponding column in the row where the corresponding missing identification column does not have data missing.

2. The method according to claim 1, characterized in that The method further comprises: Read the maximum and minimum value indexes of the first object column in the data table; Read a first row number range in the bitmap corresponding to the first object column, where the first row number range is the row number range corresponding to the maximum and minimum value indexes; If there is missing data in the first row number range, the maximum and minimum value indexes are marked as invalid, so that the first row number range in the data table is not filtered when the maximum and minimum value indexes are applied.

3. The method according to claim 1, characterized in that: The method further comprises: Reading a Bloom filter index of a second object column in the data table; Read a second row number range in the bitmap corresponding to the second object column, where the second row number range is a row number range corresponding to the Bloom filter index; If data is missing in the second row number range, the Bloom filter index is marked as invalid, so that the second row number range in the data table is not filtered when the Bloom filter index is applied.

4. The method according to claim 1, characterized in that The method further comprises: Reading an inverted index of a third object column in the data table; Read a third row number range in the bitmap corresponding to the third object column, where the third row number range is the row number range corresponding to the inverted index; A missing value inverted chain is established according to the row numbers having missing data within the third row number range, and the missing value inverted chain is added to the inverted index.

5. The method according to claim 2, characterized in that: The method further comprises: If the data read request also includes a predicate, and the predicate is a conditional expression of the first object column, when the predicate is applied to the maximum and minimum value indexes, it is determined that the maximum and minimum value indexes satisfy the predicate, and all row numbers in the first row number range are output as candidate row numbers.

6. The method according to claim 3, characterized in that The method further comprises: If the data read request also includes a predicate, and the predicate is a conditional expression of the second object column, when the predicate is applied to the Bloom filter index, it is determined that the Bloom filter index satisfies the predicate, and all row numbers in the second row number range are output as candidate row numbers.

7. The method according to claim 4, characterized in that The method further comprises: If the data read request further includes a predicate, and the predicate is a conditional expression of the third object column, when the predicate is applied to the inverted index, the row numbers in the inverted chain that satisfy the predicate and the row numbers in the missing value inverted chain are determined as candidate row numbers.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: If the data read request further includes a predicate, and the predicate is a conditional expression of the fourth object column, establishing a memory structure of the fourth object column according to a bitmap corresponding to the fourth object column and actual data included in the fourth object column; Determine a target row number where actual data included in a target data column satisfies the predicate, and determine a row number where data is missing in a target missing identification column as the target row number that satisfies the predicate; the target data column is a data column in the memory structure of the fourth object column, and the target missing identification column is a missing identification column in the memory structure of the fourth object column; The data of the data column of the to-be-read column within the target row number is obtained from the memory structure of the to-be-read column.

9. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: If the data read request further includes a predicate, and the predicate is a conditional expression of the fourth object column, establishing a memory structure of the fourth object column according to a bitmap corresponding to the fourth object column and actual data included in the fourth object column; For rows where data is missing in the target missing identification column, a default value is written into the target data column accordingly; the target data column is a data column in the memory structure of the fourth object column, and the target missing identification column is a missing identification column in the memory structure of the fourth object column; Determine a target row number where actual data included in the target data column satisfies the predicate, and determine a row number where the target data column is a default value as the target row number that satisfies the predicate; Modify the default value in the target data column to data missing; The data of the data column of the to-be-read column within the target row number is obtained from the memory structure of the to-be-read column.

10. A data processing device based on a column storage database, characterized in that: include: A storage unit is used for storing a row missing flag of a target column in a data table by using a bitmap corresponding to the target column, wherein each bit of the bitmap corresponds to whether there is data missing in each row of the target column, and the target column is any column in the data table; The storage unit is further used to store the actual data included in the target column; A receiving unit, configured to receive a data read request, wherein the data read request includes a column to be read; An establishing unit is used to establish a memory structure of the column to be read in a memory according to the bitmap corresponding to the column to be read and the actual data included in the column to be read, wherein the memory structure includes a missing identification column and a data column, the missing identification column has the same number of rows as the data column, each row of the missing identification column corresponds to each bit of the bitmap of the corresponding column, and the data column has the actual data included in the corresponding column in the row where the corresponding missing identification column does not have data missing.

11. A data processing device based on a column storage database, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the data processing method based on a column storage database as described in any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the data processing method based on a column storage database as described in any one of claims 1 to 9.

Images