Data processing method and apparatus, electronic device, and storage medium

Abstract

One or more embodiments of the present specification provide a data processing method and device, an electronic device and a storage medium, the method comprising: in the process of writing target data in the memory to the disk, in response to receiving a query instruction for the target data, querying data in at least one first local data and at least one second local data according to the query instruction and intermediate index layer information of the at least one first local data and intermediate index layer information of the at least one second local data, and sorting the queried data as a data query result; wherein the first local data comprises part of the data stored in the disk in the target data, the second local data comprises part of the data stored in the memory in the target data, the target data is in column storage form, and each column group of the target data corresponds to a second local data.

Description

Technical Field

[0001] This specification relates to the field of database technology, and more particularly to a data processing method and apparatus, electronic device and storage medium. Background Technology

[0002] In today's era of rapid development in the internet and information technology, the generation of data is exploding, thus placing increasingly higher demands on databases and their management. Data processing requires the use of DML (Data Manipulation Language) to manipulate data tables, such as adding, deleting, querying, and modifying data; it also requires the use of DDL (Data Definition Language) to reorganize data, such as creating new tables, deleting columns, and changing column types.

[0003] In related technologies, during the process of reorganizing data using DDL, the relevant data cannot be provided for external query services, which affects the database's business processing and causes business problems such as business request timeouts. Summary of the Invention

[0004] In view of the above, this specification provides a data processing method and apparatus, an electronic device and a storage medium through one or more embodiments.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a data processing method is provided, the method comprising:

[0007] During the process of writing target data from memory to disk, in response to receiving a query instruction for the target data, according to the query instruction, and intermediate index layer information of at least one first local data and at least one second local data, data is queried in the at least one first local data and the at least one second local data respectively, and the queried data is sorted and used as the data query result;

[0008] The first local data includes a portion of the target data stored on the disk, and the second local data includes a portion of the target data stored in memory. The target data is in columnar storage format, and each column group of the target data corresponds to a second local data.

[0009] In one embodiment of this specification, the query instruction includes a query range and query conditions;

[0010] The step of querying data in the at least one first local data and the at least one second local data according to the query instruction, and intermediate index layer information of at least one first local data and intermediate index layer information of at least one second local data respectively, and sorting the queried data as the data query result, includes:

[0011] Based on the intermediate index layer information of at least one first local data and the intermediate index layer information of at least one second local data, data within the query range are queried in the at least one first local data and the at least one second local data respectively, and the queried data is sorted and used as the data to be queried;

[0012] The query results are determined by querying the data to be queried according to the query conditions.

[0013] In one embodiment of this specification, the query range includes a column query range for at least one column group;

[0014] The step of querying data within the query range based on the query range, intermediate index layer information of at least one first local data, and intermediate index layer information of at least one second local data, and sorting the queried data to serve as the data to be queried, includes:

[0015] For each column query range in the query range, the first row offset range corresponding to the column query range of the column group is determined based on the intermediate index layer information of the first local data and the intermediate index layer information of the second local data of the column group, as well as the column query range of the column group.

[0016] The second row offset range is determined based on the first row offset range corresponding to the query range of each column in the query range;

[0017] Based on the second row offset range, and the intermediate index layer information of at least one first local data and at least one second local data, data within the second row offset range are queried in the at least one first local data and the at least one second local data respectively, and the queried data is sorted and used as the data to be queried.

[0018] In one embodiment of this specification, determining the first row offset range corresponding to the column query range of the column group based on the intermediate index layer information of the first local data and the intermediate index layer information of the second local data of the column group, and the column query range of the column group, includes:

[0019] Based on the intermediate index layer information of the first local data and the intermediate index layer information of the second local data of the column group, the data within the third row offset range of the query range is queried in the first local data and the second local data of the column group respectively, and the queried data is sorted according to the row offset and used as the column query result of the column group.

[0020] The column query results of the column group are filtered according to the column query range of the column group to obtain the first row offset range corresponding to the column query range of the column group.

[0021] In one embodiment of this specification, the query range includes the primary key range;

[0022] The method further includes:

[0023] Based on the intermediate index layer information of the first local data and the intermediate index layer information of the second local data of the primary key column, the data within the primary key range is queried in the first local data and the second local data of the primary key column respectively, and the queried data is sorted according to the row offset and used as the primary key query result;

[0024] Based on the row offset of the data in the primary key query result, determine the second row offset range within the query range.

[0025] In one embodiment of this specification, the data within the first local data and the second local data are stored in the form of data blocks, with each database storing a portion of the data;

[0026] The step of querying data within the query range based on intermediate index layer information of at least one first local data and intermediate index layer information of at least one second local data, and sorting the queried data as the data to be queried, includes:

[0027] Based on the intermediate index layer information of at least one first local data and the intermediate index layer information of at least one second local data, the data blocks to which the data within the query range belongs are queried in the at least one first local data and the at least one second local data, respectively. The queried data blocks are sorted and used as the data to be queried.

[0028] In one embodiment of this specification, the data block includes macroblocks and microblocks.

[0029] In one embodiment of this specification, the target data includes:

[0030] Data generated in the memory according to the Data Definition Language (DDL).

[0031] In one embodiment of this specification, different column groups of the target data correspond to the same first local data; or, different column groups of the target data correspond to different first local data.

[0032] According to a second aspect of one or more embodiments of this specification, a data processing apparatus is provided, the apparatus comprising:

[0033] The real-time query module is used to respond to a query instruction for the target data during the process of writing target data in memory to disk, and query data in the at least one first local data and the at least one second local data according to the query instruction, as well as intermediate index layer information of at least one first local data and intermediate index layer information of at least one second local data respectively, and sort the queried data as the data query result.

[0034] The first local data includes a portion of the target data stored on the disk, and the second local data includes a portion of the target data stored in memory. The target data is in columnar storage format, and each column group of the target data corresponds to a second local data.

[0035] In one embodiment of this specification, the query instruction includes a query range and query conditions;

[0036] The real-time query module is used for:

[0037] Based on the intermediate index layer information of at least one first local data and the intermediate index layer information of at least one second local data, data within the query range are queried in the at least one first local data and the at least one second local data respectively, and the queried data is sorted and used as the data to be queried;

[0038] The query results are determined by querying the data to be queried according to the query conditions.

[0039] In one embodiment of this specification, the query range includes a column query range for at least one column group;

[0040] The real-time query module is used to query data within the query range based on the query range, intermediate index layer information of at least one first local data, and intermediate index layer information of at least one second local data, respectively. When sorting the queried data and using it as the data to be queried, it is used for:

[0041] For each column query range in the query range, the first row offset range corresponding to the column query range of the column group is determined based on the intermediate index layer information of the first local data and the intermediate index layer information of the second local data of the column group, as well as the column query range of the column group.

[0042] The second row offset range is determined based on the first row offset range corresponding to the query range of each column in the query range;

[0043] Based on the second row offset range, and the intermediate index layer information of at least one first local data and at least one second local data, data within the second row offset range are queried in the at least one first local data and the at least one second local data respectively, and the queried data is sorted and used as the data to be queried.

[0044] In one embodiment of this specification, when the real-time query module determines the first row offset range corresponding to the column query range of the column group based on the intermediate index layer information of the first local data and the intermediate index layer information of the second local data of the column group, and the column query range of the column group, it is used to:

[0045] Based on the intermediate index layer information of the first local data and the intermediate index layer information of the second local data of the column group, the data within the third row offset range of the query range is queried in the first local data and the second local data of the column group respectively, and the queried data is sorted according to the row offset and used as the column query result of the column group.

[0046] The column query results of the column group are filtered according to the column query range of the column group to obtain the first row offset range corresponding to the column query range of the column group.

[0047] In one embodiment of this specification, the query range includes the primary key range;

[0048] The device further includes a key module for:

[0049] Based on the intermediate index layer information of the first local data and the intermediate index layer information of the second local data of the primary key column, the data within the primary key range is queried in the first local data and the second local data of the primary key column respectively, and the queried data is sorted according to the row offset and used as the primary key query result;

[0050] Based on the row offset of the data in the primary key query result, determine the second row offset range within the query range.

[0051] In one embodiment of this specification, the data within the first local data and the second local data are stored in the form of data blocks, with each database storing a portion of the data;

[0052] The real-time query module is used to query data within the query range based on intermediate index layer information of at least one first local data and intermediate index layer information of at least one second local data, respectively, and when sorting the queried data as the data to be queried, it is used for:

[0053] Based on the intermediate index layer information of at least one first local data and the intermediate index layer information of at least one second local data, the data blocks to which the data within the query range belongs are queried in the at least one first local data and the at least one second local data, respectively. The queried data blocks are sorted and used as the data to be queried.

[0054] In one embodiment of this specification, the data block includes macroblocks and microblocks.

[0055] In one embodiment of this specification, the target data includes:

[0056] Data generated in the memory according to the Data Definition Language (DDL).

[0057] In one embodiment of this specification, different column groups of the target data correspond to the same first local data; or, different column groups of the target data correspond to different first local data.

[0058] According to a third aspect of one or more embodiments of this specification, an electronic device is provided, comprising:

[0059] processor;

[0060] Memory used to store processor-executable instructions;

[0061] The processor implements the method as described in the first aspect by running the executable instructions.

[0062] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0063] The technical solutions provided in the embodiments of this specification may include the following beneficial effects:

[0064] The data processing method provided in this specification can, in response to receiving a query instruction for the target data during the process of writing target data from memory to disk, query data in the at least one first local data and the at least one second local data according to the query instruction, intermediate index layer information of at least one first local data, and intermediate index layer information of at least one second local data, respectively, and sort the queried data as the data query result. The first local data includes the portion of the target data stored on disk, and the second local data includes the portion of the target data stored in memory. The target data is in columnar storage format, and each column group of the target data corresponds to one second local data. In other words, this method can provide query services in real time during the process of writing columnar storage data from memory to disk. Since each column group of columnar storage data is stored as an independent second local data in memory, this method sorts the queried data after querying multiple local data in disk and memory, thereby ensuring the accuracy of the data query results. Furthermore, this method can provide query services for relevant data during the process of defining data tables using DDL, thereby ensuring the business processing response speed of the database and avoiding problems such as business request timeouts. Attached Figure Description

[0065] Figure 1 This is a flowchart of a data processing method provided in an exemplary embodiment.

[0066] Figure 2 This is a schematic diagram of intermediate index layer information provided in an exemplary embodiment.

[0067] Figure 3 This is a flowchart of a data processing method provided in an exemplary embodiment.

[0068] Figure 4 This is a schematic diagram of a multi-way merge query method provided in an exemplary embodiment.

[0069] Figure 5 This is a schematic diagram of the intermediate index layer information of the first local data and the second local data of column group C2 provided in an exemplary embodiment.

[0070] Figure 6 This is a schematic diagram of the intermediate index layer information of the first local data and the second local data of column group C3 provided in an exemplary embodiment.

[0071] Figure 7 This is a schematic diagram of the structure of a device provided in an exemplary embodiment.

[0072] Figure 8This is a block diagram of a data processing apparatus provided in yet another exemplary embodiment. Detailed Implementation

[0073] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0074] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0075] In today's era of rapid development in the internet and information technology, the generation of data is exploding, thus placing increasingly higher demands on databases and their management. Data processing requires the use of DML (Data Manipulation Language) to manipulate data, such as adding, deleting, querying, and modifying data; it also requires the use of DDL (Data Definition Language) to restructure data, such as creating new tables, deleting columns, and changing column types.

[0076] In related technologies, during the data reorganization process using DDL, the data cannot be queried externally, affecting database business processing and causing business issues such as request timeouts. For example, some DDL operations (such as deleting columns or changing column types) require writing the reorganized data to a hidden table in memory and making it effective on disk before providing read / write services. When dealing with large amounts of data, this process can be time-consuming, preventing the reorganized data from being fully available for read / write services within a certain period. This is especially true when reorganizing column-oriented data using DDL. Since the reorganized data may exist in multiple column groups, each column group executes the above process independently (i.e., data from each column group is written to different hidden tables and written to disk independently, meaning the disk writing progress cannot be guaranteed to be synchronized), the reorganized data is more dispersed in the process, making it more difficult to provide read / write services.

[0077] Based on this, in a first aspect, at least one embodiment of this specification provides a data processing method that can provide real-time read and write services such as query services during the process of transferring data from memory to disk (i.e., disk write process). That is, when a certain data, especially columnar storage data containing multiple column groups, is scattered in memory and disk during the disk write process, this method can provide real-time read and write services for data that is scattered during the disk write process, especially columnar storage data containing multiple column groups.

[0078] Please refer to the appendix. Figure 1 The example illustrates the flow of the data processing method, including step S101.

[0079] In step S101, during the process of writing target data from memory to disk, in response to receiving a query instruction for the target data, data is queried in the at least one first local data and the at least one second local data according to the query instruction, as well as intermediate index layer information of at least one first local data and intermediate index layer information of at least one second local data, and the queried data is sorted and used as the data query result.

[0080] For example, the target data may include (reorganized) data generated in memory according to a Data Definition Language (DDL). For instance, the target data may be reorganized data generated by operations such as creating new tables, deleting columns, and changing column types according to the DDL. The reorganized data obtained from the DDL needs to be written into memory first, and then written from memory to disk (i.e., disk write-in). This step occurs during the disk write-in process of the reorganized data.

[0081] For another example, the target data can be in columnar or row-based format. Row-based format means that data is stored in rows, with each row containing the values ​​of all fields in the table. Columnar format means that data is stored in columns, with data from the same column stored together to facilitate data aggregation and compression. Because columnar data may contain multiple column groups, its dispersion during disk write-to-disk process is higher (the reasons for which have been explained in detail above and will not be repeated here). Therefore, the following explanation will primarily use columnar data as an example to illustrate the method's flow, in order to broaden its applicability. However, this does not limit the form of the target data used in this method.

[0082] The first local data includes a portion of the target data stored on the disk (e.g., it may be referred to as SSTable).

[0083] The second local data includes a portion of the target data stored in memory (e.g., it may be referred to as DDLKV).

[0084] It should be understood that if the target data is in columnar storage format, then the target data includes multiple column groups (a column group is a general form of columnar storage containing one or more columns of data). That is, when the target data is generated in memory, a second local data DDLKV containing multiple column groups can be generated. Preferably, each column group corresponds to one second local data DDLKV. Furthermore, during the process of writing the target data in memory to disk, the target data will be distributed into at least one first local data and at least one column group of second local data. The number of first local data is determined by the form of the partial data written to disk by each column group. For example, if the partial data written to disk by each column group is stored in a single first local data SSTable, then the number of first local data is 1, meaning that different column groups of the target data correspond to the same first local data SSTable. Alternatively, if the partial data written to disk by each column group is stored in different first local data SSTables, then the number of first local data is the same as the number of column groups, meaning that different column groups of the target data correspond to different first local data. Preferably, the partial data written to disk by each column group is stored in a single first local data SSTable.

[0085] The number of second local data DDLKV is determined by the disk write progress of each column group. For example, if all the data of a column group has been written to disk, then there is no second local data DDLKV for that column group in memory; on the other hand, if not all the data of a column group has been written to disk, then there is second local data DDLKV for that column group in memory.

[0086] The second local data SSTable, persisted on disk, consists of its own metadata and a series of data macroblocks. Each data macroblock can be further divided into multiple microblocks. A data macroblock contains multiple rows of sorted data, is persisted on disk, has a fixed size of 2MB, and is the basic unit of SSTable. A microblock, also a basic unit of data macroblock, contains multiple rows of sorted data, is persisted on disk, and has a variable size, typically a few KB. It is the smallest unit for reading SSTable data from disk. The first local data, stored in memory, can also be stored in memory as second local data, consisting of its own metadata and a series of data macroblocks, each of which can be further divided into multiple microblocks. The macroblocks and microblocks within the first and second local data can be organized using intermediate index layer information to accelerate queries. This intermediate index layer information represents the internal data organization, specifically the distribution of the first local data across different macroblocks and microblocks. Please refer to the appendix. Figure 2 The diagram exemplifies the intermediate index layer information of a first local data set. As shown, the intermediate index layer information has a tree structure, including at least one layer of index information and a bottom-level macroblock. Each macroblock contains microblocks. The index information is progressively refined, meaning each layer includes multiple sub-local data sets derived from the first local data set. The number of sub-local data sets contained in the lower-level index information is greater than the number contained in the upper-level index information. A sub-local data set in the upper-level index information is divided into at least one sub-local data set in the lower-level index information. Each sub-local data set in the bottom-level index information corresponds to a data macroblock. Sub-local data sets can be represented by key value ranges and row offset ranges, for example, as shown in the attached diagram. Figure 2 Sub-local data within a macroblock is represented by the end value of a key value range and the end value of a row offset range. The row offset within a sub-local data segment is the absolute offset of the data within the first local data segment. A macroblock consists of multiple microblocks, which can be represented by a key value range and a row offset range, for example, appended... Figure 2 Microblocks are represented by the end value of the key value range and the end value of the line offset range, but the line offset within a microblock is the relative offset of the microblock within the macroblock.

[0087] The query instruction may include the query scope and query conditions. For example, this step can be implemented as shown in the attached document. Figure 3 The process is executed as shown, including sub-steps S1011 to S1012.

[0088] In sub-step S1011, based on the intermediate index layer information of at least one first local data and the intermediate index layer information of at least one second local data, data within the query range are queried in the at least one first local data and the at least one second local data respectively, and the queried data is sorted and used as the data to be queried.

[0089] The query range can include the column query range of at least one column group; that is, the query range can be represented by the column query range of at least one column group, such as the range of values ​​of a column in a column group. This sub-step can be performed as follows:

[0090] First, for each column query range in the query range, the first row offset range corresponding to the column query range of the column group is determined based on the intermediate index layer information of the first local data and the intermediate index layer information of the second local data of the column group, as well as the column query range of the column group.

[0091] In other words, the first local data SSTable and the second local data DDLKV involved in this column group can utilize intermediate index layer information to simulate an intermediate index iterator to perform data queries separately. The queried data is then sorted according to row offsets to obtain the column query results for the column group. Furthermore, the column query results are filtered based on the column query range of the column group to obtain the first row offset range corresponding to the column query range of the column group. Please refer to the appendix. Figure 4 When the first local data SSTable and the second local data DDLKV involved in this column group perform data queries, a multi-way merge method can be used to complete the data query and data sorting simultaneously. For example, when the first local data SSTable or the second local data DDLKV queries a certain macroblock, if the row offset of the macroblock shows that it is the macroblock with the smallest row offset among the remaining data to be queried, then the macroblock is output to be added to the query results. Alternatively, the macroblock is not output until it is the macroblock with the smallest row offset among the remaining data to be queried, at which point it is output to be added to the output results.

[0092] For example, the column query result of the column group can be determined as follows: based on the intermediate index layer information of the first local data and the intermediate index layer information of the second local data of the column group, all data of the column group are queried in the first local data and the second local data of the column group respectively, and the queried data is sorted according to the row offset and used as the column query result of the column group. For another example, the column query result of the column group can be determined as follows: Based on the intermediate index layer information of the first local data and the intermediate index layer information of the second local data of the column group, query the data within the second row offset range of the query range in the first local data and the second local data of the column group respectively, and sort the queried data according to the row offset to obtain the column query result of the column group; wherein, the second row offset range involved in this example can be the row offset range included in the query range, or it can be the offset range determined according to the primary key range included in the query range as follows: First, based on the intermediate index layer information of the first local data and the intermediate index layer information of the second local data of the primary key column (i.e., the column where the primary key is located), query the data within the primary key range in the first local data and the second local data of the primary key column respectively, and sort the queried data according to the row offset to obtain the primary key query result; determine the second row offset range of the query range based on the row offset of the data in the primary key query result (i.e., determine the union of the offsets of each row of data in the primary key query result as the second row offset range).

[0093] For example, the first row offset range corresponding to the column query range of the column group can be determined as follows: the union of the offsets of each row of data within the column query range that conforms to the column query range (which is characterized by the values ​​within the columns of the column group) is determined as the first row offset range.

[0094] Next, the second row offset range is determined based on the first row offset range corresponding to each column query range in the query range. For example, based on the relationship between each column query range in the query range, the intersection and union of the first row offset ranges corresponding to each column query range are taken to obtain the second row offset range. If the query range contains three column query ranges, and the relationship between the three column query ranges is AND, then the intersection of the first row offset ranges corresponding to the three column query ranges can be determined as the second row offset range; if the query range contains three column query ranges, and the relationship between the three column query ranges is OR, then the union of the first row offset ranges corresponding to the three types of query ranges can be determined as the second row offset range.

[0095] Finally, based on the second row offset range, and the intermediate index layer information of at least one first local data and at least one second local data, data within the second row offset range is queried in both the at least one first local data and the at least one second local data. The queried data is then sorted and used as the data to be queried. In other words, a multi-way merge cross-local data query is performed on at least one first local data (SSTable) and at least one second local data (DDLKV) of the target data using the second row offset range to achieve simultaneous querying and sorting, ultimately outputting the data to be queried sorted according to the row offset.

[0096] It should be understood that the data within the first and second local data are stored in the form of data blocks, which include macroblocks and microblocks. Details of macroblocks and microblocks have been described in detail above and will not be repeated here. Based on this, this sub-step can, according to the intermediate index layer information of at least one first local data and at least one second local data, query the data blocks to which the data within the query range belongs in the at least one first local data and the at least one second local data, respectively. The queried data blocks are then sorted and used as the data to be queried. That is, at least one macroblock sorted by row offset is output from at least one first local data and at least one second local data as the data to be queried.

[0097] In sub-step S1012, the data to be queried is queried according to the query conditions to determine the data query result.

[0098] The query conditions can be for at least one column. If the query conditions are empty, the data with the query result can be directly determined as the data query result.

[0099] The following section will describe in detail the data processing method obtained by combining the above-mentioned multiple embodiments using a specific example.

[0100] Suppose there is a table t1(c1 int primary key, c2 int, c3 int) with two column groups, cg_c2(c2) and cg_c3(c3). This table has 200 rows of data, where the data in c1 is (1..200), the data in c2 is (1001..1200), and the data in c3 is (2001..2200). The parallelism during DDL execution is 2. For each row-stored column group, the first thread is responsible for the data with the primary key (1..100), and the second thread is responsible for the data with the primary key (101..200). At some point during the process of writing the table from memory to disk, for Column Group cg_c2, when the first thread executes to c2 at value 1030 and the second thread executes to value 1130, the first partial data SSTable1 is generated. The remaining data is stored in the second partial data DDLKV1. The layout of SSTable1 and DDLKV1 can be seen in the appendix. Figure 5 (Since the key and row offset are the same for each row in t1, the index information in the diagram only includes the row offset); For ColumnGroup cg_c3, when the first thread executes to c3 at 2040, and the second thread executes to 2140, the first local data SSTable2 is generated, and the remaining data is stored in the second local data DDLKV2. The layout of SSTable2 and DDLKV2 can be seen in the appendix. Figure 6 (Since the key value and row offset are the same for each row in t1, the index information in the figure only contains the row offset).

[0101] If a query command is received for table t1: select * from t1 where c2>1035 and c3<2120 and c1>=0 and c1<150, then the query can be completed across SSTable1, DDLKV1, SSTable2, and DDLKV2.

[0102] First: Based on the primary key range [0, 150) in the query command, and the appended... Figure 5 The intermediate index layer information of SSTable1 and DDLKV1 shown are or appended. Figure 6The intermediate index layer information of SSTable2 and DDLKV2 shown can be iterated sequentially using the intermediate index iterator to obtain [25,0,25], [50,26,50], [75,51,75], [100,76,100], [125,101,125], [150,126,150] (Note: the values ​​in square brackets above are the end value of the key value, the start value of the row offset, and the end value of the row offset, respectively). Therefore, the second row offset range row_offset in the query instruction can be obtained as follows: start_row_offset is 0, and end_row_offset is 150.

[0103] Next, based on the offset range `row_offset` in the second row, query Column Group C2 and Column Group C3 respectively. For the query of Column Group C2, based on the offset range `row_offset` in the second row, and the appended... Figure 5 The intermediate index layer information of SSTable1 and DDLKV1 shown are used to iterate through [0,15], [16,30], [31,45], [46,100], [101,115], [116,130], [131,145], and [146,150] sequentially using the intermediate index iterator (Note: the values ​​in square brackets above are the start and end values ​​of the row offset, respectively). During the above iteration process, apply_filter can be applied according to a certain batch size. Since the column query range of Column Group C2 is >1035, in the result map generated by Column Group C2, the first [0,35] is 0 and the last [36,150] is 1. The column query result of Column Group C2 is [36,150]. For the query of Column Group C3, based on the second row offset range row_offset and the appended... Figure 6The intermediate index layer information of SSTable2 and DDLKV2 shown are used to iterate through [0,15], [16,40], [41,55], [56,100], [101,115], [116,140], and [141,150] in sequence using the intermediate index iterator (Note: the values ​​in square brackets above are the starting and ending values ​​of the row offset, respectively). During the above iteration process, the filter can be applied according to a certain batch size. Since the column query range of Column Group C3 is <2020, in the result map generated by Column Group C3, the first [0,119] is 1 and the last [120,150] is 0, that is, the column query result of Column Group C3 is [0,119].

[0104] Next, since the column query ranges of C2 and C3 are related by AND, the intersection of the column query range [36,150] of Column Group C2 and the column query result [0,119] of Column Group C3 is taken to obtain the first row offset range [36,119].

[0105] Finally, based on the offset range of the first row [36,119], the intermediate layer index rows are iterated in Column Group C2 and Column Group C3 respectively to determine the micro-blocks involved in the data to be queried; then, based on the micro-blocks involved in the data to be queried and the query condition result bitmap, it is determined whether a row of a micro-block should be projected to the data query result.

[0106] The data processing method provided in this specification can, in response to receiving a query instruction for the target data during the process of writing target data from memory to disk, query data in the at least one first local data and the at least one second local data according to the query instruction, intermediate index layer information of at least one first local data, and intermediate index layer information of at least one second local data, respectively, and sort the queried data as the data query result. The first local data includes the portion of the target data stored on disk, and the second local data includes the portion of the target data stored in memory. The target data is in columnar storage format, and each column group of the target data corresponds to one second local data. In other words, this method can provide query services in real time during the process of writing columnar storage data from memory to disk. Since each column group of columnar storage data is stored as an independent second local data in memory, this method sorts the queried data after querying multiple local data in disk and memory, thereby ensuring the accuracy of the data query results. Furthermore, this method can provide query services for relevant data during the process of defining data tables using DDL, thereby ensuring the business processing response speed of the database and avoiding problems such as business request timeouts.

[0107] The data processing method provided in the embodiments of this specification addresses the issue that during the process of writing reorganized data in columnar storage format from memory to disk, the progress of SSTable and DDLKV corresponding to different column groups is different. In order to avoid blocking playback and provide external query services as soon as possible, a fusion query scheme is proposed based on the intermediate layer index information. By merging and aggregating the queried data in the intermediate layer index iterator, the required data can be accurately and efficiently iterated out.

[0108] Figure 7 This is a schematic structural diagram of a device provided in an exemplary embodiment. Please refer to... Figure 7 At the hardware level, the device includes a processor 702, an internal bus 704, a network interface 706, memory 708, and non-volatile memory 710, and may also include other hardware required for tasks. One or more embodiments of this specification can be implemented in software, such as the processor 702 reading the corresponding computer program from the non-volatile memory 710 into memory 708 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0109] Please refer to Figure 8 Data processing devices can be applied to, for example Figure 7The device shown is used to implement the technical solution of this specification. The device includes:

[0110] The real-time query module 801 is used to, in the process of writing target data in memory to disk, respond to receiving a query instruction for the target data, query data in the at least one first local data and the at least one second local data according to the query instruction, as well as intermediate index layer information of at least one first local data and intermediate index layer information of at least one second local data respectively, and sort the queried data as the data query result.

[0111] The first local data includes a portion of the target data stored on the disk, and the second local data includes a portion of the target data stored in memory. The target data is in columnar storage format, and each column group of the target data corresponds to a second local data.

[0112] In one embodiment of this specification, the query instruction includes a query range and query conditions;

[0113] The real-time query module is used for:

[0114] Based on the intermediate index layer information of at least one first local data and the intermediate index layer information of at least one second local data, data within the query range are queried in the at least one first local data and the at least one second local data respectively, and the queried data is sorted and used as the data to be queried;

[0115] The query results are determined by querying the data to be queried according to the query conditions.

[0116] In one embodiment of this specification, the query range includes a column query range for at least one column group;

[0117] The real-time query module is used to query data within the query range based on the query range, intermediate index layer information of at least one first local data, and intermediate index layer information of at least one second local data, respectively. When sorting the queried data and using it as the data to be queried, it is used for:

[0118] For each column query range in the query range, the first row offset range corresponding to the column query range of the column group is determined based on the intermediate index layer information of the first local data and the intermediate index layer information of the second local data of the column group, as well as the column query range of the column group.

[0119] The second row offset range is determined based on the first row offset range corresponding to the query range of each column in the query range;

[0120] Based on the second row offset range, and the intermediate index layer information of at least one first local data and at least one second local data, data within the second row offset range are queried in the at least one first local data and the at least one second local data respectively, and the queried data is sorted and used as the data to be queried.

[0121] In one embodiment of this specification, when the real-time query module determines the first row offset range corresponding to the column query range of the column group based on the intermediate index layer information of the first local data and the intermediate index layer information of the second local data of the column group, and the column query range of the column group, it is used to:

[0122] Based on the intermediate index layer information of the first local data and the intermediate index layer information of the second local data of the column group, the data within the third row offset range of the query range is queried in the first local data and the second local data of the column group respectively, and the queried data is sorted according to the row offset and used as the column query result of the column group.

[0123] The column query results of the column group are filtered according to the column query range of the column group to obtain the first row offset range corresponding to the column query range of the column group.

[0124] In one embodiment of this specification, the query range includes the primary key range;

[0125] The device further includes a key module for:

[0126] Based on the intermediate index layer information of the first local data and the intermediate index layer information of the second local data of the primary key column, the data within the primary key range is queried in the first local data and the second local data of the primary key column respectively, and the queried data is sorted according to the row offset and used as the primary key query result;

[0127] Based on the row offset of the data in the primary key query result, determine the second row offset range within the query range.

[0128] In one embodiment of this specification, the data within the first local data and the second local data are stored in the form of data blocks, with each database storing a portion of the data;

[0129] The real-time query module is used to query data within the query range based on intermediate index layer information of at least one first local data and intermediate index layer information of at least one second local data, respectively, and when sorting the queried data as the data to be queried, it is used for:

[0130] Based on the intermediate index layer information of at least one first local data and the intermediate index layer information of at least one second local data, the data blocks to which the data within the query range belongs are queried in the at least one first local data and the at least one second local data, respectively. The queried data blocks are sorted and used as the data to be queried.

[0131] In one embodiment of this specification, the data block includes macroblocks and microblocks.

[0132] In one embodiment of this specification, the target data includes:

[0133] Data generated in the memory according to the Data Definition Language (DDL).

[0134] In one embodiment of this specification, different column groups of the target data correspond to the same first local data; or, different column groups of the target data correspond to different first local data.

[0135] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0136] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0137] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0138] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0139] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0140] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0141] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0142] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0143] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0144] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."

[0145] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

Claims

1. A data processing method, the method comprising: During the process of writing target data from memory to disk, for each column group to which the query range of the target data belongs in the query instruction received for the target data, the first row offset range corresponding to the query range of the column is determined based on the intermediate index layer information of the first local data and the intermediate index layer information of the second local data of the column group, as well as the column query range. The intersection or union of the first row offset ranges corresponding to the query range of each column in the query instruction is determined as the second row offset range; Based on the second row offset range, and the intermediate index layer information of at least one first local data and at least one second local data, query the data within the second row offset range in the at least one first local data and the at least one second local data respectively, and sort the queried data as the data query result; The first local data includes a portion of the target data stored on the disk, and the second local data includes a portion of the target data stored in memory. The target data is in columnar storage format, and each column group of the target data corresponds to a second local data.

2. The data processing method according to claim 1, wherein the query instruction includes query conditions; The retrieved data is sorted and presented as the data query results, including: The retrieved data is used as the data to be queried. The data to be queried is then queried according to the query conditions to determine the data query result.

3. The data processing method according to claim 1, wherein determining the first row offset range corresponding to the column query range of the column group based on the intermediate index layer information of the first local data and the intermediate index layer information of the second local data of the column group, and the column query range of the column group, comprises: Based on the intermediate index layer information of the first local data and the intermediate index layer information of the second local data of the column group, the data within the third row offset range of the query range is queried in the first local data and the second local data of the column group respectively, and the queried data is sorted according to the row offset and used as the column query result of the column group. The column query results of the column group are filtered according to the column query range of the column group to obtain the first row offset range corresponding to the column query range of the column group.

4. The data processing method according to claim 3, wherein the query range includes the primary key range; The method further includes: Based on the intermediate index layer information of the first local data and the intermediate index layer information of the second local data of the primary key column, the data within the primary key range is queried in the first local data and the second local data of the primary key column respectively, and the queried data is sorted according to the row offset and used as the primary key query result; Based on the row offset of the data in the primary key query result, determine the second row offset range within the query range.

5. The data processing method according to claim 1, wherein the data in the first local data and the second local data are stored in the form of data blocks, and each database stores a portion of the data; The step of querying data within the second row offset range based on the second row offset range, intermediate index layer information of at least one first local data, and intermediate index layer information of at least one second local data, and sorting the queried data as the data query result, includes: Based on the second row offset range, and the intermediate index layer information of at least one first local data and at least one second local data, query the data block to which the data within the second row offset range belongs in the at least one first local data and the at least one second local data respectively, and sort the queried data blocks as the data to be queried.

6. The data processing method according to claim 5, wherein the data block comprises macroblocks and microblocks.

7. The data processing method according to claim 1, wherein the target data includes: Data generated in the memory according to the Data Definition Language (DDL).

8. In the data processing method according to claim 1, different column groups of the target data correspond to the same first local data; or, different column groups of the target data correspond to different first local data.

9. A data processing apparatus, the apparatus comprising: The real-time query module is used to, during the process of writing target data from memory to disk, determine the first row offset range corresponding to each column query range in the column group to which the column query range belongs, based on the intermediate index layer information of the first local data and the intermediate index layer information of the second local data of the column group, and the column query range; determine the second row offset range by the intersection or union of the first row offset ranges corresponding to each column query range in the query instruction; and query the data within the second row offset range in the at least one first local data and the at least one second local data, respectively, based on the second row offset range, the intermediate index layer information of at least one first local data and the intermediate index layer information of at least one second local data, and sort the queried data as the data query result. The first local data includes a portion of the target data stored on the disk, and the second local data includes a portion of the target data stored in memory. The target data is in columnar storage format, and each column group of the target data corresponds to a second local data.

10. An electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1-8 by executing the executable instructions.

11. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1-8.

Images