Database data reading methods, devices and storage media

Abstract

This disclosure provides a data reading method, device, and storage medium for a database. The method involves receiving query conditions for the Value column of an LSM-Tree database; retrieving first candidate data from the LSM-Tree database whose Values ​​satisfy the query conditions; constructing a right table based on the Keys of the first candidate data; using the full data of the LSM-Tree database as a left table; performing a semi-join between the left and right tables; obtaining second candidate data from the left table that matches each Key in the right table; and retrieving target data from the second candidate data that satisfies the query conditions. This disclosure implements predicate pushdown in a Merge-On-Read scenario for LSM-Tree databases, avoiding extensive data scanning while balancing write and query performance.

Description

Technical Field

[0001] This disclosure relates to the field of computer and network communication technology, and in particular to a data reading method, device and storage medium for a database. Background Technology

[0002] LSM-Tree (log-structured merge-tree) databases are based on a hierarchical, ordered, disk-based data structure that uses a key-value pair structure for data storage. In some multidimensional analysis scenarios, users are more concerned with ensuring the uniqueness of the keys. Most LSM-Tree databases provide a Unique table model to support this requirement. This table model requires users to specify the key and value columns when creating the table. The engine uses a merge-on-read approach, aggregating data with the same key into a single row during the final data read, and replacing the values ​​in previously written data with the values ​​from the later batch of data.

[0003] For LSM-Tree databases, when using Merge-On-Read to implement Unique tables, it is quite difficult to support pushing down the Value column predicates (i.e., filtering conditions or query conditions) to the storage layer, because directly pushing down the Value column predicates to the storage layer may result in incorrect results. Summary of the Invention

[0004] This disclosure provides a database data reading method, device, and storage medium to enable predicate pushdown in a Merge-On-Read scenario for an LSM-Tree database.

[0005] In a first aspect, embodiments of this disclosure provide a method for reading data from a database, including:

[0006] Receive query conditions for the Value column in the LSM-Tree database;

[0007] Query the LSM-Tree database to find the first candidate data whose Value satisfies the query conditions;

[0008] A right table is constructed based on the key of the first candidate data. The full data of the LSM-Tree database is used as the left table. A semi-join is performed based on the left table and the right table to obtain the second candidate data that matches each key of the right table in the left table.

[0009] Query the target data from the second candidate data whose Value satisfies the query conditions.

[0010] Secondly, embodiments of this disclosure provide a database data reading device, comprising:

[0011] The receiving unit is used to receive query conditions for the Value column in the LSM-Tree database;

[0012] The right table construction unit is used to query the LSM-Tree database for first candidate data whose values ​​satisfy the query conditions; and to construct the right table based on the key of the first candidate data.

[0013] A semi-join unit is used to take the full data of the LSM-Tree database as the left table, and perform a semi-join based on the left table and the right table to obtain second alternative data that matches each key of the right table in the left table;

[0014] The result correction unit is used to query target data from the second candidate data whose Value satisfies the query conditions.

[0015] Thirdly, embodiments of this disclosure provide an electronic device, including: at least one processor and a memory;

[0016] The memory stores computer-executable instructions;

[0017] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the database data reading method described in the first aspect and various possible designs of the first aspect.

[0018] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the database data reading method described in the first aspect and various possible designs of the first aspect.

[0019] Fifthly, embodiments of this disclosure provide a computer program product, including computer execution instructions, which, when executed by a processor, implement the database data reading method described in the first aspect and various possible designs of the first aspect.

[0020] The database data reading method, device, and storage medium provided in this disclosure receive query conditions for the Value column of an LSM-Tree database; query first candidate data whose Values ​​satisfy the query conditions from the LSM-Tree database; construct a right table based on the Keys of the first candidate data, and use the full data of the LSM-Tree database as a left table; perform a semi-join based on the left and right tables to obtain second candidate data in the left table that matches each Key of the right table; and query target data whose Values ​​satisfy the query conditions from the second candidate data. By performing preliminary filtering based on the query conditions of the Value column to narrow down the range of Keys, and then determining the latest data corresponding to each Key after the narrowing down range based on the semi-join process, and then searching for target data that satisfies the query conditions of the Value column, the predicate pushdown of the LSM-Tree database in the Merge-On-Read scenario is realized, avoiding a large amount of data scanning, while taking into account both write performance and query performance. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1a Example diagram of retrieving correct data from an LSM-Tree database;

[0023] Figure 1b An example diagram showing incorrect data retrieved after direct predicate pushdown in an LSM-Tree database;

[0024] Figure 2 This is a schematic flowchart of a database data reading method provided in an embodiment of the present disclosure;

[0025] Figure 3 This is a schematic flowchart of a database data reading method provided in another embodiment of the present disclosure;

[0026] Figure 4 A structural block diagram of a database data reading device provided in an embodiment of this disclosure;

[0027] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0029] For LSM-Tree databases, when using Merge-On-Read to implement Unique tables, supporting the pushdown of Value column predicates (i.e., filtering or query conditions) to the storage layer is quite challenging, as directly pushing down Value column predicates to the storage layer may yield incorrect results. Predicate pushdown aims to execute filtering conditions as early as possible without affecting the final result.

[0030] The following example illustrates why LSM-Tree databases using Merge-On-Read cannot push the predicate of the Value column directly to the storage layer.

[0031] Suppose an LSM-Tree database has a Unique table with two columns: Key and Value. Two batches of data are written to the table sequentially: [(1,1),(2,2)] and [(1,2),(2,3)]. The written data has not yet been merged. The query condition is Value = 1. Figure 1a As shown, this is the correct data query process: first, the data is merged, then a query is performed based on the query condition Value=1, and the final correct result is empty; while Figure 1b The query condition (predicate) of the Value column was pushed down to the storage layer directly, resulting in an incorrect result (1,1).

[0032] To enable predicate pushdown for unique tables in LSM-Tree databases, existing industry solutions involve adjustments on the write side, for example:

[0033] Merge-On-Write checks whether the corresponding key already exists and locates the corresponding file when writing data. It then modifies the delete bitmap of the corresponding file to mark the relevant key as deleted. Its disadvantage is that it requires additional maintenance of the primary key index and delete bitmap, and additional queries and delete bitmap updates are required during writing.

[0034] Copy-On-Write (COW) locates all files affected by the newly written data during the writing process, merges the newly written data with the original files, and then commits the new version. Its drawback is that write amplification is very severe.

[0035] While the above approach ultimately supports predicate pushdown for the Value column, it introduces additional overhead on the write side, making it less performance-friendly for high-throughput write scenarios. Currently, no solution in the industry has directly implemented Value column predicate pushdown for Unique tables in LSM-Tree databases using Merge-On-Read implementations without adjusting the write process.

[0036] To address the aforementioned technical problems, this disclosure provides a database data reading method. The method involves receiving query conditions for the Value column of an LSM-Tree database; querying first candidate data from the LSM-Tree database whose Values ​​satisfy the query conditions; constructing a right table based on the Keys of the first candidate data; using the full data of the LSM-Tree database as a left table; performing a semi-join between the left and right tables to obtain second candidate data from the left table that matches each Key in the right table; and querying target data from the second candidate data whose Values ​​satisfy the query conditions. By initially filtering based on the query conditions of the Value column to narrow down the range of Keys, and then determining the latest data corresponding to each Key after the narrowing process using a semi-join, the method then searches for target data that satisfies the query conditions of the Value column. This achieves predicate pushdown in a Merge-On-Read scenario for the LSM-Tree database, avoiding extensive data scanning while balancing write and query performance.

[0037] The data reading method of the database disclosed herein will be described in detail below with reference to specific embodiments.

[0038] refer to Figure 2 , Figure 2 This is a schematic flowchart illustrating a database data reading method according to an embodiment of the present disclosure. The method of this embodiment can be applied to a terminal device or a server, and the database data reading method includes:

[0039] S201. Receive the query conditions for the Value column in the LSM-Tree database.

[0040] In this embodiment, when a user needs to query target data in the LSM-Tree database where the Value satisfies a certain query condition (e.g., Value equals a certain value, or Value is greater than or less than a certain value), the user can input the query condition for the Value column, specifically using SQL statements.

[0041] S202. Query the LSM-Tree database for the first candidate data whose Value satisfies the query conditions.

[0042] In this embodiment, the Value column in the LSM-Tree database is filtered based on query conditions to find data that meets the query conditions, which are then used as the first candidate data. For example, if the query condition is Value = 1, then data with Value = 1 in the Value column of the LSM-Tree database is searched and used as the first candidate data. Since the LSM-Tree database may contain data with the same key in a Merge-On-Read scenario, and the Value written earlier in the data with the same key is invalid (and needs to be deleted), the first candidate data retrieved based on the Value column query conditions includes data corresponding to valid Values ​​that meet the query conditions and data corresponding to invalid Values ​​that meet the query conditions.

[0043] Optionally, the index of the Value column in the LSM-Tree database can be filtered according to the query conditions to obtain the index corresponding to the first candidate data that meets the query conditions; the first candidate data can then be queried based on the index corresponding to the first candidate data.

[0044] In this embodiment, when writing data to the LSM-Tree database, an index is created for the Value column, which records the location of the data corresponding to each Value in the database. In this way, when querying data based on the query conditions of the Value column, it is only necessary to query the index to determine the location of the first candidate data that meets the query conditions in the database. There is no need to read the Value column of the LSM-Tree database for scanning and querying, which can avoid scanning a large amount of data and improve the speed of querying the first candidate data.

[0045] S203. Construct a right table based on the Key of the first candidate data, use the full data of the LSM-Tree database as the left table, and perform a semi-join based on the left table and the right table to obtain the second candidate data that matches each Key of the right table in the left table.

[0046] In this embodiment, after retrieving the first candidate data, a table can be constructed based on the keys of the first candidate data, thus narrowing down the keys to be of interest to a small range. This table is then used as the right table, and the entire data of the LSM-Tree database is used as the left table. A semi-join is performed based on the left and right tables. A semi-join matches data from the two tables, returning only the data associated with one of the tables. Furthermore, even when one data point is associated with multiple data points, only one result is returned. In this embodiment, a semi-join is used to find data in the left table that matches each key in the right table. If multiple keys in the left table match the same key in the right table, only the latest key is returned. This allows the latest data corresponding to each key in the right table to be used as the second candidate data, meaning the value in the second candidate data is the latest (valid) value. The right table is constructed based on the keys of the first candidate data. The constructed right table is a superset of the keys of the final result set (including the keys of the target data in the final result set, as well as some keys of non-target data). After the semi-join process, the second candidate data is the latest data (valid data) of each key in the right table. The invalid data of each key in the right table is filtered out. At this time, the second candidate data is also a superset of the final result set.

[0047] For example, suppose the data in the LSM-Tree database is shown in Table 1 below:

[0048] Table 1

[0049] Key Value 1 1 2 2 1 2 3 1

[0050] Using the query condition Value=1, the first candidate data can be retrieved as shown in Table 2 below:

[0051] Table 2

[0052] Key Value 1 1 3 1

[0053] The right table is constructed based on the Key of the first candidate data, as shown in Table 3 below:

[0054] Table 3

[0055] Key 1 3

[0056] After performing a semi-join based on the left and right tables, the second alternative data is shown in Table 4 below:

[0057] Table 4

[0058] Key Value 1 2 3 1

[0059] Optionally, in this embodiment, when performing a semi-join based on the left and right tables, a runtime filter can be used. That is, by filtering out input data that will not be matched in advance, the data transmission and computation of the semi-join can be greatly reduced, thereby reducing the overall execution time. In other words, the runtime filter filters the left table based on each key of the right table to obtain the second candidate data that matches each key of the right table, instead of directly reading all the data in the left table and then matching them one by one with the right table, thus speeding up the semi-join process.

[0060] Optionally, when writing data to the LSM-Tree database, an index is created on the Key column, recording the location of the data corresponding to each Key in the database. This allows the runtime filter to filter the left table based on the Keys of the right table, while simultaneously filtering the LSM-Tree database based on the Keys of the right table. This results in the latest index corresponding to the second candidate data that matches each Key in the right table. The second candidate data is then retrieved based on this index. By using the Key column index, there is no need to scan and query the Key column in the LSM-Tree database, avoiding the scanning of large amounts of data and improving the filtering speed of the runtime filter.

[0061] S204. Query the target data from the second candidate data whose Value satisfies the query conditions.

[0062] In this embodiment, since the second candidate data is a superset of the final result set, further filtering is required based on the second candidate data. The target data that satisfies the query conditions of the Value column is queried from the second candidate data, which is the final result set, and the redundant data in the second candidate data is removed.

[0063] In the example above, Table 4 is filtered based on Value=1 to obtain the final target data (Key=3, Value=1).

[0064] Since the above process is for the Merge-On-Read scenario, after querying the target data whose value meets the query conditions from the second candidate data, a Merge operation is also required. That is, based on the key of the target data, all data with the same key in the LSM-Tree database are merged so that only the target data (that is, the latest value of the same key) is retained in the data with the same key, thereby ensuring that the key uniquely corresponds to one value.

[0065] This embodiment of the database data reading method involves receiving query conditions for the Value column of an LSM-Tree database; querying first candidate data from the LSM-Tree database whose Values ​​satisfy the query conditions; constructing a right table based on the Keys of the first candidate data; using the full data of the LSM-Tree database as a left table; performing a semi-join based on the left and right tables to obtain second candidate data from the left table that matches each Key in the right table; and querying target data from the second candidate data whose Values ​​satisfy the query conditions. By performing preliminary filtering based on the query conditions of the Value column to narrow down the range of Keys, and then determining the latest data corresponding to each Key after the narrowing process based on the semi-join process, and finally searching for target data that satisfies the query conditions of the Value column, this method achieves predicate pushdown in the Merge-On-Read scenario of the LSM-Tree database, avoiding large-scale data scanning while balancing write and query performance.

[0066] Based on any of the above embodiments, before executing S202-S204, the predicate pushdown function of the Value column can be enabled or disabled. If the predicate pushdown function of the Value column is not enabled, S202-S204 will not be executed; instead, a normal Merge-On-Read query process will be executed. That is, a data merging process will be performed first, and then data will be queried according to the query conditions of the Value column. Figure 3 On the left, ScanStep → FilterStep performs data scanning and merges data during the scan. S202-S204 only continue execution if the predicate pushdown function for the Value column is enabled. Figure 3 On the right side, first, ScanStep is executed based on the query condition (Value=1) of the Value column to obtain the first candidate data. The right table (RHS) is constructed based on the Key of the first candidate data. The left table (LHS) is scanned by another ScanStep, and a semi-join of LHS.Key=RHS.Key is executed. The scanning of the left table (LHS) uses Runtime Filter. The second candidate data is obtained after the semi-join. Then, the target data whose Value satisfies the query condition (Value=1) is queried from the second candidate data through the FilterStep process.

[0067] Optionally, when a user needs to enable predicate pushdown for the Value column, they can send a predicate pushdown enable command for the Value column. This command will then enable the predicate pushdown function for the Value column.

[0068] After receiving the data, enable the predicate pushdown function of the Value column and generate a task to query the target data from the LSM-Tree database based on the query conditions of the Value column, which is the task S202-S204 mentioned above.

[0069] Corresponding to the database data reading method in the above embodiment, Figure 4 This is a structural block diagram of a database data reading device provided in an embodiment of this disclosure. For ease of explanation, only the parts relevant to the embodiments of this disclosure are shown. (Refer to...) Figure 4 The database data reading device 400 includes: a receiving unit 401, a right table construction unit 402, a semi-join unit 403, and a result correction unit 404.

[0070] The receiving unit 401 is used to receive query conditions for the Value column in the LSM-Tree database.

[0071] The right table construction unit 402 is used to query the first candidate data whose Value satisfies the query conditions from the LSM-Tree database; and construct the right table based on the Key of the first candidate data.

[0072] The semi-join unit 403 is used to take the full data of the LSM-Tree database as the left table, and perform a semi-join based on the left table and the right table to obtain second alternative data that matches each key of the right table in the left table.

[0073] The result correction unit 404 is used to query target data from the second candidate data whose Value satisfies the query condition.

[0074] In one or more embodiments of this disclosure, the device further includes a merging unit, configured to, after querying target data whose Value satisfies the query condition from the second candidate data, merge all data with the same Key in the LSM-Tree database according to the Key of the target data, so that only the target data is retained among the data with the same Key.

[0075] In one or more embodiments of this disclosure, when the right table construction unit 402 queries the LSM-Tree database for first alternative data whose Value satisfies the query conditions, it is configured to:

[0076] The indexes of the Value column in the LSM-Tree database are filtered according to the query conditions to obtain the indexes corresponding to the first candidate data that meet the query conditions.

[0077] The first candidate data is retrieved based on the index corresponding to the first candidate data.

[0078] In one or more embodiments of this disclosure, the semi-join unit 403, when performing a semi-join based on the left table and the right table to obtain second alternative data matching each Key in the right table from the left table, is configured to:

[0079] The left table is filtered based on each key in the right table using a runtime filter to obtain second candidate data that matches each key in the right table.

[0080] In one or more embodiments of this disclosure, when the semi-join unit 403 filters the left table based on each key of the right table using a runtime filter to obtain second candidate data matching each key of the right table, it is configured to:

[0081] By using a runtime filter to filter the indexes of the Key columns in the LSM-Tree database based on each Key in the right table, the latest index corresponding to the second candidate data that matches each Key in the right table is obtained.

[0082] The second alternative data is obtained based on the index corresponding to the second alternative data.

[0083] In one or more embodiments of this disclosure, before querying the LSM-Tree database for first candidate data whose Value satisfies the query conditions, the receiving unit 401 is further configured to:

[0084] After receiving the command to enable predicate pushdown for the Value column, the predicate pushdown function for the Value column is enabled, and a task is generated to query the target data from the LSM-Tree database based on the query conditions.

[0085] The device provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0086] refer to Figure 5 The diagram illustrates a structural schematic of an electronic device 500 suitable for implementing embodiments of the present disclosure. The electronic device 500 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0087] like Figure 5 As shown, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0088] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0089] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0090] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0091] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0092] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0093] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0094] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0095] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0096] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0097] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0098] In a first aspect, according to one or more embodiments of this disclosure, a method for reading data from a database is provided, comprising:

[0099] Receive query conditions for the Value column in the LSM-Tree database;

[0100] Query the LSM-Tree database to find the first candidate data whose Value satisfies the query conditions;

[0101] A right table is constructed based on the key of the first candidate data. The full data of the LSM-Tree database is used as the left table. A semi-join is performed based on the left table and the right table to obtain the second candidate data that matches each key of the right table in the left table.

[0102] Query the target data from the second candidate data whose Value satisfies the query conditions.

[0103] According to one or more embodiments of this disclosure, after querying target data whose Value satisfies the query conditions from the second candidate data, the method further includes:

[0104] Based on the key of the target data, all data in the LSM-Tree database with the same key are merged so that only the target data is retained among the data with the same key.

[0105] According to one or more embodiments of this disclosure, querying the LSM-Tree database for first candidate data whose Value satisfies the query conditions includes:

[0106] The indexes of the Value column in the LSM-Tree database are filtered according to the query conditions to obtain the indexes corresponding to the first candidate data that meet the query conditions.

[0107] The first candidate data is retrieved based on the index corresponding to the first candidate data.

[0108] According to one or more embodiments of this disclosure, the step of performing a semi-join based on the left table and the right table to obtain second alternative data matching each key in the right table from the left table includes:

[0109] The left table is filtered based on each key in the right table using a runtime filter to obtain second candidate data that matches each key in the right table.

[0110] According to one or more embodiments of this disclosure, the step of filtering the left table based on each key of the right table using a runtime filter to obtain second candidate data matching each key of the right table includes:

[0111] By using a runtime filter to filter the indexes of the Key columns in the LSM-Tree database based on each Key in the right table, the latest index corresponding to the second candidate data that matches each Key in the right table is obtained.

[0112] The second alternative data is obtained based on the index corresponding to the second alternative data.

[0113] According to one or more embodiments of this disclosure, before querying the first candidate data whose Value satisfies the query conditions from the LSM-Tree database, the method further includes:

[0114] After receiving the command to enable predicate pushdown for the Value column, the predicate pushdown function for the Value column is enabled, and a task is generated to query the target data from the LSM-Tree database based on the query conditions.

[0115] Secondly, according to one or more embodiments of this disclosure, a database data reading device is provided, comprising:

[0116] The receiving unit is used to receive query conditions for the Value column in the LSM-Tree database;

[0117] The right table construction unit is used to query the LSM-Tree database for first candidate data whose values ​​satisfy the query conditions; and to construct the right table based on the key of the first candidate data.

[0118] A semi-join unit is used to take the full data of the LSM-Tree database as the left table, and perform a semi-join based on the left table and the right table to obtain second alternative data that matches each key of the right table in the left table;

[0119] The result correction unit is used to query target data from the second candidate data whose Value satisfies the query condition.

[0120] According to one or more embodiments of this disclosure, the device further includes a merging unit, configured to, after querying target data whose Value satisfies the query condition from the second candidate data, merge all data with the same Key in the LSM-Tree database according to the Key of the target data, so that only the target data is retained among the data with the same Key.

[0121] According to one or more embodiments of this disclosure, when the right table construction unit queries the LSM-Tree database for first candidate data whose Value satisfies the query conditions, it is configured to:

[0122] The indexes of the Value column in the LSM-Tree database are filtered according to the query conditions to obtain the indexes corresponding to the first candidate data that meet the query conditions.

[0123] The first candidate data is retrieved based on the index corresponding to the first candidate data.

[0124] According to one or more embodiments of this disclosure, when the semi-join unit performs a semi-join based on the left table and the right table to obtain second alternative data matching each key in the right table from the left table, it is configured to:

[0125] The left table is filtered based on each key in the right table using a runtime filter to obtain second candidate data that matches each key in the right table.

[0126] According to one or more embodiments of this disclosure, when the semi-join unit filters the left table based on each key of the right table using a runtime filter to obtain second candidate data matching each key of the right table, it is configured to:

[0127] By using a runtime filter to filter the indexes of the Key columns in the LSM-Tree database based on each Key in the right table, the latest index corresponding to the second candidate data that matches each Key in the right table is obtained.

[0128] The second alternative data is obtained based on the index corresponding to the second alternative data.

[0129] According to one or more embodiments of this disclosure, before querying the LSM-Tree database for first candidate data whose Value satisfies the query conditions, the receiving unit is further configured to:

[0130] After receiving the command to enable predicate pushdown for the Value column, the predicate pushdown function for the Value column is enabled, and a task is generated to query the target data from the LSM-Tree database based on the query conditions.

[0131] Thirdly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;

[0132] The memory stores computer-executed instructions;

[0133] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the database data reading method described in the first aspect and various possible designs of the first aspect.

[0134] Fourthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when a processor executes the computer-executable instructions, the data reading method of the database described in the first aspect and various possible designs of the first aspect is implemented.

[0135] Fifthly, according to one or more embodiments of the present disclosure, a computer program product is provided, including computer execution instructions that, when executed by a processor, implement the data reading method of the database as described in the first aspect and various possible designs of the first aspect.

[0136] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0137] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0138] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for reading data from a database, characterized in that, include: Receive query conditions for the Value column in the LSM-Tree database; Query the LSM-Tree database to find the first candidate data whose Value satisfies the query conditions; A right table is constructed based on the key of the first candidate data. The full data of the LSM-Tree database is used as the left table. A semi-join is performed based on the left table and the right table to obtain the second candidate data that matches each key of the right table in the left table. The second candidate data is the valid value of each key of the right table after filtering out the invalid values ​​of each key of the right table. Query the target data from the second candidate data whose Value satisfies the query conditions.

2. The method according to claim 1, characterized in that, After querying the target data whose Value satisfies the query conditions from the second candidate data, the process further includes: Based on the key of the target data, all data in the LSM-Tree database with the same key are merged so that only the target data is retained among the data with the same key.

3. The method according to claim 1, characterized in that, The first candidate data for querying the LSM-Tree database whose Value satisfies the query conditions includes: The indexes of the Value column in the LSM-Tree database are filtered according to the query conditions to obtain the indexes corresponding to the first candidate data that meet the query conditions. The first candidate data is retrieved based on the index corresponding to the first candidate data.

4. The method according to any one of claims 1-3, characterized in that, The step of performing a semi-join based on the left and right tables to obtain second candidate data that matches each key in the right table from the left table includes: The left table is filtered based on each key in the right table using a runtime filter to obtain second candidate data that matches each key in the right table.

5. The method according to claim 4, characterized in that, The step of filtering the left table based on each key of the right table using a runtime filter to obtain second candidate data that matches each key of the right table includes: By using a runtime filter to filter the indexes of the Key columns in the LSM-Tree database based on each Key in the right table, the latest index corresponding to the second candidate data that matches each Key in the right table is obtained. The second alternative data is obtained based on the index corresponding to the second alternative data.

6. The method according to claim 1, characterized in that, Before querying the LSM-Tree database for the first candidate data whose Value satisfies the query conditions, the process also includes: After receiving the command to enable predicate pushdown for the Value column, the predicate pushdown function for the Value column is enabled, and a task is generated to query the target data from the LSM-Tree database based on the query conditions.

7. A database data reading device, characterized in that, include: The receiving unit is used to receive query conditions for the Value column in the LSM-Tree database; The right table construction unit is used to query the LSM-Tree database for the first alternative data whose Value satisfies the query conditions; Construct the right table based on the Key of the first candidate data; A semi-join unit is used to take the full data of the LSM-Tree database as the left table, and perform a semi-join based on the left table and the right table to obtain second candidate data that matches each key of the right table in the left table. The second candidate data is the valid value of each key of the right table after filtering out invalid values ​​of each key of the right table. The result correction unit is used to query target data from the second candidate data whose Value satisfies the query conditions.

8. An electronic device, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes computer execution instructions, which, when executed by a processor, implement the method as described in any one of claims 1-6.

Images