A data reading method, device, apparatus and storage medium

Abstract

The application provides a data reading method and device, equipment and a storage medium, which can efficiently and stably read data in an Hbase database. The method comprises the following steps: receiving a data reading request, wherein the data reading request comprises an identifier of a target data table and data filtering condition information; dividing a physical partition of the target data table into a plurality of logical partitions; and obtaining target data in each of the plurality of logical partitions, wherein the target data in each of the logical partitions is obtained by scanning data in the logical partition by a reader of the logical partition according to the data filtering condition information.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of big data analysis processing, and particularly relates to a data reading method and device, equipment and a storage medium. BACKGROUND

[0002] Spark is a general computing engine designed for processing large-scale data. Hbase is a distributed non-relational (Not Only Structured Query Language, NoSQL) database, which has the characteristics of high reliability, high performance and scalability. The combination of the computing engine Spark and the database Hbase has a wide range of applications in the fields of user portrait analysis, task relationship analysis, vehicle trajectory analysis, and vehicle flow prediction.

[0003] However, when Spark reads data in Hbase, the region server (RegionServer) of Hbase usually needs to scan data from the physical partition (Region), and then the data is transmitted to Spark through the Scan / Get Application Programming Interface (API). Hbase data needs to pass through the Region Server to be transmitted to Spark. Moreover, when there is a large amount of data on the Hbase database that needs to be transmitted to multiple Sparks, all the data needs to be transmitted from the Region Server to the corresponding Spark, which is low in transmission efficiency and can cause the RegionServer to be unstable or even down. In another way, although the Spark uses the TableSnapshootInputFormat class to directly read the data (Hfile) of the Hbase database stored in the distributed file system (Hadoop Distributed File System, HDFS), if the user only needs to process part of the data in Hbase, the method still reads all the data in Hbase into Spark in the form of Resilient Distributed Datasets (RDD), which wastes computing resources. Therefore, there is a lack of an efficient and stable method for reading data in Hbase. SUMMARY

[0004] The application provides a data reading method, device, equipment and storage medium. By dividing a plurality of logical partitions, target data in each logical partition is directly obtained according to data filtering condition information, avoiding the problem of low transmission efficiency and unstable transmission caused by the fact that all data needs to be transmitted through a Region Server of Hbase, thereby efficiently and stably reading data in Hbase.

[0005] In a first aspect, the application provides a data reading method, which comprises:

[0006] receiving a data reading request, wherein the data reading request comprises an identifier of a target data table and data filtering condition information;

[0007] dividing a physical partition of the target data table into a plurality of logical partitions;

[0008] obtaining target data in each logical partition of the plurality of logical partitions, wherein the target data in each logical partition is obtained by a reader of the logical partition from scanning data of the logical partition according to the data filtering condition information.

[0009] Further, the filtering condition information comprises:

[0010] a row filtering condition for screening rows of the target data table, and an identifier of a column of the target data table that needs to be read.

[0011] Further, dividing the physical partition of the target data table into a plurality of logical partitions comprises:

[0012] obtaining a snapshot of the target data table and a user reading parallelism setting; wherein the snapshot of the target data table comprises information of one or more physical partitions of the target data table;

[0013] dividing each physical partition of the one or more physical partitions into a plurality of logical partitions according to the user reading parallelism setting.

[0014] Further, the method further comprises:

[0015] constructing a data scanner according to the data filtering condition information;

[0016] the target data in each logical partition is obtained by a reader of the logical partition from processing data of the logical partition according to the data filtering condition information, comprising:

[0017] the target data in each logical partition is obtained by a reader of the logical partition from scanning data of the logical partition using the data scanner.

[0018] Further, the method further comprises:

[0019] creating a target data reading task for each of the plurality of logical partitions, to obtain a plurality of target data reading tasks corresponding to the plurality of logical partitions;

[0020] the obtaining of the target data in each of the plurality of logical partitions comprises:

[0021] running at least two of the plurality of target data reading tasks in parallel, to obtain the target data in the logical partition under each of the at least two target data reading tasks.

[0022] In a second aspect, the present application provides a data reading device, the device comprising:

[0023] an obtaining module, configured to receive a data reading request, the data reading request comprising an identification of a target data table and data filtering condition information;

[0024] a processing module, configured to divide a physical partition of the target data table into a plurality of logical partitions, and to obtain target data in each of the plurality of logical partitions, wherein the target data in each of the logical partitions is obtained by a reader of the logical partition from data in the logical partition according to the data filtering condition information.

[0025] Further, the filtering condition information comprises a row filtering condition for filtering rows of the target data table, and an identification of columns of the target data table that need to be read.

[0026] Further, the processing module divides the physical partition of the target data table into a plurality of logical partitions, and is specifically configured to: obtain a snapshot of the target data table and a user reading parallelism setting; wherein the snapshot of the target data table comprises information of one or more physical partitions of the target data table; and divide each of the one or more physical partitions into a plurality of logical partitions according to the user reading parallelism setting.

[0027] Further, the processing module is further configured to: construct a data scanner according to the data filtering condition information; and obtain the target data in each of the logical partitions by the reader of the logical partition from data in the logical partition according to the data filtering condition information, comprising: obtaining the target data in each of the logical partitions by the reader of the logical partition from scanning of data in the logical partition by the data scanner.

[0028] Further, the processing module is further configured to: create a target data reading task for each of the plurality of logical partitions, to obtain a plurality of target data reading tasks corresponding to the plurality of logical partitions; and when the processing module obtains the target data in each of the plurality of logical partitions, the processing module is specifically configured to: run at least two of the plurality of target data reading tasks in parallel, and obtain the target data of the logical partition under each of the at least two target data reading tasks.

[0029] In a third aspect, the present application provides an electronic device, the electronic device comprising at least a processor and a memory, wherein the processor executes a computer program or instructions stored in the memory to implement the method of the first aspect.

[0030] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program or instructions, wherein the computer program or instructions are executed by a processor to implement the method of the first aspect.

[0031] In the embodiments of the present application, by dividing a plurality of logical partitions, the target data in each logical partition can be directly obtained according to the data filtering condition information, avoiding the problem of low transmission efficiency and unstable transmission caused by the transmission of all data through the Region Server of Hbase, so that the data in Hbase can be efficiently and stably read, and the target data in a plurality of logical partitions can be obtained in parallel, further improving the data reading efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0033] Figure 1 A process diagram for reading an Hbase table by a Spark through a Region Server is provided in the present application.

[0034] Figure 2 A flowchart of a data reading method is provided in the present application.

[0035] Figure 3 A predicate pushdown mechanism diagram is provided in the present application.

[0036] Figure 4 A diagram for reading Hbase data using the method of the present application is provided in the present application.

[0037] Figure 5 A specific flow chart of a data reading method provided in the present application.

[0038] Figure 6 A structural schematic diagram of a data reading device provided in the present application.

[0039] Figure 7 A structural schematic diagram of an electronic device provided in the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose and implementation of the present application more clear, the following will combine the drawings in the exemplary embodiments of the present application to clearly and completely describe the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application.

[0041] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the following described embodiments, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0042] The terms "first", "second", "third" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar or similar objects or text entities, and do not necessarily mean to limit the specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchanged under appropriate circumstances.

[0043] The terms "include" and "have" and any variations thereof are intended to cover but not exclusive inclusion, for example, a product or device including a series of components is not necessarily limited to all the components clearly listed, but can include other components that are not clearly listed or inherent to these products or devices.

[0044] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware or / and software code capable of performing functions related to the element.

[0045] When Spark reads data in Hbase, the region server of Hbase usually needs to scan data from the physical partition (Region) and then transmit the data to Spark through Scan / Get API. Hbase data needs to pass through the region server before being transmitted to Spark. When there is a large amount of data on the Hbase database that needs to be transmitted to multiple Sparks, all data needs to be transmitted from the region server to the corresponding Spark, which is low in transmission efficiency and can cause the region server to be unstable or even down. Figure 1 FIG. 1 is a schematic diagram of a process in which Spark reads an Hbase table through a region server. As shown in FIG. 1, a Spark application interacts with the region server of Hbase through SCAN / GET API or TableInputFormat API. The region server of Hbase establishes multiple scanners to scan and read all Hfile (data storage format of Hbase database) files under the Nth physical partition (Region_N). The region server of Hbase returns all data of all Hfile files under all physical partitions to the Spark application through SCAN / GET API interface or TableInputFormat API. Figure 1

[0046] In another way, Spark uses the TableSnapshootInputFormat class to read data. Although it can directly read the data (Hfile) of the Hbase database stored in the distributed file system (Hadoop Distributed File System, HDFS), if the user only needs to process part of the data in Hbase, this method still reads all data in Hbase into Spark in the form of a resilient distributed dataset (Resilient Distributed Datasets, RDD), which can cause waste of computing resources. Therefore, the existing Spark method for reading Hbase cannot efficiently and stably read Hbase data. In order to efficiently and stably read Hbase data, the present application provides a data reading method, device, equipment and storage medium.

[0047] Figure 2 FIG. 2 is a flowchart of a data reading method. The method can be applied to an electronic device. The method comprises the following steps. ​

[0048] S201: receiving a data read request, the data read request comprising an identification of a target data table and data filtering condition information.

[0049] The identification of the target data table can be the namespace, table name, etc. of the Hbase data table. The data filtering condition information of the target data table can be a series of filtering condition statements set by the user, which are used to filter the target data table to obtain the target data required by the user.

[0050] Further, the filtering condition information comprises:

[0051] a row filtering condition for filtering rows of the target data table, and an identification of columns of the target data table that need to be read.

[0052] The row filtering condition for filtering rows of the target data table can be a condition statement such as "WHERE table1.age>50" (age greater than 50 in table1), "WHERE table2.score>90" (score greater than 90 in table2), etc. The identification of the columns of the target data table that need to be read can be the field name (i.e. column cluster name + column modifier) of the target data table. The user can set the field name of the target data table that needs to be read so that the reader only reads the required columns, reducing the amount of data actually read.

[0053] S202: dividing a physical partition of the target data table into a plurality of logical partitions;

[0054] In order to efficiently scan the Hbase data table, the electronic device can first divide the physical partition of the Hbase data table to obtain a plurality of logical partitions, so that the data scanners under each logical partition can perform scanning in parallel. The algorithm for dividing the logical partitions can use methods such as DecimalStringSplit, NumberStringSplit, and UniformSplit.

[0055] Optionally, the dividing of the physical partition of the target data table into a plurality of logical partitions comprises:

[0056] obtaining a snapshot of the target data table and a user read parallelism setting; wherein the snapshot of the target data table comprises one or more physical partition information of the target data table;

[0057] According to the user read parallelism setting, each physical partition of the one or more physical partitions is divided into a plurality of logical partitions.

[0058] In order to realize user personalized settings, the user can divide the logical partitions according to his own needs. The electronic device can obtain a snapshot of the target data table, and obtain one or more physical partition information in the target data table from the snapshot, each physical partition information including the row key range of the physical partition. According to the read parallelism setting of the target user, each physical partition of the target data table is divided into multiple logical partitions. For example: the user sets the read parallelism to M, and the target data table has P physical partitions, each physical partition is divided into M logical partitions, and a total of M*P logical partitions are obtained. Each logical partition carries its own logical partition name, and the start row key and end row key information of the logical partition. M and P are positive integers greater than or equal to 1.

[0059] S203: Obtain target data in each logical partition of the plurality of logical partitions, wherein the target data in each logical partition is obtained by scanning the data in the logical partition according to the data filtering condition information by a reader of the logical partition.

[0060] In order to stably obtain target data in each logical partition of the plurality of logical partitions, a reader can be created under each logical partition; each reader independently scans the data in the corresponding logical partition according to the data condition filtering information obtained from the user instruction, and only reads the data required by the user.

[0061] In this application, by dividing multiple logical partitions, target data in each logical partition is directly obtained according to the data filtering condition information, avoiding the problem of low transmission efficiency and unstable transmission caused by the fact that all data needs to be transmitted through the Region Server of Hbase, so that the data in Hbase can be efficiently and stably read, and the target data in multiple logical partitions can be obtained in parallel, further improving the data read efficiency.

[0062] Further, in order to efficiently read the target data, the method further comprises:

[0063] Creating a target data reading task for each logical partition of the plurality of logical partitions to obtain a plurality of target data reading tasks corresponding to the plurality of logical partitions;

[0064] The obtaining of the target data in each logical partition of the plurality of logical partitions comprises:

[0065] Parallelly running at least two target data reading tasks in the plurality of target data reading tasks to obtain target data of the logical partition under each target data reading task of the at least two target data reading tasks.

[0066] In order to efficiently read the target data, Spark can create a task for each logical partition to read the target data of the logical partition, and when Spark reads the target data, a certain number of tasks are run in parallel to read a certain number of logical partitions at the same time. The number of tasks to be run can be set according to the carrying capacity of the electronic device used at the time and the actual situation, so as to efficiently read the target data while ensuring the stability of the electronic device.

[0067] Optionally, in each task, the reader corresponding to the logical partition of the task can be used to process the data of the logical partition according to the data filtering condition information.

[0068] Further, the target data in each of the plurality of logical partitions can also be obtained in the following manner: a data scanner is constructed according to the data filtering condition information; and the target data in each logical partition is obtained by scanning the data of the logical partition using the data scanner by the reader of the logical partition.

[0069] In order to quickly read the target data in each logical partition, a data scanner with the same start row key and end row key as each logical partition can be constructed according to the start row key and end row key information of each logical partition. And using the idea of Spark predicate pushdown mechanism, the filtering condition set in Spark (i.e. the row filtering condition used to filter the rows of the target data table) is constructed as a row filter of the data scanner, and the column pruning information (i.e. the identification of the columns of the target data table that need to be read) is constructed as a column filter of the data scanner. In each logical partition, the data scanner with the row filter and the column filter is used to scan the logical partition, so as to realize the transfer of the filtering process of the data in Spark to the scanning of the data table, thereby reducing the amount of data actually read.

[0070] Referring to Figure 3 the predicate pushdown mechanism diagram shown in Figure 3In the above, "SELECT table1.name, table2.score" is column pruning information "select a column of record name in table 1 and a column of score in table 2", "WHERE table1.age>50 AND table2.score>90" is a filter condition set in Spark "select a row in table 1 with age greater than 50 and a row in table 2 with score greater than 90", and "FROM table1 JOIN table2 ON (table1.id=table2.id)" is to aggregate corresponding information in the same serial number in the two tables. The left side is a flowchart without using a predicate pushdown mechanism, and the flow is to first join table 1 (table1) and table 2 (table2), and then obtain the final result (result) after condition filtering and column pruning. The right side is a flowchart using a predicate pushdown mechanism, and the flow is to first filter and prune table 1 and table 2 respectively, and then join to obtain the final result (result).

[0071] Figure 4 A schematic diagram of reading Hbase data using the method of the present application. As shown in the figure, an Hbase table is divided into M logical partitions, each of which contains a set number of Hfile files, the Row Key of logical partition_1 is [a, b] and contains Hfile_1 and Hfile_i, and the Row Key of logical partition_M is [y, z] and contains Hfile_j and Hfile_k. Then, there is a corresponding task for each logical partition, and a data scanner (Region Scanner) with a row filter and a column filter under the task scans the logical partition, and returns the target data (the target data is the data required by the user obtained by scanning with the data scanner with the row filter and the column filter) to the Spark application. Figure 4

[0072] Figure 5 A specific flowchart of a data reading method provided in the present application. As shown in the figure, the specific flow is as follows: Figure 5

[0073] S501: Preconfigure environment parameters;

[0074] ​​Before the electronic device receives the user reading request, the Hbase cluster configuration file (hbase-site.xml) for the Spark job needs to be provided, which contains the address of the Hbase root on the HDFS, the IP of the distributed application coordination service software (zookeeper) node and other parameters.

[0075] S502: receiving a user request;

[0076] The electronic device receives a user data reading request, and the data reading request includes the identification of the target data table and the data filtering condition information. The identification of the data table includes the namespace, table name, and field name (i.e. column cluster name + column modifier) of the Hbase data table. The data filtering condition information of the target data table is a series of filtering condition statements set by the user to filter the target data table to obtain the target data.

[0077] S503: constructing a filter (such as a row filter, a column filter);

[0078] Using the idea of Spark predicate pushdown mechanism, the filtering conditions set in Spark (i.e. the row filtering conditions for filtering the rows of the target data table) are constructed as the row filter of the data scanner, and the column pruning information (i.e. the identification of the columns of the target data table that need to be read) is constructed as the column filter of the data scanner.

[0079] S504: dividing logical partitions;

[0080] First, the snapshot of the target data table is obtained, from which the physical partition information of the target data table is obtained, including the number of physical partitions of the target data table and the row key range of each physical partition. Then, according to the read parallelism setting of the target user, each physical partition of the target data table is divided into multiple logical partitions.

[0081] S505: creating a data scanner for each logical partition;

[0082] According to the start row key and end row key information of each logical partition, a data scanner with the same start row key and end row key as each logical partition is constructed, and the row filter and column filter constructed in step S503 are configured into the data scanner.

[0083] S506: reading each logical partition;

[0084] In each logical partition, the data scanner with the row filter and column filter conditionally scans the logical partition. And the reader under each logical partition directly reads the result scanned by the data scanner (i.e. the target data of the logical partition).

[0085] Based on the above data reading method, the application provides a data reading device, Figure 6 A data reading device provided by the embodiment of the application has the structure as shown in the figure, which comprises:

[0086] The acquisition module 601 is configured to receive a data reading request, wherein the data reading request comprises an identification of a target data table and data filtering condition information.

[0087] The processing module 602 is configured to divide a physical partition of the target data table into a plurality of logical partitions, and acquire target data in each logical partition of the plurality of logical partitions, wherein the target data in each logical partition is obtained by a reader of the logical partition from data in the logical partition according to the data filtering condition information.

[0088] Further, the filtering condition information comprises a row filtering condition for screening rows of the target data table, and an identification of columns of the target data table that need to be read.

[0089] Further, when dividing the physical partition of the target data table into a plurality of logical partitions, the processing module 602 is specifically configured to acquire a snapshot of the target data table and a read parallelism setting of a user, wherein the snapshot of the target data table comprises physical partition information of the target data table; and divide each physical partition of the target data table into a plurality of logical partitions according to the physical partition information of the target data table and the read parallelism setting of the user.

[0090] Further, the processing module 602 is further configured to construct a data scanner according to the data filtering condition information; and the target data in each logical partition is obtained by the reader of the logical partition from data in the logical partition according to the data filtering condition information, comprising that the target data in each logical partition is obtained by the reader of the logical partition from scanning of data in the logical partition by using the data scanner.

[0091] Further, the processing module 602 is further configured to create a target data reading task for each logical partition of the plurality of logical partitions, to obtain a plurality of target data reading tasks corresponding to the plurality of logical partitions; and when acquiring the target data in each logical partition of the plurality of logical partitions, the processing module 602 is specifically configured to run at least two target data reading tasks of the plurality of target data reading tasks in parallel, and acquire target data of a logical partition under each target data reading task of the at least two target data reading tasks.

[0092] Figure 7 A structure of an electronic device is shown in the figure. Figure 7As shown, the electronic device includes a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702 and the memory 703 complete communication with each other through the communication bus 704.

[0093] The memory 703 stores a computer program, and when the program is executed by the processor 701, the processor 701 implements the steps of any one of the above-mentioned data reading methods.

[0094] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0095] The communication interface 702 is used for communication between the above-mentioned electronic device and other devices.

[0096] The memory can include a Random Access Memory (RAM) and can also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0097] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (NP), etc.; can also be a Digital Signal Processing (DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0098] On the basis of the above-mentioned embodiments, the embodiments of the present application further provide a computer readable storage medium, which stores a computer program executable by an electronic device, and when the program runs on the electronic device, the electronic device implements the steps of any one of the above-mentioned data reading methods.

[0099] The computer readable storage medium described above can be any available medium or data storage that can be accessed by a processor in the electronic device, including but not limited to a magnetic memory such as a floppy disk, a hard disk, a magnetic tape, a magneto-optical disk (MO), etc., an optical memory such as a CD, a DVD, a BD, a HVD, etc., and a semiconductor memory such as a ROM, an EPROM, an EEPROM, a non-volatile memory (NAND FLASH), a solid state disk (SSD), etc.

[0100] Finally, it should be noted that the above embodiments are merely used to illustrate the technical solutions of the present application, rather than limit the technical solutions of the present application; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the above embodiments, or make equivalent replacements for part or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0101] For the convenience of explanation, the above description has been made in combination with specific embodiments. However, the above exemplary discussion is not intended to exhaust or limit the embodiments to the specific forms disclosed above. Various modifications and variations can be derived according to the above teachings. The selection and description of the above embodiments are for better explanation of the principles and practical applications, so that those skilled in the art can better use the embodiments and various different modified embodiments suitable for specific use considerations.

[0102] Those skilled in the art will appreciate that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0103] Those skilled in the art will appreciate that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0104] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 one or more flow or blocks.

[0105] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 one or more flow or blocks.

[0106] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 one or more flow or blocks.

[0107] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A data reading method characterized by, The method is applied to an electronic device, and the method comprises: receiving a data reading request, the data reading request comprising an identification of a target data table and data filtering condition information; dividing a physical partition of the target data table into a plurality of logical partitions; constructing a data scanner according to the data filtering condition information; wherein, according to start row key and end row key information of each logical partition, a data scanner with the same start row key and end row key as each logical partition is constructed; and by using a Spark predicate pushdown mechanism, a filtering condition set in Spark is constructed as a row filter of the data scanner, and column clipping information is constructed as a column filter of the data scanner; obtaining target data in each of the plurality of logical partitions, wherein the target data in each of the logical partitions is obtained by a reader of the logical partition by scanning data of the logical partition by using the data scanner.

2. The method of claim 1, wherein, The filtering condition information comprises: a row filtering condition for screening rows of the target data table, and an identification of a column of the target data table that needs to be read.

3. The method of claim 1, wherein, The dividing of the physical partition of the target data table into a plurality of logical partitions comprises: obtaining a snapshot of the target data table and a user reading parallelism setting; wherein the snapshot of the target data table comprises information of one or more physical partitions of the target data table; dividing each of the one or more physical partitions into a plurality of logical partitions according to the user reading parallelism setting.

4. The method of claim 1, wherein, The method further comprises: creating a target data reading task for each of the plurality of logical partitions, to obtain a plurality of target data reading tasks corresponding to the plurality of logical partitions; The obtaining of the target data in each of the plurality of logical partitions comprises: running at least two of the plurality of target data reading tasks in parallel, and obtaining target data in a logical partition under each of the at least two target data reading tasks.

5. A data reading device, characterized by The device comprises: an obtaining module configured to receive a data reading request, the data reading request comprising an identification of a target data table and data filtering condition information; a processing module configured to divide a physical partition of the target data table into a plurality of logical partitions; construct a data scanner according to the data filtering condition information; wherein, according to start row key and end row key information of each logical partition, a data scanner with the same start row key and end row key as each logical partition is constructed; and by using a Spark predicate pushdown mechanism, a filtering condition set in Spark is constructed as a row filter of the data scanner, and column clipping information is constructed as a column filter of the data scanner; and obtain target data in each of the plurality of logical partitions, wherein the target data in each of the logical partitions is obtained by a reader of the logical partition by scanning data of the logical partition by using the data scanner.

6. The apparatus of claim 5, wherein, When the processing module divides the physical partition of the target data table into a plurality of logical partitions, the processing module is specifically configured to: obtaining a snapshot of the target data table and a user read parallelism setting; wherein the snapshot of the target data table contains information of one or more physical partitions of the target data table; and dividing each of the one or more physical partitions into a plurality of logical partitions according to the user read parallelism setting.

7. An electronic device, comprising: The electronic device at least includes a processor and a memory, and the processor implements the method as claimed in any one of claims 1-4 when executing the computer program or instructions stored in the memory.

8. A computer-readable storage medium, characterized in that, The computer program or instructions stored in the memory are executed by the processor to implement the method as claimed in any one of claims 1-4.

Images