Data processing method and apparatus

Abstract

This invention provides a data processing method and apparatus. The method includes: identifying an offline data warehouse and a real-time data warehouse to be connected; determining whether the offline and real-time data warehouses meet connection conditions; when the connection conditions are met, using Presto to connect the offline and real-time data warehouses to obtain a Hive data warehouse; establishing a unified query view for Hive in Presto; and having an external program process data query requests through the API provided by Presto and return the target data to the client. By applying the method provided by this invention, connecting the offline and real-time data warehouses through Presto to form a unified Hive data warehouse, the data that the user needs to query can be quickly found.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data processing method and apparatus. Background Technology

[0002] Multiple data warehouses are set up in various business systems to store various data generated in real time. In these systems, historically generated data is typically stored in offline data warehouses, while data generated in real time is stored in real-time data warehouses. Because multiple offline and real-time data warehouses may exist within a business system, data retrieval requires analysis and calculations for each warehouse, which can lead to lengthy query times for a single piece of data, impacting the service requirements of various business functions. Summary of the Invention

[0003] In view of this, the present invention provides a data processing method that connects offline and real-time data warehouses through Presto to form a unified Hive data warehouse, enabling users to quickly find the data they need to query.

[0004] The present invention also provides a data processing apparatus to ensure the implementation and application of the above method in practice.

[0005] A data processing method, comprising:

[0006] Identify at least one offline data warehouse and one real-time data warehouse to be connected, wherein the offline data warehouse is used to store historical data and the real-time data warehouse is used to store real-time data generated within a preset time period;

[0007] Determine whether the offline data warehouse and the real-time data warehouse meet the preset connection conditions;

[0008] When the offline data warehouse and the real-time data warehouse meet the preset connection conditions, the preset open-source multi-data source distributed SQL query engine Presto is used to connect the offline data warehouse and the real-time data warehouse to obtain the Hive data warehouse.

[0009] In Presto, a query view corresponding to the Hive data warehouse is created, and when an external program sends a data query request through the API provided by Presto, the target data corresponding to the data query request is returned to the client.

[0010] Optionally, in the above method, determining whether the offline data warehouse and the real-time data warehouse meet preset connection conditions includes:

[0011] Determine the first table structure of the data table in the offline data warehouse and the second table structure of the data table in the real-time data warehouse;

[0012] Determine whether the first table structure is consistent with the second table structure, and whether each piece of real-time data in the real-time data warehouse has been separated for reading and writing;

[0013] When the first table structure is consistent with the second table structure, and the real-time data in the real-time data warehouse has been separated for reading and writing, it is determined that the offline data warehouse and the real-time data warehouse meet the connection conditions.

[0014] When the first table structure is inconsistent with the second table structure, or when the real-time data in the real-time data warehouse has been separated for reading and writing, it is determined that the offline data warehouse and the real-time data warehouse do not meet the connection conditions.

[0015] The above methods may also include:

[0016] When the offline data warehouse and the real-time data warehouse do not meet the connection conditions, the second table structure is adjusted according to the first table structure, and / or, the read and write of each real-time data in the real-time data warehouse is separated to obtain the first data cluster and the second data cluster in the real-time data warehouse.

[0017] Optionally, in the above method, the step of returning the target data corresponding to the data query request to the client when receiving a data query request sent by an external program through the API provided by Presto includes:

[0018] Enable the Presto coordinator in the Presto to parse the data query request and obtain the request information of the data query request;

[0019] Obtain the write time corresponding to the target data contained in the request information;

[0020] Based on the write time, the target data warehouse to which the target data belongs is determined, and the target data warehouse is either the offline data warehouse or the real-time data warehouse;

[0021] Access the Hive data warehouse through the warehouse connector between the Presto and the Hive data warehouse, and query the target data in the target data warehouse through the Hive data warehouse.

[0022] The above methods may also include:

[0023] Determine whether the current time point has entered the next timing period;

[0024] When the current time point has entered the next time period, all the data stored in the real-time data warehouse are migrated to the offline data warehouse to update the Hive data warehouse.

[0025] A data processing method, comprising:

[0026] A determining unit is used to determine at least one offline data warehouse and a real-time data warehouse to be connected, wherein the offline data warehouse is used to store historical data and the real-time data warehouse is used to store real-time data generated in real time within a preset time period.

[0027] The judgment unit is used to determine whether the offline data warehouse and the real-time data warehouse meet the preset connection conditions;

[0028] The connection unit is used to connect the offline data warehouse and the real-time data warehouse to obtain the Hive data warehouse when the offline data warehouse and the real-time data warehouse meet the preset connection conditions;

[0029] The query unit is used to create a query view corresponding to the Hive data warehouse in the Presto, and when it receives a data query request sent by an external program through the API provided by the Presto, it returns the target data corresponding to the data query request to the client.

[0030] Optionally, in the aforementioned apparatus, the determining unit includes:

[0031] The first determining subunit is used to determine the first table structure of the data table in the offline data warehouse and the second table structure of the data table in the real-time data warehouse;

[0032] The judgment subunit is used to determine whether the first table structure and the second table structure are consistent, and whether each piece of real-time data in the real-time data warehouse has been read and written separately.

[0033] The second determining subunit is used to determine that the offline data warehouse and the real-time data warehouse meet the connection conditions when the first table structure is consistent with the second table structure and the real-time data in the real-time data warehouse has been separated from read and write.

[0034] The third determining subunit is used to determine that the offline data warehouse and the real-time data warehouse do not meet the connection conditions when the first table structure is inconsistent with the second table structure, or when the real-time data in the real-time data warehouse has been read and written separately.

[0035] The aforementioned apparatus may optionally further include:

[0036] The adjustment subunit is used to adjust the second table structure according to the first table structure when the offline data warehouse and the real-time data warehouse do not meet the connection conditions, and / or to separate the read and write operations of each real-time data in the real-time data warehouse to obtain the first data cluster and the second data cluster in the real-time data warehouse.

[0037] Optionally, the query unit in the aforementioned apparatus includes:

[0038] The parsing subunit is used to enable the presto coordinator in the presto to parse the data query request and obtain the request information of the data query request;

[0039] The acquisition subunit is used to acquire the write time corresponding to the target data contained in the request information;

[0040] The fourth determining subunit is used to determine the target data warehouse to which the target data belongs based on the write time, wherein the target data warehouse is the offline data warehouse or the real-time data warehouse;

[0041] The query subunit is used to access the Hive data warehouse through the warehouse connector between the presto and the Hive data warehouse, and to query the target data in the target data warehouse through the Hive data warehouse.

[0042] The aforementioned apparatus may optionally further include:

[0043] The fifth determining sub-unit is used to determine whether the current time point has entered the next timing period;

[0044] The data migration subunit is used to migrate the data stored in the real-time data warehouse to the offline data warehouse when the current time point has entered the next time period, so as to update the Hive data warehouse.

[0045] A storage medium comprising stored instructions, wherein, when the instructions are executed, the device in which the storage medium resides controls the execution of the data processing method described above.

[0046] An electronic device includes a memory and one or more instructions, wherein one or more instructions are stored in the memory and configured to be executed by one or more processors using the data processing method described above.

[0047] Compared with the prior art, the present invention has the following advantages:

[0048] This invention provides a data processing method, comprising: determining at least one offline data warehouse and a real-time data warehouse to be connected, wherein the offline data warehouse is used to store historical data, and the real-time data warehouse is used to store real-time data generated within a preset time period; determining whether the offline data warehouse and the real-time data warehouse meet preset connection conditions; when the offline data warehouse and the real-time data warehouse meet the preset connection conditions, applying Presto to connect the offline data warehouse and the real-time data warehouse to obtain a Hive data warehouse; establishing a query view corresponding to the Hive data warehouse in Presto, and returning the target data corresponding to the data query request to the client when receiving a data query request sent by an external program through the API provided by Presto. By applying the method provided by this invention, connecting the offline data warehouse and the real-time data warehouse into a unified Hive data warehouse through Presto, the data that the user needs to query can be quickly found. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0050] Figure 1 A flowchart of a data processing method provided in an embodiment of the present invention;

[0051] Figure 2 Another flowchart of a data processing method provided in an embodiment of the present invention;

[0052] Figure 3 A further flowchart of a data processing method provided in an embodiment of the present invention;

[0053] Figure 4 This is a structural diagram of a data processing apparatus provided in an embodiment of the present invention;

[0054] Figure 5 This is a schematic diagram of an electronic device structure provided in an embodiment of the present invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0057] This invention can be used in a wide variety of general-purpose or special-purpose computing environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor devices, distributed computing environments including any of the above devices, etc.

[0058] This invention provides a data processing method that can be applied to various system platforms. The executing entity can be a computer terminal or a processor of various mobile devices. The method flowchart is shown below. Figure 1 As shown, it specifically includes:

[0059] S101: Identify at least one offline data warehouse and one real-time data warehouse to be connected.

[0060] The offline data warehouse stores historical data, while the real-time data warehouse stores real-time data generated within a preset time period. For example, the real-time data warehouse stores data generated on the current day, while the offline data warehouse stores historical data generated before the current date.

[0061] It should be noted that a business system can have multiple offline data warehouses and multiple real-time data warehouses. Different offline and real-time data warehouses can be used to store business data corresponding to different business operations. If a business system has only one offline data warehouse and one real-time data warehouse, then the offline data warehouse stores historical business data for all business operations, and the real-time data warehouse stores business data generated by all business operations on that day.

[0062] Understandably, offline data warehouses and real-time data warehouses are divided based on time range.

[0063] In this embodiment of the invention, the offline data warehouse can be a data warehouse using the distributed system infrastructure Hadoop, and the real-time data warehouse can be a data warehouse using Kudu. Kudu is a storage solution for fast data analysis; combined with fast insertion and update and efficient histogram scanning, it can implement multiple real-time analysis workloads on a single storage layer.

[0064] S102: Determine whether the offline data warehouse and the real-time data warehouse meet the preset connection conditions.

[0065] It's important to note that the table structures of data tables in the offline data warehouse and the real-time data warehouse may differ. Before establishing a connection, the table structures of the data warehouses to be connected must be consistent. Furthermore, since the real-time data warehouse receives real-time data, read and write operations in the real-time data warehouse need to be separated to avoid data queries affecting real-time data writing. Therefore, consistent table structures between data warehouses and read / write analysis of the real-time data warehouse are connection requirements that must be met between the offline and real-time data warehouses.

[0066] S103: When the offline data warehouse and the real-time data warehouse meet the preset connection conditions, the preset browser layout engine Presto is applied to connect the offline data warehouse and the real-time data warehouse to obtain the Hive data warehouse.

[0067] It's important to note that connecting offline and real-time data warehouses using Presto involves connecting each warehouse through various Presto data source connectors. Because the data sources for offline and real-time data warehouses differ, and Presto includes multiple data source connectors, the various data warehouses can be connected based on their respective data sources. The connected offline and real-time data warehouses then constitute the Hive data warehouse. Hive is a data warehouse tool based on Hadoop, and Hive data warehouses provide SQL query capabilities.

[0068] S104: Establish a query view corresponding to the Hive data warehouse in the Presto, and when an external program sends a data query request through the API provided by the Presto, return the target data corresponding to the data query request to the client.

[0069] In this invention, a query view that can span two data warehouses is created in Presto. External programs can connect to Presto through this view and send data query requests through the API provided by Presto.

[0070] In the method provided by this invention, the offline and real-time data warehouses to be connected in the business system are determined, and it is determined whether each data warehouse meets the connection conditions. If so, Presto is used to connect the data warehouses to form a Hive data warehouse. A query view corresponding to the Hive data warehouse is established in Presto. External programs can send data query requests through Presto from the query view, including the data information of the target data to be queried. The target data corresponding to the data query request in the Hive data warehouse can be found based on the data information, and the found target data is returned to the client.

[0071] By applying the method provided in this embodiment of the invention, offline and real-time data warehouses can be connected through Presto to form a unified Hive data warehouse, which can quickly find the data that users need to query.

[0072] In the method provided in the embodiments of the present invention, reference is made to Figure 2 Determining whether the offline data warehouse and the real-time data warehouse meet preset connection conditions includes:

[0073] S201: Determine the first table structure of the data table in the offline data warehouse and the second table structure of the data table in the real-time data warehouse.

[0074] S202: Determine whether the structure of the first table is consistent with the structure of the second table.

[0075] Specifically, when the first table structure is consistent with the second table structure, S203 is executed; when the first table structure is inconsistent with the second table structure, S205 is executed.

[0076] Furthermore, when the first table structure is inconsistent with the second table structure, the second table structure is adjusted according to the first table structure.

[0077] S203: Determine whether the read and write operations of each piece of real-time data in the real-time data warehouse have been separated.

[0078] Specifically, when the real-time data in the real-time data warehouse has been separated from reads, S204 is executed; when the real-time data in the real-time data warehouse has not been separated from reads, S205 is executed.

[0079] Furthermore, when the real-time data in the real-time data warehouse has been separated into read and write operations, the real-time data in the real-time data warehouse is further separated into read and write operations to obtain the first data cluster and the second data cluster in the real-time data warehouse.

[0080] It's important to note that both the first and second data clusters are Kudu clusters. In high-concurrency, high-traffic applications with large data volumes, Presto will query the Kudu cluster (a read operation), while Flink itself will write to Kudu for real-time data processing. Without read-write separation, there will be an IO bottleneck. However, with separation, and given that Flink and Kudu are distributed and scalable, theoretically, they can be expanded indefinitely to support the business's query and write operations.

[0081] S204: Determine that the offline data warehouse and the real-time data warehouse meet the connection conditions.

[0082] S205: It is determined that the offline data warehouse and the real-time data warehouse do not meet the connection conditions.

[0083] In this invention, after executing the judgments in S202 and S203 above, if the first table structure is inconsistent with the second table structure, and the real-time data in the real-time data warehouse has already undergone read-write separation, only the second table structure needs to be adjusted according to the first table structure. If the first table structure is consistent with the second table structure, but the real-time data in the real-time data warehouse has not undergone read-write separation, only the real-time data in the real-time data warehouse needs to undergo read-write separation. If the first table structure is inconsistent with the second table structure, and the real-time data in the real-time data warehouse has not undergone read-write separation, the second table structure needs to be adjusted according to the first table structure before the real-time data in the real-time data warehouse undergoes read-write separation.

[0084] It should be noted that after obtaining two data clusters through read-write separation, the real-time data warehouse can write data to both data clusters when writing data.

[0085] In the method provided by this invention, before connecting the offline data warehouse and the real-time data warehouse, it is necessary to ensure that the tables in each data warehouse are completely consistent to ensure that data queries can be performed quickly. At the same time, it is necessary to separate the read and write operations of the real-time data in the real-time data warehouse to prevent the hot spots from all falling in the real-time data warehouse due to interactive queries while writing massive amounts of real-time data.

[0086] In the method provided in the embodiments of the present invention, reference is made to Figure 3 When a data query request is received from an external program via the API provided by Presto, the target data corresponding to the data query request is returned to the client, including:

[0087] S301: Enable the presto coordinator in the presto to parse the data query request and obtain the request information of the data query request.

[0088] It should be noted that the Presto coordinator is an information parsing tool.

[0089] S302: Obtain the write time corresponding to the target data contained in the request information.

[0090] It should be noted that write time refers to the time it takes for data to be written to the real-time data warehouse after it is generated in the business system.

[0091] Optionally, the method provided in the embodiments of the present invention further includes:

[0092] Determine whether the current time point has entered the next timing period;

[0093] When the current time point has entered the next time period, all the data stored in the real-time data warehouse are migrated to the offline data warehouse to update the Hive data warehouse.

[0094] Understandably, data is written to the real-time data warehouse when it is generated. When the set time is reached, the data in the real-time data warehouse will be migrated to the offline data warehouse, and the real-time data in the real-time data warehouse will be changed to offline data.

[0095] For example, a real-time data warehouse stores data generated between 0:00 and 24:00 on a given day. When the time jumps to 0:00 on the next day, the real-time data previously stored in the real-time data warehouse is migrated to an offline data warehouse. During the data migration, the offline and real-time data warehouses remain connected.

[0096] S303: Based on the write time, determine the target data warehouse to which the target data belongs, wherein the target data warehouse is the offline data warehouse or the real-time data warehouse.

[0097] In this invention, if the write time does not belong to the time period of the day, the target data to be acquired is offline data in the offline data warehouse; if it belongs to the time period of the day, the target data to be acquired is real-time data in the real-time data warehouse.

[0098] S304: Access the Hive data warehouse through the warehouse connector between the presto and the Hive data warehouse, and query the target data in the target data warehouse through the Hive data warehouse.

[0099] In this invention, when querying data, the target data is queried through the warehouse connector.

[0100] It should also be noted that if the target data warehouse is a real-time data warehouse, then when reading data, Flink is used to process the "hot" data portion of the data warehouse, as well as the recent real-time data.

[0101] In the method provided by this invention, the request information is parsed through the Presto coordinator to further determine the write time of the target data to be queried, and thus determine the data warehouse to which the target data currently belongs. The target data is then obtained by connecting to the Hive data warehouse through a warehouse connector.

[0102] The specific implementation processes and derivative methods of the above embodiments are all within the protection scope of this invention.

[0103] and Figure 1 Corresponding to the method described above, embodiments of the present invention also provide a data processing apparatus for processing data... Figure 1 The specific implementation of the method, the data processing device provided in this embodiment of the invention can be applied to computer terminals or various mobile devices, and its structural schematic diagram is shown below. Figure 4 As shown, it specifically includes:

[0104] The determining unit 401 is used to determine at least one offline data warehouse and a real-time data warehouse to be connected, wherein the offline data warehouse is used to store historical data and the real-time data warehouse is used to store real-time data generated in real time within a preset time period.

[0105] Judgment unit 402 is used to determine whether the offline data warehouse and the real-time data warehouse meet the preset connection conditions;

[0106] The connection unit 403 is used to connect the offline data warehouse and the real-time data warehouse using a preset browser layout engine Presto to obtain the Hive data warehouse when the offline data warehouse and the real-time data warehouse meet the preset connection conditions.

[0107] The query unit 404 is used to establish a query view corresponding to the Hive data warehouse in the Presto, and when it receives a data query request sent by an external program through the API provided by the Presto, it returns the target data corresponding to the data query request to the client.

[0108] In the apparatus provided by this invention, offline and real-time data warehouses to be connected in the business system are identified, and it is determined whether each data warehouse meets the connection conditions. If so, Presto is used to connect the data warehouses to form a Hive data warehouse. A query view corresponding to the Hive data warehouse is established in Presto. External programs can send data query requests through Presto from the query view, including the data information of the target data to be queried. The target data corresponding to the data query request in the Hive data warehouse can be found based on the data information, and the found target data is returned to the client.

[0109] By using the apparatus provided in this embodiment of the invention, offline and real-time data warehouses can be connected through Presto to form a unified Hive data warehouse, which can quickly find the data that the user needs to query.

[0110] In the apparatus provided in this embodiment of the invention, the determining unit 402 includes:

[0111] The first determining subunit is used to determine the first table structure of the data table in the offline data warehouse and the second table structure of the data table in the real-time data warehouse;

[0112] The judgment subunit is used to determine whether the first table structure and the second table structure are consistent, and whether each piece of real-time data in the real-time data warehouse has been read and written separately.

[0113] The second determining subunit is used to determine that the offline data warehouse and the real-time data warehouse meet the connection conditions when the first table structure is consistent with the second table structure and the real-time data in the real-time data warehouse has been separated from read and write.

[0114] The third determining subunit is used to determine that the offline data warehouse and the real-time data warehouse do not meet the connection conditions when the first table structure is inconsistent with the second table structure and / or when each real-time data in the real-time data warehouse has been read and written separately.

[0115] The apparatus provided in this embodiment of the invention further includes:

[0116] The adjustment subunit is used to adjust the second table structure according to the first table structure when the offline data warehouse and the real-time data warehouse do not meet the connection conditions, and / or to separate the read and write operations of each real-time data in the real-time data warehouse to obtain the first data cluster and the second data cluster in the real-time data warehouse.

[0117] In the apparatus provided in this embodiment of the invention, the query unit 404 includes:

[0118] The parsing subunit is used to enable the presto coordinator in the presto to parse the data query request and obtain the request information of the data query request;

[0119] The acquisition subunit is used to acquire the write time corresponding to the target data contained in the request information;

[0120] The fourth determining subunit is used to determine the target data warehouse to which the target data belongs based on the write time, wherein the target data warehouse is the offline data warehouse or the real-time data warehouse;

[0121] The query subunit is used to access the Hive data warehouse through the warehouse connector between the presto and the Hive data warehouse, and to query the target data in the target data warehouse through the Hive data warehouse.

[0122] The apparatus provided in this embodiment of the invention further includes:

[0123] The fifth determining sub-unit is used to determine whether the current time point has entered the next timing period;

[0124] The data migration subunit is used to migrate the data stored in the real-time data warehouse to the offline data warehouse when the current time point has entered the next time period, so as to update the Hive data warehouse.

[0125] The specific working process of each unit and subunit in the data processing device disclosed in the above embodiments of the present invention can be found in the corresponding content of the data processing method disclosed in the above embodiments of the present invention, and will not be repeated here.

[0126] This invention also provides a storage medium, which includes stored instructions, wherein the execution of the instructions controls the device where the storage medium is located to perform the above-described data processing method.

[0127] This invention also provides an electronic device, the structural schematic of which is shown below. Figure 5 As shown, it specifically includes a memory 501 and one or more instructions 502, wherein one or more instructions 502 are stored in the memory 501 and configured to be executed by one or more processors 503 to perform the following operations:

[0128] Identify at least one offline data warehouse and one real-time data warehouse to be connected, wherein the offline data warehouse is used to store historical data and the real-time data warehouse is used to store real-time data generated within a preset time period;

[0129] Determine whether the offline data warehouse and the real-time data warehouse meet the preset connection conditions;

[0130] When the offline data warehouse and the real-time data warehouse meet the preset connection conditions, the preset browser layout engine Presto is applied to connect the offline data warehouse and the real-time data warehouse to obtain the Hive data warehouse;

[0131] In the Presto, a task scheduling platform Airflow corresponding to the Hive data warehouse is established. When a data query request is received from a client through Airflow, the platform accesses the Hive data warehouse to query the target data corresponding to the data query request and returns the target data to the client.

[0132] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0133] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both.

[0134] To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality above. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0135] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method, characterized in that, include: Identify at least one offline data warehouse and one real-time data warehouse to be connected, wherein the offline data warehouse is used to store historical data and the real-time data warehouse is used to store real-time data generated within a preset time period; Determine the first table structure of the data tables in the offline data warehouse and the second table structure of the data tables in the real-time data warehouse; Determine whether the first table structure is consistent with the second table structure, and whether each piece of real-time data in the real-time data warehouse has been separated for reading and writing; When the first table structure is consistent with the second table structure, and the real-time data in the real-time data warehouse has been separated for reading and writing, it is determined that the offline data warehouse and the real-time data warehouse meet the preset connection conditions. When the first table structure is inconsistent with the second table structure, or when the real-time data in the real-time data warehouse is not read-write separated, it is determined that the offline data warehouse and the real-time data warehouse do not meet the connection conditions. When the offline data warehouse and the real-time data warehouse meet the preset connection conditions, the preset open-source multi-data source distributed SQL query engine Presto is used to connect the offline data warehouse and the real-time data warehouse to obtain the Hive data warehouse. In Presto, a query view corresponding to the Hive data warehouse is created, and when a data query request is received from an external program through the API provided by Presto, the target data is returned to the client according to the write time of the target data to be queried contained in the data query request.

2. The method according to claim 1, characterized in that, Also includes: When the offline data warehouse and the real-time data warehouse do not meet the connection conditions, the second table structure is adjusted according to the first table structure, and / or, the read and write of each real-time data in the real-time data warehouse is separated to obtain the first data cluster and the second data cluster in the real-time data warehouse.

3. The method according to claim 1, characterized in that, The step of returning the target data corresponding to the data query request to the client when receiving a data query request sent by an external program through the API provided by Presto includes: Enable the Presto coordinator in the Presto to parse the data query request and obtain the request information of the data query request; Obtain the write time corresponding to the target data contained in the request information; Based on the write time, the target data warehouse to which the target data belongs is determined, and the target data warehouse is either the offline data warehouse or the real-time data warehouse; Access the Hive data warehouse through the warehouse connector between the Presto and the Hive data warehouse, and query the target data in the target data warehouse through the Hive data warehouse.

4. The method according to any one of claims 1 to 3, characterized in that, Also includes: Determine whether the current time point has entered the next timing period; When the current time point has entered the next time period, all the data stored in the real-time data warehouse are migrated to the offline data warehouse to update the Hive data warehouse.

5. A data processing apparatus, characterized in that, include: A determining unit is used to determine at least one offline data warehouse and a real-time data warehouse to be connected, wherein the offline data warehouse is used to store historical data and the real-time data warehouse is used to store real-time data generated in real time within a preset time period. The judgment unit is used to determine the first table structure of the data table in the offline data warehouse and the second table structure of the data table in the real-time data warehouse; to determine whether the first table structure and the second table structure are consistent, and whether each real-time data in the real-time data warehouse has been read and written separately; when the first table structure and the second table structure are consistent, and each real-time data in the real-time data warehouse has been read and written separately, to determine that the offline data warehouse and the real-time data warehouse meet the preset connection conditions; When the first table structure is inconsistent with the second table structure, or when the real-time data in the real-time data warehouse is not read-write separated, it is determined that the offline data warehouse and the real-time data warehouse do not meet the connection conditions. The connection unit is used to connect the offline data warehouse and the real-time data warehouse to obtain the Hive data warehouse when the offline data warehouse and the real-time data warehouse meet the preset connection conditions; The query unit is used to establish a query view corresponding to the Hive data warehouse in the Presto, and when it receives a data query request sent by an external program through the API provided by the Presto, it returns the target data corresponding to the data query request to the client according to the write time of the target data to be queried contained in the data query request.

6. The apparatus according to claim 5, characterized in that, Also includes: The adjustment subunit is used to adjust the second table structure according to the first table structure when the offline data warehouse and the real-time data warehouse do not meet the connection conditions, and / or to separate the read and write operations of each real-time data in the real-time data warehouse to obtain the first data cluster and the second data cluster in the real-time data warehouse.

7. The apparatus according to claim 5, characterized in that, The query unit includes: The parsing subunit is used to enable the presto coordinator in the presto to parse the data query request and obtain the request information of the data query request; The acquisition subunit is used to acquire the write time corresponding to the target data contained in the request information; The fourth determining subunit is used to determine the target data warehouse to which the target data belongs based on the write time, wherein the target data warehouse is the offline data warehouse or the real-time data warehouse; The query subunit is used to access the Hive data warehouse through the warehouse connector between the presto and the Hive data warehouse, and to query the target data in the target data warehouse through the Hive data warehouse.

8. The apparatus according to any one of claims 5 to 7, characterized in that, Also includes: The fifth determining sub-unit is used to determine whether the current time point has entered the next timing period; The data migration subunit is used to migrate the data stored in the real-time data warehouse to the offline data warehouse when the current time point has entered the next time period, so as to update the Hive data warehouse.

Images