Efficient data processing method for hundred million-level heterogeneous database based on lake and warehouse

Abstract

The invention relates to an efficient data processing method of a hundred million-level heterogeneous database based on a lake and a warehouse, and belongs to the technical field of data processing, and adopts the following technical scheme: receiving a data processing task which at least comprises to-be-processed data and specific processing operation; based on the data processing task, determining a data source, calling the to-be-processed data from the data source, and storing the to-be-processed data in a data lake; and calling the to-be-processed data from the data lake, performing corresponding specific processing operation on the to-be-processed data to generate processed data, and storing the processed data in a data warehouse. The method has the effect of improving the data processing efficiency.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an efficient data processing method for a billion-level heterogeneous database based on a lake warehouse. Background Art

[0002] Data processing methods refer to a series of operations to collect, clean, transform, analyze, store and interpret data in order to extract valuable information. These methods include statistical analysis, data mining, and learning.

[0003] Existing data processing methods generally require first retrieving the data to be processed from a data source (such as a database), and storing the data to be processed in memory, and then performing the above-mentioned specific data operations on it. However, with the rapid development of the computer field, the demand for data processing is constantly increasing, and the corresponding amount of data to be processed is also constantly increasing. When the amount of data to be processed exceeds the capacity that the memory can carry, it may be necessary to retrieve and process it in batches, that is, it cannot be stored in the memory at one time, which affects the efficiency of data processing, so it needs to be improved. Summary of the invention

[0004] In order to improve the technical problem of low efficiency in processing large amounts of data, the present application provides an efficient data processing method for a billion-level heterogeneous database based on Lake Warehouse.

[0005] In the first aspect, the present application provides an efficient data processing method for a billion-level heterogeneous database based on a lake warehouse, which adopts the following technical solutions: Receiving a data processing task, wherein the data processing task at least includes data to be processed and a specific processing operation; Based on the data processing task, determine a data source, retrieve the data to be processed from the data source, and store the data to be processed in a data lake; The data to be processed is retrieved from the data lake, and corresponding specific processing operations are performed on the data to be processed to generate processed data, and the processed data is stored in the data warehouse.

[0006] By adopting the above technical solution, data processing tasks can be proposed manually, and specific processing operations can be data cleaning, data type conversion, data statistical analysis, etc. When the amount of data to be processed is large, this application proposes to store the data to be processed in a data lake. Since the database is a large-scale data storage and processing system, it can store data of various types and formats, including structured, semi-structured and unstructured data, and the data lake has large-scale storage capabilities and can store PB-level data. Therefore, this application uses the data lake to temporarily store the data to be processed, realize one-time retrieval and processing of the data to be processed, and improve the overall efficiency of data processing.

[0007] Optionally, each of the data processing tasks also includes a task ID; The step of determining a data source based on the data processing task, retrieving the data to be processed from the data source, and storing the data to be processed in a data lake includes: Based on the data processing task, determining whether there is first data in the current data lake, where the first data refers to data shared by the data to be processed and all data stored in the current data lake; If the first data exists, establishing a first association relationship between the current data processing task and the first data; Determine a data source of all data except the first data in the data to be processed, retrieve all data except the first data from the data source, and store the retrieved data in the data lake; The retrieving the to-be-processed data from the data lake includes: Data corresponding to the data processing task and first data having a first association relationship with the data processing task are retrieved from the data lake, and the retrieved data are used as data to be processed.

[0008] By adopting the above technical solution, when the present application retrieves the data to be processed from the data source and stores it in the data lake, it will perform a preliminary analysis on the data to be processed to determine whether there is data (i.e., the first data) consistent with the data to be processed in the current data lake. If so, it is only necessary to retrieve other data except the first data from the data source, and establish a first association relationship between the data processing task and the first data. When it is necessary to retrieve the data to be processed from the data lake for specific processing operations, the data corresponding to the data processing task (i.e., all other data except the first data in the data to be processed as mentioned above) and the first data having a first association relationship with the data processing task are retrieved, thereby reducing the problem of repeated storage of the first data and increasing the storage space occupied by the first data in the data lake, and improving the large-scale storage capacity of the data lake.

[0009] Optionally, the data processing task includes a processing status, which is unprocessed or processed; and determining whether the first data exists in the current data lake based on the data processing task further includes: Add all data processing tasks before the current time whose processing status is unprocessed to the task list; Whenever a new data processing task is added to the task list, determine whether there is a corresponding associated processing task for each data processing task in the task list; wherein the associated processing task refers to a data processing task that has a second associated relationship with a target data processing task among any data processing tasks received before the current time; the target data processing task refers to any data processing task in the task list; two data processing tasks that have a second associated relationship satisfy: the processed data of one of the data processing tasks contains the same data as the data to be processed of the other data processing task; and among the two data processing tasks that have a second associated relationship, the data processing task whose processed data contains the same data is called a preceding processing task, and the data processing task whose data to be processed contains the same data is called a subsequent processing task; If the target data processing task has a corresponding associated processing task, a second association relationship is established between the associated processing task and the target data processing task, the same data is stored as the first data in the data lake, and a first association relationship is established between the first data and the subsequent processing task.

[0010] By adopting the above technical solution, among the multiple data processing tasks received, there is easily a second association relationship between two data processing tasks, that is, the processed data of one data processing task (that is, the preceding processing task) is the to-be-processed data corresponding to the other data processing task (that is, the subsequent processing task), that is, there is common data between the processed data of the preceding processing task and the to-be-processed data of the subsequent processing task, and the common data can be stored in the data lake as the first data. When it is necessary to perform a specific processing operation on the subsequent processing task, there is no need to retrieve the common data again from the data warehouse corresponding to the storage of the processed data of the preceding processing task, but the first data having a first association relationship with the current data processing task (that is, the subsequent processing task) can be directly retrieved from the data lake, thereby achieving efficient retrieval of data and further improving data processing efficiency.

[0011] Optionally, the data processing task further includes a pre-order ID and / or a post-order ID, wherein the pre-order ID refers to the task ID of the data processing task that has a second association relationship with the data processing task to which it belongs and serves as the pre-order processing task in the second association relationship; the post-order ID refers to the task ID of the data processing task that has a second association relationship with the data processing task to which it belongs and serves as the post-order processing task in the second association relationship; The step of determining whether there is a corresponding associated processing task for each data processing task in the task list comprises: If the preset first condition or second condition is met, it is considered that the target data processing task has a corresponding associated processing task; wherein the first condition means: in the task list, among other data processing tasks except the target data processing task, there exists a data processing task whose task ID is consistent with the pre-order ID and / or post-order ID contained in the target data processing task; the second condition means: in any data processing task received before the current time, there exists a data processing task whose corresponding processed data contains the to-be-processed data of the target data processing task; If the target data processing task has a corresponding associated processing task, a second association relationship is established between the associated processing task and the target data processing task, the same data is stored as first data in the data lake, and a first association relationship is established between the first data and the subsequent processing task, including: If the target data processing task has a corresponding associated processing task, a second associated relationship is established between the associated processing task and the target data processing task; if the associated processing task corresponding to the target data processing task is included in the task list, a processing order of all target data processing tasks with associated processing tasks in the task list and their corresponding associated processing tasks is determined based on the second associated relationship; If the associated processing task corresponding to the target data processing task does not exist in the task list, the common data between the processed data of the associated processing task and the to-be-processed data of the target data processing task is stored in the data lake as first data, and a first association relationship is established between the first data and the target data processing task; The determining, based on the data processing task, whether the first data exists in the current data lake includes: For the data processing tasks in the task list that have a processing order, determining in sequence according to the processing order whether the corresponding data processing tasks have corresponding first data in the data lake; For data processing tasks in the task list that do not have a processing order, it is determined for each data processing task whether corresponding first data exists in the data lake according to preset rules.

[0012] By adopting the above technical solution, all data processing tasks that have not been processed will be centrally stored in the task list. Whenever a new data processing task is added to the task list, it will be re-determined whether there are corresponding associated processing tasks for all data processing tasks in the task list. If so, the corresponding processing order will be determined. For the data processing tasks in the task list that have a processing order, the specific processing operations of the data processing tasks will be performed in sequence according to the processing order, thereby shortening the temporary storage time of the first data in the data lake, so as to release the storage space occupied by the first data in the data lake as soon as possible, and ensure the data lake's storage capacity for the to-be-processed data corresponding to subsequent data processing tasks.

[0013] Optionally, for the data processing tasks in the task list having a processing order, determining in sequence according to the processing order whether corresponding first data exists in the data lake for the corresponding data processing tasks includes: The data processing tasks in the task list that have the second association relationship are sequentially added to the preset task queue according to the corresponding processing order, and whether the corresponding data processing tasks have corresponding first data in the data lake is sequentially determined according to the stacking and popping order of the task queue.

[0014] By adopting the above technical solution, data processing tasks with processing order in the task list are managed in the form of a queue, so that the stacking and popping order of the task queue can be consistent with the processing order, so that the data processing tasks in the task queue can be processed in an orderly and efficient manner during the stacking and popping process, and the system does not need to frequently confirm the order of data processing tasks.

[0015] Optionally, the method further includes: Deleting the data processing task that generates processed data from the task list, and forming a data stream according to the data processing task, wherein the data stream includes at least a task ID of the data processing task, a data source corresponding to the data to be processed, and a data warehouse address for storing the processed data of the data processing task; A query instruction is obtained, and according to a task ID included in the query instruction, the to-be-processed data and the processed data included in the data processing task corresponding to the task ID are retrieved and fed back.

[0016] By adopting the above-mentioned technical scheme, the present application will realize the trace processing of all data processing tasks in the form of data stream, so as to facilitate users to query the corresponding processed data in the subsequent process. When users need to use the processed data of historical data processing tasks in the subsequently triggered data processing tasks, they can use the above-mentioned method to query the corresponding processed data, and add the task ID of the corresponding data processing task to the currently triggered data processing task, and limit it to the preceding ID or the following ID, so as to facilitate the present application system to determine whether the data processing task has the first data.

[0017] Optionally, for the data processing tasks in the task list that do not have a processing order, determining whether corresponding first data exists in the data lake for each data processing task according to a preset rule includes: Comprehensively determining the arrangement order of all data processing tasks that do not have a processing order in the task list according to the order of receiving time of the received data processing tasks and the processing urgency of the data processing tasks; According to the arrangement order, it is sequentially determined whether corresponding first data exists in the data lake for all data processing tasks that do not have a processing order.

[0018] In a second aspect, the present application provides an efficient data processing system for a billion-level heterogeneous database based on a lake warehouse, characterized in that it includes: A processing task receiving module, used for receiving a data processing task, wherein the data processing task at least includes data to be processed and a specific processing operation; A data retrieval and temporary storage module is used to determine a data source based on the data processing task, retrieve the data to be processed from the data source, and store the data to be processed in a data lake; The data processing and storage module is used to retrieve the data to be processed from the data lake, perform corresponding specific processing operations on the data to be processed to generate processed data, and store the processed data in the data warehouse.

[0019] In a third aspect, the present application provides an efficient data processing device for a billion-level heterogeneous database based on Hucang, comprising a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and execute any one of the methods described in the first aspect.

[0020] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program that can be loaded by a processor and execute any one of the methods described in the first aspect.

[0021] In summary, this application includes the following beneficial technical effects: This application proposes to store the data to be processed in a data lake. Since the database is a large-scale data storage and processing system, it can store data of various types and formats, including structured, semi-structured and unstructured data. The data lake has large-scale storage capabilities and can store PB-level data. Therefore, this application uses the data lake to temporarily store the data to be processed, realize one-time retrieval and processing of the data to be processed, and improve the overall efficiency of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 It is a flow chart of an efficient data processing method for a billion-level heterogeneous database based on a lake warehouse disclosed in an embodiment of the present application.

[0024] Figure 2 It is a structural block diagram of an efficient data processing system for a billion-level heterogeneous database based on a lake warehouse disclosed in an embodiment of the present application.

[0025] Explanation of the accompanying drawings: 1. Processing task receiving module; 2. Data retrieval and temporary storage module; 3. Data processing and storage module. DETAILED DESCRIPTION

[0026] The following is combined with Figure 1-2 This application is described in further detail.

[0027] The embodiment of the present application discloses an efficient data processing method for a billion-level heterogeneous database based on a lake warehouse. The efficient data processing method for a billion-level heterogeneous database based on a lake warehouse (hereinafter referred to as the data processing method) is used to provide storage space for temporary storage of the aforementioned data when processing data with a large amount of data, so that the aforementioned data can be retrieved at one time and the processing operation can be completed at one time, without the need to retrieve and process the data in batches due to the large amount of data, ultimately achieving the effect of improving data processing efficiency. The executor of the data processing method of the present application is an efficient data processing system for a billion-level heterogeneous database based on a lake warehouse (hereinafter referred to as the data processing system), which will be described in conjunction with the attached Figure 1 The process steps of the data processing method are explained in detail.

[0028] S101, receiving a data processing task, wherein the data processing task at least includes data to be processed and a specific processing operation.

[0029] In implementation, a data processing system may include a processor and a client, which are communicatively connected to each other. The client is used to interact with a user, obtain user requirements, and generate corresponding instructions or tasks, and then pass the corresponding instructions or tasks to the processor. The processor is used to store data and process data. Accordingly, the processor includes a data source (such as a database) for storing data or is communicatively connected to the data source so as to retrieve data from the data source or store data in the data source.

[0030] Specifically, users can access the data processing system in the form of a web page, and trigger data processing requirements by touching the data processing button preset on the display interface of the data processing system. The data processing system will generate data processing tasks for the corresponding data processing requirements, and automatically assign task IDs to the data processing tasks. The task IDs are unique and are used to distinguish different data processing tasks. A search box and an entry box will also be displayed. The search box is used for users to search and select data to be processed, and the entry box is used for users to enter specific processing operations. The specific processing operations can be data cleaning, data prediction, etc.

[0031] S102: Based on the data processing task, determine the data source, retrieve the data to be processed from the data source, and store the data to be processed in the data lake.

[0032] S103, retrieve the data to be processed from the data lake, perform corresponding specific processing operations on the data to be processed to generate processed data, and store the processed data in the data warehouse.

[0033] During implementation, the data processing system will determine the data source storing the data to be processed based on the data to be processed contained in the data processing task. The data source in the embodiment of the present application exists in the form of a database. Then the data processing system retrieves the data to be processed from the data source and stores the data to be processed in a preset data lake. Since the data lake has a large storage capacity, it can be used to store large amounts of data, so that the retrieval and temporary storage of the data to be processed can be completed at one time before the specific processing operations are performed on the aforementioned data to be processed.

[0034] The data processing system is also used to retrieve the data to be processed temporarily stored in the data lake, and perform specific processing operations on it to obtain processed data, and then store the processed data in a preset data warehouse, thereby completing the specific processing operations of the data processing task. Afterwards, the data processing system is also used to generate and store data streams, which include the data source of the corresponding data processing task, the data warehouse address of the processed data, and the task ID; and then delete the aforementioned data to be processed from the data lake.

[0035] Optionally, the data processing task also includes a receiving time and a processing status. The receiving time refers to the time when the data processing task is generated, and the processing status is used to distinguish whether the data processing task has obtained the processed data. If so, the processing status is processed, if not, the processing status is unprocessed.

[0036] S102 specifically includes the following sub-steps: S1021, adding all data processing tasks before the current time and in a processing status of unprocessed to the task list; S1022, whenever a new data processing task is added to the task list, determine whether there is a corresponding associated processing task for each data processing task in the task list; wherein the associated processing task refers to a data processing task that has a second associated relationship with the target data processing task among any data processing tasks received before the current time; the target data processing task refers to any data processing task in the task list; two data processing tasks that have a second associated relationship satisfy: the processed data of one of the data processing tasks contains the same data as the data to be processed of the other data processing task; and among the two data processing tasks that have a second associated relationship, the data processing task that contains the same data in the processed data is called a preceding processing task, and the data processing task that contains the same data in the data to be processed is called a subsequent processing task; S1023, if the preset first condition or second condition is met, it is considered that the target data processing task has a corresponding associated processing task; wherein the first condition means: in the task list, among other data processing tasks except the target data processing task, there exists a data processing task whose task ID is consistent with the pre-order ID and / or post-order ID contained in the target data processing task; the second condition means: in any data processing task received before the current time, there exists a data processing task whose corresponding processed data contains the to-be-processed data of the target data processing task; S1024, if the target data processing task has a corresponding associated processing task, establish a second association relationship between the associated processing task and the target data processing task, store the same data as the first data in the data lake, and establish a first association relationship between the first data and the subsequent processing task; S1025, based on the data processing task, determining whether there is first data in the current data lake, where the first data refers to data shared by the data to be processed and all data stored in the current data lake; S1026, if the first data exists, establishing a first association relationship between the current data processing task and the first data; S1027, determining the data source of all data except the first data in the data to be processed, retrieving all data except the first data from the data source, and storing the retrieved data in the data lake; S103 includes the following steps: Data corresponding to the data processing task and first data having a first association relationship with the data processing task are retrieved from the data lake, and the retrieved data are used as data to be processed.

[0037] In implementation, the data processing system is used to add all data processing tasks whose processing status is unprocessed before the current time to the task list in real time, and each time a new data processing task is added to the task list, the data processing system will determine whether there is an associated processing task for each data processing task. An associated processing task refers to a data processing task that has a second association relationship with the corresponding data processing task. As can be seen from the foregoing, two data processing tasks with a second association relationship satisfy the following condition: the processed data of one of the data processing tasks (i.e., the preceding processing task) is included in the data to be processed of the other data processing task (i.e., the subsequent processing task), that is, only after the preceding processing task completes the specific processing operation can the subsequent processing task perform the specific processing operation, and there is a sequence of processing between the two.

[0038] Correspondingly, the user can manually enter the predecessor ID and / or the successor ID. Specifically, a plus button can be displayed on the display interface of the data processing system to allow the user to trigger multiple data processing requirements at the same time, and display the task ID of the data processing task corresponding to each data processing requirement. An entry box, a search box, and an input box for entering the predecessor ID and the successor ID can be displayed at the data processing task corresponding to each data processing requirement. On this basis, the user can select the predecessor ID and / or the successor ID for the data processing task corresponding to each data processing requirement, thereby establishing a second association relationship between multiple data processing tasks.

[0039] In addition, in other embodiments, when a user triggers a data processing requirement and selects data to be processed, the data processing system may also automatically determine from the data stream stored in the historical period whether there is a corresponding processed data containing the aforementioned data processing task based on the data to be processed selected by the user; if so, the task ID of the data processing task is added as a preceding ID to the data processing task corresponding to the data processing requirement triggered by the current user.

[0040] The above-mentioned associated processing tasks that meet the first condition refer to data processing tasks that exist in the task list and are in an unprocessed state, which can be considered as data processing tasks that are triggered simultaneously with the target data processing task; and the associated processing tasks that meet the second condition can be considered as data processing tasks that have just completed specific processing operations and generated processed data.

[0041] If there is an associated processing task that meets the second condition, that is, the associated processing task plays the role of the previous processing task in the second association relationship, the data in the processed data generated by the associated processing task that is shared with the data to be processed of the target data processing task (that is, the same data) is stored as the first data in the data lake, and a first association relationship is established between the first data and the target data processing task; If there is an associated processing task that meets the first condition, it means that the associated processing task and the target data processing task are both in an unprocessed state. Then, the roles played by the associated processing task and the target data processing task in the second association relationship (i.e., the preceding processing task or the subsequent processing task) are determined. When the data processing task as the preceding processing task completes the specific processing operation and generates processed data, the data in the corresponding processed data that is shared with the data to be processed by the subsequent processing task (i.e., the same data) is stored as the first data in the data lake, and a first association relationship is established between the first data and the data processing task as the subsequent processing task.

[0042] Thereafter, whenever the data processing system determines the data source for a data processing task in the task list and retrieves the data to be processed, it will pre-determine whether there is first data in the data lake that has established a first association relationship with the data processing task. If so, it is only necessary to determine the data source for other data in the data to be processed except the first data, and retrieve the aforementioned other data from the corresponding data source and temporarily store them in the data lake, that is, the data to be processed = the first data + other data except the first data. The first data has been pre-stored in the data lake, which saves the time spent on finding the data source for the first data and retrieving the first data.

[0043] Correspondingly, when performing specific processing operations on the data to be processed later, it is necessary to retrieve the first data and other data except the first data from the data lake to form the data to be processed, and then perform specific processing operations on the data to be processed.

[0044] Optionally, S1024 specifically includes the following sub-steps: If the target data processing task has a corresponding associated processing task, a second associated relationship is established between the associated processing task and the target data processing task; if the associated processing task corresponding to the target data processing task is included in the task list, a processing order of all target data processing tasks with associated processing tasks in the task list and their corresponding associated processing tasks is determined based on the second associated relationship; If the associated processing task corresponding to the target data processing task does not exist in the task list, the common data between the processed data of the associated processing task and the to-be-processed data of the target data processing task is stored in the data lake as the first data, and a first association relationship is established between the first data and the target data processing task; “Determining whether the first data exists in the current data lake based on the data processing task” in S1025 includes the following sub-steps: For the data processing tasks in the task list that have a processing order, determine in sequence according to the processing order whether the corresponding data processing tasks have corresponding first data in the data lake; For data processing tasks in the task list that do not have a processing order, it is determined for each data processing task whether corresponding first data exists in the data lake according to preset rules.

[0045] In implementation, if the target data processing task has an associated processing task, and the associated processing task is included in the task list, that is, the associated relationship satisfies the first condition, then the data processing system of the present application is also used to determine the processing order for the data processing tasks in the task list that form a second associated relationship due to satisfying the first condition, and the data processing system will process the aforementioned data processing tasks sequentially according to the determined processing order.

[0046] For target data processing tasks that do not have associated processing tasks in the task list or whose corresponding associated processing tasks do not exist in the task list, the data processing system will process them sequentially according to preset rules.

[0047] Optionally, the aforementioned “for the data processing tasks in the task list having a processing order, determining in sequence according to the processing order whether the corresponding data processing tasks have corresponding first data in the data lake” specifically includes the following sub-steps: The data processing tasks in the task list that have the second association relationship are sequentially added to the preset task queue according to the corresponding processing order, and according to the stacking and popping order of the task queue, it is sequentially determined whether the corresponding data processing tasks have corresponding first data in the data lake.

[0048] In implementation, the data processing system will generate a task queue for each group of data processing tasks in the task list that has a determined processing order, and add the corresponding data processing tasks to the task queue in sequence according to the processing order, and the stacking and popping order of the task queue is consistent with the corresponding processing order. When the data processing tasks in the task queue need to be processed (i.e., determining whether the first data exists, determining the data source, and storing the data to be processed in the data lake), the data to be processed is taken out from the head of the task queue in sequence according to the stacking and popping order. In other embodiments, the first data with which the data processing tasks in the task queue are associated can be directly stored in the storage location of the corresponding data processing task in the task queue.

[0049] Optionally, the aforementioned “for data processing tasks in the task list that do not have a processing order, determining, according to a preset rule, for each data processing task whether corresponding first data exists in the data lake” includes: The order of arranging all data processing tasks that do not have a processing order in the task list is determined comprehensively according to the order of receiving time of the received data processing tasks and the processing urgency of the data processing tasks; According to the arrangement order, for all data processing tasks that do not have a processing order, it is determined in sequence whether corresponding first data exists in the data lake.

[0050] In implementation, in the embodiment of the present application, the degree of urgency corresponds to different levels, and different levels of urgency can correspond to a duration, and the higher the level of urgency, the more urgent the corresponding data processing task, and the longer the corresponding duration; the receiving time is subtracted from the duration corresponding to the corresponding degree of urgency to obtain a new receiving time, and then sorted according to the order of the new receiving time to obtain the arrangement order.

[0051] Optionally, the data processing method further includes the following steps: The data processing task that generates the processed data is deleted from the task list, and a data stream is formed according to the data processing task, the data stream at least including the task ID of the data processing task, the data source corresponding to the data to be processed, and the data warehouse address for storing the processed data of the data processing task; The query instruction is obtained, and according to the task ID contained in the query instruction, the to-be-processed data and the processed data contained in the data processing task corresponding to the task ID are retrieved and fed back.

[0052] During implementation, the user can touch the query button preset on the data processing system display interface, and the data processing system can display a query box for the user to enter keywords. The keywords can be task ID, receipt time, user identity information that triggers the data processing task, etc., to query the data stream corresponding to the data processing tasks in the historical period, and then find the corresponding data source and data warehouse address based on the data stream, so as to query the corresponding data.

[0053] The present application also discloses an efficient data processing system for a billion-level heterogeneous database based on a lake warehouse. Figure 2 ,include: The processing task receiving module 1 is used to receive a data processing task, wherein the data processing task at least includes data to be processed and a specific processing operation; A data retrieval and temporary storage module 2 is used to determine a data source based on the data processing task, retrieve the data to be processed from the data source, and store the data to be processed in a data lake; The data processing and storage module 3 is used to retrieve the data to be processed from the data lake, perform corresponding specific processing operations on the data to be processed to generate processed data, and store the processed data in the data warehouse.

[0054] Optionally, the data retrieval temporary storage module 2 is further used to determine whether there is first data in the current data lake based on the data processing task, where the first data refers to data shared by the data to be processed and all data stored in the current data lake; if the first data exists, a first association relationship is established between the current data processing task and the first data; determine the data source of all data other than the first data in the data to be processed, retrieve all data other than the first data from the data source, and store the retrieved data in the data lake; The data processing storage module 3 is also used to retrieve data corresponding to the data processing task and first data having a first association relationship with the data processing task from the data lake, and use the retrieved data as data to be processed.

[0055] Optionally, the data retrieval temporary storage module 2 is also used to add all data processing tasks whose processing status is unprocessed before the current time to the task list; and is also used to determine whether there is a corresponding associated processing task for each data processing task in the task list whenever a new data processing task is added to the task list; wherein the associated processing task refers to a data processing task that has a second association relationship with the target data processing task among any data processing tasks received before the current time; the target data processing task refers to any data processing task in the task list; two data processing tasks with a second association relationship satisfy: the processed data of one of the data processing tasks contains the same data as the data to be processed of the other data processing task; and among the two data processing tasks with the second association relationship, the data processing task whose processed data contains the same data is called a preceding processing task, and the data processing task whose data to be processed contains the same data is called a subsequent processing task; It is also used to establish a second association relationship between the associated processing task and the target data processing task if there is a corresponding associated processing task for the target data processing task, store the same data as first data in the data lake, and establish a first association relationship between the first data and the subsequent processing task.

[0056] Optionally, the data retrieval temporary storage module 2 is also used to consider that the target data processing task has a corresponding associated processing task if a preset first condition or a second condition is met; wherein the first condition refers to: in the task list, among other data processing tasks except the target data processing task, there exists a data processing task whose task ID is consistent with the pre-order ID and / or post-order ID contained in the target data processing task; the second condition refers to: in any data processing task received before the current time, there exists a data processing task whose corresponding processed data contains the to-be-processed data of the target data processing task; The data retrieval temporary storage module 2 is also used to establish a second association relationship between the associated processing task and the target data processing task if there is a corresponding associated processing task for the target data processing task; if the associated processing task corresponding to the target data processing task is included in the task list, then based on the second association relationship, determine the processing order of all target data processing tasks with associated processing tasks in the task list and their corresponding associated processing tasks; if the associated processing task corresponding to the target data processing task does not exist in the task list, then store the common data between the processed data of the associated processing task and the to-be-processed data of the target data processing task as first data in the data lake, and establish a first association relationship between the first data and the target data processing task.

[0057] The data retrieval and temporary storage module 2 is also used to determine, for data processing tasks in the task list that have a processing order, whether corresponding first data exists in the data lake according to the processing order; for data processing tasks in the task list that do not have a processing order, determine whether corresponding first data exists in the data lake for each data processing task according to preset rules.

[0058] Optionally, the data retrieval and temporary storage module 2 is also used to add the data processing tasks in the task list that have a second association relationship to a preset task queue in sequence according to the corresponding processing order, and to determine in sequence whether the corresponding data processing tasks have corresponding first data in the data lake in accordance with the stacking and popping order of the task queue.

[0059] Optionally, it also includes a data flow generation module, which is used to delete the data processing task that generates processed data from the task list, and form a data flow based on the data processing task, and the data flow includes at least the task ID of the data processing task, the data source corresponding to the data to be processed, and the address of the data warehouse for storing the processed data of the data processing task; it is also used to obtain query instructions, and according to the task ID contained in the query instruction, retrieve and feedback the data to be processed and processed data contained in the data processing task corresponding to the task ID.

[0060] Optionally, the data retrieval temporary storage module 2 is also used to comprehensively determine the arrangement order of all data processing tasks that do not have a processing order in the task list according to the order of receipt time of the data processing tasks and the processing urgency of the data processing tasks; according to the arrangement order, for all data processing tasks that do not have a processing order, determine in sequence whether there is corresponding first data in the data lake.

[0061] An embodiment of the present application also discloses an efficient data processing device for a billion-level heterogeneous database based on a lake warehouse. The efficient data processing device for a billion-level heterogeneous database based on a lake warehouse includes a memory and a processor. The memory stores a computer program that can be loaded by the processor and executes the efficient data processing method for a billion-level heterogeneous database based on a lake warehouse as described above.

[0062] An embodiment of the present application also discloses a computer-readable storage medium, which stores a computer program that can be loaded by a processor and executes the above-mentioned efficient data processing method for a billion-level heterogeneous database based on a lake warehouse. The computer-readable storage medium includes, for example: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0063] It should be noted that, in this document, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0064] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit the protection scope of the application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on these embodiments, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

Claims

1. An efficient data processing method for a billion-level heterogeneous database based on Lake Warehouse, characterized in that: include: Receiving a data processing task, wherein the data processing task at least includes data to be processed and a specific processing operation; Based on the data processing task, determine a data source, retrieve the data to be processed from the data source, and store the data to be processed in a data lake; The data to be processed is retrieved from the data lake, and corresponding specific processing operations are performed on the data to be processed to generate processed data, and the processed data is stored in the data warehouse.

2. The efficient data processing method for a billion-level heterogeneous database based on a lake warehouse according to claim 1 is characterized in that: Each of the data processing tasks also includes a task ID; The step of determining a data source based on the data processing task, retrieving the data to be processed from the data source, and storing the data to be processed in a data lake includes: Based on the data processing task, determining whether there is first data in the current data lake, where the first data refers to data shared by the data to be processed and all data stored in the current data lake; If the first data exists, establishing a first association relationship between the current data processing task and the first data; Determine a data source of all data except the first data in the data to be processed, retrieve all data except the first data from the data source, and store the retrieved data in the data lake; The retrieving the to-be-processed data from the data lake includes: Data corresponding to the data processing task and first data having a first association relationship with the data processing task are retrieved from the data lake, and the retrieved data are used as data to be processed.

3. The efficient data processing method for a billion-level heterogeneous database based on a lake warehouse according to claim 2 is characterized in that: The data processing task includes a processing status, which is unprocessed or processed; and determining whether there is first data in the current data lake based on the data processing task, further comprising: Add all data processing tasks before the current time whose processing status is unprocessed to the task list; Whenever a new data processing task is added to the task list, determine whether there is a corresponding associated processing task for each data processing task in the task list; wherein the associated processing task refers to a data processing task that has a second associated relationship with a target data processing task among any data processing tasks received before the current time; the target data processing task refers to any data processing task in the task list; two data processing tasks that have a second associated relationship satisfy: the processed data of one of the data processing tasks contains the same data as the data to be processed of the other data processing task; and among the two data processing tasks that have a second associated relationship, the data processing task whose processed data contains the same data is called a preceding processing task, and the data processing task whose data to be processed contains the same data is called a subsequent processing task; If the target data processing task has a corresponding associated processing task, a second association relationship is established between the associated processing task and the target data processing task, the same data is stored as the first data in the data lake, and a first association relationship is established between the first data and the subsequent processing task.

4. The efficient data processing method for a billion-level heterogeneous database based on a lake warehouse according to claim 3 is characterized in that: The data processing task further includes a pre-order ID and / or a post-order ID, wherein the pre-order ID refers to the task ID of the data processing task that has a second association relationship with the data processing task to which it belongs and serves as the pre-order processing task in the second association relationship; the post-order ID refers to the task ID of the data processing task that has a second association relationship with the data processing task to which it belongs and serves as the post-order processing task in the second association relationship; The step of determining whether there is a corresponding associated processing task for each data processing task in the task list comprises: If the preset first condition or second condition is met, it is considered that the target data processing task has a corresponding associated processing task; wherein the first condition means: in the task list, among other data processing tasks except the target data processing task, there exists a data processing task whose task ID is consistent with the pre-order ID and / or post-order ID contained in the target data processing task; the second condition means: in any data processing task received before the current time, there exists a data processing task whose corresponding processed data contains the to-be-processed data of the target data processing task; If the target data processing task has a corresponding associated processing task, a second association relationship is established between the associated processing task and the target data processing task, the same data is stored as first data in the data lake, and a first association relationship is established between the first data and the subsequent processing task, including: If the target data processing task has a corresponding associated processing task, a second associated relationship is established between the associated processing task and the target data processing task; if the associated processing task corresponding to the target data processing task is included in the task list, a processing order of all target data processing tasks with associated processing tasks in the task list and their corresponding associated processing tasks is determined based on the second associated relationship; If the associated processing task corresponding to the target data processing task does not exist in the task list, the common data between the processed data of the associated processing task and the to-be-processed data of the target data processing task is stored as first data in the data lake, and a first association relationship is established between the first data and the target data processing task; The determining, based on the data processing task, whether the first data exists in the current data lake includes: For the data processing tasks in the task list that have a processing order, determining in sequence according to the processing order whether the corresponding data processing tasks have corresponding first data in the data lake; For data processing tasks in the task list that do not have a processing order, it is determined for each data processing task whether corresponding first data exists in the data lake according to preset rules.

5. The efficient data processing method for a billion-level heterogeneous database based on a lake warehouse according to claim 4 is characterized in that: For the data processing tasks in the task list that have a processing order, determining in sequence according to the processing order whether the corresponding data processing tasks have corresponding first data in the data lake includes: The data processing tasks in the task list that have the second association relationship are sequentially added to the preset task queue according to the corresponding processing order, and whether the corresponding data processing tasks have corresponding first data in the data lake is sequentially determined according to the stacking and popping order of the task queue.

6. The efficient data processing method for a billion-level heterogeneous database based on a lake warehouse according to claim 4 is characterized in that: The method further comprises: Deleting the data processing task that generates processed data from the task list, and forming a data stream according to the data processing task, wherein the data stream includes at least a task ID of the data processing task, a data source corresponding to the data to be processed, and a data warehouse address for storing the processed data of the data processing task; A query instruction is obtained, and according to a task ID included in the query instruction, the to-be-processed data and the processed data included in the data processing task corresponding to the task ID are retrieved and fed back.

7. The efficient data processing method for a billion-level heterogeneous database based on a lake warehouse according to claim 1 is characterized in that: For the data processing tasks in the task list that do not have a processing order, determining whether corresponding first data exists in the data lake for each data processing task according to a preset rule includes: Comprehensively determining the arrangement order of all data processing tasks that do not have a processing order in the task list according to the order of receiving time of the received data processing tasks and the processing urgency of the data processing tasks; According to the arrangement order, it is sequentially determined whether corresponding first data exists in the data lake for all data processing tasks that do not have a processing order.

8. An efficient data processing system for a billion-level heterogeneous database based on Hucang, characterized in that: include: A processing task receiving module (1) is used to receive a data processing task, wherein the data processing task at least includes data to be processed and a specific processing operation; A data retrieval and temporary storage module (2) is used to determine a data source based on the data processing task, retrieve the data to be processed from the data source, and store the data to be processed in a data lake; The data processing and storage module (3) is used to retrieve the data to be processed from the data lake, perform corresponding specific processing operations on the data to be processed to generate processed data, and store the processed data in the data warehouse.

9. An efficient data processing device for a billion-level heterogeneous database based on Hucang, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and execute any one of the methods according to claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute any one of the methods as claimed in claims 1 to 7.

Images