Table service management method, system, device, and electronic device based on data lake

Abstract

The present application discloses a table service management method, system, and device based on a data lake. The table service management system based on the data lake is completely independent of the data lake, and the working principle of the table service management system is: when it is determined that at least one data table to be managed in the data lake has completed registration, the managed task type of each data table to be managed is determined from the registration description information of each data table to be managed, so that after the timeline of each data table to be managed is pulled, according to the managed task type of each data table to be managed, at least one to-be-executed task of each data table to be managed is pulled from the timeline of each data table to be managed, and these to-be-executed tasks are managed, so that the purpose of having the table service management system manage the table services of a large number of data tables in the data lake can be achieved, thereby effectively avoiding the impact of the table service on the stability of the writing tasks in the data lake.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a table service management method, system, device, electronic device, and computer-readable medium based on a data lake. Background Art

[0002] In the data lake scenario, in order to better improve the reading performance, some table services can be used to process the stored data tables (for example, merging, categorizing, etc.). Table services refer to table-level data operations taken on data tables; and the table services can include table-level data operations such as merging (compaction) and categorizing (clustering).

[0003] However, when the data lake has a large amount of data, the table service usually has higher requirements for resources, which will affect the stability of the write task. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides a table service management method, system, device, electronic device, and computer-readable medium based on a data lake, which can effectively avoid the impact of table services on the stability of write tasks.

[0005] In order to achieve the above purpose, the technical solutions provided by the embodiments of the present application are as follows:

[0006] The embodiment of the present application provides a table service management method based on a data lake, which is applied to a table service management system based on a data lake. The method includes:

[0007] When it is determined that at least one data table to be managed has completed registration, determining the hosting task type of each data table to be managed from the registration description information of each data table to be managed;

[0008] After the timeline of each of the data tables to be managed is pulled, at least one to-be-executed task of each of the data tables to be managed is pulled from the timeline of each of the data tables to be managed according to the managed task type of each of the data tables to be managed;

[0009] At least one to-be-executed task of each of the to-be-managed data tables is managed.

[0010] In a possible implementation manner, managing at least one to-be-executed task of each of the to-be-managed data tables includes:

[0011] Execute at least one task to be executed of each of the data tables to be managed.

[0012] In a possible implementation, the number of the data tables to be managed is N; the at least one task to be executed of the nth data table to be managed includes M tasks to be executed; wherein n is a positive integer, n≤N, and N is a positive integer;

[0013] The method further comprises:

[0014] Determine, from the registration description information of the nth data table to be managed, the execution device corresponding to the mth task to be executed of the nth data table to be managed; wherein m is a positive integer, m≤M, and M is a positive integer;

[0015] The managing at least one task to be executed of each of the data tables to be managed includes:

[0016] The mth task to be executed in the nth data table to be managed is sent to the execution device corresponding to the mth task to be executed, so that the execution device corresponding to the mth task to be executed executes the mth task to be executed in the nth data table to be managed.

[0017] In a possible implementation manner, after pulling at least one to-be-executed task of each of the data tables to be managed from the timeline of each of the data tables to be managed, the method further includes:

[0018] Storing at least one to-be-executed task of each of the to-be-managed data tables into a preset storage space;

[0019] After scanning at least one candidate task and task description information of the at least one candidate task from the preset storage space, screening at least one target task that meets a preset execution condition from the at least one candidate task according to the task description information of the at least one candidate task;

[0020] The managing at least one task to be executed of each of the data tables to be managed includes:

[0021] The at least one target task is managed.

[0022] In a possible implementation manner, the number of the data tables to be managed is N;

[0023] The method further comprises:

[0024] After obtaining a registration request for the nth data table to be managed, register the nth data table to be managed according to the registration description information carried in the registration request; wherein n is a positive integer, n≤N, and N is a positive integer.

[0025] In a possible implementation manner, the registration request is triggered when the nth data table to be managed is created;

[0026] or,

[0027] The registration request is triggered when starting a write task for the nth data table to be managed.

[0028] In a possible implementation, the data lake-based table service management system is independent of the data lake; the data lake is used to store the at least one data table to be managed.

[0029] The embodiment of the present application also provides a table service management system based on a data lake, the system comprising an information layer and a service layer;

[0030] The information layer is used to determine the hosting task type of each data table to be managed from the registration description information of each data table to be managed when it is determined that at least one data table to be managed has completed registration;

[0031] The information layer is further used to, after pulling the timeline of each of the data tables to be managed, pull at least one to-be-executed task of each of the data tables to be managed from the timeline of each of the data tables to be managed according to the managed task type of each of the data tables to be managed;

[0032] The service layer is used to manage at least one task to be executed in each of the data tables to be managed.

[0033] In a possible implementation, the service layer is specifically configured to execute at least one task to be executed for each of the data tables to be managed.

[0034] In a possible implementation manner, the number of the data tables to be managed is N;

[0035] The information layer is also used to register the nth data table to be managed according to the registration description information carried in the registration request after obtaining the registration request for the nth data table to be managed; wherein n is a positive integer, n≤N, and N is a positive integer.

[0036] In a possible implementation, the number of the data tables to be managed is N; the at least one task to be executed of the nth data table to be managed includes M tasks to be executed; wherein n is a positive integer, n≤N, and N is a positive integer;

[0037] The service layer is further used to determine the execution device corresponding to the mth task to be executed in the nth data table to be managed from the registration description information of the nth data table to be managed; wherein m is a positive integer, m≤M, and M is a positive integer;

[0038] The service layer is specifically used to send the mth task to be executed in the nth data table to be managed to the execution device corresponding to the mth task to be executed, so that the execution device corresponding to the mth task to be executed executes the mth task to be executed in the nth data table to be managed.

[0039] In a possible implementation manner, the information layer is further used to store at least one to-be-executed task of each of the to-be-managed data tables into a preset storage space;

[0040] The service layer is further configured to, after scanning at least one candidate task and task description information of the at least one candidate task from the preset storage space, screen out at least one target task that meets a preset execution condition from the at least one candidate task according to the task description information of the at least one candidate task;

[0041] The service layer is specifically used to manage the at least one target task.

[0042] In a possible implementation, the system further includes a storage layer; the storage layer is used to store metadata of the at least one data table to be managed.

[0043] In a possible implementation, the system is independent of a data lake; the data lake is used to store the at least one data table to be managed.

[0044] The embodiment of the present application also provides a table service management device based on a data lake, including:

[0045] A first determining unit is used to determine the hosting task type of each of the data tables to be managed from the registration description information of each of the data tables to be managed when it is determined that at least one of the data tables to be managed has completed registration;

[0046] A task pulling unit is used to pull at least one to-be-executed task of each data table to be managed from the timeline of each data table to be managed according to the managed task type of each data table to be managed after pulling the timeline of each data table to be managed;

[0047] The task execution unit is used to manage at least one task to be executed of each of the data tables to be managed.

[0048] An embodiment of the present application further provides an electronic device, the device comprising: a processor and a memory;

[0049] The memory is used to store instructions or computer programs;

[0050] The processor is used to execute the instructions or computer programs in the memory so that the electronic device executes any implementation of the table service management method based on the data lake provided in the embodiments of the present application.

[0051] An embodiment of the present application also provides a computer-readable medium, in which instructions or computer programs are stored. When the instructions or computer programs are executed on a device, the device executes any implementation of the table service management method based on the data lake provided in the embodiment of the present application.

[0052] Compared with the prior art, the embodiments of the present application have at least the following advantages:

[0053] The embodiment of the present application provides a table service management system based on a data lake, and the working principle of the table service management system is as follows: when determining that at least one data table to be managed in the data lake has completed registration, the managed task type of each data table to be managed is determined from the registration description information of each data table to be managed, so that after the timeline of each data table to be managed is pulled, at least one to-be-executed task of each data table to be managed is pulled from the timeline of each data table to be managed according to the managed task type of each data table to be managed, and at least one to-be-executed task of each data table to be managed is managed (for example, directly executing these to-be-executed tasks, or using some engines to complete these to-be-executed tasks, etc.), so that the purpose of the table service management system to manage the table services of a large number of data tables in the data lake can be achieved. Among them, because the table service management system is completely independent of the data lake, when the table service management system provides table services to a large number of data tables in the data lake, these table services will not occupy the resource data of the data lake, so that the influence of the table service on the stability of the writing task in the data lake can be effectively avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 A flowchart of a table service management method based on a data lake provided in an embodiment of the present application;

[0056] Figure 2 A schematic diagram of a table service hosting process provided in an embodiment of the present application;

[0057] Figure 3A schematic diagram of the structure of a service management system based on a data lake provided in an embodiment of the present application;

[0058] Figure 4 A schematic diagram of the structure of another table service management system based on a data lake provided in an embodiment of the present application;

[0059] Figure 5 A schematic diagram of the structure of a table service management device based on a data lake provided in an embodiment of the present application;

[0060] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] The inventors found in their research on table services that in order to avoid the impact of table services on the stability of write tasks, an additional application (Application, APP) can be started for each data table to perform table service scheduling (for example, scheduling compaction or clustering tasks); however, this solution has the following defects:

[0062] ① High cost of use. That is, because the user needs to manually configure an additional offline scheduling task for each data table, the user needs to maintain two tasks for the data table, which results in a relatively high cost of use.

[0063] ② High management cost. That is, each data table is configured with a table service scheduling task, which easily leads to the task manager having to manage too many overly scattered tasks, resulting in high task management costs.

[0064] ③ Unable to converge. That is, with the substantial increase in data tables, the above method will lead to a substantial increase in user operation and maintenance costs, thereby significantly increasing task response pressure.

[0065] Based on the above findings, in order to solve the technical problems shown in the background technology part, the embodiment of the present application provides a table service management system based on a data lake, and the working principle of the table service management system is as follows: when it is determined that at least one data table to be managed in the data lake has completed registration, the type of the managed task of each data table to be managed is determined from the registration description information of each data table to be managed, so that after the timeline of each data table to be managed is pulled, according to the managed task type of each data table to be managed, at least one task to be executed of each data table to be managed is pulled from the timeline of each data table to be managed, and at least one task to be executed of each data table to be managed is managed (for example, directly executing these tasks to be executed, or using some engines to complete these tasks to be executed, etc.), so that the purpose of the table service management system to manage the table services of a large number of data tables in the data lake can be achieved. Among them, because the table service management system is completely independent of the data lake, when the table service management system provides table services to a large number of data tables in the data lake, these table services will not occupy the resource data of the data lake, so that the influence of the table service on the stability of the writing task in the data lake can be effectively avoided.

[0066] In addition, the data lake-based table service management system provided in the embodiment of the present application can provide table services for multiple data tables in the data lake, so that the table service management system can overcome the defects of the above solution, thereby better improving the table service performance for a large number of data tables in the data lake.

[0067] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0068] In order to facilitate the understanding of the table service management system based on the data lake provided in the embodiment of the present application, Figure 1 The working principle of the table service management system (that is, the table service management method based on the data lake) is explained and illustrated. Figure 1 A flowchart of a table service management method based on a data lake is provided in an embodiment of the present application.

[0069] like Figure 1 As shown, the table service management method based on the data lake provided in the embodiment of the present application can be applied to the table service management system based on the data lake, and the table service management method can specifically include S1-S3:

[0070] S1: When it is determined that at least one data table to be managed has completed registration, the trusteeship task type of each data table to be managed is determined from the registration description information of each data table to be managed.

[0071] The data table to be managed is used to represent the data table in the data lake that has been registered in the table service management system based on the data lake; and the embodiment of the present application does not limit the number of data tables to be managed, for example, it can be N. N is a positive integer. It can be seen that the data lake can be used to store the above-mentioned "at least one data table to be managed". In addition, the table service management system based on the data lake provided in the embodiment of the present application is completely independent of the data lake.

[0072] The embodiments of the present application do not limit the registration method of the nth data table to be managed. For example, it can be specifically as follows: after obtaining a registration request for the nth data table to be managed, the nth data table to be managed is registered according to the registration description information carried in the registration request, so that the table service management system based on the data lake records the registration description information of the nth data table to be managed (for example, the registration description information of the nth data table to be managed can be stored in the storage layer of the table service management system), so that the table service management system can subsequently manage some table services (especially, some table-level data operations that are designated to be hosted, etc.) for the nth data table to be managed. Among them, the registration request is used to request that the nth data table to be managed be registered on the table service management system based on the data lake. n is a positive integer, n≤N, and N is a positive integer.

[0073] In addition, the embodiment of the present application does not limit the triggering method of the above-mentioned "registration request", for example, it can be automatically triggered by a pre-generated script. For another example, it can be manually triggered by a user through a user interface.

[0074] In addition, the embodiments of the present application do not limit the triggering time of the above-mentioned "registration request". For example, in some application scenarios, the registration request for the nth data table to be managed can be triggered when the write task for the nth data table to be managed is started. For another example, in order to further improve the user experience, the registration request for the nth data table to be managed can be triggered when the nth data table to be managed is created.

[0075] The registration description information of the nth data table to be managed is used to describe the registration-related information of the nth data table to be managed on the table service management system based on the data lake; and the embodiment of the present application does not limit the registration description information of the nth data table to be managed, for example, it may include the hosting task description information of the nth data table to be managed.

[0076] The managed task description information of the nth data table to be managed is used to indicate which table services of the nth data table to be managed are managed to the table service management system based on the data lake; and the embodiment of the present application does not limit the managed task description information of the nth data table to be managed, for example, it may include a managed task type (for example, an action type, etc.).

[0077] The hosting task type of the nth data table to be managed is used to indicate which table service of the nth data table to be managed should be hosted to the table service management system based on the data lake; and the embodiment of the present application does not limit the hosting task type of the nth data table to be managed, for example, it may include: Compaction, Clustering, etc.

[0078] Based on the relevant content of S1 above, it can be known that for a table service management system based on a data lake, after the nth data table to be managed has been registered on the table service management system, the table service management system can determine the type of managed task of the nth data table to be managed from the registration description information of the nth data table to be managed, so that the table service management system can subsequently understand which table service the table service management system can manage for the nth data table to be managed instead of the data lake based on the type of managed task of the nth data table to be managed. Where n is a positive integer, n≤N, and N is a positive integer.

[0079] S2: After the timeline of each data table to be managed is pulled, at least one to-be-executed task of each data table to be managed is pulled from the timeline of each data table to be managed according to the managed task type of each data table to be managed.

[0080] The timeline of the nth data table to be managed is used to maintain all operations on the nth data table to be managed at different time points. n is a positive integer, n≤N, and N is a positive integer.

[0081] In addition, the embodiment of the present application does not limit the method for pulling the timeline of the nth data table to be managed. For example, the timeline of the nth data table to be managed can be pulled from the metadata system of the nth data table to be managed. The metadata system is used to perform metadata management on each data table in the data lake including the nth data table to be managed.

[0082] In fact, because the timeline of a data table is constantly changing, in order to further improve the table service effect, the timeline of the data table can be pulled regularly. Based on this, the embodiment of the present application also provides another way to pull the timeline of the nth data table to be managed, which can be specifically: pull the timeline of the nth data table to be managed from the metadata system of the nth data table to be managed according to the preset pulling frequency. Among them, the preset pulling frequency can be pre-set, and the embodiment of the present application does not limit the preset pulling frequency. For example, it can be pulled once every first time interval.

[0083] The at least one to-be-executed task of the nth data table to be managed refers to a table-level data operation under a managed task type belonging to the nth data table to be managed that is screened from the timeline of the nth data table to be managed.

[0084] In addition, the embodiment of the present application does not limit the number of tasks in the above-mentioned "at least one task to be executed in the nth data table to be managed", for example, it can be M.

[0085] In addition, the embodiments of the present application do not limit the timing of pulling the above-mentioned "at least one task to be executed of the nth data table to be managed". For example, in the scenario of periodically pulling the timeline of a data table, at least one task to be executed of the nth data table to be managed can be pulled from the timeline of the nth data table to be managed immediately after each timeline of the nth data table to be managed is pulled.

[0086] Based on the relevant content of S2 above, it can be known that for a table service management system based on a data lake, if the nth data table to be managed has been registered on the table service management system, the table service management system can periodically pull the timeline of the nth data table to be managed (such as Figure 2 As shown), and filter out the various table-level data operations under the managed task type belonging to the nth data table to be managed from the pulled timeline as the various tasks to be executed for the nth data table to be managed, so that the table service management system can obtain the managed task for the nth data table to be managed from the timeline of the nth data table to be managed, thereby achieving the purpose of the table service management system hosting all table services under the managed task type for the nth data table to be managed. Where n is a positive integer, n≤N, and N is a positive integer.

[0087] S3: managing at least one task to be executed in each data table to be managed.

[0088] In the embodiment of the present application, for a table service management system based on a data lake, after the table service management system obtains at least one task to be executed of the nth data table to be managed, the table service management system can manage these tasks to be executed, which can effectively reduce the pressure of the table service of the data lake, thereby effectively reducing the impact of these table services on the stability of the writing tasks in the data lake. Wherein, n is a positive integer, n≤N, and N is a positive integer.

[0089] In addition, the embodiments of the present application do not limit the management method of the table service management system based on the data lake for the tasks to be executed. For example, the table service management system can determine when to execute the task to be executed, how to execute the task to be executed (for example, it can be executed directly by the table service management system itself, or it can be executed with the help of some external devices, etc.), and what information of the task to be executed is paid attention to and stored (for example, priority information, execution status information, etc.).

[0090] To better understand how the table service management system based on the data lake manages tasks to be executed, the following uses three execution methods of tasks to be executed as examples for explanation.

[0091] Method 1: The table service management system based on the data lake executes the tasks to be executed.

[0092] Based on the method 1, it can be known that S3 may specifically include: executing at least one task to be executed of each data table to be managed.

[0093] It can be seen that for the table service management system based on the data lake, after the table service management system obtains at least one task to be executed for the nth data table to be managed, the table service management system can execute these tasks by itself, so that the table service management system can complete some table services for the nth data table to be managed instead of the data lake, thereby effectively reducing the impact of these table services on the stability of the writing tasks in the data lake. Where n is a positive integer, n≤N, and N is a positive integer.

[0094] Method 2: The table service management system based on the data lake uses some additional equipment (for example, Figure 2 The execution devices 1 to G shown in the figure are used to execute the tasks to be executed. Wherein, G is a positive integer, and G represents the number of execution devices that can be used by the table service management system.

[0095] For method 2, in order to further improve the table service effect, the table service management system based on the data lake can use multiple engines to complete the managed table service for a data table. Based on this, the embodiment of the present application also provides the execution method of the above-mentioned "at least one task to be executed for the nth data table to be managed", which can specifically include steps 11-12:

[0096] Step 11: Determine the execution device corresponding to the mth task to be executed in the nth data table to be managed from the registration description information of the nth data table to be managed, where m is a positive integer, m≤M, M is a positive integer, and M represents the number of tasks to be executed in the nth data table to be managed.

[0097] In fact, the registration description information of the nth data table to be managed may include not only the managed task description information of the nth data table to be managed, but also the task execution device description information of the nth data table to be managed, so that the task execution device description information can indicate which execution device (for example, Spark engine or Flink engine, etc.) is used to complete the table-level data operations of each type of the nth data table to be managed.

[0098] Based on this, it can be known that for a table service management system based on a data lake, after the table service management system obtains the mth task to be executed of the nth data table to be managed, the table service management system can determine the execution device corresponding to the type (for example, Compaction, etc.) of the mth task to be executed from the task execution device description information of the nth data table to be managed as the execution device corresponding to the mth task to be executed, so that the table service management system can subsequently complete the mth task to be executed with the help of the execution device. Wherein, m is a positive integer, m≤M, and M is a positive integer.

[0099] Step 12: Send the mth task to be executed in the nth data table to be managed to the execution device corresponding to the mth task to be executed, so that the execution device corresponding to the mth task to be executed executes the mth task to be executed in the nth data table to be managed, where m is a positive integer, m≤M, and M is a positive integer.

[0100] In the embodiment of the present application, for a table service management system based on a data lake, after the table service management system determines the execution device corresponding to the mth task to be executed of the nth data table to be managed, the table service management system can directly submit the mth task to be executed to the execution device corresponding to the mth task to be executed (for example, Figure 2The execution device 1 to the execution device G shown in the figure are a certain execution device, etc.), so that the execution device executes the mth task to be executed of the nth data table to be managed, so that the purpose of using the user-specified device to execute the task to be executed can be achieved, so that the mth task to be executed can be completed while satisfying the user's execution device requirements for the mth task to be executed, so that not only the purpose of the table service management system supporting multiple engines can be achieved, but also the user experience can be effectively improved. Wherein, m is a positive integer, m≤M, and M is a positive integer.

[0101] Based on the relevant contents of steps 11 to 12 above, it can be known that for a table service management system based on a data lake, after the table service management system obtains at least one task to be executed for the nth data table to be managed, the table service management system can submit these tasks to be executed to the execution device specified by the user for execution. This not only enables the table service management system to support multiple engines, but also effectively improves the user experience.

[0102] Mode 3: The table service system can arrange the execution process of all the acquired managed tasks as a whole.

[0103] For method 3, since the real-time requirement of the table service is not high, there can be some time difference between the acquisition process of a hosted task and the execution process of the hosted task. Therefore, the table service system can first store a large number of hosted tasks and then uniformly arrange the execution process of these hosted tasks.

[0104] Based on this, the embodiment of the present application further provides an execution method of the above-mentioned "at least one task to be executed for each data table to be managed", which may specifically include steps 21 to 23:

[0105] Step 21: storing at least one to-be-executed task of each to-be-managed data table into a preset storage space.

[0106] The preset storage space refers to the space used for storage in the table service management system based on the data lake; and the present application embodiment does not limit the preset storage space, for example, it can be the storage layer shown below. It should be noted that the relevant content of the storage layer is shown below.

[0107] Based on the relevant content of step 21, it can be known that for a table service management system based on a data lake, after the table service management system obtains at least one pending task of the nth data table to be managed, the table service management system may not immediately execute these pending tasks, but store these pending tasks in the preset storage space of the table service management system, so that the table service management system can subsequently perform unified scheduling for all tasks stored in the preset storage space, which is conducive to improving the effect of table service implementation. Wherein, n is a positive integer, n≤N, and N is a positive integer.

[0108] Step 22: After scanning at least one candidate task and the task description information of the at least one candidate task from the preset storage space, at least one target task that meets the preset execution condition is screened out from the at least one candidate task according to the task description information of the at least one candidate task.

[0109] The candidate task is used to represent the tasks that have been stored in the preset storage space. For example, after storing at least one to-be-executed task of the nth to-be-managed data table in the preset storage space, the above-mentioned "at least one candidate task" may include at least one to-be-executed task of the nth to-be-managed data table, where n is a positive integer, n≤N, and N is a positive integer.

[0110] The task description information of the tth candidate task is used to describe the relevant execution information of the tth candidate task (for example, execution status information, execution priority information, execution resource consumption information, etc.), so that the task description information can indicate whether the tth candidate task needs to be executed. Among them, the execution status information is used to describe the current state of the tth candidate task (for example, not executed, being executed, how many times it has been retried due to execution failure, etc.); the execution priority information is used to describe the execution priority of the tth candidate task (for example, the priority of compaction is higher than the priority of clustering, etc.). The execution resource consumption information is used to describe which resources (for example, computing resources, memory resources, etc.) need to be consumed when executing the tth candidate task. t is a positive integer, t≤T, T is a positive integer, and T represents the number of candidate tasks (that is, the number of tasks stored in the preset storage space).

[0111] In addition, the embodiments of the present application do not limit the method for obtaining the task description information of the tth candidate task. For example, the execution status information of the tth candidate task can be obtained by the table service management system based on the data lake by real-time monitoring the execution status of each task; the execution priority information of the tth candidate task can be set during the registration process of its corresponding data table; and the execution resource consumption information of the tth candidate task can be set during the registration process of its corresponding data table, and can also be determined during the pulling process of the tth candidate task.

[0112] In addition, the embodiment of the present application does not limit the scanning timing of the table service management system based on the data lake for the preset storage space. For example, the table service management system can scan the preset storage space according to the preset scanning frequency. The preset scanning frequency can be pre-set; and the embodiment of the present application does not limit the preset scanning frequency. For example, it can be scanned once every second time interval.

[0113] The preset execution condition is used to indicate the condition that needs to be satisfied by the task to be executed; and the preset execution condition can be set in advance, especially the preset execution condition can be set with reference to the task scheduling method (for example, First Input First Output (FIFO) scheduling, priority-based scheduling, etc.), the available resources of the table service management system based on the data lake, priority information, etc.

[0114] The target task refers to a candidate task that meets the preset execution conditions, so that the target task can represent the task that needs to be executed.

[0115] Based on the relevant content of the above step 22, it can be known that for a table service management system based on a data lake, the table service management system can periodically scan the tasks (instant) stored in the preset storage space to obtain at least one candidate task and the task description information of these candidate tasks, so that these task description information can indicate which candidate tasks need to be executed; then, the table service management system can refer to the task description information of these candidate tasks, and filter out the target tasks that meet the preset execution conditions from these candidate tasks, so that these target tasks represent the tasks that need to be executed by the table service management system.

[0116] Step 23: Manage at least one target task.

[0117] In an embodiment of the present application, for a table service management system based on a data lake, after the table service management system finds at least one target task from a preset storage space, the table service management system can manage the execution of these target tasks (for example, the table service management system itself can directly execute these target tasks, or some external devices can be used to execute these target tasks, etc.), which can effectively reduce the pressure on the table services of the data lake, thereby effectively reducing the impact of these table services on the stability of write tasks in the data lake.

[0118] In addition, the embodiments of the present application do not limit the management method of the target task based on the table service management system of the data lake. For example, the table service management system can determine when to execute the target task, how to execute the target task (for example, it can be executed directly by the table service management system itself, or it can be executed with the help of some external devices, etc.), and what information of the target task is paid attention to and stored (for example, priority information, execution status information, etc.).

[0119] In addition, the management method of the table service management system based on the data lake for the target task is similar to the management method for the data table to be managed shown in the above method 1 and method 2. It can be seen that the table service management system can execute at least one target task in a certain way.

[0120] It should be noted that the embodiments of the present application do not limit the execution method of the qth target task. For example, the qth target task can be directly executed by a table service management system based on a data lake. For another example, the execution process of the qth target task can be specifically as follows: after the table service management system obtains the execution device corresponding to the qth target task, the table service management system submits the qth target task to the execution device so that the execution device executes the qth target task. Wherein, q is a positive integer, q≤Q, Q is a positive integer, and Q represents the number of target tasks.

[0121] It should also be noted that the above-mentioned "execution device corresponding to the qth target task" is similar to the above-mentioned "execution device corresponding to the mth task to be executed in the nth data table to be managed", and for the sake of brevity, it will not be repeated here.

[0122] Based on the relevant contents of steps 21 to 23 above, it can be known that for a table service management system based on a data lake, after the table service management system obtains at least one task to be executed for each data table to be managed, the table service management system may first store these tasks to be executed in a preset storage space; then, the table service management system periodically scans the preset storage space (for example, Figure 2The scanning method shown in the figure) submits these tasks to be executed in a certain order to the execution device specified during registration, so that these tasks can be executed better, thereby improving the table service effect.

[0123] Based on the relevant contents of S1 to S3 above, it can be known that for the table service management system based on the data lake provided in the embodiment of the present application, when it is determined that at least one data table to be managed in the data lake has completed registration, the type of the managed task of each data table to be managed is determined from the registration description information of each data table to be managed, so that after the timeline of each data table to be managed is pulled, at least one task to be executed of each data table to be managed is pulled from the timeline of each data table to be managed according to the type of the managed task of each data table to be managed, and at least one task to be executed of each data table to be managed is managed (for example, directly executing these tasks to be executed, or using some engines to complete these tasks to be executed, etc.), so that the purpose of the table service management system to manage the table services of a large number of data tables in the data lake can be achieved. Among them, because the table service management system is completely independent of the data lake, when the table service management system provides table services to a large number of data tables in the data lake, these table services will not occupy the resource data of the data lake, so that the influence of the table service on the stability of the writing task in the data lake can be effectively avoided.

[0124] In order to further understand the table service management system based on the data lake provided by the embodiment of the present application, Figure 3-Figure 4 The structure of the table service management system is described. Figure 3 A schematic diagram of the structure of a table service management system based on a data lake provided in an embodiment of the present application; Figure 4 A schematic diagram of another table service management system based on a data lake provided in an embodiment of the present application. It should be noted that the technical details of the table service management system can be found in the above-mentioned table service management method based on a data lake. For the sake of brevity, it will not be repeated here.

[0125] In a possible implementation, the data lake-based table service management system 300 provided in the embodiment of the present application can be used to execute any implementation of the data lake-based table service management method provided in the embodiment of the present application.

[0126] In one possible implementation, Figure 3 As shown, the data lake-based table service management system 300 provided in the embodiment of the present application may include an information layer 301 and a service layer 302;

[0127] The information layer 301 is used to determine the hosting task type of each data table to be managed from the registration description information of each data table to be managed when it is determined that at least one data table to be managed has completed registration;

[0128] The information layer 301 is further used to, after pulling the timeline of each of the data tables to be managed, pull at least one to-be-executed task of each of the data tables to be managed from the timeline of each of the data tables to be managed according to the managed task type of each of the data tables to be managed;

[0129] The service layer 302 is used to manage at least one task to be executed in each of the data tables to be managed.

[0130] In a possible implementation, the service layer 302 is specifically configured to execute at least one task to be executed for each of the data tables to be managed.

[0131] In one possible implementation, the number of data tables to be managed is N; the timeline of the nth data table to be managed is pulled by the information layer 301 from the metadata system of the nth data table to be managed; or, the timeline of the nth data table to be managed is pulled by the information layer 301 from the metadata system of the nth data table to be managed according to a preset pulling frequency; wherein n is a positive integer, n≤N, and N is a positive integer.

[0132] In a possible implementation, the data lake-based table service management system 300 further includes a storage layer 303 (such as Figure 4 shown);

[0133] The number of the data tables to be managed is N;

[0134] The information layer 301 is also used to, after obtaining a registration request for the nth data table to be managed, perform registration processing on the nth data table to be managed according to the registration description information carried in the registration request, so as to store the registration description information in the storage layer 303, so that the information layer 301 can subsequently obtain the registration description information from the storage layer 303; wherein n is a positive integer, n≤N, and N is a positive integer.

[0135] In a possible implementation manner, the registration request for the nth data table to be managed is triggered when the nth data table to be managed is created.

[0136] In a possible implementation manner, the registration request for the nth data table to be managed is triggered when a write task for the nth data table to be managed is started.

[0137] In a possible implementation manner, the number of the data tables to be managed is N; the at least one task to be executed of the nth data table to be managed includes M tasks to be executed; wherein n is a positive integer, n≤N, and N is a positive integer;

[0138] The service layer 302 is further used to determine the execution device corresponding to the mth task to be executed in the nth data table to be managed from the registration description information of the nth data table to be managed; wherein m is a positive integer, m≤M, and M is a positive integer;

[0139] The service layer 302 is specifically used to send the mth task to be executed in the nth data table to be managed to the execution device corresponding to the mth task to be executed, so that the execution device corresponding to the mth task to be executed executes the mth task to be executed in the nth data table to be managed.

[0140] In a possible implementation manner, the information layer 301 is further used to store at least one to-be-executed task of each of the to-be-managed data tables into a preset storage space (for example, Figure 4 Storage layer 303 shown);

[0141] The service layer 302 is further configured to, after scanning at least one candidate task and task description information of the at least one candidate task from the preset storage space, screen out at least one target task that meets a preset execution condition from the at least one candidate task according to the task description information of the at least one candidate task;

[0142] The service layer 302 is specifically used to manage the at least one target task (for example, directly execute these target tasks, or execute these target tasks with the help of execution devices corresponding to each target task).

[0143] In one possible implementation, Figure 4 As shown, the data lake-based table service management system 300 also includes a storage layer 303; the storage layer is used to store metadata of the at least one data table to be managed.

[0144] The metadata of the nth data table to be managed is used to describe the relevant information of the nth data table to be managed; and the embodiment of the present application does not limit the metadata of the nth data table to be managed, for example, it may include the table-level metadata of the nth data table to be managed and the task execution record of the nth data table to be managed. n is a positive integer, n≤N, and N is a positive integer.

[0145] The above-mentioned "table-level metadata of the nth data table to be managed" may include one or more of the creator (owner) of the nth data table to be managed, the execution queue of the nth data table to be managed, the managed task type of the nth data table to be managed, the task execution priority of the nth data table to be managed, the task execution engine of the nth data table to be managed, etc.

[0146] The above-mentioned “task execution record of the nth data table to be managed” includes the execution record information (for example, execution time, execution status, number of retries, etc.) of each managed task of the nth data table to be managed.

[0147] In a possible implementation, the storage layer 303 may store the table-level metadata of the nth data table to be managed and the task execution record of the nth data table to be managed separately. The execution record information of each managed task of the nth data table to be managed may be stored separately, and the execution record information of each managed task of the nth data table to be managed and the table-level metadata of the nth data table to be managed may be associated with the data table identifier (Identity document, ID) of the nth data table to be managed.

[0148] In a possible implementation, the storage layer 303 may be implemented using a pluggable storage device.

[0149] In a possible implementation, the storage layer 303 may use a distributed file system and / or key-value pairs to store metadata of the at least one data table to be managed, so that the data lake-based table service management system 300 supports multiple metadata storage methods.

[0150] In a possible implementation, the service layer 302 includes a scheduling module (Scheduler Service); the number of the data tables to be managed is N; the at least one task to be executed of the nth data table to be managed includes M tasks to be executed; wherein n is a positive integer, n≤N, and N is a positive integer;

[0151] The scheduling module is used to determine the execution device corresponding to the mth task to be executed in the nth data table to be managed from the registration description information of the nth data table to be managed; wherein m is a positive integer, m≤M, and M is a positive integer;

[0152] The scheduling module is also used to send the mth task to be executed in the nth data table to be managed to the execution device corresponding to the mth task to be executed, so that the execution device corresponding to the mth task to be executed executes the mth task to be executed in the nth data table to be managed.

[0153] In a possible implementation manner, the scheduling module is further configured to, after scanning at least one candidate task and task description information of the at least one candidate task from the preset storage space, screen out at least one target task that meets a preset execution condition from the at least one candidate task according to the task description information of the at least one candidate task;

[0154] The scheduling module is further used to manage the at least one target task (for example, directly execute the target tasks, or execute the target tasks with the help of execution devices corresponding to each target task).

[0155] In a possible implementation manner, the scheduling module is further configured to scan the preset storage space according to a preset scanning frequency.

[0156] Based on the relevant content of the above-mentioned scheduling module, it can be known that for the scheduling module in the service layer 302, the scheduling module is responsible for periodically scanning a large number of candidate tasks stored in the storage layer 303, and screening out some target tasks that need to be executed from these candidate tasks; then, the scheduling module can schedule these target tasks through a preset scheduling method (for example, FIFO scheduling, Priority-based scheduling, etc.); finally, the scheduling module submits these target tasks to the execution devices corresponding to each target task respectively, so that these target tasks can be completed with the help of these execution devices.

[0157] In a possible implementation manner, the service layer may further include at least one of a monitoring module (Monitor Service) and a retry module (Retry Service);

[0158] The monitoring module is used to obtain the execution status of each candidate task in real time and update the execution status to the storage layer 303;

[0159] The monitoring module is further used to provide these execution states to the user according to a preset metric, so that the user can learn the execution state of each candidate task from these execution states;

[0160] The retry module is used to scan the failed tasks in the storage layer 303 according to a preset restart frequency, and resubmit all the failed tasks to the scheduling module for scheduling and execution.

[0161] In a possible implementation, the data lake-based table service management system 300 is independent of the data lake; the data lake is used to store the at least one data table to be managed.

[0162] Based on the above-mentioned relevant contents of the table service management system 300 based on the data lake, it can be known that for the table service management system 300 including the information layer 301, the service layer 302 and the storage layer 303, when the information layer 301 in the table service management system 300 determines that at least one data table to be managed has completed registration, the information layer 301 determines the managed task type of each data table to be managed from the registration description information of each data table to be managed, so that after the information layer 301 pulls the timeline of each data table to be managed each time, the information layer 301 pulls at least one to-be-executed task of each data table to be managed from the timeline of each data table to be managed according to the managed task type of each data table to be managed, and stores these to-be-executed tasks in the storage layer 303; finally, the scheduling module in the service layer 302 obtains these to-be-executed tasks by periodically scanning the storage layer 303, and submits these to-be-executed tasks to the execution device specified during registration for execution, so that the purpose of providing table services to a large number of data tables in the data lake by the table service management system 300 can be achieved. Among them, because the table service management system 300 is completely independent of the data lake, when the table service management system 300 provides table services to a large number of data tables in the data lake, these table services will not occupy the resource data of the data lake, thereby effectively avoiding the impact of the table services on the stability of the writing tasks in the data lake.

[0163] In addition, the table service management system 300 based on the data lake can provide table services for all data tables in the data lake, so that the final consistency of table-level data operations can be guaranteed. In addition, the table service management system can execute table services by calling multiple execution devices (for example, Spark engine or Flink engine, etc.), so that the table service management system supports multiple engines. In addition, the table service management system can solve the execution problem of a large number of table-level data operations under the large data scale by adding instances.

[0164] Based on the table service management method based on data lake provided by the above method embodiment, the present application embodiment also provides a table service management device based on data lake, which is explained and illustrated in conjunction with the accompanying drawings. It should be noted that for the technical details of the table service management device, please refer to the relevant content of the table service management method based on data lake.

[0165] See also Figure 5 , which is a structural diagram of a table service management device based on a data lake provided in an embodiment of the present application.

[0166] The table service management device 500 based on the data lake provided in the embodiment of the present application includes:

[0167] A first determining unit 501 is configured to determine, when determining that at least one data table to be managed has completed registration, a trusteeship task type of each data table to be managed from registration description information of each data table to be managed;

[0168] The task pulling unit 502 is used to pull at least one to-be-executed task of each data table to be managed from the timeline of each data table to be managed according to the managed task type of each data table to be managed after pulling the timeline of each data table to be managed;

[0169] The task management unit 503 is used to manage at least one to-be-executed task of each of the to-be-managed data tables.

[0170] In a possible implementation, the task management unit 503 is specifically configured to execute at least one to-be-executed task of each of the to-be-managed data tables.

[0171] In a possible implementation, the number of the data tables to be managed is N; the at least one task to be executed of the nth data table to be managed includes M tasks to be executed; wherein n is a positive integer, n≤N, and N is a positive integer;

[0172] The data lake-based table service management device 500 further includes:

[0173] A second determining unit is used to determine, from the registration description information of the nth data table to be managed, the execution device corresponding to the mth task to be executed in the nth data table to be managed; wherein m is a positive integer, m≤M, and M is a positive integer;

[0174] The task management unit 503 is specifically used to send the mth task to be executed in the nth data table to be managed to the execution device corresponding to the mth task to be executed, so that the execution device corresponding to the mth task to be executed executes the mth task to be executed in the nth data table to be managed.

[0175] In a possible implementation manner, the data lake-based table service management device 500 further includes:

[0176] A task storage unit, used for storing at least one to-be-executed task of each of the to-be-managed data tables into a preset storage space;

[0177] a task scanning unit, configured to, after scanning at least one candidate task and task description information of the at least one candidate task from the preset storage space, screen out at least one target task that meets a preset execution condition from the at least one candidate task according to the task description information of the at least one candidate task;

[0178] The task management unit 503 is specifically configured to manage the at least one target task.

[0179] In a possible implementation manner, the number of the data tables to be managed is N;

[0180] The data lake-based table service management device 500 further includes:

[0181] The data table registration unit is used to register the nth data table to be managed according to the registration description information carried in the registration request after obtaining the registration request for the nth data table to be managed; wherein n is a positive integer, n≤N, and N is a positive integer.

[0182] In a possible implementation manner, the registration request is triggered when the nth data table to be managed is created.

[0183] In a possible implementation manner, the registration request is triggered when a write task for the nth data table to be managed is started.

[0184] In a possible implementation, the data lake-based table service management device 500 is independent of the data lake; the data lake is used to store the at least one data table to be managed.

[0185] Based on the relevant content of the above-mentioned data lake-based table service management device 500, it can be known that for the data lake-based table service management device 500 provided in the embodiment of the present application, when it is determined that at least one data table to be managed in the data lake has completed registration, the managed task type of each data table to be managed is determined from the registration description information of each data table to be managed, so that after the timeline of each data table to be managed is pulled, according to the managed task type of each data table to be managed, at least one to-be-executed task of each data table to be managed is pulled from the timeline of each data table to be managed, and at least one to-be-executed task of each data table to be managed is managed (for example, directly executing these to-be-executed tasks, or completing these to-be-executed tasks with the help of some engines, etc.), so that the purpose of having the table service management device 500 host the table services of a large number of data tables in the data lake can be achieved. Among them, because the table service management device 500 is completely independent of the data lake, when the table service management device 500 provides table services to a large number of data tables in the data lake, these table services will not occupy the resource data of the data lake, thereby effectively avoiding the impact of the table services on the stability of the writing tasks in the data lake.

[0186] In addition, an embodiment of the present application also provides an electronic device, which includes a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device executes any implementation of the table service management method based on the data lake provided in the embodiment of the present application.

[0187] See also Figure 6 , which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0188] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0189] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0190] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0191] The electronic device provided by the embodiment of the present disclosure and the method provided by the above embodiment belong to the same inventive concept, and the technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0192] An embodiment of the present application also provides a computer-readable medium, in which instructions or computer programs are stored. When the instructions or computer programs are executed on a device, the device executes any implementation of the table service management method based on the data lake provided in the embodiment of the present application.

[0193] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0194] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0195] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0196] The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device can execute the method.

[0197] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0198] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0199] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit / module does not, in some cases, constitute a limitation on the unit itself.

[0200] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0201] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0202] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0203] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0204] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0205] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0206] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A table service management method based on a data lake, characterized in that: Applied to a table service management system based on a data lake, the table service management system is independent of the data lake, and the method includes: When it is determined that at least one data table to be managed has completed registration, determining the hosting task type of each data table to be managed from the registration description information of each data table to be managed; the at least one data table to be managed is stored in the data lake; the number of the data tables to be managed is N; After the timeline of each of the data tables to be managed is pulled, at least one to-be-executed task of each of the data tables to be managed is pulled from the timeline of each of the data tables to be managed according to the managed task type of each of the data tables to be managed, the timeline of the nth data table to be managed is used to maintain operations on the nth data table to be managed at different time points, and the at least one to-be-executed task of the nth data table to be managed refers to table-level data operations under the managed task type of the nth data table to be managed that are screened from the timeline of the nth data table to be managed; wherein n is a positive integer, n≤N, and N is a positive integer; Managing at least one task to be executed in each of the data tables to be managed; The method further comprises: After obtaining the registration request for the nth data table to be managed, the nth data table to be managed is registered according to the registration description information carried in the registration request, so as to store the registration description information in the table service management system, wherein the registration description information of the nth data table to be managed includes the hosting task type of the nth data table to be managed, and the hosting task type of the nth data table to be managed is used to indicate the type of table service hosted to the table service management system for executing the nth data table to be managed.

2. The method according to claim 1, characterized in that The managing at least one to-be-executed task of each of the to-be-managed data tables includes: Execute at least one task to be executed of each of the data tables to be managed.

3. The method according to claim 1, characterized in that The number of the data tables to be managed is N; the at least one task to be executed of the nth data table to be managed includes M tasks to be executed; wherein n is a positive integer, n≤N, and N is a positive integer; The method further comprises: Determine, from the registration description information of the nth data table to be managed, the execution device corresponding to the mth task to be executed of the nth data table to be managed; wherein m is a positive integer, m≤M, and M is a positive integer; The managing at least one to-be-executed task of each of the to-be-managed data tables includes: The mth task to be executed in the nth data table to be managed is sent to the execution device corresponding to the mth task to be executed, so that the execution device corresponding to the mth task to be executed executes the mth task to be executed in the nth data table to be managed.

4. The method according to claim 1, characterized in that After pulling at least one to-be-executed task of each of the data tables to be managed from the timeline of each of the data tables to be managed, the method further includes: Storing at least one to-be-executed task of each of the to-be-managed data tables into a preset storage space; After scanning at least one candidate task and task description information of the at least one candidate task from the preset storage space, screening out at least one target task that meets a preset execution condition from the at least one candidate task according to the task description information of the at least one candidate task; The managing at least one to-be-executed task of each of the to-be-managed data tables includes: The at least one target task is managed.

5. The method according to claim 1, characterized in that The registration request is triggered when the nth data table to be managed is created; or, The registration request is triggered when starting a write task for the nth data table to be managed.

6. A table service management system based on a data lake, characterized in that: The table service management system is independent of the data lake, and the system includes an information layer and a service layer; The information layer is used to determine the hosting task type of each data table to be managed from the registration description information of each data table to be managed when it is determined that at least one data table to be managed has completed registration; The at least one data table to be managed is stored in the data lake; the number of the data tables to be managed is N; The information layer is further used to, after pulling the timeline of each of the data tables to be managed, pull at least one to-be-executed task of each of the data tables to be managed from the timeline of each of the data tables to be managed according to the managed task type of each of the data tables to be managed, the timeline of the nth data table to be managed being used to maintain operations on the nth data table to be managed at different time points, the at least one to-be-executed task of the nth data table to be managed being a table-level data operation under the managed task type of the nth data table to be managed selected from the timeline of the nth data table to be managed; wherein n is a positive integer, n≤N, and N is a positive integer; The service layer is used to manage at least one task to be executed of each of the data tables to be managed; The table service management system also includes a storage layer; The information layer is also used to, after obtaining a registration request for the nth data table to be managed, perform registration processing on the nth data table to be managed according to the registration description information carried in the registration request, so as to store the registration description information to the storage layer, the registration description information of the nth data table to be managed includes the hosting task type of the nth data table to be managed, and the hosting task type of the nth data table to be managed is used to indicate the type of table service hosted to the table service management system for execution on the nth data table to be managed.

7. The system according to claim 6, characterized in that The service layer is specifically used to execute at least one task to be executed of each of the data tables to be managed.

8. The system according to claim 6, characterized in that The number of the data tables to be managed is N; the at least one task to be executed of the nth data table to be managed includes M tasks to be executed; wherein n is a positive integer, n≤N, and N is a positive integer; The service layer is further used to determine the execution device corresponding to the mth task to be executed in the nth data table to be managed from the registration description information of the nth data table to be managed; wherein m is a positive integer, m≤M, and M is a positive integer; The service layer is specifically used to send the mth task to be executed in the nth data table to be managed to the execution device corresponding to the mth task to be executed, so that the execution device corresponding to the mth task to be executed executes the mth task to be executed in the nth data table to be managed.

9. The system according to claim 6, characterized in that The information layer is further used to store at least one to-be-executed task of each of the to-be-managed data tables into a preset storage space; The service layer is further configured to, after scanning at least one candidate task and task description information of the at least one candidate task from the preset storage space, screen out at least one target task that meets a preset execution condition from the at least one candidate task according to the task description information of the at least one candidate task; The service layer is specifically used to manage the at least one target task.

10. The system according to any one of claims 6 to 9, characterized in that: The system further includes a storage layer; the storage layer is used to store metadata of the at least one data table to be managed.

11. A table service management device based on a data lake, characterized in that: The table service management device is independent of the data lake, and the device includes: a first determining unit, configured to determine, when determining that at least one data table to be managed has completed registration, a type of hosting task for each data table to be managed from registration description information of each data table to be managed; the at least one data table to be managed is stored in the data lake; the number of the data tables to be managed is N; A task pulling unit is used to pull at least one to-be-executed task of each data table to be managed from the timeline of each data table to be managed according to the managed task type of each data table to be managed after pulling the timeline of each data table to be managed, wherein the timeline of the nth data table to be managed is used to maintain operations on the nth data table to be managed at different time points, and the at least one to-be-executed task of the nth data table to be managed refers to table-level data operations under the managed task type of the nth data table to be managed selected from the timeline of the nth data table to be managed; wherein n is a positive integer, n≤N, and N is a positive integer; A task management unit, used for managing at least one to-be-executed task of each of the to-be-managed data tables; The table service management device also includes: A data table registration unit is used to, after obtaining a registration request for the nth data table to be managed, register the nth data table to be managed according to the registration description information carried in the registration request, so as to store the registration description information in the table service management system, wherein the registration description information of the nth data table to be managed includes the hosting task type of the nth data table to be managed, and the hosting task type of the nth data table to be managed is used to indicate the type of table service hosted to the table service management system for execution on the nth data table to be managed.

12. An electronic device, characterized in that: The device comprises: a processor and a memory; The memory is used to store instructions or computer programs; The processor is used to execute the instructions or computer programs in the memory so that the electronic device executes the method according to any one of claims 1 to 5.

13. A computer readable medium, characterized in that The computer-readable medium stores instructions or computer programs, and when the instructions or computer programs are executed on a device, the device is enabled to execute the method according to any one of claims 1 to 5.

14. A computer program product, characterized in that It comprises a computer program carried on a non-transitory computer-readable medium, the computer program comprising a program code for executing the method according to any one of claims 1 to 5.

Images