Task merging processing method and device, equipment and storage medium

Abstract

The invention relates to a task merging processing method and device, equipment and a storage medium, and the method comprises the steps: obtaining first metadata in a currently running first task, the first task being a data synchronization task; according to the first metadata, detecting whether a second task which can be combined with the first task exists in a to-be-run task or not; when a second task which can be combined with the first task exists, combining the first task with the second task to obtain a third task; and executing a replacement operation on the first task by utilizing the third task. According to the method, the use of the running kernel can be reduced, so that the use of running resources is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a task merging processing method and device, equipment and a storage medium. BACKGROUND

[0002] With the development of technology, it is more and more common to synchronize data tables in a certain system with data tables in other systems, for example, to synchronize data tables in a data management system with data in a data aggregation system.

[0003] In the related art, when synchronizing data tables in a data management system to a data aggregation system, the data aggregation system establishes a data synchronization task according to the data tables to be synchronized, and performs data synchronization on the data tables based on the data synchronization task, so as to synchronize the data tables to the data aggregation system.

[0004] In practice, since the above-mentioned data synchronization task will monitor whether the data tables in the data management system are updated in real time, the data synchronization task will occupy a running kernel for a long time. When there are multiple tasks, these data synchronization tasks will occupy multiple running kernels, and thus occupy too much computing resources. SUMMARY

[0005] The present application provides a task merging processing method, device, equipment and a storage medium, which can reduce the use of running kernels, and thus reduce the use of running resources.

[0006] In a first aspect, the present application provides a task merging processing method, which comprises:

[0007] Obtaining first metadata in a first task currently running, the first task being a data synchronization task;

[0008] According to the first metadata, detecting whether there is a second task capable of being merged with the first task in a task to be run;

[0009] When there is a second task capable of being merged with the first task, merging the first task and the second task to obtain a third task;

[0010] Performing a replacement operation on the first task by using the third task.

[0011] Optionally, the step of detecting whether there is a second task capable of being merged with the first task in a task to be run according to the first metadata comprises:

[0012] Obtaining a preset merging information identification strategy;

[0013] According to the merging information identification strategy, determining first merging information in the first metadata;

[0014] According to the first merging information, it is detected whether there is a second task capable of being merged with the first task in the to-be-run task.

[0015] Optionally, the detecting whether there is a second task capable of being merged with the first task in the to-be-run task according to the first merging information comprises:

[0016] acquiring second metadata in the to-be-run task;

[0017] detecting whether there is a second task capable of being merged with the first task according to whether the second metadata in the to-be-run task comprises the first merging information.

[0018] Optionally, the detecting whether there is a second task capable of being merged with the first task in the to-be-run task according to the first metadata comprises:

[0019] acquiring second metadata in the to-be-run task;

[0020] determining first merging information in the first metadata according to a preset merging information recognition strategy;

[0021] determining second merging information corresponding to the to-be-run task in second metadata in the to-be-run task according to the merging information recognition strategy;

[0022] detecting whether there is a second task capable of being merged with the first task according to whether the second merging information corresponding to the to-be-run task is same as the first merging information.

[0023] Optionally, before the acquiring first metadata in the first task currently running, the method further comprises:

[0024] detecting whether there is a newly generated data synchronization task;

[0025] acquiring first metadata in the first task currently running when it is detected that there is a newly generated data synchronization task;

[0026] the detecting whether there is a second task capable of being merged with the first task in the to-be-run task according to the first metadata comprises:

[0027] determining a newly generated data synchronization task in the to-be-run task;

[0028] detecting whether there is a second task capable of being merged with the first task in the newly generated data synchronization task according to the first metadata.

[0029] Optionally, the detecting whether there is a second task capable of being combined with the first task in the to-be-run task according to the first metadata comprises:

[0030] determining the data synchronization task to be run in the to-be-run task;

[0031] detecting whether there is a second task capable of being combined with the first task in the data synchronization task to be run according to the first metadata.

[0032] Optionally, before the obtaining the first metadata in the currently running first task, the method further comprises:

[0033] obtaining an IP address and a port number of a database to which the to-be-collected managed data table belongs in the data management system;

[0034] collecting metadata of the managed data table according to a preset metadata collection rule to obtain first metadata;

[0035] creating a first task according to the first metadata, the IP address and the port number.

[0036] In a second aspect, the present application provides a task combination processing device, the device comprising:

[0037] an obtaining unit configured to obtain first metadata in a currently running first task, the first task being a data synchronization task;

[0038] a first detecting unit configured to detect whether there is a second task capable of being combined with the first task in a to-be-run task according to the first metadata;

[0039] a combining unit configured to combine the first task and the second task to obtain a third task when there is a second task capable of being combined with the first task;

[0040] a replacing unit configured to perform a replacing operation on the first task by using the third task.

[0041] Optionally, the first detecting unit is configured to:

[0042] obtain a preset combination information identification strategy;

[0043] determine first combination information in the first metadata according to the combination information identification strategy;

[0044] detect whether there is a second task capable of being combined with the first task in the to-be-run task according to the first combination information.

[0045] Optionally, the first detecting unit is configured to:

[0046] acquire second metadata in the to-be-run task;

[0047] detect whether there is a second task capable of being merged with the first task according to whether the second metadata in the to-be-run task comprises the first merging information.

[0048] Optionally, the first detection unit is configured to:

[0049] acquire second metadata in the to-be-run task;

[0050] determine first merging information in the first metadata according to a preset merging information identification strategy;

[0051] determine second merging information corresponding to the to-be-run task in the second metadata in the to-be-run task according to the merging information identification strategy;

[0052] detect whether there is a second task capable of being merged with the first task according to whether the second merging information corresponding to the to-be-run task is same as the first merging information.

[0053] Optionally, the apparatus further comprises a second detection unit configured to:

[0054] detect whether there is a newly generated data synchronization task;

[0055] when it is detected that there is a newly generated data synchronization task, acquire first metadata in a currently running first task;

[0056] the first detection unit is configured to:

[0057] determine the newly generated data synchronization task in the to-be-run task;

[0058] detect whether there is a second task capable of being merged with the first task in the newly generated data synchronization task according to the first metadata.

[0059] Optionally, the first detection unit is configured to:

[0060] determine a to-be-run data synchronization task in the to-be-run task;

[0061] detect whether there is a second task capable of being merged with the first task in the to-be-run data synchronization task according to the first metadata.

[0062] Optionally, the apparatus further comprises an acquisition unit configured to:

[0063] acquire an IP address and a port number of a database to which a to-be-acquired data table in a data management system belongs;

[0064] According to a preset metadata collection rule, metadata of the pipe-in data table is collected to obtain first metadata;

[0065] According to the first metadata, the IP address and the port number, a first task is created.

[0066] In a third aspect, the present application provides a task merging processing device, comprising: at least one communication interface; at least one bus connected with the at least one communication interface; at least one processor connected with the at least one bus; and at least one memory connected with the at least one bus, wherein the processor is configured to:

[0067] Obtain first metadata in a first task currently running, the first task being a data synchronization task;

[0068] According to the first metadata, detect whether there is a second task capable of being merged with the first task in a task to be run;

[0069] When there is a second task capable of being merged with the first task, merge the first task and the second task to obtain a third task;

[0070] Perform a replacement operation on the first task by using the third task.

[0071] In a fourth aspect, the present application provides a computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the above task merging processing method.

[0072] Compared with the prior art, the above technical solution provided by the embodiments of the present application has the following advantages: in the embodiments of the present application, first metadata in a first task currently running is obtained, the first task being a data synchronization task; according to the first metadata, it is detected whether there is a second task capable of being merged with the first task in a task to be run; when there is a second task capable of being merged with the first task, the first task and the second task are merged to obtain a third task; and a replacement operation on the first task is performed by using the third task. It can be seen that, in the present application, it can be detected in a task to be run whether other tasks can be merged with a current data synchronization task, and if so, the other tasks and the current data synchronization task are merged to reduce the number of occupied running kernels, thereby avoiding too many computing resources caused by occupying too many running contents and reducing the use of running resources. BRIEF DESCRIPTION OF DRAWINGS

[0073] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0074] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without any creative effort.

[0075] One or more embodiments are illustrated by way of example in the drawings that are not necessarily drawn to scale, and which will be described in detail herein below. In the drawings, like reference numerals refer to like elements unless otherwise indicated. The drawings are intended to be illustrative, and not limiting of the embodiments.

[0076] Figure 1 A flowchart of a task merging processing method provided by an embodiment of the present application;

[0077] Figure 2 A flowchart of a second task detection method provided by an embodiment of the present application;

[0078] Figure 3 A flowchart of another second task detection method provided by an embodiment of the present application;

[0079] Figure 4 A flowchart of still another second task detection method provided by an embodiment of the present application;

[0080] Figure 5 A flowchart of yet another second task detection method provided by an embodiment of the present application;

[0081] Figure 6 A flowchart of still another data table creation method provided by an embodiment of the present application;

[0082] Figure 7 A flowchart of a task merging processing device provided by an embodiment of the present application;

[0083] Figure 8 A schematic diagram of a task merging processing apparatus provided by an embodiment of the present application. DETAILED DESCRIPTION

[0084] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will combine the drawings in the embodiments of the present application to make a clear and complete description of the technical solutions in the embodiments of the present application. Obviously, the described embodiments are a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without any creative effort fall within the scope of protection of the present application.

[0085] The following disclosure provides many different embodiments, or examples, for implementing different structures of the application. For the purpose of simplifying the present application disclosure, the components and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the application. In addition, the application can refer to different examples and / or letters. Such repetition is for the purpose of simplification and clarity, which itself does not indicate the relationship between the various embodiments and / or settings discussed.

[0086] The application scenario involved in the present application includes a plurality of business systems, a data management system and a data aggregation system. Each business system sends the business data generated by itself to the data management system, so that the data management system receives the business data sent by each business system. After the data management system receives the business data, the data management system sorts these business data, generates a management data table based on this, and stores it in a certain database. At the same time, the data aggregation system synchronizes the data table. In order to realize data table synchronization, the data aggregation system can collect metadata based on the pre-set metadata collection rule, obtain the collected metadata, and then generate a data synchronization task based on the metadata, so as to synchronize the data in the management data table to the data aggregation system based on the data synchronization task, and monitor whether the management data table is updated in real time through the data synchronization task, and if so, synchronize the updated data to the data aggregation system.

[0087] The data aggregation system uses FLINK cluster at the bottom, uses open source FLINK technology to collect and analyze real-time data, and calls the FLINK environment to generate a data synchronization task. Since the FLink cluster can connect to multiple data sources and process data, and the invasion to the original business system is relatively small during real-time data collection, the data aggregation system based on the FLINK cluster can better access the data management system.

[0088] Based on the above application scenario, the data aggregation system in the present application can detect the currently running data synchronization task to detect whether there is another data synchronization task that can be merged with the data synchronization task in the to-be-run task. When there is another data synchronization task that can be merged with the data synchronization task, the data synchronization task and the other data synchronization task are merged to obtain a merged data synchronization task, and then the currently running data synchronization task is set to the merged data synchronization task. Since the technical solution of the present application can merge the data synchronization tasks that can be merged, the use of running kernels can be reduced, and the use of running resources can be reduced.

[0089] In summary, the embodiment of the present application provides a task merging processing method, which can reduce the use of running kernels and reduce the use of running resources, such as Figure 1As shown, the specific steps include:

[0090] Step 101, obtaining first metadata in a first task currently running.

[0091] The first task is a data synchronization task currently running, which can be an unmerged data synchronization task or a merged data synchronization task. As long as the task currently running is a data synchronization task, it is determined as the first task. The metadata in the first task is determined as the first metadata. The first metadata is related data of a data table in the data onboarding system, such as the database name of a database, the IP address of the database, the field in the data table, and the like.

[0092] In this step, the data aggregation system can also run other tasks, therefore, the data aggregation system needs to detect whether the task currently running is a data synchronization task, and when it is a data synchronization task, the task is determined as the first task. Since the data synchronization task must include metadata, the data aggregation system can detect whether the task includes metadata to determine whether the task is a data synchronization task. Or a preset label is set for the data synchronization task, which is used to indicate that the task is a data synchronization task. Or a corresponding relationship between each task and the task category is set in the data aggregation system, therefore, the data aggregation system can determine the category of the task according to the corresponding relationship, and then determine whether the task belongs to the data synchronization category according to the category of the task, when the task belongs to the data synchronization category, the task is determined as the data synchronization task. Of course, other methods can also be used to determine whether the task is a data synchronization task in this step, which is not limited here.

[0093] Then, the data aggregation system can periodically obtain the first metadata in the first task currently running, or periodically detect whether there is a newly generated task, when there is a newly generated task, detect whether the task is a data synchronization task, when the task is a data synchronization task, the task is determined as the first task, and the first metadata in the first task currently running is obtained.

[0094] Step 102, according to the first metadata, detecting whether there is a second task capable of being merged with the first task in the task to be run.

[0095] The second task is also a data synchronization task, and the task and the first task are both for synchronizing the data table of the same database.

[0096] In this step, the data aggregation system detects whether there is a second task that can be merged with the first task by detecting whether the data table monitored by the to-be-run task and the data table monitored by the first task belong to the same database. For example, for each task in the to-be-run task, when it is detected that the data table monitored by the task and the data table monitored by the first task belong to the same database, the task is determined as the second task, and it is determined that there is a second task that can be merged with the first task. When it is detected that the data table monitored by all tasks and the data table monitored by the first task do not belong to the same database, it is determined that there is no second task that can be merged with the first task. Since the metadata includes the library name of the database and the IP address and port number of the database, it can be determined whether two data tables belong to the same database by using the two data.

[0097] In this step, it can be determined whether there is a second task in the to-be-run task, or the to-be-run task can be filtered first, and then it is determined whether there is a second task in the filtered task. Specifically, the data synchronization task is selected from the to-be-run task first, and then it is determined whether there is a second task from the data synchronization task. Or when there is a newly generated task, the new task needs to be selected from the to-be-run task when step 101 is performed, and it is determined whether there is a second task from the new task. Or when there is a newly generated task, the new data synchronization task needs to be selected from the to-be-run task when step 101 is performed, and it is determined whether there is a second task from the new data synchronization task.

[0098] Step 103, when there is a second task that can be merged with the first task, the first task and the second task are merged to obtain a third task.

[0099] In this step, when the data aggregation system detects the second task, the data aggregation system can merge the first task and the second task to obtain a third task.

[0100] It should be noted that the task merging in this step can be simply merging two tasks to obtain one task, or generating a task based on two tasks, as long as one running kernel can run a merged task.

[0101] Step 104, using the third task to perform a replacement operation on the first task.

[0102] In this step, the data aggregation system suspends the use of the first task and starts running the third task, so as to replace the first task with the third task.

[0103] In the embodiment of the present application, the first metadata in the first task currently running is acquired, wherein the first task is a data synchronization task; whether there is a second task capable of being merged with the first task in the task to be run is detected according to the first metadata; when there is a second task capable of being merged with the first task, the first task and the second task are merged to obtain a third task; and the second task is replaced by the third task. It can be seen that the present application can detect whether other tasks capable of being merged with the current data synchronization task exist in the task to be run, and if so, the other tasks and the current data synchronization task are merged to reduce the number of running kernels occupied, thereby avoiding too many computing resources caused by occupying too many running contents and reducing the use of running resources.

[0104] In the embodiment of the present application, a merging information identification strategy is arranged in the data aggregation system. When the data aggregation system needs to identify a second task capable of being merged with a first task, the data aggregation system needs to acquire a pre-stored merging information identification strategy, and identify the first task based on the merging information identification strategy. Therefore, the embodiment of the present application provides a second task detection method which further limits step 102. As shown in the method, Figure 2

[0105] Step 201, acquiring a preset merging information identification strategy.

[0106] In this step, the data aggregation system stores the merging information identification strategy, which includes which data needs to be merged, for example, the library name of the library to which the data table belongs, IP, etc.

[0107] Step 202, determining first merging information in the first metadata according to the merging information identification strategy.

[0108] In this step, when the merging information identification strategy is based on the library name of the library to which the data table belongs for merging, the data aggregation system can find the library name of the library to which the data table belongs in the first metadata, and determine it as the first merging information. When the merging information identification strategy is based on the IP of the library to which the data table belongs for merging, the data aggregation system can find the IP of the library to which the data table belongs in the first metadata, and determine it as the first merging information. Of course, other information can also be used as the first merging information in this step, which is not limited here.

[0109] Step 203, detecting whether there is a second task capable of being merged with the first task in the task to be run according to the first merging information.

[0110] In this step, the data aggregation system determines whether there is a second task capable of being merged with the first task in the task to be run by detecting whether the first merging information is included in the task to be run.

[0111] ​In the embodiment of the present application, the data aggregation system can detect whether there is a second task that can be merged with the first task by whether the to-be-run task includes the first merge information, and the specific steps include:

[0112] Obtaining the second metadata in the to-be-run task; and detecting whether there is a second task that can be merged with the first task according to whether the second metadata in the to-be-run task includes the first merge information.

[0113] In this step, for each to-be-run task, the data aggregation system obtains the metadata in the task, determines the metadata as the second metadata, detects whether the second metadata includes the first merge information, and when the second metadata includes the first merge information, determines that the task can be merged with the first task and takes the task as the second task. When the second metadata of all to-be-run tasks does not include the first merge information, the data aggregation system determines that there is no second task that can be merged with the first task.

[0114] In the embodiment of the present application, the data aggregation system is provided with a merge information identification strategy. When the data aggregation system needs to identify the second task that can be merged with the first task, the data aggregation system needs to obtain the pre-stored merge information identification strategy and identify the first task and the to-be-run task based on the merge information identification strategy. The embodiment of the present application also provides a second task detection method, which further limits step 102, and the method is as shown in Figure 3 The specific steps include:

[0115] Step 301: Obtaining the second metadata in the to-be-run task.

[0116] In this step, for each task in the to-be-run task, the data aggregation system needs to obtain the metadata in each task and determine the metadata as the second metadata.

[0117] Step 302: Determining the first merge information in the first metadata according to the preset merge information identification strategy.

[0118] This step is similar to step 202, and will not be described here.

[0119] Step 303: Determining the second merge information corresponding to the to-be-run task in the second metadata in the to-be-run task according to the merge information identification strategy.

[0120] In this step, for each of the to-be-run tasks, when the merging information identification strategy is to merge based on the library name of the library to which the data table belongs, the data aggregation system can perform a lookup on the second metadata in the task, find the library name of the library to which the data table monitored by the task belongs, and determine the library name as the second merging information. For each of the to-be-run tasks, when the merging information identification strategy is to merge based on the IP of the library to which the data table belongs, the data aggregation system can perform a lookup on the second metadata in the task, find the IP of the library to which the data table monitored by the task belongs, and determine the IP as the second merging information.

[0121] Step 304: According to whether the second merging information corresponding to the to-be-run task is the same as the first merging information, it is detected whether there is a second task that can be merged with the first task.

[0122] In this step, for each of the to-be-run tasks, when the second merging information corresponding to the task is the same as the first merging information, the task is determined as the second task, and it is determined that there is a second task that can be merged with the first task. When the second merging information corresponding to the task is not the same as the first merging information, it is determined that there is no second task that can be merged with the first task.

[0123] In the embodiment of the present application, the data aggregation system can detect the second task in the to-be-run task, and can also first screen the to-be-run task, and then detect the second task in the screened task. Therefore, the embodiment of the present application provides another second task detection method, which is as shown in Figure 4 , and specifically includes:

[0124] Step 401: In the to-be-run task, a to-be-run data synchronization task is determined.

[0125] In this step, if there are various tasks in the data aggregation system, the to-be-run task can include a data synchronization task, or can include other tasks. Since only the data synchronization task can be merged, it is necessary to screen all data synchronization tasks from all to-be-run tasks.

[0126] Step 402: According to the first metadata, it is detected whether there is a second task that can be merged with the first task in the to-be-run data synchronization task.

[0127] In this step, the data aggregation system detects whether there is a second task that can be merged with the first task in all data synchronization tasks according to the first metadata.

[0128] Further, the data aggregation system can also only detect the newly added data synchronization task to determine whether the newly added data synchronization task is a second task that can be merged with the first task. For example, when step 101 is periodically executed, it is detected whether there is a new task in the to-be-run task, and if so, it is detected whether the task is a data synchronization task. If so, it is detected whether the new data synchronization task is a second task that can be merged with the first task. When step 101 is executed when a new data synchronization task is detected, it is detected whether the new data synchronization task is a second task that can be merged with the first task. Therefore, the embodiment of the present application provides a second task detection method, as shown in Figure 5 The method comprises the following steps:

[0129] Step 501, detecting whether there is a newly generated data synchronization task.

[0130] In this step, the data aggregation system can periodically detect whether there is a newly generated task, and if so, continue to detect whether the newly generated task is a data synchronization task.

[0131] Step 502, when a newly generated data synchronization task is detected, acquiring first metadata in a currently running first task.

[0132] In this step, when a newly generated data synchronization task is detected, the data aggregation system can acquire a currently running first task and acquire first metadata in the first task.

[0133] Step 503, determining the newly generated data synchronization task in the to-be-run task.

[0134] In this step, the data aggregation system can determine the newly generated data synchronization task in the to-be-run task.

[0135] Step 504, according to the first metadata, detecting whether there is a second task that can be merged with the first task in the newly generated data synchronization task.

[0136] In this step, the data aggregation system can detect whether there is a second task that can be merged with the first task in the newly generated data synchronization task according to the first metadata.

[0137] In the embodiment of the present application, before step 101 is executed, the first task also needs to be acquired. The sources of the first task acquisition include two kinds, one is that the first task is based on the merging of other tasks, and the other is that the first task is created by the data aggregation system based on the data table in the data management system. For the second kind, the embodiment of the present application provides a data table creation method, as shown in Figure 6 The method comprises the following steps:

[0138] Step 601, in the data management system, the IP address and port number of the database to which the to-be-collected management data table belongs are acquired.

[0139] The data management system is configured to collect data in the business system and store the data in the management data table.

[0140] In this step, when the data management system generates a new management data table, the data management system acquires the IP address and port number of the database to which the management data table belongs, and sends the IP address and port number to the data aggregation system, so that the data aggregation system acquires the IP address and port number of the database to which the to-be-collected management data table belongs.

[0141] Step 602, according to a preset metadata collection rule, metadata of the management data table is collected to obtain first metadata.

[0142] In this step, the data aggregation system stores the metadata collection rule, and when the data aggregation system needs to perform step 401, the data aggregation system acquires the pre-stored metadata collection rule, collects metadata of the management data table in the data management system, and obtains first metadata.

[0143] Further, the metadata collection rule can collect the library name to which the monitored data table belongs, the field in the monitored data table, and the like, to create a data table based on the metadata.

[0144] Step 603, a first task is created according to the first metadata, the IP address, and the port number.

[0145] When the cluster used in this step is a FLINK cluster, the data aggregation system can mobilize the underlying FLINK environment to create a real-time task according to the first metadata, the IP address, and the port number, to obtain the first task.

[0146] In addition, the data aggregation system involved in the embodiments of the present application can also monitor the state of the task in real time, and send the monitored state to the terminal, so that the technical personnel can view the task state on the terminal. At the same time, when the task is in a failed state, a message notification is generated based on the failed task, and is sent to the terminal. When the terminal receives the message notification, the message notification is displayed based on the message notification, so that the technical personnel can process the failed task based on the display content.

[0147] The terminal can also display the data table after synchronization, and perform data operations on the data table, such as data addition operation, data deletion operation, data modification operation, and the like. The technical personnel can add data in the data table through the data addition operation, delete data in the data table through the data deletion operation, and modify data in the data table through the data modification operation.

[0148] Meanwhile, the terminal also displays buttons related to task management, such as a task start button, a task stop button, and a task delete button. A technician can start a task in the data aggregation system through the task start button, stop a task in the data aggregation system through the task stop button, and delete a task in the data aggregation system through the task delete button.

[0149] In the embodiments of the present application, a task merging processing apparatus is also provided, which corresponds to the method embodiments, and specifically as shown in Figure 7 the figure, the apparatus comprises:

[0150] The acquisition unit 701 is configured to acquire first metadata in a first task currently running, the first task being a data synchronization task.

[0151] The first detection unit 702 is configured to detect, according to the first metadata, whether there is a second task capable of being merged with the first task in tasks to be run.

[0152] The merging unit 703 is configured to merge the first task and the second task to obtain a third task when there is a second task capable of being merged with the first task.

[0153] The replacing unit 704 is configured to perform a replacing operation on the first task by using the third task.

[0154] Optionally, the first detection unit 702 is configured to:

[0155] acquire a preset merging information identification strategy;

[0156] determine first merging information in the first metadata according to the merging information identification strategy;

[0157] detect, according to the first merging information, whether there is a second task capable of being merged with the first task in tasks to be run.

[0158] Optionally, the first detection unit 702 is configured to:

[0159] acquire second metadata in tasks to be run;

[0160] detect, according to whether the second metadata in tasks to be run includes the first merging information, whether there is a second task capable of being merged with the first task.

[0161] Optionally, the first detection unit 702 is configured to:

[0162] acquire second metadata in tasks to be run;

[0163] According to the preset merging information identification strategy, first merging information is determined in the first metadata;

[0164] According to the merging information identification strategy, second merging information corresponding to the to-be-run task is determined in second metadata in the to-be-run task;

[0165] According to whether the second merging information corresponding to the to-be-run task is same as the first merging information, it is detected whether there is a second task capable of being merged with the first task.

[0166] Optionally, the apparatus further includes a second detection unit 705, configured to:

[0167] detect whether there is a newly generated data synchronization task;

[0168] when it is detected that there is a newly generated data synchronization task, acquire first metadata in a currently running first task;

[0169] The first detection unit 702 is configured to:

[0170] determine, in to-be-run tasks, a newly generated data synchronization task;

[0171] According to the first metadata, it is detected, in the newly generated data synchronization task, whether there is a second task capable of being merged with the first task.

[0172] Optionally, the first detection unit 702 is configured to:

[0173] determine, in to-be-run tasks, a to-be-run data synchronization task;

[0174] According to the first metadata, it is detected, in the to-be-run data synchronization task, whether there is a second task capable of being merged with the first task.

[0175] Optionally, the apparatus further includes an acquisition unit 706, configured to:

[0176] acquire, in a data management system, an IP address and a port number of a database to which a to-be-acquired managed data table belongs;

[0177] According to a preset metadata acquisition rule, metadata acquisition is performed on the managed data table to obtain first metadata;

[0178] According to the first metadata, the IP address and the port number, a first task is created.

[0179] As Figure 8As shown, the embodiment of the present application provides a task merging processing device, comprising a processor 801, a communication interface 802, a memory 803 and a communication bus 804, wherein the processor 801, the communication interface 802 and the memory 803 complete mutual communication through the communication bus 804,

[0180] The memory 803 is used for storing a computer program.

[0181] In an embodiment of the present application, the processor 801 is used for executing the program stored in the memory 803, and a task merging processing method provided by any one of the foregoing method embodiments is realized, comprising the following steps:

[0182] obtaining first metadata in a first task currently running, wherein the first task is a data synchronization task;

[0183] detecting whether there is a second task capable of being merged with the first task in a task to be run according to the first metadata;

[0184] when there is a second task capable of being merged with the first task, merging the first task and the second task to obtain a third task;

[0185] performing a replacement operation on the first task by using the third task.

[0186] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the steps of the task merging processing method provided by any one of the foregoing method embodiments.

[0187] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0188] Through the description of the foregoing embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software plus a general hardware platform, and of course can also be realized by hardware. Based on such understanding, the foregoing technical solutions essentially or in other words the part that contributes to the related art can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0189] It is to be understood that the terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms "comprises", "comprising", "includes", "including" and "has" are inclusive and therefore specify the presence of stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring their performance in the particular order

[0190] The above description is merely that of particular embodiments of the application and is not intended to limit the application to these particular embodiments. Various modifications within the spirit and scope of the application will become readily apparent to those skilled in the art from this disclosure, which defines one or more embodiments of the application. The scope of the application is to be accorded the broadest scope of the principles and novel features disclosed herein and / or defined by the following claims and their equivalents.

Claims

1. A task merging processing method, characterized in that, The method includes: Obtain the first metadata from the currently running first task, where the first task is a data synchronization task; Based on the first metadata, detect whether there is a second task that can be merged with the first task among the tasks to be run; When there is a second task that can be merged with the first task, the first task and the second task are merged to obtain the third task; The third task is used to perform a replacement operation on the first task.

2. The method according to claim 1, characterized in that, The step of detecting whether there is a second task that can be merged with the first task in the tasks to be run, based on the first metadata, includes: Obtain the preset merging information identification strategy; According to the merged information identification strategy, the first merged information is determined in the first metadata; Based on the first merging information, among the tasks to be run, it is detected whether there is a second task that can be merged with the first task.

3. The method according to claim 2, characterized in that, The step of detecting whether there is a second task that can be merged with the first task among the tasks to be run, based on the first merging information, includes: Retrieve the second metadata from the task to be run; Based on whether the second metadata in the task to be run includes the first merging information, detect whether there is a second task that can be merged with the first task.

4. The method according to claim 1, characterized in that, The step of detecting whether there is a second task that can be merged with the first task in the tasks to be run, based on the first metadata, includes: Retrieve the second metadata from the task to be run; According to the preset merging information identification strategy, the first merging information is determined in the first metadata; According to the merged information identification strategy, the second merged information corresponding to the task to be run is determined from the second metadata in the task to be run; Based on whether the second merge information corresponding to the task to be run is the same as the first merge information, it is detected whether there is a second task that can be merged with the first task.

5. The method according to claim 1, characterized in that, Before retrieving the first metadata from the currently running first task, the method further includes: Check if any newly generated data synchronization tasks exist; When a newly generated data synchronization task is detected, retrieve the first metadata from the currently running first task; The step of detecting whether there is a second task that can be merged with the first task in the tasks to be run, based on the first metadata, includes: In the tasks to be run, identify the newly generated data synchronization task; Based on the first metadata, in the newly generated data synchronization task, it is detected whether there is a second task that can be merged with the first task.

6. The method according to claim 1, characterized in that, The step of detecting whether there is a second task that can be merged with the first task in the tasks to be run, based on the first metadata, includes: In the tasks to be run, identify the data synchronization tasks to be run; Based on the first metadata, in the data synchronization task to be run, it is detected whether there is a second task that can be merged with the first task.

7. The method according to claim 1, characterized in that, Before obtaining the first metadata from the currently running first task, the method further includes: In the data management system, obtain the IP address and port number of the database to which the data table to be managed belongs; According to the preset metadata collection rules, the metadata of the managed data table is collected to obtain the first metadata; Create a first task based on the first metadata, the IP address, and the port number.

8. A task merging processing apparatus, characterized in that, The device includes: The acquisition unit is used to acquire the first metadata in the currently running first task, where the first task is a data synchronization task. The detection unit is used to detect, based on the first metadata, whether there is a second task that can be merged with the first task in the task to be run; The merging unit is used to merge the first task and the second task to obtain a third task when there is a second task that can be merged with the first task. The replacement unit is used to perform a replacement operation on the first task using the third task.

9. A task merging processing device, characterized in that, include: At least one communication interface; At least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; At least one memory connected to the at least one bus, wherein the processor is configured to: Obtain the first metadata from the currently running first task, where the first task is a data synchronization task; Based on the first metadata, detect whether there is a second task that can be merged with the first task among the tasks to be run; When there is a second task that can be merged with the first task, the first task and the second task are merged to obtain the third task; The third task is used to perform a replacement operation on the first task.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the task merging processing method according to any one of claims 1 to 7.

Images