A data integration, processing task allocation method and device and electronic equipment

Abstract

The application provides a data integration task allocation method, comprising: obtaining at least one data integration task which needs to be executed in a target database; obtaining a data integration estimated duration of the at least one data integration task according to database performance information of the target database and data characteristic information of the at least one data integration task; and allocating target database resources for executing data integration tasks for the at least one data integration task according to the data integration estimated duration. The data integration task allocation method allocates target database resources for executing data integration tasks for the at least one data integration task based on the data integration estimated duration, solves the problem of reasonably allocating different data integration tasks, and improves the utilization rate of target database resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a data integration task allocation method. The present application also relates to a data integration task allocation device, a data processing task allocation method, a data processing task allocation device, an electronic device, a storage medium and a computer program product. BACKGROUND

[0002] In order to integrate system data of different systems of an enterprise, break through the barriers between system data of different systems, and make it convenient for employees, customers and partners of the enterprise to apply system data of different systems, it is often necessary to perform data processing operations such as data aggregation integration, data purification processing, data service visualization and data value realization on system data of different systems of the enterprise. Nowadays, data aggregation integration, data purification processing, data service visualization and data value realization and other data processing operations on system data of different systems of an enterprise can generally be implemented through a data middle platform. However, when implementing data aggregation integration on system data of different systems of an enterprise through a data middle platform, for a system, it is often necessary for the system to first perform data integration on its own multiple system data in a database and then synchronize to the data middle platform.

[0003] Since the database is very important for system operation and is already under high load, the timing of performing a data integration task becomes a problem that the database has to face when performing data integration. In addition, since there are often more than one data integration task to be performed in the database, how to reasonably allocate different data integration tasks also becomes a problem that the database has to face when performing data integration.

[0004] In addition, in other scenarios, such as scenarios of integrating system data of different systems of different enterprises and breaking through the barriers between different system data of different systems of different enterprises, there may also be the problem of how to reasonably allocate different data integration tasks. SUMMARY

[0005] The present application provides a data integration task allocation method, a data integration task allocation device, a data processing task allocation method, a data processing task allocation device, an electronic device, a storage medium and a computer program product to solve the problem of reasonably allocating different data integration tasks.

[0006] The present application provides a data integration task allocation method, comprising:

[0007] obtaining at least one data integration task that needs to be performed in a target database;

[0008] obtaining data integration estimated time length of the at least one data integration task according to the database performance information of the target database and the data characteristic information of the at least one data integration task;

[0009] allocating target database resources for executing data integration tasks for the at least one data integration task according to the data integration estimated time length.

[0010] Optionally, the application provides a data integration task allocation method, further comprising:

[0011] obtaining identification information of the target database;

[0012] obtaining database performance information of the target database according to the identification information of the target database and a first corresponding relationship between the identification information and the database performance information.

[0013] Optionally, the obtaining of the at least one data integration task that needs to be executed in the target database comprises:

[0014] obtaining a target data integration task list corresponding to the target database according to the identification information of the target database and a second corresponding relationship between the identification information and the data integration task list;

[0015] obtaining the at least one data integration task according to the target data integration task list.

[0016] Optionally, the obtaining of the at least one data integration task according to the target data integration task list comprises: obtaining the at least one data integration task and data characteristic information of the at least one data integration task according to the target data integration task list.

[0017] Optionally, the database performance information of the target database comprises bandwidth information of the target database, and the data characteristic information of the at least one data integration task comprises data capacity information of the at least one data integration task;

[0018] the obtaining of the data integration estimated time length of the at least one data integration task according to the database performance information of the target database and the data characteristic information of the at least one data integration task comprises: obtaining the data integration estimated time length of the at least one data integration task according to the bandwidth information of the target database and the data capacity information of the at least one data integration task.

[0019] Optionally, the obtaining of the data integration estimated time length of the at least one data integration task according to the bandwidth information of the target database and the data capacity information of the at least one data integration task comprises:

[0020] obtaining bandwidth corresponding to bandwidth information of the target database;

[0021] obtaining data capacity corresponding to data capacity information of the at least one data integration task;

[0022] obtaining data integration estimated duration of the at least one data integration task according to a ratio between data capacity corresponding to data capacity information of the at least one data integration task and bandwidth corresponding to bandwidth information of the target database.

[0023] Optionally, the allocating target database resources for performing data integration tasks for the at least one data integration task according to the data integration estimated duration comprises:

[0024] creating a target resource pool for providing resource load management service for database resources for performing data integration tasks;

[0025] associating the at least one data integration task to the target resource pool;

[0026] allocating the target database resources for the at least one data integration task through the target resource pool according to the data integration estimated duration.

[0027] Optionally, the allocating the target database resources for the at least one data integration task through the target resource pool according to the data integration estimated duration comprises: allocating the target database resources for the at least one data integration task in a descending order of the data integration estimated duration in database resources corresponding to the target resource pool.

[0028] Optionally, the allocating the target database resources for the at least one data integration task in a descending order of the data integration estimated duration in database resources corresponding to the target resource pool comprises:

[0029] obtaining an available time interval of database resources corresponding to the target resource pool;

[0030] judging whether database resources corresponding to the target resource pool can complete the at least one data integration task in a serial manner within the available time interval;

[0031] if yes, allocating the target database resources for the at least one data integration task in a serial manner in database resources corresponding to the target resource pool in a descending order of the data integration estimated duration.

[0032] Optionally, the allocating the target database resources to the at least one data integration task in the target resource pool in a concurrent manner according to the data integration estimated time length from small to large comprises:

[0033] Optionally, the method further comprises: if the target resource pool corresponding database resources cannot complete the at least one data integration task in a serial manner within the available time interval, allocating the target database resources to the at least one data integration task in the target resource pool in a concurrent manner according to the data integration estimated time length from small to large.

[0034] Optionally, the allocating the target database resources to the at least one data integration task in the target resource pool in a concurrent manner according to the data integration estimated time length from small to large comprises:

[0035] obtaining a first type of data integration task and a second type of data integration task in the at least one data integration task, the first type of data integration task being a data integration task whose data integration estimated time length information corresponding time length does not reach a time length threshold, and the second type of data integration task being a data integration task whose data integration estimated time length information corresponding time length reaches the time length threshold;

[0036] allocating the target database resources to the first type of data integration task in the target resource pool in a concurrent manner according to the data integration estimated time length from small to large;

[0037] after allocating the target database resources to the first type of data integration task, allocating the target database resources to the second type of data integration task in the target resource pool in a concurrent manner according to the data integration estimated time length from small to large.

[0038] Optionally, the creating a target resource pool for providing resource load management service to database resources used for executing data integration tasks comprises:

[0039] obtaining historical database performance information of the target database in the database performance information of the target database;

[0040] determine a target historical time interval in which historical database performance of the target database meets a preset performance threshold according to historical database performance information of the target database;

[0041] divide database resources for executing the data integration task from database resources of the target database in a same time interval as the target historical time interval;

[0042] create the target resource pool for the database resources for executing the data integration task.

[0043] Optionally, the historical database performance information of the target database at least includes remaining memory information of the target database and utilization of a database server of the target database.

[0044] The determining of the target historical time interval in which the historical database performance of the target database meets the preset performance threshold according to the historical database performance information of the target database includes: determining, according to the remaining memory information and the utilization of the database server of the target database, a historical time interval in which the remaining memory information exceeds a first preset performance threshold and the utilization of the database server of the target database is lower than a second preset performance threshold as the target historical time interval.

[0045] Another aspect of the present application further provides a data integration task allocation device, which comprises:

[0046] a data integration task obtaining unit configured to obtain at least one data integration task that needs to be executed in a target database;

[0047] a data integration estimated duration obtaining unit configured to obtain a data integration estimated duration of the at least one data integration task according to database performance information of the target database and data characteristic information of the at least one data integration task;

[0048] a data integration task allocation unit configured to allocate target database resources for executing the data integration task to the at least one data integration task according to the data integration estimated duration.

[0049] Another aspect of the present application further provides a data processing task allocation method, which comprises:

[0050] obtaining at least one data processing task that needs to be executed in a target database;

[0051] obtaining a data processing estimated duration of the at least one data processing task according to database performance information of the target database and data characteristic information of the at least one data processing task;

[0052] According to the data processing estimated duration, the target database resource for executing the data processing task is allocated to the at least one data processing task.

[0053] Optionally, the database performance information of the target database comprises bandwidth information of the target database, and the data characteristic information of the at least one data processing task comprises data capacity information of the at least one data processing task.

[0054] The data processing estimated duration of the at least one data processing task is obtained according to the database performance information of the target database and the data characteristic information of the at least one data processing task, comprising: the data processing estimated duration of the at least one data processing task is obtained according to the bandwidth information of the target database and the data capacity information of the at least one data processing task.

[0055] Optionally, the data processing estimated duration of the at least one data processing task is obtained according to the bandwidth information of the target database and the data capacity information of the at least one data processing task, comprising:

[0056] The bandwidth corresponding to the bandwidth information of the target database is obtained.

[0057] The data capacity corresponding to the data capacity information of the at least one data processing task is obtained.

[0058] The data processing estimated duration of the at least one data processing task is obtained according to the ratio between the data capacity corresponding to the data capacity information of the at least one data processing task and the bandwidth corresponding to the bandwidth information of the target database.

[0059] Optionally, the target database resource for executing the data processing task is allocated to the at least one data processing task according to the data processing estimated duration, comprising:

[0060] A target resource pool for providing resource load management service to the database resource for executing the data processing task is created.

[0061] The at least one data processing task is associated to the target resource pool.

[0062] The target database resource is allocated to the at least one data processing task through the target resource pool according to the data processing estimated duration.

[0063] Optionally, the allocating the target database resource for the at least one data processing task in the target resource pool according to the data processing estimated time length comprises: allocating the target database resource for the at least one data processing task in the database resource corresponding to the target resource pool in ascending order of the data processing estimated time length.

[0064] Optionally, the allocating the target database resource for the at least one data processing task in the target resource pool according to the data processing estimated time length comprises:

[0065] obtaining an available time interval of the database resource corresponding to the target resource pool;

[0066] judging whether the database resource corresponding to the target resource pool can complete the at least one data processing task in a serial manner within the available time interval;

[0067] if yes, allocating the target database resource for the at least one data processing task in the database resource corresponding to the target resource pool in a serial manner in ascending order of the data processing estimated time length.

[0068] Optionally, the allocating the target database resource for the at least one data processing task in the target resource pool according to the data processing estimated time length comprises: allocating the target database resource for the at least one data processing task in the database resource corresponding to the target resource pool in a serial manner in ascending order of the data processing estimated time length within the available time interval.

[0069] Optionally, the method further comprises: if the database resource corresponding to the target resource pool cannot complete the at least one data processing task in a serial manner within the available time interval, allocating the target database resource for the at least one data processing task in the database resource corresponding to the target resource pool in a concurrent manner in ascending order of the data processing estimated time length.

[0070] Optionally, the data processing task at least comprises a data integration task.

[0071] Another aspect of the present application also provides a data processing task allocation apparatus, characterized in that comprising:

[0072] a data processing task obtaining unit, configured to obtain at least one data processing task needed to be executed in a target database;

[0073] a data processing estimated duration obtaining unit, configured to obtain a data processing estimated duration of the at least one data processing task according to database performance information of the target database and data characteristic information of the at least one data processing task;

[0074] a data processing task allocating unit, configured to allocate target database resources for executing the data processing task to the at least one data processing task according to the data processing estimated duration.

[0075] In another aspect, the present application provides an electronic device, comprising:

[0076] a processor;

[0077] a memory configured to store a computer program, wherein the computer program is run by the processor to execute any one of the data integration task allocating method and the data processing task allocating method provided by the present application.

[0078] In another aspect, the present application provides a storage medium, wherein the storage medium stores a computer program, and the computer program is run by a processor to execute any one of the data integration task allocating method and the data processing task allocating method provided by the present application.

[0079] In another aspect, the present application provides a computer program product, comprising a non-volatile computer readable storage medium, wherein the non-volatile computer readable storage medium contains computer program code for executing any one of the data integration task allocating method and the data processing task allocating method provided by the present application.

[0080] Compared with the prior art, the present application has the following advantages:

[0081] The data integration task allocating method provided by the present application comprises the following steps: firstly, obtaining at least one data integration task which needs to be executed in a target database; then, obtaining a data integration estimated duration of the at least one data integration task according to database performance information of the target database and data characteristic information of the at least one data integration task; finally, allocating target database resources for executing the data integration task to the at least one data integration task according to the data integration estimated duration. The data integration task allocating method allocates target database resources for executing the data integration task to the at least one data integration task based on the data integration estimated duration, and can improve the utilization rate of the target database resources.

[0082] The application provides a data processing task allocation method. First, at least one data processing task to be executed in a target database is obtained. Then, according to database performance information of the target database and data characteristic information of the at least one data processing task, data processing estimated duration of the at least one data processing task is obtained. Finally, according to the data processing estimated duration, target database resources for executing the data processing task are allocated to the at least one data processing task. The data processing task allocation method allocates target database resources for executing the data processing task to the at least one data processing task based on the data processing estimated duration, and can improve utilization of the target database resources. BRIEF DESCRIPTION OF DRAWINGS

[0083] Figure 1 An application schematic diagram of the data integration task allocation method provided in the first embodiment of the application.

[0084] Figure 2 A flowchart of the data integration task allocation method provided in the first embodiment of the application.

[0085] Figure 3 A first data integration task allocation schematic diagram provided in the first embodiment of the application.

[0086] Figure 4 A second data integration task allocation schematic diagram provided in the first embodiment of the application.

[0087] Figure 5 A first data comparison application module schematic diagram provided in the first embodiment of the application.

[0088] Figure 6 A second data comparison application module schematic diagram provided in the first embodiment of the application.

[0089] Figure 7 An application schematic diagram of the data integration task allocation device provided in the second embodiment of the application.

[0090] Figure 8 A flowchart of the data processing method provided in the third embodiment of the application.

[0091] Figure 9 An application schematic diagram of the data processing task allocation device provided in the fourth embodiment of the application.

[0092] Figure 10 An electronic device schematic diagram provided in the embodiments of the application. DETAILED DESCRIPTION

[0093] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced without the specific details. In other instances, well-known methods have not been described in detail in order not to unnecessarily obscure aspects of the present application.

[0094] First embodiment

[0095] In order to more clearly show the data integration task allocation method provided in the first embodiment of the present application, first introduce the application scenario of the data integration task allocation method provided in the first embodiment of the present application. The data integration task allocation method provided in the first embodiment of the present application can be implemented based on an application or software installed in an enterprise data platform for implementing the data integration task allocation method, at this time, the data integration task allocation method provided in the first embodiment of the present application targets at least one target database, i.e., different databases under the management of the same enterprise data platform. In addition, the data integration task allocation method provided in the first embodiment of the present application can also be implemented based on an application or software installed in a database server for implementing the data integration task allocation method, at this time, the data integration task allocation method provided in the first embodiment of the present application targets at a specified database.

[0096] The so-called data integration task is a data processing task that logically or physically organically integrates data of different sources, formats, characteristics and properties, thereby providing comprehensive data sharing for enterprises. The so-called different sources refer to that different data to be integrated can come from databases of different systems.

[0097] In the first embodiment of the present application, the execution subject of the data integration task allocation method is not specifically limited, as long as it can be used to implement the data integration task allocation method. The following specifically takes an application based on an enterprise data platform for implementing the data integration task allocation method as an example to implement the data integration task allocation method provided in the first embodiment of the present application to explain the data integration task allocation method provided in the first embodiment of the present application in detail.

[0098] When the data integration task allocation method provided in the first embodiment of the present application is implemented based on an application installed in an enterprise data platform for implementing the data integration task allocation method, the number of target databases is at least one, and since the working principle of the data integration task allocation method is the same for different target data, only one target database is taken as an example to be described in detail in the related steps of implementing the data integration task allocation method provided in the first embodiment of the present application based on an application installed in an enterprise data platform for implementing the data integration task allocation method.

[0099] The target database is generally a MySQL (MySQL Datas, open source relational) database, and can also be other types of databases. In the first embodiment of the present application, the type of the target database is not specifically limited.

[0100] The enterprise data middle platform is a data processing platform for integrating system data of different systems of an enterprise, bridging the gap between system data of different systems, so that employees, customers and partners of the enterprise can conveniently apply system data of different systems. The data processing platform is specifically used for performing data aggregation and integration, data purification processing, data service visualization, and data value realization on system data of different systems of the enterprise. Correspondingly, the data integration task is for system data. In the first embodiment of the present application, the system data includes, but is not limited to, application data in the enterprise system, platform or system running data, user historical behavior data, device running data, and enterprise operation data.

[0101] Please refer to Figure 1 , which is a schematic diagram of the application of the method for assigning data integration tasks provided in the first embodiment of the present application. The application of the method for assigning data integration tasks includes a data integration task management module 101, a data source information acquisition module 102, and a data integration task planning module 103. The data integration task management module 101 is used to manage data integration tasks, including creating data integration tasks, modifying the time of data integration tasks, etc. The data source information acquisition module 102 is used to acquire database performance information of the target database. The data integration task planning module 103 is used to acquire data integration estimated time length of at least one data integration task, and assign target database resources for executing the data integration task to the at least one data integration task according to the data integration estimated time length.

[0102] In the first embodiment of the present application, the method for assigning data integration tasks provided in the first embodiment of the present application is implemented based on the application of the method for assigning data integration tasks installed in the enterprise data middle platform. The process is as follows: First, at least one data integration task that needs to be executed in the target database is obtained through the data integration task management module 101; then, the database performance information of the target database is obtained based on the data source information acquisition module 102; finally, the data characteristic information of the at least one data integration task is obtained based on the data integration task planning module 103; the data integration estimated time length of the at least one data integration task is obtained according to the data characteristic information of the at least one data integration task and the database performance information of the target database obtained based on the data source information acquisition module 102; and the target database resources for executing the data integration task are assigned to the at least one data integration task according to the data integration estimated time length. For specific processes, please refer toFigure 2 which is a flowchart of a data integration task allocation method provided in the first embodiment of the present application.

[0103] In step S201, at least one data integration task needed to be executed in the target database is obtained.

[0104] The so-called data integration task is a task created for the data to be integrated in the target database. The so-called data to be integrated includes one or more of the application data stored in the database, the platform or system running data, the user historical behavior data, the device running data, and the enterprise operation data, etc.

[0105] The specific implementation of obtaining at least one data integration task needed to be executed in the target database is as follows: first, according to the identification information of the target database and the second correspondence relationship between the identification information and the data integration task list, the target data integration task list corresponding to the target database is obtained; then, according to the target data integration task list, at least one data integration task is obtained. Specifically, when at least one data integration task is obtained according to the target data integration task list, the data characteristic information of the at least one data integration task can be obtained at the same time.

[0106] The so-called identification information of the target database is identification information that can uniquely identify the target database, and through which the target database can be uniquely linked. The identification information includes but is not limited to the IP (Internet Protocol) address information of the target database, the port information of the target database, and the username information of the target database, etc.

[0107] In step S202, according to the database performance information of the target database and the data characteristic information of the at least one data integration task, the data integration estimated duration of the at least one data integration task is obtained.

[0108] The so-called database performance information of the target database includes the current database performance information of the target database and the historical database performance information of the target database. Specifically, the database performance information of the target database includes but is not limited to the bandwidth information of the target database, the memory information of the target database, and the utilization rate of the database server of the target database, etc. Among them, the so-called memory information of the target database usually includes the memory information of the used memory of the target database and the memory information of the remaining memory of the target database.

[0109] Before using the database performance information of the target database, the database performance information of the target database needs to be obtained. A specific implementation manner is generally as follows: first, obtaining the identification information of the target database; then, obtaining the database performance information of the target database according to the identification information of the target database and a first corresponding relationship between the identification information and the database performance information.

[0110] The database performance information of the target database is the database performance information from a database performance view pre-created for storing the database performance information. In the first embodiment of the present application, the specific implementation manner of obtaining the database performance information of the target database is not specifically limited, as long as the database performance information of the target database can be obtained.

[0111] In the first embodiment of the present application, the database performance information of the target database includes bandwidth information of the target database, and the data characteristic information of the at least one data integration task includes data capacity information of the at least one data integration task. According to the database performance information of the target database and the data characteristic information of the at least one data integration task, the implementation manner of obtaining the data integration estimated duration of the at least one data integration task is as follows: obtaining the data integration estimated duration of the at least one data integration task according to the bandwidth information of the target database and the data capacity information of the at least one data integration task. The specific implementation process is as follows: first, obtaining the bandwidth corresponding to the bandwidth information of the target database; obtaining the data capacity corresponding to the data capacity information of the at least one data integration task; obtaining the data integration estimated duration of the at least one data integration task according to the ratio between the data capacity corresponding to the data capacity information of the at least one data integration task and the bandwidth corresponding to the bandwidth information of the target database, and the corresponding formula is as follows: data integration estimated duration = data capacity / bandwidth.

[0112] In step S203, the target database resources for executing the data integration task are allocated to the at least one data integration task according to the data integration estimated duration.

[0113] In the first embodiment of the present application, the specific implementation manner of allocating the target database resources for executing the data integration task to the at least one data integration task is as follows: first, creating a target resource pool for providing resource load management services for the database resources for executing the data integration task; then, associating the at least one data integration task to the target resource pool; finally, allocating the target database resources to the at least one data integration task through the target resource pool according to the data integration estimated duration. The implementation process of the above steps is described in detail as follows:

[0114] For creating the target resource pool for providing the resource load management service to the database resource used for performing the data integration task, in the first embodiment of the present application, the following method is adopted: first, obtaining the historical database performance information of the target database in the database performance information of the target database; second, determining the target historical time interval in which the historical database performance of the target database meets the preset performance threshold according to the historical database performance information of the target database; third, taking the database resource of the target database in the same time interval as the target historical time interval as the database resource used for performing the data integration task; fourth, creating the target resource pool for the database resource used for performing the data integration task.

[0115] The historical database performance information of the target database at least includes the remaining memory information of the target database and the utilization rate of the database server of the target database. At this time, the process of determining the target historical time interval in which the historical database performance of the target database meets the preset performance threshold according to the historical database performance information of the target database is: determining the historical time interval in which the remaining memory exceeds the first preset performance threshold and the utilization rate of the database server of the target database is lower than the second preset performance threshold as the target historical time interval according to the remaining memory information and the utilization rate of the database server of the target database.

[0116] The historical database performance information of the target database can be the database performance information of the target database one day ago, the database performance information within one week, the database performance information within one month, etc. Taking the historical database performance information of the target database as the database performance information of the target database one day ago as an example, when the historical time interval in which the memory corresponding to the remaining memory information exceeds the first preset performance threshold and the utilization rate of the database server of the target database is lower than the second preset performance threshold is determined as the target historical time interval according to the remaining memory information and the utilization rate of the database server of the target database, the remaining memory information and the utilization rate of the database server of the target database in each hour within the previous 24 hours, i.e., the remaining memory information and the utilization rate of the database server of the target database in 0-1, 1-2, …, 23-24, can be obtained. If the memory corresponding to the remaining memory information in the three historical intervals of 0-5, 10-11 and 15-16 exceeds the first preset performance threshold and the utilization rate of the database server of the target database is lower than the second preset performance threshold, the three historical intervals of 0-5, 10-11 and 15-16 are taken as the target historical time interval.

[0117] The database resources for performing the data integration task are divided from the database resources of the target database in the same time interval as the target historical time interval. For example, the database resources for performing the data integration task are divided from the database resources of the target database in the time intervals of 0-5, 10-11, and 15-16. After the database resources for performing the data integration task are divided, the target resource pools, i.e., the first target resource pool (0-5), the second target resource pool (10-11), and the second target resource pool (15-16), are created for the database resources for performing the data integration task.

[0118] For allocating the target database resources for the at least one data integration task from the target resource pool according to the data integration estimated duration, in the first embodiment of the present application, the following method is used: the target database resources are allocated for the at least one data integration task in the database resources corresponding to the target resource pool in the order of the data integration estimated duration from small to large. Specifically, first, the available time interval of the database resources corresponding to the target resource pool is obtained; second, it is determined whether the database resources corresponding to the target resource pool can complete the at least one data integration task in a serial manner in the available time interval; third, if yes, the target database resources are allocated for the at least one data integration task in the database resources corresponding to the target resource pool in a serial manner in the order of the data integration estimated duration from small to large; and finally, if the database resources corresponding to the target resource pool cannot complete the at least one data integration task in a serial manner in the available time interval, the target database resources are allocated for the at least one data integration task in the database resources corresponding to the target resource pool in a concurrent manner in the order of the data integration estimated duration from small to large.

[0119] The specific implementation process of allocating the target database resources for the at least one data integration task in the database resources corresponding to the target resource pool in the order of the data integration estimated duration from small to large is as follows: in the available time interval, the target database resources are allocated for the at least one data integration task in the database resources corresponding to the target resource pool in a serial manner according to the data integration estimated duration in the order of the data integration estimated duration from small to large. Specifically, if there are one hundred data integration tasks, the data integration tasks are sequentially sorted in the order of the data integration estimated duration from large to small as follows: the first data integration task (the data integration estimated duration is 1 minute), the second data integration task (the data integration estimated duration is 2 minutes), the third data integration task (the data integration estimated duration is 4 minutes),..., and the one hundredth data integration task (the data integration estimated duration is 60 minutes).

[0120] For example, Figure 3If the data integration estimated time length corresponding to the one hundred data integration tasks does not exceed the first target resource pool, the second target resource pool, and the available time of the second target resource pool, then the first data integration task is allocated target database resources in the database resources corresponding to the first target resource pool, and the target database resources are the available database resources in the 0-1 minute time zone of the database resources corresponding to the first target resource pool; the second data integration task is allocated target database resources in the database resources corresponding to the first target resource pool, and the target database resources are the available database resources in the 2-4 minute time zone of the database resources corresponding to the second target resource pool; the third data integration task is allocated target database resources in the database resources corresponding to the first target resource pool, and the target database resources are the available database resources in the 4-8 minute time zone of the database resources corresponding to the third target resource pool; and so on until the target database resources are allocated to the at least one data integration task in the database resources corresponding to the first target resource pool, the second target resource pool, and the third target resource pool.

[0121] The specific implementation process of allocating target database resources to the at least one data integration task in the database resources corresponding to the target resource pool in a concurrent manner in order of increasing data integration estimated time length is as follows: first, obtain the first type of data integration task in the at least one data integration task, and the second type of data integration task in the at least one data integration task, the first type of data integration task being a data integration task whose data integration estimated time length information does not reach the time threshold, and the second type of data integration task being a data integration task whose data integration estimated time length information reaches the time threshold; then, allocate target database resources to the first type of data integration task in the database resources corresponding to the target resource pool in a concurrent manner in order of increasing data integration estimated time length; finally, after the target database resources are allocated to the first type of data integration task, allocate target database resources to the second type of data integration task in the database resources corresponding to the target resource pool in a concurrent manner in order of increasing data integration estimated time length.

[0122] It should be noted that the method for obtaining the first type of data integration task and the second type of data integration task is: determining a conversion value and a data integration task division threshold value for obtaining the first type of data integration task and the second type of data integration task, obtaining a ratio of a data integration estimated duration of at least one data integration task to the conversion value; comparing the ratio of the data integration estimated duration of at least one data integration task to the conversion value and the data integration task division threshold value, if the ratio of the data integration estimated duration of the data integration task to the conversion value does not exceed the data integration task division threshold value, the data integration task is taken as the first type of data integration task, otherwise the data integration task is taken as the second type of data integration task. The conversion value and the data integration task division threshold value are generally empirical values.

[0123] As shown in Figure 4 , first, in the order of data integration estimated duration from small to large, target database resources are allocated to the first type of data integration task in the target resource pool in a concurrent manner, and then, after the target database resources are allocated to the first type of data integration task, target database resources are allocated to the second type of data integration task in the target resource pool in a concurrent manner in the order of data integration estimated duration from small to large. When target database resources are allocated to at least one data integration task in the target resource pool in a concurrent manner, it is necessary to ensure that the concurrency of the database resources is not exceeded.

[0124] In the first embodiment of the present application, after the target database resources are allocated to at least one data integration task for executing the data integration task, the target database resources will be used for data integration according to the allocation to the data integration task, and after the data integration task is completed, the integrated data will be further stored in the enterprise data hub. In the process of synchronizing the integrated data to the data hub, a large amount of data will be involved. In order to ensure the consistency of the data in the target database and the data synchronized to the data hub, the data in the target database and the data synchronized to the data hub need to be compared. Therefore, in the first embodiment of the present application, the following two data comparison methods are provided:

[0125] The first data comparison method is based on Figure 5The data comparison application module is shown to be implemented, and the data comparison application module includes a comparison data acquisition module 501, which is configured to acquire data to be compared; a sampling module 502, which is configured to perform stratified sampling on the data to be compared to acquire comparison samples from the data to be compared; a comparison module 503, which is configured to perform data comparison to obtain a data comparison result; and a result writing module 504, which is configured to write the data comparison result into a local file for storage. The working process of the data comparison application module is as follows: the data synchronized to the data platform is mapped into a Hive data table, and the partition data in the Hive data table is queried; the record number cnt (Count, counter) of the partition data is obtained to acquire the primary key List and sort; 20 pieces of data are obtained from the primary key List as first data to be compared according to a step cnt / 20 (in the first embodiment of the application, the step and the number of pieces of data obtained are not specifically limited); the data table in which the data in the target database is located is queried according to the 20 primary keys to obtain second data to be compared; all fields of each piece of data in the first data to be compared and the second data to be compared are converted into strings, the strings of each piece of data are spliced, and the hash value of the first data to be compared and the second data to be compared is obtained through MD5 (Message-Digest Algorithm, information digest algorithm); the hash values of the first data to be compared and the second data to be compared are compared to obtain a comparison result, and the comparison result is written into the Hive data table. It should be noted that the stratified sampling method is used to replace full-amount comparison, which can greatly save the time of data comparison work and can maximize the rationality of sampling. In addition, the field splicing + MD5 information digest algorithm is used to complete the data, which can improve the comparison efficiency.

[0126] The second data comparison method is based on the data comparison method as Figure 6The data comparison application shown is implemented by a data comparison application module, which includes a comparison data acquisition module 601 configured to acquire data to be compared; a structured query language (SQL) construction issuing module 602 configured to construct SQL used for data comparison and issue the same; a comparison module 603 configured to perform data comparison to obtain a data comparison result; and a result writing module 604 configured to write the data comparison result into a local file for storage. The workflow of the data comparison application module is as follows: map data synchronized to the data center to a Hive data table and construct a Hive SQL; issue the Hive SQL to obtain first data to be compared from the data synchronized to the data center; obtain a data table where data in a target database is located, construct a data table check SQL of the data table where the data in the target database is located, and in the process of constructing the data table check SQL of the data table where the data in the target database is located, the SUM of a key indicator needs to use an absolute value; issue the data table check SQL of the data table where the data in the target database is located to obtain second data to be compared from the data in the target database; compare the first data to be compared with the second data to be compared to obtain a comparison result, and write the comparison result into the Hive data table. It should be noted that the SUM (SUM Function, mathematical and trigonometric functions) of the key indicator needs to use an absolute value, which can prevent errors caused by positive and negative offset.

[0127] The first embodiment of the present application provides a data integration task allocation method. First, at least one data integration task to be executed in a target database is obtained. Then, according to database performance information of the target database and data characteristic information of the at least one data integration task, data integration estimated duration of the at least one data integration task is obtained. Finally, according to the data integration estimated duration, target database resources for executing the data integration task are allocated to the at least one data integration task. The data integration task allocation method allocates target database resources for executing the data integration task to the at least one data integration task based on the data integration estimated duration, solves the problem of reasonably allocating different data integration tasks, and can improve utilization of target database resources.

[0128] Second embodiment

[0129] Corresponding to the data integration task allocation method provided by the first embodiment, the second embodiment of the present application also provides a data integration task allocation method. Since the device embodiment is basically similar to the data integration task allocation method provided by the first embodiment, the description is relatively simple, and the related parts can be referred to the part of the description of the data integration task allocation method provided by the first embodiment. The device embodiment described below is only illustrative.

[0130] Please refer toFigure 7 FIG. 1 is a schematic diagram of a data integration task allocation apparatus provided in a second embodiment of the present application.

[0131] The data integration task allocation apparatus comprises:

[0132] a data integration task obtaining unit 701 configured to obtain at least one data integration task to be executed in a target database;

[0133] a data integration estimated duration obtaining unit 702 configured to obtain a data integration estimated duration of the at least one data integration task according to database performance information of the target database and data characteristic information of the at least one data integration task;

[0134] a data integration task allocation unit 703 configured to allocate target database resources for executing data integration tasks to the at least one data integration task according to the data integration estimated duration.

[0135] The data integration task allocation apparatus provided in the second embodiment of the present application further comprises:

[0136] an identification information obtaining module configured to obtain identification information of the target database;

[0137] a database performance information obtaining module configured to obtain database performance information of the target database according to the identification information of the target database and a first corresponding relationship between the identification information and the database performance information.

[0138] Optionally, the data integration task obtaining unit 701 is specifically configured to obtain a target data integration task list corresponding to the target database according to the identification information of the target database and a second corresponding relationship between the identification information and the data integration task list; and obtain the at least one data integration task according to the target data integration task list.

[0139] Optionally, the obtaining of the at least one data integration task according to the target data integration task list comprises: obtaining the at least one data integration task and data characteristic information of the at least one data integration task according to the target data integration task list.

[0140] Optionally, the database performance information of the target database comprises bandwidth information of the target database, and the data characteristic information of the at least one data integration task comprises data capacity information of the at least one data integration task.

[0141] The data integration estimated duration obtaining unit 702 is specifically configured to obtain data integration estimated durations of the at least one data integration task according to bandwidth information of the target database and data capacity information of the at least one data integration task.

[0142] Optionally, the obtaining of the data integration estimated durations of the at least one data integration task according to the bandwidth information of the target database and the data capacity information of the at least one data integration task comprises:

[0143] obtaining bandwidth corresponding to the bandwidth information of the target database;

[0144] obtaining data capacity corresponding to the data capacity information of the at least one data integration task;

[0145] obtaining the data integration estimated durations of the at least one data integration task according to a ratio between the data capacity corresponding to the data capacity information of the at least one data integration task and the bandwidth corresponding to the bandwidth information of the target database.

[0146] Optionally, the data integration task assigning unit 703 is specifically configured to create a target resource pool for providing resource load management service to database resources used for executing data integration tasks, associate the at least one data integration task to the target resource pool, and assign the target database resources to the at least one data integration task through the target resource pool according to the data integration estimated durations.

[0147] Optionally, the assigning of the target database resources to the at least one data integration task through the target resource pool according to the data integration estimated durations comprises: assigning the target database resources to the at least one data integration task in the database resources corresponding to the target resource pool in ascending order of the data integration estimated durations.

[0148] Optionally, the assigning of the target database resources to the at least one data integration task in the database resources corresponding to the target resource pool in ascending order of the data integration estimated durations comprises:

[0149] obtaining an available time interval of the database resources corresponding to the target resource pool;

[0150] judging whether the database resources corresponding to the target resource pool can complete the at least one data integration task in a serial manner within the available time interval;

[0151] If yes, the target database resources are allocated to the at least one data integration task in a serial manner in the database resources corresponding to the target resource pool according to the data integration estimated time length in ascending order.

[0152] Optionally, the allocating the target database resources to the at least one data integration task in a serial manner in the database resources corresponding to the target resource pool according to the data integration estimated time length in ascending order comprises: allocating the target database resources to the at least one data integration task in a serial manner in the database resources corresponding to the target resource pool according to the data integration estimated time length in ascending order within the available time interval.

[0153] Optionally, the method further comprises: if the target resource pool corresponding database resources cannot complete the at least one data integration task in a serial manner within the available time interval, allocating the target database resources to the at least one data integration task in a concurrent manner in the database resources corresponding to the target resource pool according to the data integration estimated time length in ascending order.

[0154] Optionally, the allocating the target database resources to the at least one data integration task in a concurrent manner in the database resources corresponding to the target resource pool according to the data integration estimated time length in ascending order comprises:

[0155] obtaining a first type of data integration task and a second type of data integration task in the at least one data integration task, the first type of data integration task being a data integration task whose data integration estimated time length information corresponding time length does not reach a time length threshold, and the second type of data integration task being a data integration task whose data integration estimated time length information corresponding time length reaches the time length threshold;

[0156] allocating the target database resources to the first type of data integration task in a concurrent manner in the database resources corresponding to the target resource pool according to the data integration estimated time length in ascending order;

[0157] after the target database resources are allocated to the first type of data integration task, the target database resources are allocated to the second type of data integration task in a concurrent manner in the database resources corresponding to the target resource pool according to the data integration estimated time length in ascending order.

[0158] Optionally, the creating the target resource pool for providing resource load management service for database resources used for executing data integration tasks comprises:

[0159] obtain historical database performance information of the target database in the database performance information of the target database;

[0160] determine a target historical time interval in which historical database performance of the target database meets a preset performance threshold according to the historical database performance information of the target database;

[0161] divide the database resources for executing the data integration task from database resources of the target database in a same time interval as the target historical time interval;

[0162] create the target resource pool for the database resources for executing the data integration task.

[0163] Optionally, the historical database performance information of the target database at least includes remaining memory information of the target database and utilization of a database server of the target database;

[0164] The determining of the target historical time interval in which the historical database performance of the target database meets the preset performance threshold according to the historical database performance information of the target database includes: determining, according to the remaining memory information and the utilization of the database server of the target database, a historical time interval in which the remaining memory information exceeds a first preset performance threshold and the utilization of the database server of the target database is lower than a second preset performance threshold as the target historical time interval.

[0165] Third Embodiment

[0166] Corresponding to the data integration task allocation method provided by the first embodiment of the present application, the third embodiment of the present application further provides a data processing task allocation method. Since the third embodiment is basically similar to the data integration task allocation method provided by the first embodiment of the present application, it is described simply, and the related parts refer to the part of the description of the data integration task allocation method provided by the first embodiment of the present application. The third embodiment described below is only illustrative.

[0167] Please refer to Figure 8 which is a flowchart of a data processing method provided by the third embodiment of the present application.

[0168] In step S801, at least one data processing task that needs to be executed in a target database is obtained.

[0169] In the third embodiment of the present application, the so-called data processing task includes but is not limited to a data integration task, that is, the data processing task at least includes a data integration task.

[0170] The data processing task refers to a task created for the to-be-processed data in the target database. The to-be-processed data includes one or more of the following: application data stored in the database, platform or system running data, user historical behavior data, device running data, and enterprise operation data.

[0171] The specific implementation of obtaining the at least one data processing task to be executed in the target database includes the following steps: first, obtaining the target data processing task list corresponding to the target database according to the identification information of the target database and the second correspondence relationship between the identification information and the data processing task list; and then, obtaining the at least one data processing task according to the target data processing task list. Specifically, when obtaining the at least one data processing task according to the target data processing task list, the data characteristic information of the at least one data processing task can be obtained at the same time.

[0172] The identification information of the target database refers to the identification information that can uniquely identify the target database, and through which the target database can be uniquely linked. The identification information includes but is not limited to the IP address information of the target database, the port information of the target database, and the username information of the target database.

[0173] In step S802, the data processing estimated duration of the at least one data processing task is obtained according to the database performance information of the target database and the data characteristic information of the at least one data processing task.

[0174] The database performance information of the target database includes the current database performance information of the target database and the historical database performance information of the target database. Specifically, the database performance information of the target database includes but is not limited to the bandwidth information of the target database, the memory information of the target database, and the utilization rate of the database server of the target database. The memory information of the target database generally includes the memory information of the used memory of the target database and the memory information of the remaining memory of the target database.

[0175] Before using the database performance information of the target database, the database performance information of the target database needs to be obtained first. The specific implementation generally includes the following steps: first, obtaining the identification information of the target database; and then, obtaining the database performance information of the target database according to the identification information of the target database and the first correspondence relationship between the identification information and the database performance information.

[0176] The database performance information of the target database is the database performance information from a database performance view of a pre-created database storing database performance information. In the third embodiment of the present application, the specific implementation of obtaining the database performance information of the target database is not limited as long as the database performance information of the target database can be obtained.

[0177] In the third embodiment of the present application, the database performance information of the target database includes bandwidth information of the target database, and the data characteristic information of the at least one data processing task includes data capacity information of the at least one data processing task. According to the database performance information of the target database and the data characteristic information of the at least one data processing task, the implementation of obtaining the data processing estimated duration of the at least one data processing task is as follows: according to the bandwidth information of the target database and the data capacity information of the at least one data processing task, the data processing estimated duration of the at least one data processing task is obtained. The specific implementation process is as follows: first, the bandwidth corresponding to the bandwidth information of the target database is obtained; the data capacity corresponding to the data capacity information of the at least one data processing task is obtained; according to the ratio between the data capacity corresponding to the data capacity information of the at least one data processing task and the bandwidth corresponding to the bandwidth information of the target database, the data processing estimated duration of the at least one data processing task is obtained, and the corresponding formula is as follows: data processing estimated duration = data capacity / bandwidth.

[0178] In step S803, according to the data processing estimated duration, the target database resources for executing the data processing task are allocated to the at least one data processing task.

[0179] In the third embodiment of the present application, the specific implementation of allocating the target database resources for executing the data processing task to the at least one data processing task is as follows: first, a target resource pool for providing resource load management services to the database resources for executing the data processing task is created; then, the at least one data processing task is associated to the target resource pool; finally, according to the data processing estimated duration, the target database resources are allocated to the at least one data processing task through the target resource pool. The implementation process of the above steps is described in detail as follows:

[0180] For creating the target resource pool for providing the resource load management service for the database resources used for performing the data processing task, in the third embodiment of the present application, the following manner is adopted: first, obtaining the historical database performance information of the target database in the database performance information of the target database; second, determining the target historical time interval in which the historical database performance of the target database meets the preset performance threshold according to the historical database performance information of the target database; third, taking the database resources of the target database in the same time interval as the target historical time interval as the database resources used for performing the data processing task; fourth, creating the target resource pool for the database resources used for performing the data processing task.

[0181] The historical database performance information of the target database at least includes the remaining memory information of the target database and the utilization rate of the database server of the target database. At this time, the process of determining the target historical time interval in which the historical database performance of the target database meets the preset performance threshold according to the historical database performance information of the target database is: determining the historical time interval in which the remaining memory exceeds the first preset performance threshold and the utilization rate of the database server of the target database is lower than the second preset performance threshold as the target historical time interval according to the remaining memory information and the utilization rate of the database server of the target database.

[0182] The historical database performance information of the target database can be the database performance information of the target database one day ago, the database performance information within one week, the database performance information within one month, etc.

[0183] For allocating the target database resources for at least one data processing task through the target resource pool according to the data processing estimated duration, in the third embodiment of the present application, the following manner is adopted: allocating the target database resources for at least one data processing task in the database resources corresponding to the target resource pool in the order from small to large of the data processing estimated duration. Specifically, first, obtaining the available time interval of the database resources corresponding to the target resource pool; second, judging whether the database resources corresponding to the target resource pool can complete at least one data processing task in a serial manner within the available time interval; third, if yes, allocating the target database resources for at least one data processing task in the database resources corresponding to the target resource pool in a serial manner in the order from small to large of the data processing estimated duration. Finally, if the database resources corresponding to the target resource pool cannot complete at least one data processing task in a serial manner within the available time interval, allocating the target database resources for at least one data processing task in the database resources corresponding to the target resource pool in a concurrent manner in the order from small to large of the data processing estimated duration.

[0184] The specific implementation process of sequentially allocating the target database resources to the at least one data processing task in the target resource pool in a concurrent manner according to the order from small to large of the data processing estimated durations is as follows: first, obtaining a first type of data processing task in the at least one data processing task and a second type of data processing task in the at least one data processing task, the first type of data processing task being a data processing task whose duration corresponding to the data processing estimated duration information does not reach the duration threshold, and the second type of data processing task being a data processing task whose duration corresponding to the data processing estimated duration information reaches the duration threshold; then, sequentially allocating the target database resources to the first type of data processing task in the target resource pool in a concurrent manner according to the order from small to large of the data processing estimated durations; and finally, after the target database resources are allocated to the first type of data processing task, sequentially allocating the target database resources to the second type of data processing task in the target resource pool in a concurrent manner according to the order from small to large of the data processing estimated durations.

[0185] The specific implementation process of sequentially allocating the target database resources to the at least one data processing task in the target resource pool in a concurrent manner according to the order from small to large of the data processing estimated durations is as follows: first, obtaining a first type of data processing task in the at least one data processing task and a second type of data processing task in the at least one data processing task, the first type of data processing task being a data processing task whose duration corresponding to the data processing estimated duration information does not reach the duration threshold, and the second type of data processing task being a data processing task whose duration corresponding to the data processing estimated duration information reaches the duration threshold; then, sequentially allocating the target database resources to the first type of data processing task in the target resource pool in a concurrent manner according to the order from small to large of the data processing estimated durations; and finally, after the target database resources are allocated to the first type of data processing task, sequentially allocating the target database resources to the second type of data processing task in the target resource pool in a concurrent manner according to the order from small to large of the data processing estimated durations.

[0186] It should be noted that the method of obtaining the first type of data processing task and the second type of data processing task is as follows: determining a conversion value and a data processing task division threshold for obtaining the first type of data processing task and the second type of data processing task, and obtaining a ratio of the data processing estimated duration of the at least one data processing task to the conversion value; comparing the ratio of the data processing estimated duration of the at least one data processing task to the conversion value with the data processing task division threshold, if the ratio of the data processing estimated duration of the data processing task to the conversion value does not exceed the data processing task division threshold, the data processing task is regarded as the first type of data processing task, otherwise, the data processing task is regarded as the second type of data processing task. The conversion value and the data processing task division threshold are generally empirical values.

[0187] The third embodiment of the present application provides a data processing task allocation method. First, at least one data processing task to be executed in a target database is obtained. Then, according to database performance information of the target database and data characteristic information of the at least one data processing task, data processing estimated duration of the at least one data processing task is obtained. Finally, according to the data processing estimated duration, target database resources for executing the data processing task are allocated to the at least one data processing task. The data processing task allocation method allocates target database resources for executing the data processing task to the at least one data processing task based on the data processing estimated duration, solves the problem of reasonably allocating different data processing tasks, and improves utilization of target database resources.

[0188] The fourth embodiment

[0189] Corresponding to the data processing task allocation method provided by the third embodiment, the fourth embodiment of the present application further provides a data processing task allocation method. Since the device embodiment is basically similar to the data processing task allocation method provided by the third embodiment, the description is relatively simple, and the related parts can be referred to the part of the description of the data processing task allocation method provided by the third embodiment. The device embodiment described below is only illustrative.

[0190] Please refer to Figure 9 which is a schematic diagram of a data processing task allocation device provided in the fourth embodiment of the present application.

[0191] The data processing task allocation device comprises:

[0192] The data processing task obtaining unit 901 is configured to obtain at least one data processing task to be executed in a target database.

[0193] The data processing estimated duration obtaining unit 902 is configured to obtain data processing estimated duration of the at least one data processing task according to database performance information of the target database and data characteristic information of the at least one data processing task.

[0194] The data processing task allocation unit 903 is configured to allocate target database resources for executing the data processing task to the at least one data processing task according to the data processing estimated duration.

[0195] Optionally, the database performance information of the target database comprises bandwidth information of the target database, and the data characteristic information of the at least one data processing task comprises data capacity information of the at least one data processing task.

[0196] The data processing estimated duration obtaining unit 902 is specifically configured to obtain data processing estimated duration of the at least one data processing task according to bandwidth information of the target database and data capacity information of the at least one data processing task.

[0197] Optionally, the obtaining of the data processing estimated duration of the at least one data processing task according to the bandwidth information of the target database and the data capacity information of the at least one data processing task comprises:

[0198] obtaining bandwidth corresponding to the bandwidth information of the target database;

[0199] obtaining data capacity corresponding to the data capacity information of the at least one data processing task;

[0200] obtaining the data processing estimated duration of the at least one data processing task according to a ratio between the data capacity corresponding to the data capacity information of the at least one data processing task and the bandwidth corresponding to the bandwidth information of the target database.

[0201] Optionally, the data processing task assigning unit 903 is specifically configured to create a target resource pool for providing resource load management service for database resources used for executing data processing tasks, associate the at least one data processing task to the target resource pool, and assign the target database resources to the at least one data processing task through the target resource pool according to the data processing estimated duration.

[0202] Optionally, the assigning of the target database resources to the at least one data processing task through the target resource pool according to the data processing estimated duration comprises: assigning the target database resources to the at least one data processing task in the database resources corresponding to the target resource pool in ascending order of the data processing estimated duration.

[0203] Optionally, the assigning of the target database resources to the at least one data processing task in the database resources corresponding to the target resource pool in ascending order of the data processing estimated duration comprises:

[0204] obtaining an available time interval of the database resources corresponding to the target resource pool;

[0205] judging whether the database resources corresponding to the target resource pool can complete the at least one data processing task in a serial manner within the available time interval;

[0206] If yes, the target database resources are allocated to the at least one data processing task in a serial manner in the database resources corresponding to the target resource pool in an order from small to large according to the estimated data processing time length.

[0207] Optionally, if the target database resources corresponding to the target resource pool cannot complete the at least one data processing task in a serial manner within the available time interval, the target database resources are allocated to the at least one data processing task in a concurrent manner in the database resources corresponding to the target resource pool in an order from small to large according to the estimated data processing time length.

[0208] Optionally, the data processing task includes at least a data integration task.

[0209] Fifth Embodiment

[0210] Corresponding to the data integration task allocation method provided by the first embodiment of the present application and the data processing task allocation method provided by the third embodiment of the present application, the fifth embodiment of the present application further provides an electronic device. Since the fifth embodiment is basically similar to the data integration task allocation method provided by the first embodiment of the present application and the data processing task allocation method provided by the third embodiment of the present application, the description is relatively simple, and the relevant part can be referred to the part of the description of the data integration task allocation method provided by the first embodiment of the present application and the data processing task allocation method provided by the third embodiment of the present application. The fifth embodiment described below is only illustrative.

[0211] Please refer to Figure 10 which is a schematic diagram of an electronic device provided in an embodiment of the present application.

[0212] The electronic device comprises a processor 1001;

[0213] and a memory 1002 for storing a computer program, which is run by the processor to execute the data integration task allocation method provided by the first embodiment of the present application and / or the data processing task allocation method provided by the first embodiment of the present application.

[0214] It should be noted that the detailed description of the electronic device provided by the fifth embodiment of the present application can refer to the relevant description of the data integration task allocation method provided by the first embodiment of the present application and the data processing task allocation method provided by the third embodiment of the present application, which will not be repeated here.

[0215] Sixth Embodiment

[0216] Corresponding to the data integration task allocation method provided by the first embodiment of the present application and the data processing task allocation method provided by the third embodiment of the present application, the sixth embodiment of the present application further provides a storage medium. Since the sixth embodiment is basically similar to the data integration task allocation method provided by the first embodiment of the present application and the data processing task allocation method provided by the third embodiment of the present application, the description is relatively simple, and the relevant parts can be referred to the part of the description of the data integration task allocation method provided by the first embodiment of the present application and the data processing task allocation method provided by the third embodiment of the present application. The device embodiments described below are only schematic.

[0217] The storage medium stores a computer program, and the computer program is run by a processor to execute the data integration task allocation method provided by the first embodiment of the present application and / or the data processing task allocation method provided by the first embodiment of the present application.

[0218] It should be noted that the detailed description of the storage medium provided by the sixth embodiment of the present application can refer to the related description of the data integration task allocation method provided by the first embodiment of the present application and the data processing task allocation method provided by the third embodiment of the present application, which will not be repeated here.

[0219] Seventh Embodiment

[0220] Corresponding to the data integration task allocation method provided by the first embodiment of the present application and the data processing task allocation method provided by the third embodiment of the present application, the seventh embodiment of the present application further provides a computer program product. Since the seventh embodiment is basically similar to the data integration task allocation method provided by the first embodiment of the present application and the data processing task allocation method provided by the third embodiment of the present application, the description is relatively simple, and the relevant parts can be referred to the part of the description of the data integration task allocation method provided by the first embodiment of the present application and the data processing task allocation method provided by the third embodiment of the present application. The device embodiments described below are only schematic.

[0221] The computer program product includes a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium contains computer program code for executing the data integration task allocation method provided by the first embodiment of the present application and the data processing task allocation method provided by the third embodiment of the present application.

[0222] It should be noted that the detailed description of the computer program product provided by the seventh embodiment of the present application can refer to the related description of the data integration task allocation method provided by the first embodiment of the present application and the data processing task allocation method provided by the third embodiment of the present application, which will not be repeated here.

[0223] Although the present application is disclosed with reference to the preferred embodiments above, it is not intended to limit the present application, and any person skilled in the art should understand that possible variations and modifications can be made without departing from the spirit and scope of the present application, and the scope of protection of the present application should be limited by the scope of the claims.

[0224] In one typical arrangement, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0225] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) having a common memory cache, and / or non-volatile memory, such as read-only memory (ROM), EPROM, and / or a flash memory (Flash RAM). The memory is an example of computer readable media.

[0226] 1. Computer readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology for storing information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other physical categories of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic disk storage or other magnetic storage media, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0227] 2. Those skilled in the art should understand that embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

Claims

1. A method for allocating a data integration task, characterized in that, The method comprises the following steps: obtaining at least one data integration task required to be executed in a target database, wherein the data integration task is a task created for data to be integrated in the target database; obtaining a data integration estimated duration of the at least one data integration task according to database performance information of the target database and data characteristic information of the at least one data integration task; allocating target database resources for executing data integration tasks to the at least one data integration task according to the data integration estimated duration; wherein the step of allocating target database resources for executing data integration tasks to the at least one data integration task according to the data integration estimated duration comprises the following steps: obtaining an available time interval of database resources corresponding to a target resource pool, wherein the target resource pool is used for executing the database resources of the data integration task; judging whether the database resources corresponding to the target resource pool can complete the at least one data integration task in a serial manner within the available time interval; if yes, allocating the target database resources to the at least one data integration task in the serial manner in the database resources corresponding to the target resource pool in the order from small to large of the data integration estimated duration; if the database resources corresponding to the target resource pool cannot complete the at least one data integration task in the serial manner within the available time interval, allocating the target database resources to the at least one data integration task in a concurrent manner in the database resources corresponding to the target resource pool in the order from small to large of the data integration estimated duration.

2. The method of claim 1, wherein, The method further comprises the following steps: obtaining identification information of the target database; obtaining database performance information of the target database according to the identification information of the target database and a first corresponding relationship between the identification information and the database performance information.

3. The method of claim 2, wherein, The step of obtaining at least one data integration task required to be executed in a target database comprises the following steps: obtaining a target data integration task list corresponding to the target database according to the identification information of the target database and a second corresponding relationship between the identification information and the data integration task list; obtaining the at least one data integration task according to the target data integration task list.

4. The method of claim 3, wherein, The step of obtaining the at least one data integration task according to the target data integration task list comprises the following steps: obtaining the at least one data integration task and data characteristic information of the at least one data integration task according to the target data integration task list.

5. The method of claim 1, wherein, The database performance information of the target database comprises bandwidth information of the target database, and the data characteristic information of the at least one data integration task comprises data capacity information of the at least one data integration task. The data integration estimated time length of the at least one data integration task is obtained according to the bandwidth information of the target database and the data capacity information of the at least one data integration task.

6. The method of claim 5, wherein, The data integration estimated time length of the at least one data integration task is obtained according to the bandwidth information of the target database and the data capacity information of the at least one data integration task. The bandwidth corresponding to the bandwidth information of the target database is obtained. The data capacity corresponding to the data capacity information of the at least one data integration task is obtained. The data integration estimated time length of the at least one data integration task is obtained according to the ratio between the data capacity corresponding to the data capacity information of the at least one data integration task and the bandwidth corresponding to the bandwidth information of the target database.

7. The method of claim 1, wherein, The target database resource for executing the data integration task is allocated to the at least one data integration task through the target resource pool according to the order of the data integration estimated time length, including: The target resource pool providing the database resource for executing the data integration task is created to provide resource load management service; The at least one data integration task is associated to the target resource pool; The target database resource is allocated to the at least one data integration task through the target resource pool according to the order of the data integration estimated time length.

8. The method of claim 7, wherein, The target database resource is allocated to the at least one data integration task through the target resource pool according to the order of the data integration estimated time length, including: the target database resource is allocated to the at least one data integration task in the database resource corresponding to the target resource pool in the order from small to large of the data integration estimated time length.

9. The method of claim 8, wherein, The target database resource is allocated to the at least one data integration task in the database resource corresponding to the target resource pool in the order from small to large of the data integration estimated time length, including: in the available time interval, the target database resource is allocated to the at least one data integration task in the database resource corresponding to the target resource pool in the order from small to large of the data integration estimated time length in a serial manner.

10. The method of claim 9, wherein, The target database resource is allocated to the at least one data integration task in the database resource corresponding to the target resource pool in the order from small to large of the data integration estimated time length, including: The first type of data integration task in the at least one data integration task and the second type of data integration task in the at least one data integration task are obtained, the first type of data integration task is a data integration task whose time length corresponding to the data integration estimated time length information does not reach a time length threshold, and the second type of data integration task is a data integration task whose time length corresponding to the data integration estimated time length information reaches the time length threshold. In the target resource pool, the target database resources are allocated to the first type of data integration tasks in a concurrent manner in the order of the data integration estimated time length from small to large. After the target database resources are allocated to the first type of data integration tasks, the target database resources are allocated to the second type of data integration tasks in a concurrent manner in the order of the data integration estimated time length from small to large.

11. The method of claim 7, wherein, The target resource pool for providing resource load management services for database resources used for executing data integration tasks is created, including: The historical database performance information of the target database in the database performance information of the target database is obtained. According to the historical database performance information of the target database, a target historical time interval in which the historical database performance of the target database meets a preset performance threshold is determined. The database resources used for executing data integration tasks are divided from the database resources of the target database in the same time interval as the target historical time interval. The target resource pool is created for the database resources used for executing data integration tasks.

12. The method of claim 11, wherein, The historical database performance information of the target database at least includes the remaining memory information of the target database and the utilization rate of the database server of the target database. According to the historical database performance information of the target database, a target historical time interval in which the historical database performance of the target database meets a preset performance threshold is determined.

13. An apparatus for allocating a data integration task, characterized by: including: A data integration task obtaining unit is configured to obtain at least one data integration task that needs to be executed in a target database, wherein the data integration task is a task created for to-be-integrated data in the target database. A data integration estimated time length obtaining unit is configured to obtain data integration estimated time length of the at least one data integration task according to database performance information of the target database and data characteristic information of the at least one data integration task. A data integration task allocation unit is configured to allocate target database resources used for executing data integration tasks to the at least one data integration task according to the data integration estimated time length. The data integration task allocation unit is further configured to acquire an available time interval of a database resource corresponding to a target resource pool, wherein the target resource pool is used to execute a database resource of the data integration task; determine whether the database resource corresponding to the target resource pool can complete the at least one data integration task in a serial manner within the available time interval; if yes, allocate the target database resource to the at least one data integration task in the serial manner in the database resource corresponding to the target resource pool in a descending order of the data integration estimated time length; and if the database resource corresponding to the target resource pool cannot complete the at least one data integration task in the serial manner within the available time interval, allocate the target database resource to the at least one data integration task in a concurrent manner in the database resource corresponding to the target resource pool in the descending order of the data integration estimated time length.

14. An electronic device, comprising: Comprise: a processor; and a memory for storing a computer program, which is run by the processor to execute the method of any one of claims 1-12.

15. A storage medium, characterized by The storage medium stores a computer program, which is run by the processor to execute the method of any one of claims 1-12.

16. A computer program product comprising a non-transitory computer readable storage medium having computer program code embodied therein, the computer program code configured to perform the method of any one of claims 1-12.

Images