Resource dynamic scheduling method and device, equipment, storage medium and program product

Abstract

The invention relates to a resource dynamic scheduling method and device, equipment, a storage medium and a program product, and the method comprises the steps: obtaining a first time length when a target task completes a job in a first stage, a second time length when the target task is distributed and delayed in a second stage, and an optimization target, the optimization target comprises a first target time interval and a second target time interval; when the first duration does not fall into the first target duration interval or the second duration does not fall into the second target duration interval, traversing the resource allocation nodes; according to the fragment type processed by the resource allocation node, obtaining a target residual estimated total time length or an average time consumption of a write-in record; obtaining a first comparison result between the target residual estimated total duration and a first target duration interval, or a second comparison result between the average time consumption and a second target duration interval; and determining a target strategy according to the first comparison result or the second comparison result, and updating resource configuration information associated with the resource allocation node.

Description

Technical Field

[0001] The present disclosure relates to the field of big data technology, and in particular to a method, device, equipment, storage medium and program product for dynamic resource scheduling. Background Art

[0002] Flink CDC (Change Data Capture) is a batch-stream integrated collection solution that can be used for transactional relational databases. The overall collection task is divided into a full phase and an incremental phase. The full phase uses JDBC (Java DataBase Connectivity) to query and obtain historical data through table-level primary key segments, and supports multi-segment parallel queries. The incremental phase is based on the master-slave synchronization mechanism to obtain real-time data through consumption changes, and only supports single parallel synchronization.

[0003] Flink's resource schedulers can be divided into default resource schedulers (such as DefaultScheduler) and adaptive resource schedulers (such as AdaptiveScheduler). Flink CDC hopes to use resources efficiently during collection, but the default scheduler has a fixed degree of parallelism after startup and cannot be adjusted automatically. The adaptive scheduler can dynamically adjust the degree of parallelism, but the job needs to be restarted during the operation phase. At the same time, the resource requirements of CDC jobs in the full and incremental phases are very different, and resources need to be automatically and efficiently scaled.

[0004] The current default scheduler and adaptive scheduler cannot optimize the collection time of the Flink CDC full-volume phase under the constraints of limited resources to ensure the timeliness of full-volume collection, and cannot redefine the resource allocation node specifications. Summary of the invention

[0005] In view of this, the present disclosure provides a method, apparatus, device, storage medium and program product for dynamic resource scheduling to solve the problem that the default scheduler and the adaptive scheduler cannot optimize the collection time of the full-volume phase of Flink CDC under the constraints of limited resources to ensure the timeliness of the full-volume collection, and cannot redefine the resource allocation node specifications.

[0006] In a first aspect, the present disclosure provides a method for dynamic resource scheduling, the method comprising:

[0007] Obtaining a first duration of the target task when the task is completed in the first stage, a second duration of the target task when the task is delayed in the second stage, and an optimization target, wherein the optimization target includes a first target time interval of any task when the task is completed in the first stage and a second target time interval of any task when the task is delayed in the second stage;

[0008] When the first duration does not fall within the first target duration interval or the second duration does not fall within the second target duration interval, traversing the resource allocation nodes, wherein the resource allocation nodes are used to calculate and allocate resources;

[0009] According to the shard type processed by the resource allocation node, the target remaining estimated total duration for completing the job when the target task is in the first stage or the average duration for writing records when the target task is in the second stage is obtained;

[0010] Obtain a first comparison result between the target remaining estimated total duration and the first target duration interval, or a second comparison result between the average duration and the second target duration interval;

[0011] According to the first comparison result or the second comparison result, a target policy is determined, and resource configuration information associated with the resource allocation node is updated.

[0012] In the disclosed embodiment, by setting the optimization target, the first duration of the target task when completing the job in the first stage is compared with the first target time interval in the optimization target, and the second duration of the target task when the distribution delay is in the second stage is compared with the second target time interval in the optimization target, and then the resource allocation node is obtained, and then the target strategy is determined according to the first comparison result of the target remaining estimated total duration and the first target duration interval and the second comparison result of the average time and the second target duration interval, so as to update the resource allocation information of the resource allocation node. By defining resource configuration files and optimization targets, and combining indicator monitoring and prediction, adaptive resource scheduling with job processing duration close to the target is achieved, which not only ensures processing timeliness, but also does not cause waste of resources. It successfully guides the elastic resource scheduling of Flink jobs and greatly improves job performance and resource utilization.

[0013] In an optional implementation, according to the shard type processed by the resource allocation node, obtaining the target remaining estimated total duration for completing the job when the target task is in the first stage or the average time consumed for writing records when the target task is in the second stage includes:

[0014] When the target task is still in the first stage, obtaining an average completion time of the current resource allocation node when processing the first shard type, wherein the current resource allocation node is included in the resource allocation node;

[0015] Determine the remaining estimated total time for the current resource allocation node to complete the job based on the average completion time and the remaining resource allocation nodes;

[0016] After traversing all resource allocation nodes, obtain the remaining estimated total time for each resource allocation node to complete the job;

[0017] According to the remaining estimated total time for each resource allocation node to complete the job, the target remaining estimated total time for the target task to complete the job is obtained;

[0018] or,

[0019] When the target task is in the second stage, the average time consumed to write records when the resource allocation node processes the second shard type is obtained.

[0020] In the disclosed embodiment, job processing progress and time consumption indicators are collected regularly to facilitate subsequent comparison with optimization goals and form different strategies to determine the resource adjustment sequence and adjustment range.

[0021] In an optional implementation, obtaining an average completion time when the current resource allocation node processes the first shard type includes:

[0022] Obtain multiple completion times corresponding to the current resource allocation node processing the first shard type;

[0023] Subtracting the preset value from the completion time to obtain a plurality of remaining completion times;

[0024] Get the average completion time based on the remaining completion time.

[0025] In an optional implementation, obtaining an average time consumed for writing records when the resource allocation node processes the second shard type includes:

[0026] Obtain multiple write record times corresponding to when the current resource allocation node processes the second shard type;

[0027] Subtracting a preset value from the write record time to obtain a plurality of remaining write record times;

[0028] The average time consumed is obtained based on the remaining write record time.

[0029] In an optional implementation, determining a target strategy according to the first comparison result or the second comparison result, and updating resource configuration information associated with the resource allocation node include:

[0030] When the target remaining estimated total duration is greater than or equal to the upper limit of the duration within the first target duration interval or less than the lower limit of the duration within the first target duration interval, the number of data processing task nodes, the number of resource allocation nodes, and the configuration parameters of the resource allocation nodes are updated according to the first target strategy, wherein a data processing task node includes multiple resource allocation nodes, and the target strategy includes the first target strategy;

[0031] or,

[0032] When the average duration is greater than or equal to the upper limit of the second target duration interval or less than the lower limit of the second target duration interval, the number of data processing task nodes, the number of resource allocation nodes and the configuration parameters of the resource allocation nodes are updated according to the second target strategy, wherein the target policy includes the second target policy.

[0033] In the disclosed embodiment, two types of resource optimization strategies are supported: resource shortage priority and time shortage priority. The resource shortage priority strategy pursues resource conservation, while the time shortage priority strategy pursues processing efficiency. The order of resource optimization under the two strategies is different to adapt to the demands of different business scenarios.

[0034] In an optional implementation, updating the number of data processing task nodes, the number of resource allocation nodes, and configuration parameters of the resource allocation nodes according to the first target strategy includes:

[0035] Obtaining the initial number of data processing task nodes, the initial number of resource allocation nodes and the initial configuration parameters of the resource allocation nodes, the upper limit of the number of data processing task nodes, the upper limit of the number of resource allocation nodes and the upper limit of the configuration parameters of the resource allocation nodes;

[0036] When the first target strategy is the resource shortage priority strategy, the initial configuration parameters, the initial number of resource allocation nodes, and the initial number of data processing task nodes are adjusted in sequence according to the upper limit of the number of data processing task nodes, the upper limit of the number of resource allocation nodes, and the upper limit of the configuration parameters of the resource allocation nodes, to obtain the adjusted configuration parameters of the resource allocation nodes, the number of resource allocation nodes, and the number of data processing task nodes;

[0037] When the first target strategy is the time priority strategy, the initial number of data processing task nodes, the initial number of resource allocation nodes and the initial configuration parameters are adjusted in sequence according to the upper limit of the number of data processing task nodes, the upper limit of the number of resource allocation nodes and the upper limit of the configuration parameters of the resource allocation nodes to obtain the adjusted number of data processing task nodes, the number of resource allocation nodes and the configuration parameters of the resource allocation nodes.

[0038] In an optional implementation, updating the number of data processing task nodes, the number of resource allocation nodes, and configuration parameters of the resource allocation nodes according to the second target strategy includes:

[0039] The data processing task nodes that have completed the task are released, and the number of idle data processing task nodes and the number of resource allocation nodes are adjusted to preset values;

[0040] Get the current configuration parameters of the resource allocation node;

[0041] The current configuration parameters are adjusted according to the duration upper limit and the duration lower limit within the second target duration interval to obtain the configuration parameters of the resource allocation node.

[0042] In a second aspect, the present disclosure provides a device for dynamic resource scheduling, the device comprising:

[0043] A first acquisition module is used to acquire a first duration of a target task when it is in the first stage to complete the job, a second duration of the target task when it is in the second stage to be distributed and an optimization target, wherein the optimization target includes a first target time interval of any task when it is in the first stage to complete the job and a second target time interval of any task when it is in the second stage to be distributed and delayed;

[0044] A traversal module, used for traversing resource allocation nodes when the first duration does not fall within the first target duration interval or the second duration does not fall within the second target duration interval, wherein the resource allocation nodes are used for calculating and allocating target tasks;

[0045] The second acquisition module is used to obtain the target remaining estimated total time for completing the job when the target task is in the first stage or the average time consumed for writing the record when the target task is in the second stage according to the shard type processed by the resource allocation node;

[0046] A third acquisition module is used to obtain a first comparison result between the target remaining estimated total duration and the first target duration interval, or a second comparison result between the average duration and the second target duration interval;

[0047] The updating module is used to determine the target strategy according to the first comparison result or the second comparison result, and to update the resource configuration information associated with the resource allocation node.

[0048] In a third aspect, the present disclosure provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for dynamic resource scheduling of the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0049] In a fourth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method for dynamic resource scheduling of the above-mentioned first aspect or any corresponding embodiment thereof.

[0050] In a fifth aspect, the present disclosure provides a computer program product, including computer instructions, which are used to enable a computer to execute the method for dynamic resource scheduling of the above-mentioned first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 is a flowchart of a method for dynamic resource scheduling according to an embodiment of the present disclosure;

[0053] Figure 2 This is a Flink CDC collection job architecture diagram according to an embodiment disclosed in the present invention;

[0054] Figure 3 It is a schematic diagram of the collection process of the full phase and incremental phase of Flink CDC according to the disclosed embodiment of the present invention;

[0055] Figure 4 It is a complete flow chart of Flink CDC resource dynamic scheduling according to the disclosed embodiment of the present invention;

[0056] Figure 5 is a structural block diagram of a device for dynamic resource scheduling according to an embodiment of the present disclosure;

[0057] Figure 6 It is a schematic diagram of the hardware structure of the computer device of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0059] In conjunction with the application scenarios on which the execution of the resource dynamic scheduling method depends, the application scenarios are described here.

[0060] The resource dynamic scheduling method of the disclosed embodiment is applicable to Flink CDC real-time collection jobs that need to frequently perform resource elastic scheduling according to business load fluctuations, especially for scenarios with obvious high and low peak business cycles and scenarios that are sensitive to job execution resources and time consumption.

[0061] Take the example of an e-commerce platform that needs to collect user browsing behavior data in real time. Since e-commerce business has obvious peak and off-peak periods, if the default scheduler and adaptive scheduler are used, processing delays will occur during peak periods and resource waste will occur during off-peak periods. To solve this problem, according to an embodiment of the present disclosure, a method embodiment of dynamic resource scheduling is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0062] In this embodiment, a method for dynamic resource scheduling is provided. Figure 1 is a flow chart of a method for dynamic resource scheduling according to an embodiment of the present disclosure, which can be applied to a server side, such as Figure 1 As shown, the method flow includes the following steps:

[0063] Step S101, obtaining a first duration of a target task when it is in the first stage to complete the job, a second duration of a target task when it is in the second stage to be distributed and an optimization target, wherein the optimization target includes a first target time interval of any task when it is in the first stage to complete the job and a second target time interval of any task when it is in the second stage to be distributed and delayed;

[0064] Step S102, when the first duration does not fall within the first target duration interval or the second duration does not fall within the second target duration interval, traverse the resource allocation nodes, wherein the resource allocation nodes are used to calculate and allocate resources;

[0065] Step S103, according to the shard type processed by the resource allocation node, obtain the target remaining estimated total duration of the target task when it is in the first stage to complete the job or the average time consumed to write the record when the target task is in the second stage;

[0066] Step S104, obtaining a first comparison result between the target remaining estimated total duration and the first target duration interval, or a second comparison result between the average duration and the second target duration interval;

[0067] Step S105: determine the target policy according to the first comparison result or the second comparison result, and update the resource configuration information associated with the resource allocation node.

[0068] Optionally, the disclosed embodiment refines the definition of resources based on the Flink adaptive scheduler and supplements the definition of optimization objectives. Resource definition: Referring to the design of the adaptive scheduler, improvements are made based on the ResourceProfile object. In addition to supporting the setting of upper and lower limits of job parallelism (determined by the total number of TaskManagers (data processing task nodes) * slots (resource allocation nodes)), it also supports the upper and lower limits of the number of slots and the total amount of memory in a single TaskManager of the job.

[0069] Optimization target: independently set a new configuration object Objective, which supports the upper and lower limits of the completion time of the full data processing stage (that is, the upper and lower limits corresponding to the first target time interval when any task completes the job in the first stage), and the upper and lower limits of the distribution delay in the incremental data processing stage (that is, the upper and lower limits corresponding to the second target time interval when any task is in the second stage of distribution delay). ResourceProfile is equivalent to an independent variable, while Objective is a dependent variable, such as Figure 2 As shown, compared with the related art, the embodiment of the present disclosure has an additional Job Requirement (task requirement module), which includes two objects: ResourceProfile and Objective. Figure 2 In addition to the task requirement module, other modules, such as Source (the starting point of the data flow, responsible for obtaining data from various sources and inputting it into the Flink program), FlinkCDC Job (Flink CDC task), Runtime Container (an important component of the Flink runtime framework, mainly responsible for task execution and data flow buffering and exchange), Sink (a key component of data flow processing, which is responsible for outputting processed data to external systems) are all existing technologies and will not be elaborated here.

[0070] For example, in the application scenario of real-time collection of user browsing behavior data, the disclosed embodiment sets the resource allocation ResourceProfile of the task based on historical data: the upper limit of resources is 10 degrees of parallelism, each TaskManager has a maximum of 2 slots and 8GB of memory, and the lower limit of resources is 4 degrees of parallelism, each TaskManager has a minimum of 1 slot and 2GB of memory. At the same time, the optimization target Objective of the job processing time is set: the upper limit of the full data processing time is 2 hours and the lower limit is 0, and the upper limit of the incremental data distribution delay is 100ms and the lower limit is 60ms.

[0071] It should be noted that the above process will be carried out in a fixed cycle, and a deviation threshold can be set for the optimization target to avoid repeated optimization under small fluctuations.

[0072] In the disclosed embodiment, the server regularly adjusts the configuration according to the current resource allocation status and the time consumption of the job. Furthermore, the server will periodically collect the first duration of the target task (such as the above-mentioned task of real-time collection of user browsing behavior data) when the job is completed in the first stage (i.e., the full data processing stage) (here, referring to the collection of user browsing behavior data), and the second duration of the target task when the distribution delay is in the second stage (i.e., the incremental data processing stage).

[0073] Then the first duration is compared with the first target duration interval, and the second duration is compared with the second target duration interval. If the first duration does not fall into the first target duration interval or the second duration does not fall into the second target duration interval, all resource allocation nodes in the target task are traversed. The resource allocation node here is the slot, which is the resource allocation unit in Flink. It is used to execute parallel computing tasks, represents the physical or virtual execution resources in the Flink cluster, and is usually associated with the data processing task node (TaskManager). Each TaskManager can contain one or more slots, the specific number is defined by the parameter taskmanager.numberOfTaskSlots, and each slot can execute a parallel subtask.

[0074] When traversing each slot, if it is still in the full collection stage, and the shard type processed by the current resource allocation node is SnapshotSplit, the target remaining estimated total duration for the target task to complete the job is obtained; when traversing each slot, if it is still in the incremental collection stage, and the shard type processed by the current resource allocation node is BinlogSplit, the average time consumed for the target task to write records is obtained.

[0075] Then, the target remaining estimated total duration is compared with the upper and lower limit values ​​of the first target duration interval to obtain a first comparison result, and the average duration is compared with the upper and lower limit values ​​of the second target duration interval to obtain a second comparison result. Then, based on the first comparison result, a target strategy is determined to change the resource configuration information of the resource allocation node, and based on the second comparison result, a target strategy is determined to change and adjust the resource configuration information related to the resource allocation node. Among them, the resource configuration information related to the resource allocation node includes the number of resource allocation nodes (slots), the memory of the resource allocation node, and the number of data processing task nodes (TaskManager).

[0076] In the disclosed embodiment, by setting the optimization target, the first duration of the target task when completing the job in the first stage is compared with the first target time interval in the optimization target, and the second duration of the target task when the distribution delay is in the second stage is compared with the second target time interval in the optimization target, and then the resource allocation node is obtained, and then the target strategy is determined according to the first comparison result of the target remaining estimated total duration and the first target duration interval and the second comparison result of the average time and the second target duration interval, so as to update the resource allocation information of the resource allocation node. By defining resource configuration files and optimization targets, and combining indicator monitoring and prediction, adaptive resource scheduling with job processing duration close to the target is achieved, which not only ensures processing timeliness, but also does not cause waste of resources. It successfully guides the elastic resource scheduling of Flink jobs and greatly improves job performance and resource utilization.

[0077] In some optional implementations, according to the shard type processed by the resource allocation node, obtaining the target remaining estimated total duration for completing the job when the target task is in the first stage or the average time consumed for writing records when the target task is in the second stage includes:

[0078] When the target task is still in the first stage, obtaining an average completion time of the current resource allocation node when processing the first shard type, wherein the current resource allocation node is included in the resource allocation node;

[0079] Determine the remaining estimated total time for the current resource allocation node to complete the job based on the average completion time and the remaining resource allocation nodes;

[0080] After traversing all resource allocation nodes, obtain the remaining estimated total time for each resource allocation node to complete the job;

[0081] According to the remaining estimated total time for each resource allocation node to complete the job, the target remaining estimated total time for the target task to complete the job is obtained;

[0082] or,

[0083] When the target task is in the second stage, the average time consumed to write records when the resource allocation node processes the second shard type is obtained.

[0084] Optionally, if the target task is currently in the full collection stage, when processing the first shard type of SnapshotSplit, obtain the average completion time of the current resource allocation node when processing the first shard type, and then obtain the number of remaining resource allocation nodes in the resource allocation node except the current resource allocation node. At this time, the remaining estimated total duration of the current resource allocation node = the remaining number of slots to be collected × the average completion time (the remaining number of slots to be collected can be determined when the SourceReader partition is queried in the full stage. If there are additions or deletions to TaskManager, re-collection is required).

[0085] After traversing all resource allocation nodes of the job, the remaining estimated total duration for each resource allocation node to complete the job is obtained, and then the maximum estimated duration is taken as the target remaining estimated total duration.

[0086] If the target task is already in the incremental collection phase, when the resource allocation node processes the second shard type of BinlogSplit, obtain the average time consumed to write the current record.

[0087] In the disclosed embodiment, job processing progress and time consumption indicators are collected regularly to facilitate subsequent comparison with optimization goals and form different strategies to determine the resource adjustment sequence and adjustment range.

[0088] In some optional implementations, obtaining an average completion time when the current resource allocation node processes the first shard type includes:

[0089] Obtain multiple completion times corresponding to the current resource allocation node processing the first shard type;

[0090] Subtracting the preset value from the completion time to obtain a plurality of remaining completion times;

[0091] Get the average completion time based on the remaining completion time.

[0092] Optionally, when determining the average completion time of the current resource node processing the first shard type, the average completion time can be obtained by taking out the corresponding completion times of the most recent n current resource allocation nodes processing the first shard type, removing the preset values ​​therein, such as the maximum and minimum values ​​(of course, some set values ​​can also be removed), and then averaging the multiple remaining completion times.

[0093] In some optional implementations, obtaining an average time consumed by a resource allocation node to write a record when processing the second shard type includes:

[0094] Obtain multiple write record times corresponding to when the current resource allocation node processes the second shard type;

[0095] Subtracting a preset value from the write record time to obtain a plurality of remaining write record times;

[0096] The average time consumed is obtained based on the remaining write record time.

[0097] Optionally, when determining the average time consumed by the current resource node to process writes of the second shard type, the average time can be obtained by taking the completion time of the most recent n write records, removing preset values ​​therein, such as the maximum and minimum values ​​(of course, some set values ​​can also be removed), and then averaging the remaining multiple write record times.

[0098] In some optional implementations, determining a target strategy according to the first comparison result or the second comparison result, and updating resource configuration information associated with the resource allocation node include:

[0099] When the target remaining estimated total duration is greater than or equal to the upper limit of the duration within the first target duration interval or less than the lower limit of the duration within the first target duration interval, the number of data processing task nodes, the number of resource allocation nodes, and the configuration parameters of the resource allocation nodes are updated according to the first target strategy, wherein a data processing task node includes multiple resource allocation nodes, and the target strategy includes the first target strategy;

[0100] or,

[0101] When the average duration is greater than or equal to the upper limit of the second target duration interval or less than the lower limit of the second target duration interval, the number of data processing task nodes, the number of resource allocation nodes and the configuration parameters of the resource allocation nodes are updated according to the second target strategy, wherein the target policy includes the second target policy.

[0102] Optionally, the target remaining estimated total duration obtained for the resource allocation node under the first sharding type is compared with the upper and lower limits of the first target duration interval. If the target remaining estimated total duration is greater than or equal to the upper limit of the first target duration interval or less than the lower limit of the first target duration interval, it is necessary to call the first target strategy to adjust the resource configuration information associated with the resource allocation node, including adjusting the number of data processing task nodes, the number of resource allocation nodes, and the configuration parameters of the resource allocation nodes (such as memory).

[0103] The average time consumed in writing records obtained by the resource allocation node under the second sharding type is compared with the upper and lower limits of the second target duration interval. If the average time consumed is greater than or equal to the upper limit of the second target duration interval or less than the lower limit of the second target duration interval, it is necessary to call the second target strategy to adjust the resource configuration information associated with the resource allocation node, including adjusting the number of data processing task nodes, the number of resource allocation nodes, and the configuration parameters of the resource allocation nodes (such as memory).

[0104] It should be noted that the first target strategy includes a resource shortage priority strategy and a time shortage priority strategy. If the resource shortage priority strategy is adopted, the memory usage of the unit slot (i.e., the initial configuration parameters) is first determined through binary search, and the initial number of resource allocation nodes is adjusted without exceeding the upper limit of the number of data processing task nodes. If the adjusted initial number of resource allocation nodes exceeds the upper limit of the configuration parameters of the resource allocation nodes, the number of data processing task nodes is adjusted, and finally the number of data processing task nodes is determined.

[0105] If the time-criticality priority strategy is adopted, the number of data processing task nodes is determined first, and then the number of resource allocation nodes is confirmed based on the upper limit of the number of data processing task nodes, the upper limit of the number of resource allocation nodes, and the upper limit of the configuration parameters of the resource allocation nodes. Finally, the memory usage is fine-tuned according to the configuration parameters, such as memory utilization.

[0106] For example, when the job is first started, the initial resource allocation strategy of resource shortage priority is adopted. This strategy prioritizes resource conservation and allocates only the minimum necessary resources. Specifically, one TaskManager is started in the full data processing stage, and each TaskManager is allocated one slot and 2GB of memory. After the job is started, the optimizer begins to regularly monitor the processing progress and time consumption of each Task. Through estimation analysis, it is found that under the current resource conditions, the full data processing is predicted to take another 3 hours to complete, which has exceeded the set upper limit of 2 hours. Therefore, the optimizer uses the resource shortage priority strategy and binary search to step by step confirm that the optimal memory is 3GB under the premise of 1 slot parallelism, but the full processing time still cannot meet the requirements. The slot configuration is increased to 2, and the memory needs to be adjusted to 6GB (<8GB) at this time, but the processing time still cannot meet the requirements. Then the number of TaskManagers is increased to 2 (with a parallelism of 4, <10), which controls the full data processing time within 2 hours.

[0107] In the disclosed embodiment, two types of resource optimization strategies are supported: resource shortage priority and time shortage priority. The resource shortage priority strategy pursues resource conservation, while the time shortage priority strategy pursues processing efficiency. The order of resource optimization under the two strategies is different to adapt to the demands of different business scenarios.

[0108] In some optional implementations, updating the number of data processing task nodes, the number of resource allocation nodes, and configuration parameters of the resource allocation nodes according to the second target strategy includes:

[0109] The data processing task nodes that have completed the task are released, and the number of idle data processing task nodes and the number of resource allocation nodes are adjusted to preset values;

[0110] Get the current configuration parameters of the resource allocation node;

[0111] The current configuration parameters are adjusted according to the duration upper limit and the duration lower limit within the second target duration interval to obtain the configuration parameters of the resource allocation node.

[0112] Optionally, when the job enters the incremental data processing phase, the optimizer detects that the TaskManager that has completed the task can be released due to the decrease in business volume. The idle TaskManagers are gradually reduced to a preset value, such as 1, and the slot of each TaskManager also needs to be reduced to 1. The job parallelism is further modified to 1. At this time, the 100ms delay requirement of the second target duration interval of the incremental processing phase is met, and resource utilization is also improved.

[0113] However, the distribution delay is reduced to 50ms. At this time, the storage and computing power can be controlled as long as the lower limit of the duration in the second target duration interval is above 60ms. Therefore, the current configuration parameters (i.e. memory) are fine-tuned: the memory is finally reduced to 4GB through binary search, and the average delay is within the interval. As the business further enters the off-peak period, the memory in the incremental processing stage is reduced to 2GB, saving resources to the greatest extent.

[0114] In some optional embodiments, such as Figure 3 As shown, Figure 3 : This is a schematic diagram of the collection process of the Flink CDC full phase and incremental phase according to the disclosed embodiment of the present invention. The specific process is as follows:

[0115] Collect indicators from TaskManager, process them, estimate the duration, obtain the first duration of the indicator when the job is completed in the first stage, and the second duration when the distribution is delayed in the second stage, then obtain the optimization goals and resource definitions in the job requirements, compare the first duration, the second duration with the optimization goals, and then adjust the strategy based on the resource definition, optimization goals, the average time consumed in writing records in the incremental collection phase obtained by the splitter, and the target remaining estimated total duration of the full collection phase obtained by the splitter, and obtain the updated number of TaskManagers and slot resource configuration information (including the number and memory), which are output through the data output component.

[0116] In some optional embodiments, such as Figure 4 As shown, Figure 4 This is a complete flow diagram of Flink CDC resource dynamic scheduling according to the disclosed embodiment of the present invention. The specific process is as follows:

[0117] start.

[0118] Indicators are collected regularly and the targets are checked; it should be noted that the target here refers to the optimization target in the above embodiment.

[0119] Compare the collected index with the target threshold in the target. If the threshold is not met, traverse to obtain the slot. If it is met, continue to collect the index regularly. It should be noted that the target threshold here refers to the upper and lower limits of the optimization target, such as the upper and lower limits of the completion time of the full data processing stage, and the upper and lower limits of the distribution delay in the incremental data processing stage.

[0120] Determine the shard type; if it is still in the full collection stage, when processing SnapshotSplit, take out the completion time of the most recent n SnapshotSplits, remove the maximum and minimum values ​​and take the average; estimate the total remaining time required for the current slot: the number of remaining slots to be collected × the average completion time.

[0121] Determine whether the traversal of all slots is completed; if so, take the maximum estimated duration as the remaining total duration required; if not, continue to traverse to obtain slots.

[0122] If the remaining total time required exceeds the upper limit of the time required to complete the full data processing stage or is lower than the lower limit of the time required to complete the full data processing stage, select the priority strategy to adjust the current resources of the slot.

[0123] If the resource shortage priority strategy is adopted, the memory usage of each slot is first determined through binary search, the number of slots is adjusted without exceeding the memory resources of a single TaskManager, and finally the number of TaskManagers is determined; if the time shortage priority strategy is adopted, the number of TaskManagers is determined first, then the number of slots is confirmed, and finally the memory usage is fine-tuned according to the memory utilization. Finally, the updated slot resource configuration is obtained.

[0124] If the current phase is incremental collection, when processing BinlogSplit, take the completion time of the most recent n write records, remove the maximum and minimum values, and calculate the average time consumption.

[0125] If the average time consumption exceeds the upper limit of the incremental data processing phase distribution delay or is lower than the lower limit of the incremental data processing phase distribution delay, the total number of completed idle TaskManagers that are automatically recycled will drop to 1. At this time, the slot also needs to be adjusted to 1, and the memory usage is determined through binary search. Finally, the updated slot resource configuration is obtained.

[0126] In the disclosed embodiment, by defining resource configuration files and optimization goals, regularly collecting job processing progress and time consumption indicators, predicting the remaining time consumption based on comparison of the optimization goals, and determining the resource adjustment sequence and adjustment range according to different strategies, a closed loop of indicator feedback and resource adjustment is formed, thereby achieving adaptive resource scheduling with job processing time close to the target, thereby ensuring processing efficiency and achieving rational use of resources.

[0127] In the present embodiment, a device for dynamic resource scheduling is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0128] This embodiment provides a device for dynamic resource scheduling, such as Figure 5 As shown, including:

[0129] The first acquisition module 501 is used to acquire a first duration of the target task when the task is completed in the first stage, a second duration of the target task when the task is delayed in the second stage, and an optimization target, wherein the optimization target includes a first target time interval of any task when the task is completed in the first stage and a second target time interval of any task when the task is delayed in the second stage;

[0130] A traversal module 502, configured to traverse a resource allocation node when the first duration does not fall within the first target duration interval or the second duration does not fall within the second target duration interval, wherein the resource allocation node is used to calculate and allocate a target task;

[0131] The second acquisition module 503 is used to acquire the target remaining estimated total duration of the target task in the first stage of completing the job or the average time consumed for writing the record of the target task in the second stage according to the shard type processed by the resource allocation node;

[0132] The third acquisition module 504 is used to obtain a first comparison result between the target remaining estimated total duration and the first target duration interval, or a second comparison result between the average duration and the second target duration interval;

[0133] The updating module 505 is used to determine the target strategy according to the first comparison result or the second comparison result, and update the resource configuration information associated with the resource allocation node.

[0134] In some optional embodiments, the second acquisition module 503 is also used to obtain the average completion time of the current resource allocation node when processing the first shard type when the target task is still in the first stage, wherein the current resource allocation node is included in the resource allocation node; determine the remaining estimated total time for the current resource allocation node to complete the job based on the average completion time and the remaining resource allocation nodes; after traversing the resource allocation nodes, obtain the remaining estimated total time for each resource allocation node to complete the job; based on the remaining estimated total time for each resource allocation node to complete the job, obtain the target remaining estimated total time for the target task to complete the job; or, when the target task is in the second stage, obtain the average time consumed to write records when the resource allocation node processes the second shard type.

[0135] In some optional implementations, the second acquisition module 503 is also used to obtain multiple completion times corresponding to the current resource allocation node processing the first shard type; remove the preset value from the completion time to obtain multiple remaining completion times; and obtain the average completion time based on the remaining completion time.

[0136] In some optional embodiments, the second acquisition module 503 is also used to obtain multiple write record times corresponding to the current resource allocation node processing the second shard type; remove the preset value from the write record time to obtain multiple remaining write record times; and obtain the average time consumption based on the remaining write record time.

[0137] In some optional embodiments, the update module 505 is used to update the number of data processing task nodes, the number of resource allocation nodes, and the configuration parameters of the resource allocation nodes according to the first target policy when the target remaining estimated total duration is greater than or equal to the upper limit of the duration within the first target duration interval or less than the lower limit of the duration within the first target duration interval, wherein one data processing task node includes multiple resource allocation nodes, and the target policy includes the first target policy; or, when the average duration is greater than or equal to the upper limit of the duration within the second target duration interval or less than the lower limit of the duration within the second target duration interval, update the number of data processing task nodes, the number of resource allocation nodes, and the configuration parameters of the resource allocation nodes according to the second target policy, wherein the target policy includes the second target policy.

[0138] In some optional implementations, the update module 505 is also used to obtain the initial number of data processing task nodes, the initial number of resource allocation nodes and the initial configuration parameters of the resource allocation nodes, the upper limit of the number of data processing task nodes, the upper limit of the number of resource allocation nodes and the upper limit of the configuration parameters of the resource allocation nodes; when the first target strategy is a resource shortage priority strategy, the initial configuration parameters, the initial number of resource allocation nodes and the initial number of data processing task nodes are adjusted in sequence according to the upper limit of the number of data processing task nodes, the upper limit of the number of resource allocation nodes and the upper limit of the configuration parameters of the resource allocation nodes to obtain the adjusted configuration parameters of the resource allocation nodes, the number of resource allocation nodes and the number of data processing task nodes; when the first target strategy is a time shortage priority strategy, the initial number of data processing task nodes, the initial number of resource allocation nodes and the initial configuration parameters are adjusted in sequence according to the upper limit of the number of data processing task nodes, the upper limit of the number of resource allocation nodes and the upper limit of the configuration parameters of the resource allocation nodes to obtain the adjusted number of data processing task nodes, the number of resource allocation nodes and the configuration parameters of the resource allocation nodes.

[0139] In some optional embodiments, the update module 505 is also used to release the data processing task nodes that have completed the job tasks, adjust the number of idle data processing task nodes and the number of resource allocation nodes to preset values; obtain the current configuration parameters of the resource allocation node; adjust the current configuration parameters according to the upper and lower duration limits within the second target duration interval, and obtain the configuration parameters of the resource allocation node.

[0140] The functional description of each of the above modules is the same as that of the above corresponding embodiments and will not be repeated here.

[0141] The present disclosure also provides a computer device having the above Figure 5 The resource dynamic scheduling device is shown.

[0142] See also Figure 6 , Figure 6 is a schematic diagram of a computer device provided by an optional embodiment of the present disclosure, such as Figure 6 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 10 is taken as an example.

[0143] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0144] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0145] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0146] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0147] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0148] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0149] A part of the present disclosure may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present disclosure through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0150] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for dynamic resource scheduling, characterized in that: The method comprises: Obtaining a first duration of a target task when completing the operation in the first stage, a second duration of the target task when being delayed in the second stage, and an optimization target, wherein the optimization target includes a first target time interval of any task when completing the operation in the first stage and a second target time interval of any task when being delayed in the second stage; When the first duration does not fall within the first target duration interval or the second duration does not fall within the second target duration interval, traverse a resource allocation node, wherein the resource allocation node is used to calculate and allocate resources; According to the shard type processed by the resource allocation node, the target remaining estimated total duration for completing the job when the target task is in the first stage or the average time consumed for writing records when the target task is in the second stage is obtained; Obtaining a first comparison result between the target remaining estimated total duration and the first target duration interval, or a second comparison result between the average duration and the second target duration interval; According to the first comparison result or the second comparison result, a target policy is determined, and resource configuration information associated with the resource allocation node is updated.

2. The method according to claim 1, characterized in that The obtaining, according to the shard type processed by the resource allocation node, the target remaining estimated total duration for completing the job when the target task is in the first stage or the average time consumed for writing records when the target task is in the second stage includes: When the target task is still in the first stage, obtaining an average completion time of a current resource allocation node when processing a first slice type, wherein the current resource allocation node is included in the resource allocation node; Determine the remaining estimated total time for the current resource allocation node to complete the job based on the average completion time and the remaining resource allocation nodes; After traversing the resource allocation nodes, obtaining the remaining estimated total time for each resource allocation node to complete the job; According to the remaining estimated total time for each resource allocation node to complete the job, the target remaining estimated total time for the target task to complete the job is obtained; or, When the target task is in the second stage, an average time consumed for writing records when the resource allocation node processes the second shard type is obtained.

3. The method according to claim 2, characterized in that The obtaining the average completion time when the current resource allocation node processes the first shard type includes: Obtaining multiple completion times corresponding to when the current resource allocation node processes the first shard type; Subtracting a preset value from the completion time to obtain a plurality of remaining completion times; The average completion time is obtained according to the remaining completion time.

4. The method according to claim 2, characterized in that: The obtaining an average time consumption for writing records when the resource allocation node processes the second shard type includes: Obtaining a plurality of write record times corresponding to when the current resource allocation node processes the second shard type; Subtracting a preset value from the write record time to obtain a plurality of remaining write record times; The average time consumption is obtained according to the remaining writing record time.

5. The method according to claim 1, characterized in that The determining a target strategy according to the first comparison result or the second comparison result, and updating resource configuration information associated with the resource allocation node, includes: When the target remaining estimated total duration is greater than or equal to the upper limit of the duration within the first target duration interval or less than the lower limit of the duration within the first target duration interval, the number of data processing task nodes, the number of resource allocation nodes and the configuration parameters of the resource allocation nodes are updated according to the first target strategy, wherein one data processing task node includes multiple resource allocation nodes, and the target strategy includes the first target strategy; or, When the average duration is greater than or equal to the upper limit of the second target duration interval or less than the lower limit of the second target duration interval, the number of data processing task nodes, the number of resource allocation nodes and the configuration parameters of the resource allocation nodes are updated according to the second target strategy, wherein the target strategy includes the second target strategy.

6. The method according to claim 5, characterized in that The updating of the number of data processing task nodes, the number of resource allocation nodes and the configuration parameters of the resource allocation nodes according to the first target strategy includes: Obtaining the initial number of the data processing task nodes, the initial number of the resource allocation nodes and the initial configuration parameters of the resource allocation nodes, the upper limit of the number of the data processing task nodes, the upper limit of the number of the resource allocation nodes and the upper limit of the configuration parameters of the resource allocation nodes; In the case where the first target strategy is a resource shortage priority strategy, the initial configuration parameters, the initial number of resource allocation nodes, and the initial number of data processing task nodes are adjusted in sequence according to the upper limit of the number of data processing task nodes, the upper limit of the number of resource allocation nodes, and the upper limit of the configuration parameters of the resource allocation nodes to obtain the adjusted configuration parameters of the resource allocation nodes, the number of resource allocation nodes, and the number of data processing task nodes; In the case where the first target strategy is a time priority strategy, the initial number of data processing task nodes, the initial number of resource allocation nodes and the initial configuration parameters are adjusted in sequence according to the upper limit of the number of data processing task nodes, the upper limit of the number of resource allocation nodes and the upper limit of the configuration parameters of the resource allocation nodes to obtain the adjusted number of data processing task nodes, the number of resource allocation nodes and the configuration parameters of the resource allocation nodes.

7. The method according to claim 5, characterized in that The updating of the number of the data processing task nodes, the number of the resource allocation nodes and the configuration parameters of the resource allocation nodes according to the second target strategy includes: The data processing task nodes that have completed the job tasks are released, and the number of idle data processing task nodes and the number of resource allocation nodes are adjusted to preset values; Obtaining current configuration parameters of the resource allocation node; The current configuration parameters are adjusted according to the upper and lower duration limits within the second target duration interval to obtain the configuration parameters of the resource allocation node.

8. A device for dynamic resource scheduling, characterized in that: The device comprises: A first acquisition module is used to acquire a first duration of a target task when it is in the first stage to complete the job, a second duration of the target task when it is in the second stage to be distributed and an optimization target, wherein the optimization target includes a first target time interval when any task is in the first stage to complete the job and a second target time interval when any task is in the second stage to be distributed and delayed; A traversal module, configured to traverse a resource allocation node when the first duration does not fall within the first target duration interval or the second duration does not fall within the second target duration interval, wherein the resource allocation node is used to calculate and allocate the target task; A second acquisition module is used to acquire, according to the shard type processed by the resource allocation node, the target remaining estimated total time for completing the job when the target task is in the first stage or the average time consumed for writing records when the target task is in the second stage; A third acquisition module, used to obtain a first comparison result between the target remaining estimated total duration and the first target duration interval, or a second comparison result between the average duration and the second target duration interval; An updating module is used to determine a target strategy according to the first comparison result or the second comparison result, and to update resource configuration information associated with the resource allocation node.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for dynamic resource scheduling according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method for dynamic resource scheduling according to any one of claims 1 to 7.

Images