Data platform task time planning method, equipment and medium

Abstract

The invention relates to the technical field of platform task scheduling, and discloses a data platform task time planning method and device and a medium, and the method comprises the steps: determining the task type of a new task and a dependent task at the upstream of the new task when the new task of a data platform is received at a specified time; on the basis of the task type, determining resource information and slot position information required by operation of the new task; obtaining the latest completion time of the dependent task; and based on the latest completion time, the specified time, the resource information and the slot position information, performing resource matching of the corresponding time period in a pre-constructed available resource network of each time period, and determining the optimal execution time of the new task. According to the method, the first future time window which simultaneously meets the requirements of dependency ready and resource sufficiency is directly matched in the available resource network, and the mechanism can skip a fluctuation link of real-time resource detection in traditional scheduling, so that the task can be quickly linked from ready to start.

Description

Technical Field

[0001] This specification relates to the technical field of platform task scheduling, and in particular to a data platform task time planning method, device, and medium. Background Art

[0002] As enterprises deepen their digital transformation, data has become a core asset. As the hub of enterprise-level data capabilities, the data center's core component, the data platform, is responsible for integrating, processing, and analyzing massive amounts of data from various business systems and domains. One of the platform's core functions is to efficiently and reliably schedule and execute massive amounts of data processing tasks, including offline batch processing and near-real-time tasks (such as hourly tasks), providing timely and accurate data support for downstream reporting, analysis, decision support, and other applications.

[0003] However, with the continued expansion of business scale and increasingly complex data requirements, the existing data platform task scheduling system is facing severe challenges. The platform needs to handle a large number of offline batch tasks and hourly near-real-time tasks on a daily basis. This massive task volume places extremely high demands on the platform's task scheduling capabilities. Summary of the Invention

[0004] In view of the above problems, the present disclosure provides a data platform task time planning method, device and medium, which are used to solve the technical problem of the lack of effective task time planning capability in the data platform of the data middle station due to the large number of tasks. The technical solution is as follows:

[0005] A data platform task time planning method, comprising:

[0006] When a new task is received from the data platform at a specified time, the task type of the new task and the upstream dependent tasks of the new task are determined;

[0007] Based on the task type, determining resource information and slot information required for running the new task;

[0008] Get the latest completion time of the dependent task;

[0009] Based on the latest completion time, the designated time, the resource information, and the slot information, an optimal execution time of the new task is determined in a pre-constructed available resource network for each time period.

[0010] It should be noted that this application builds an available resource network based on the slots and resource usage distribution in historical time periods. At the moment a new task arrives, it can be combined with the latest completion time of its upstream dependencies and its own resource requirements to directly match the first future time window in the resource network that satisfies both dependency readiness and resource sufficiency. This mechanism skips the volatility of real-time resource detection in traditional scheduling, allowing tasks to be quickly connected from readiness to startup. At the same time, as the business scale expands, the periodic laws of the historical resource network can continuously enhance prediction accuracy, thereby forming a steady-state scheduling capability that can resist a surge in task volume, and ultimately ensure that time-sensitive data processing such as hourly tasks is always delivered within a deterministic time window.

[0011] Optionally, before matching resources for the corresponding time period in the pre-built available resource network for each time period and determining the optimal execution time of the new task, the method further includes:

[0012] Collect the slot usage distribution and running resource usage distribution in each time period;

[0013] Based on the slot usage distribution and the running resource usage distribution in each time period, an available resource network for each time period is constructed.

[0014] It's important to note that this application builds a time-based available resource network based on the system's continuous collection of historical slot and resource usage distribution. This essentially transforms the periodic fluctuations of cluster resources into a calculable, deterministic supply capacity. Before a new task is scheduled, this network already reveals the probability of resource idleness in each future time period, enabling the subsequent calculation of optimal execution times based on dependency time and resource demand to provide forward-looking scheduling capabilities.

[0015] Optionally, matching resources for corresponding time periods in a pre-built available resource network for each time period based on the latest completion time, the specified time, the resource information, and the slot information to determine the optimal execution time of the new task includes:

[0016] Determine whether the latest completion time is less than the specified time;

[0017] If so, based on the resource information and the slot information, scanning the available resource network to see whether the slot usage and the running resource usage in the time period of the specified time meet the preset requirements;

[0018] If so, the specified time is used as the optimal execution time of the new task.

[0019] It's important to note that when the latest completion time of an upstream dependent task is earlier than the current time, the system immediately triggers a check on resource availability for the current period. The resource network, built based on historical patterns, is pre-loaded with the normal slots and resource status for that period. By scanning this available resource network, we can instantly confirm whether the task's resource requirements are currently met. If the pre-set conditions are met, the current time is directly locked as the optimal execution time.

[0020] Optionally, if the slot usage and running resource usage in the time period of the specified time in the available resource network do not meet preset requirements, the method further includes:

[0021] Delaying the specified time based on a preset time window to obtain a delay time;

[0022] Scan the available resource network to determine whether the slot usage and running resource usage in the time period of the delay time meet the preset requirements;

[0023] If so, the delay time is used as the optimal execution time of the new task.

[0024] It should be noted that when the system detects that the resources at the current moment do not meet the demand (such as the new task dependency is ready but the slot is temporarily tight), this application does not adopt a fixed-interval polling or passive waiting strategy, but instead uses the available resource network to directionally detect the next high-confidence idle period of resources along the time axis. For example, 30 minutes is preset as the time window step, and after the specified time is moved from 02:00 to 02:30, the historical rules of this period in the available resource network are immediately scanned. Once the demand is matched, 02:30 is immediately anchored as the optimal time to avoid invalid waiting of more than half an hour in the traditional retry mechanism. In the case of a large number of tasks being scheduled on a daily basis, this mechanism enables each blocked task to regain the deterministic execution window with only a small amount of computing power, ultimately ensuring that all time-sensitive tasks always glide forward on a predictable time track, supporting the uninterrupted downstream business decision-making chain.

[0025] Optionally, before determining the optimal execution time of the new task, the method further includes: matching resources for the corresponding time period in a pre-built available resource network for each time period based on the latest completion time, the specified time, the resource information, and the slot information; and:

[0026] Based on historical execution records, predict the runtime required for the new task;

[0027] The matching of resources for the corresponding time period in the pre-built available resource network for each time period based on the latest completion time, the designated time, the resource information, and the slot information to determine the optimal execution time of the new task includes:

[0028] Determine whether the latest completion time is less than the specified time;

[0029] If yes, determine one or more time periods occupied by the new task based on the specified time and the running time, and scan the slot usage and running resource usage of the one or more time periods in the available resource network based on the resource information and the slot information to determine whether they all meet the preset usage requirements;

[0030] If so, the specified time is used as the optimal execution time of the new task.

[0031] It should be noted that this application predicts the running time of new tasks based on historical execution records, and starts the verification of continuous supply of resources across time periods when the latest completion time of upstream dependencies is earlier than the current moment. Based on the available resource network, the slots and resource history patterns of each segment within the time window required for the task from start to finish are scanned to see if they meet the preset requirements. Only when the certainty of resource supply throughout the process is met can the current moment be set as the optimal execution time to avoid unexpected interruptions caused by resource exhaustion during the execution of the new task.

[0032] Optionally, if at least one time period of the slot usage and the running resource usage in the one or more time periods in the available resource network does not meet the preset requirements, the method further includes:

[0033] Scanning the slot usage and running resource usage of each time period after the specified time in the available resource network to obtain an execution time period closest to the specified time in which one or more consecutive time periods all meet the preset usage requirements;

[0034] A designated execution time corresponding to the execution time period is determined, and the designated execution time is used as the optimal execution time of the new task.

[0035] It should be noted that this application detects resource gaps across multiple time periods spanning the task's execution time and immediately initiates a directional scan of the resource network model along the time axis. Based on historical slot and resource distribution patterns, it quickly locates the first nearest time window that can meet continuous resource supply requirements throughout the task. In business scenarios with an average of tens of thousands of tasks per day and competing resources, this mechanism ensures that hourly critical tasks maintain sufficient operating resources and slots.

[0036] Optionally, if at least one time period of the slot usage and the running resource usage in the one or more time periods in the available resource network does not meet the preset requirements, the method further includes:

[0037] Identify specific time periods during which pre-set requirements are not met;

[0038] The designated time is used as the optimal execution time of the new task, and the execution of the new task is suspended during the specific time period.

[0039] It should be noted that this application detects that there are resource gaps in local time periods during the long-term execution of tasks. This application does not postpone the start time of the entire task, but splits the task execution process into discontinuous units. Start the task in the time period that currently meets the conditions, and actively pause when entering a high-risk period, and automatically resume when the available resource network shows that the subsequent time period is safe. In scenarios where resources fluctuate drastically during business peak hours, this mechanism can enable hourly time-sensitive tasks to make full use of fragmented resource gaps to advance calculations, ensure that full data processing is completed before the agreed delivery time point, and support the pressure-resistant operation of the downstream business decision-making chain.

[0040] Optionally, when the new task involves multiple tasks, and the slot usage and running resource usage in the one or more time periods in the available resource network do not meet preset requirements in at least one time period, the method further includes:

[0041] Determine the priority of each new task;

[0042] Identify specific time periods during which pre-set requirements are not met;

[0043] Scanning the slot usage and running resource usage in the available resource network during the specific time period to determine whether both meet the preset usage requirements of the new high-priority task;

[0044] If so, the specified time is used as the optimal execution time of the new task, and execution of new tasks of other priorities is suspended during the specific time period.

[0045] It should be noted that when this application detects that multiple tasks need to be executed concurrently during a high-risk resource period, it immediately starts a priority-driven resource time-space partitioning mechanism. Based on the available resource network scanning of this period, if it is confirmed that the resource requirements of high-priority tasks can be guaranteed, an execution channel will be immediately opened for the high-priority tasks, and low-priority tasks will be automatically suspended and their status will be retained. When an enterprise faces sudden business peaks, this mechanism can ensure that the core decision-making data is produced on time for the data center, and avoid the resource avalanche caused by the failed retry of low-priority tasks through flexible suspension, and ultimately maintain the high response resilience of the data center at all times.

[0046] A data platform task time planning device, comprising:

[0047] at least one processor; and,

[0048] a memory communicatively connected to the at least one processor; wherein,

[0049] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0050] When a new task is received from the data platform at a specified time, the task type of the new task and the upstream dependent tasks of the new task are determined;

[0051] Based on the task type, determining resource information and slot information required for running the new task;

[0052] Get the latest completion time of the dependent task;

[0053] Based on the latest completion time, the designated time, the resource information and the slot information, resources of the corresponding time period are matched in a pre-built available resource network of each time period to determine the optimal execution time of the new task.

[0054] A non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a computer, can implement:

[0055] When a new task is received from the data platform at a specified time, the task type of the new task and the upstream dependent tasks of the new task are determined;

[0056] Based on the task type, determining resource information and slot information required for running the new task;

[0057] Get the latest completion time of the dependent task;

[0058] Based on the latest completion time, the designated time, the resource information and the slot information, resources of the corresponding time period are matched in a pre-built available resource network of each time period to determine the optimal execution time of the new task.

[0059] The above description is only an overview of the technical solution of the present disclosure. In order to more clearly understand the technical means of the present disclosure, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present disclosure more obvious and easy to understand, the specific implementation methods of the present disclosure are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some of the embodiments described in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without inventive work. In the drawings:

[0061] Figure 1 A flowchart of a data platform task time planning method provided by an embodiment of the present disclosure is shown;

[0062] Figure 2 A flow chart of another data platform task time planning method provided by an embodiment of the present disclosure is shown;

[0063] Figure 3 A flow chart of a big data platform task optimization method provided by an embodiment of the present disclosure is shown;

[0064] Figure 4 A structural diagram of a data platform task time planning device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0065] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0066] During peak hours, the platform's resource utilization is high, even reaching saturation. This overloaded state can lead to serious consequences. During off-peak hours, as some critical tasks are completed, the platform's load begins to decrease. However, the platform's computing resource utilization plummets, leaving most operating resources idle. This idleness results in significant resource waste.

[0067] Under the dual pressures of continued business expansion and surging data complexity, the core contradiction facing the current data platform scheduling system is that the concentrated outbreak of massive offline batch processing and hourly quasi-real-time tasks during peak hours pushes cluster resource utilization to the saturation threshold, inducing systemic risks such as task backlog avalanche, dependency chain cascading delays, and even quasi-real-time task timeliness breakdown; at the same time, when the platform load drops sharply during the off-peak period after the completion of critical tasks, a large amount of computing resources fall into an idle state. However, due to the lack of cross-time resource prediction and elastic supply capabilities, the traditional scheduling mechanism can neither avoid task deadlock caused by resource competition during peak hours, nor actively pull tasks to fill resource gaps during off-peak hours. Ultimately, the computing resources of the data platform may be in a state of overload paralysis or idle waste.

[0068] To this end, the present disclosure provides a flowchart of a method for time planning of data platform tasks, as shown in FIG. Figure 1 As shown in Figure 2, the process can be executed by a task scheduling system. Certain input parameters or intermediate results in the process can be manually adjusted to help improve accuracy.

[0069] The method flow steps of the embodiments of the present application are as follows:

[0070] S101, when receiving a new task of a data platform at a specified time, determining a task type of the new task and a dependent task upstream of the new task.

[0071] In S101 of the embodiments of the present application, a task registration or metadata management system can be established, and each task needs to explicitly indicate its task type identifier and upstream dependent task list when being submitted or defined. The subsequent new tasks of the embodiments of the present application are all taken as vehicle production tasks for example, and the dependent task of the vehicle production task can be a vehicle order aggregation task.

[0072] The new task (vehicle production task) is submitted to the data platform through a specific interface. The submission information contains its task type identifier. After the platform receives the new task, the upstream dependent task list preset for the task type is automatically obtained by querying the task metadata management system according to the task type identifier. Based on the upstream dependent task list, it can be determined that the dependent task of the production task is the vehicle order aggregation task.

[0073] S102, determining resource information and slot information required by the new task based on the task type.

[0074] In S102 of the embodiments of the present application, a preset resource template library can be matched according to the task type, and the resource demand is dynamically adjusted combined with the task parameters, and the resource information (such as memory size and CPU size) and slot information (such as slot type and slot number) required by the new task are output, wherein the slot type can include a first-level slot and a second-level slot.

[0075] S103, obtaining the latest completion time of the dependent task.

[0076] In S103 of the embodiments of the present application, the latest execution state of all upstream dependent tasks can be queried from the task scheduling state database, and if the dependent task has not been completed, the preset latest completion time is obtained, and if it has been completed, the actual completion time is taken.

[0077] S104, determining the optimal execution time of the new task in the available resource network of each time period pre-constructed based on the latest completion time, the specified time, the resource information and the slot information.

[0078] In Example S104 of this specification, the latest completion time of the dependent task, the specified time (which can be understood as the current system time), resource information (which can be understood as the resource requirements of the new task), and slot information (which can be understood as the slots required for the new task) are used as input to load a pre-generated time-based resource network (which can store historical slot distribution and resource distribution at minute granularity). Based on the dependent completion time constraint (the task cannot be started earlier than this time), the resource network is scanned in chronological order for the first time period that meets the following conditions at the same time:

[0079] The start time ≥ the latest completion time of the dependency; the number of historical slots available in the period ≥ the slots required by the task; and the historical remaining resources in the period ≥ the resource requirements of the task. Output the starting time of the matching period as the optimal execution time.

[0080] It should be noted that this application builds an available resource network based on the slots and resource usage distribution in historical time periods. At the moment a new task arrives, it can be combined with the latest completion time of its upstream dependencies and its own resource requirements to directly match the first future time window in the resource network that satisfies both dependency readiness and resource sufficiency. This mechanism skips the volatility of real-time resource detection in traditional scheduling, allowing tasks to be quickly connected from readiness to startup. At the same time, as the business scale expands, the periodic laws of the historical resource network can continuously enhance prediction accuracy, thereby forming a steady-state scheduling capability that can resist a surge in task volume, and ultimately ensure that time-sensitive data processing such as hourly tasks is always delivered within a deterministic time window.

[0081] Optionally, before matching resources for the corresponding time period in the pre-constructed available resource network for each time period and determining the optimal execution time for the new task, the slot usage distribution and the running resource usage distribution for each time period may be collected, where the slot usage distribution and the running resource usage distribution for each time period are historical data. Based on the slot usage distribution and the running resource usage distribution for each time period, the available resource network for each time period is then constructed.

[0082] In the embodiments of this specification, a unified time segment can be determined (e.g., 00:00-8:00 every day, with each segment being 2 minutes). The slot usage distribution and the operating resource usage distribution for each segment over a sufficient number of days (e.g., the past 60 days) can be extracted from the platform monitoring system or database.

[0083] For the slot usage distribution in each time period, the slot information can be calculated from alisa_task every two minutes and the statistics are written into the slot_minute_distribute table. According to the historical execution records of the node, the calculated slot information may not match the actual available slots. Therefore, the embodiment of this specification can use a self-collection method to obtain the slot usage distribution.

[0084] For the distribution of running resource usage cu (control unit) in each time period, cu information can be obtained from the interface every two minutes and written into the table (resource_minute_distribute). cu is a general term for CPU and memory resources.

[0085] The embodiments of this specification can construct a minute-level available resource network based on slot usage and CU usage, where the fields include minutes, primary slots, secondary slots, and CUs.

[0086] When the available resource network is implemented, the embodiment of this specification can allocate 0-8 o'clock by 2 minutes, and layer it into 8*60 / 2 time slices (i.e., time periods). The slot usage distribution and running resource (CU) usage distribution of each time period in the past sufficient number of days are extracted from the platform monitoring system or database, and allocated to each time slice. The created available resource network is shown in Table 1, which is as follows from the 0th column: used first-level slots, used second-level slots, current workspace used CU, current workspace used memory, total CU used, total memory used; from the 0th row: 0th, 2nd, 4th, 6th...480 minutes.

[0087] Table 1 Available resource networks

[0088] 0 1 2 3 4 5 0 471 671 320 620 71775 134230 1 201 207 116 300 88076 175811 2 175 133 392 945 77754 160990 3 189 152 1262 2720 130254 160957 4 200 194 731 1834 85986 190715 …… 239 150 148 15 47 67706 133047

[0089] It's important to note that this application builds a time-based available resource network based on the system's continuous collection of historical slot and resource usage distribution. This essentially transforms the periodic fluctuations of cluster resources into a calculable, deterministic supply capacity. Before a new task is scheduled, this network already reveals the probability of resource idleness in each future time period, enabling the subsequent calculation of optimal execution times based on dependency time and resource demand to provide forward-looking scheduling capabilities.

[0090] Optionally, in determining the optimal execution time of the new task, the specification embodiment can first determine whether the latest completion time is less than the specified time, to avoid the dependence node not being executed, resulting in the new task being unable to execute normally; if so, based on the resource information and the slot information, the slot usage and running resource usage of the time period in which the specified time is located in the available resource network are scanned to determine whether there is sufficient slot and running resource for the new task to run in the current time period; if so, the specified time is taken as the optimal execution time of the new task.

[0091] It should be noted that when the latest completion time of the upstream dependent task is earlier than the current time, the system immediately triggers the determination of the availability of resources in the current period. The resource network constructed based on historical regularity has preloaded the normal slot and resource state in this period, and the availability of resources in the current period can be determined instantaneously by scanning the available resource network. If the preset condition is met, the current time is directly locked as the optimal execution time.

[0092] Optionally, if the slot usage and running resource usage of the time period in which the specified time is located in the available resource network do not meet the preset requirements, the specified time can be delayed based on a preset time window to obtain a delay time. The slot usage and running resource usage of the time period in which the delay time is located in the available resource network are scanned to determine whether they meet the preset requirements; if so, the delay time is taken as the optimal execution time of the new task.

[0093] In the specification embodiment, when it is determined that the slot usage and running resource usage of the time period in which the specified time is located in the available resource network do not meet the preset requirements, the specified time is taken as the delay starting point. A fixed delay step (such as 30 minutes each time) and a maximum allowed delay range (such as allowing a maximum of 4 attempts / 120 minutes from the specified time) are defined. The specified time is added by a delay step to obtain a first delay time (for example, the specified time 02:00 is added by a 30 min step to obtain a first delay time 02:30), and the delay is recorded as the 1st attempt.

[0094] The time slice to which the delay time belongs in the available resource network is found. The resource conditions corresponding to the new time period are queried: the number of used slots, the used cu of the current workspace, the used memory of the current workspace, the total used cu, and the total used content. The required slot information and resource information of the new task are compared with the queried resource conditions corresponding to the new time period. If the resource conditions corresponding to the new time period in the available resource network meet the required slot information and resource information of the new task, the delay time is taken as the optimal execution time of the new task.

[0095] It should be noted that when the system detects that the resources at the current moment do not meet the demand (such as the new task dependency is ready but the slot is temporarily tight), this application does not adopt a fixed-interval polling or passive waiting strategy, but instead uses the available resource network to directionally detect the next high-confidence idle period of resources along the time axis. For example, 30 minutes is preset as the time window step, and after the specified time is moved from 02:00 to 02:30, the historical rules of this period in the available resource network are immediately scanned. Once the demand is matched, 02:30 is immediately anchored as the optimal time to avoid invalid waiting of more than half an hour in the traditional retry mechanism. In the case of a large number of tasks being scheduled on a daily basis, this mechanism enables each blocked task to regain the deterministic execution window with only a small amount of computing power, ultimately ensuring that all time-sensitive tasks always glide forward on a predictable time track, supporting the uninterrupted downstream business decision-making chain.

[0096] Optionally, the embodiment of this specification also needs to consider the running time required for the new task. The embodiment of this disclosure provides a flow chart of another data platform task time planning method, such as Figure 2 As shown in Figure 2, this process can also be executed by a task scheduling system. Certain input parameters or intermediate results in the process can be manually adjusted to help improve accuracy.

[0097] The steps of the method of the embodiment of the present application are as follows:

[0098] S201 , when a new task is received from a data platform at a specified time, the task type of the new task and the upstream dependent tasks of the new task are determined.

[0099] In the embodiment S201 of this specification, please refer to the above-mentioned S101 for details, which will not be repeated here.

[0100] S202: Determine resource information and slot information required for running the new task based on the task type.

[0101] In the embodiment S202 of this specification, please refer to the above-mentioned S102 for details, which will not be repeated here.

[0102] S203: Obtain the latest completion time of the dependent tasks.

[0103] In the embodiment S203 of this specification, please refer to the above S103 for details, which will not be repeated here.

[0104] S204: Predicting the running time required for the new task based on historical execution records.

[0105] In the embodiment S204 of the present specification, the specific type of the new task can be determined first (such as "A type vehicle production task" or "B type vehicle customized production task"). From the platform historical task execution record library, the actual execution completion time (time consumption) of all completed tasks of the specific task type is extracted. The collected time consumption data sequence is sorted to find the value at the 75% position (i.e., the 75th percentile value). This calculation can be completed offline periodically (such as once a day), and the result is stored in the task metadata management library for query. When the scheduling decision needs to be made for the new task, the 75th percentile time consumption corresponding to the task type is directly obtained from the metadata database as the required running time of the new task.

[0106] S205, determining whether the latest completion time is less than the specified time, if yes, performing S206; if no, ending the process.

[0107] S206, determining one or more time periods occupied by the new task according to the specified time and the running time, and scanning the slot usage and running resource usage of the one or more time periods in the available resource network based on the resource information and the slot information, whether they all meet the preset usage requirement, if yes, performing S207; if no, ending the process.

[0108] In the embodiment S206 of the present specification, the running time predicted by S204 is used as the time window required by the task. According to the specified time and the running time, the predicted end time of the new task (specified time + running time) is calculated. It is determined that in the time slice division of the available resource network, from the time slice containing the specified time to the end of the time slice containing the predicted end time, all the time periods (time slices) continuously covered in this time range are determined. For example: the time length is 5 minutes, and the time slice is 2 minutes, which may cover 3 continuous time slices.

[0109] For each continuously covered time period determined above, the available resource network is queried to obtain the predicted available resource amount and the available slot number of the time period. It is determined whether the slot amount occupied by the new task in the time period is less than the available slot capacity of the time period, and whether each resource requirement in the resource information of the new task is less than the corresponding available resource amount of the time period.

[0110] S207, taking the specified time as the optimal execution time of the new task.

[0111] In the embodiment S207 of the present specification, if the slot amount occupied by the new task in the time period is less than the available slot capacity of the time period, and each resource requirement in the resource information of the new task is less than the corresponding available resource amount of the time period, the specified time can be taken as the optimal execution time of the new task.

[0112] It should be noted that this application predicts the running time of new tasks based on historical execution records, and starts the verification of continuous supply of resources across time periods when the latest completion time of upstream dependencies is earlier than the current moment. Based on the available resource network, the slots and resource history patterns of each segment within the time window required for the task from start to finish are scanned to see if they meet the preset requirements. Only when the certainty of resource supply throughout the process is met can the current moment be set as the optimal execution time to avoid unexpected interruptions caused by resource exhaustion during the execution of the new task.

[0113] Optionally, if there is at least one time period in which the slot usage and the running resource usage of the one or more time periods in the available resource network do not meet the preset requirements, scan the slot usage and the running resource usage of each time period after the specified time in the available resource network to obtain the execution time period closest to the specified time in which the one or more consecutive time periods all meet the preset usage requirements; determine the specified execution time corresponding to the execution time period, and use the specified execution time as the optimal execution time of the new task.

[0114] It should be noted that, following the previous step S206, the result is: scanning the slot usage and running resource usage of one or more time periods in the available resource network reveals that at least one time period is resource-constrained. The starting point of the scan is set to the starting boundary time of the first time period after the specified time. For example, if the specified time is 02:10 and the time period is a 2-minute time slice, the scan starting point is 02:12. Based on the predicted runtime of the new task (e.g., the 75th percentile) and the time slice length of the available resource network, the number of consecutive time periods (N) required for the task to run is calculated. For example, if the runtime is 5 minutes and the time slice is 2 minutes, then N = 3. Each possible starting time period is examined in chronological order to see if the N consecutive time periods that follow meet the resource requirements. That is, based on the current candidate start time, the corresponding time period is found in the available resource network. Starting from the current candidate start time period, N consecutive time periods are selected. The available resource network is queried to obtain the available CPU, available memory, etc. for that time period. If (all N time periods meet the requirements), the current candidate start time is recorded as a feasible starting point. The scan stops (since the target is the most recent, the first one found is sufficient). If not (does not match any of the N time periods): Move the candidate start time to the start boundary of the next time period. Select the closest feasible start point found and set it as the task start time.

[0115] It should be noted that this application detects resource gaps across multiple time periods spanning the task's execution time and immediately initiates a directional scan of the resource network model along the time axis. Based on historical slot and resource distribution patterns, it quickly locates the first nearest time window that can meet continuous resource supply requirements throughout the task. In business scenarios with an average of tens of thousands of tasks per day and competing resources, this mechanism ensures that hourly critical tasks maintain sufficient operating resources and slots.

[0116] Optionally, if there is at least one time period in which the slot usage and the running resource usage in the one or more time periods in the available resource network do not meet the preset requirements, a specific time period that does not meet the preset requirements can be determined first; the specified time is used as the optimal execution time of the new task, and the execution of the new task is suspended during the specific time period.

[0117] It should be noted that the result of the previous S206 is: scan the slot usage and running resource usage of the one or more time periods in the available resource network, and there is insufficient resources in at least one time period. Traverse the continuous time period sequence planned for the new task (starting from the specified time), and find the first (or all) specific time periods (such as time slice a) marked as insufficient resources (not meeting the preset requirements). Record this specific resource-deficient time period (time slice a) and its precise start and end time as the basis for subsequent suspension operations. Bind this information to the new task instance. Maintain the specified time as the optimal execution time of the new task (i.e., the planned start time).

[0118] After a new task is started, the actual running time of the task is monitored in real time. When it is monitored that the task execution time is approaching or entering a specific resource-deficient time period (time slice a) previously recorded, the scheduler / resource manager issues a pause instruction. The task execution engine saves the runtime state of the task (such as memory data, intermediate calculation results, breakpoint location) at a safe point. Actively release the CPU computing resources, memory resources, etc. currently occupied by the task. The task status can be marked as "manual intervention pause" or "waiting for resource recovery". The scheduler continuously monitors the available resource network or actual resource load, waiting for the resource-deficient time period (time slice a) to pass. After time slice a ends, the scheduler rescans resources. Find a new starting time period (usually after time slice a ends) for the paused task to resume. Based on the saved state, reapply for resources and resume task execution.

[0119] It should be noted that this application detects that there are resource gaps in local time periods during the long-term execution of tasks. This application does not postpone the start time of the entire task, but splits the task execution process into discontinuous units. Start the task in the time period that currently meets the conditions, and actively pause when entering a high-risk period, and automatically resume when the available resource network shows that the subsequent time period is safe. In scenarios where resources fluctuate drastically during business peak hours, this mechanism can enable hourly time-sensitive tasks to make full use of fragmented resource gaps to advance calculations, ensure that full data processing is completed before the agreed delivery time point, and support the pressure-resistant operation of the downstream business decision-making chain.

[0120] Optionally, when the new task involves multiple tasks, and there is at least one time period in which the slot usage and the running resource usage in the one or more time periods in the available resource network do not meet the preset requirements, the priority of each new task can be determined first; the specific time period that does not meet the preset requirements can be determined; the slot usage and the running resource usage in the specific time period in the available resource network can be scanned to see whether they all meet the preset usage requirements of the high-priority new task; if so, the specified time can be used as the optimal execution time of the new task, and the execution of new tasks of other priorities can be suspended during the specific time period.

[0121] It should be noted that the result of the previous S206 is: scanning the slot usage and running resource usage of one or more time periods in the available resource network, at least one time period is short of resources. Within the required time period starting at the specified time, the resources of a specific time period (such as time slice b) are insufficient (not meeting the preset requirements of the task). Confirm that there are currently multiple new tasks to be scheduled, and at least some of them (including the target task) need to run or occupy resources in time slice b. Clarify the list of all new tasks that need to be executed within this overlapping time window (especially time slice b) or have been scheduled to be executed in time slice b. Obtain the preset priority attributes of all relevant new tasks (such as querying from task metadata). According to business rules (such as: "urgent orders" are high priority, "ordinary orders" are other priorities), the relevant tasks can be divided into at least two groups: high priority group and other priority groups.

[0122] Scan the slot usage and running resource usage in the available resource network during the specified time period to see if they both meet the preset usage requirements for the high-priority new task. If so, the specified time is used as the optimal execution time for the new task, and new tasks of other priorities are suspended during the specified time period. If not (any resource does not meet the requirements), suspending new tasks of other priorities alone cannot solve the problem. Try other strategies (such as delaying high-priority tasks, staggering scheduling, rejecting some tasks, etc.) and exit the solution process.

[0123] It should be noted that when this application detects that multiple tasks need to be executed concurrently during a high-risk resource period, it immediately starts a priority-driven resource time-space partitioning mechanism. Based on the available resource network scanning of this period, if it is confirmed that the resource requirements of high-priority tasks can be guaranteed, an execution channel will be immediately opened for the high-priority tasks, and low-priority tasks will be automatically suspended and their status will be retained. When an enterprise faces sudden business peaks, this mechanism can ensure that the core decision-making data is produced on time for the data center, and avoid the resource avalanche caused by the failed retry of low-priority tasks through flexible suspension, and ultimately maintain the high response resilience of the data center at all times.

[0124] It should be noted that the big data platform of the data center, as the core hub of enterprise data processing, is responsible for scheduling and executing massive offline and near-real-time tasks in various systems and fields. The current platform faces the following problems:

[0125] 1. Huge task scale: The average daily offline tasks reach over 35,000, and the hourly tasks exceed 30,000. Due to the business's reliance on data and the requirement for daily system stability, over 80% of tasks must be completed between 00:00 and 8:00.

[0126] 2. Resource utilization imbalance: During peak hours (02:00-04:00), CPU / memory utilization is ≥95%, with resource oversold reaching 100%, leading to task queue timeouts and OOM (Out of Memory) interruptions. During off-peak hours (05:00-07:00), resource utilization is ≤30%, with over 40% of computing resources idle and wasted.

[0127] 3. Low cost-efficiency: Resource scramble during peak periods coexists with idle resources during low periods.

[0128] To address the aforementioned issues of "the large number of tasks on the big data platform, the concentrated execution time of tasks leading to unbalanced resource utilization, low task timeliness, and insufficient system stability," this solution proposes the following technical measures:

[0129] 1. Slot usage distribution collection: Slot information is calculated from alisa_task every two minutes and the statistics are written to the slot_minute_distribute table. (The slot information calculated from the node's historical execution records does not match the actual available slots, so self-collection is used instead to obtain slot usage distribution.)

[0130] 2. CU usage distribution collection: CU information is obtained from the interface every two minutes and written to the table (resource_minute_distribute).

[0131] 3. Estimating the duration of a new task: Select the 75th percentile of the execution time of all nodes as the predicted duration (DataWorks tasks, calculated in advance).

[0132] 4. Resources required to run the new task

[0133] a. Non-data integration nodes require one first-level slot;

[0134] b. For a data integration node, in addition to one primary slot, two secondary slots are required, equal to the node concurrency number * 2 (the node concurrency number can be obtained from the node configuration).

[0135] 5. Resource modeling and building a network of available resources

[0136] Build a minute-level resource network based on slot usage and CU usage. The fields include minutes, first-level slots, second-level slots, and CUs.

[0137] Implementation process: Allocate points 0-8 by 2 minutes, layered into 8*60 / 2 time slices, and allocate the slots, CUs, and other resources calculated in 1 and 2 to each time slice. The created resource network is shown in Table 1: Column 0 to 1 are: used first-level slots, used second-level slots, CUs used in the current workspace, memory used in the current workspace, total CUs used, and total memory used; each row represents minutes 0, 2, 4, 6, ..., 480.

[0138] 6. Node dependency processing

[0139] a. Calculate the estimated start time and estimated running time of all nodes;

[0140] Take the execution records of all tasks within 15-30 days, remove the 10% records with abnormally low execution time and the 10% records with abnormally high execution time, and record the 75th percentile of these records as the expected start time and expected execution time of the node.

[0141] b. The SDK obtains the name of the dependent node of the node to be put online, and obtains the node ID based on the name;

[0142] c. Based on a) and b), obtain the latest completion time of the dependent upstream node.

[0143] 7. Recommend optimal execution time

[0144] a. First scan the time points of resource usage (used CPU / project space CPU < 0.2). If there are any, select the earliest time as the optimal execution time;

[0145] b. If the above conditions are not met, scan the 30 minutes before and after the preset execution time and scan the "Available Resource Network" to find the time point (minutes) when resources (slots, CUs) can be satisfied;

[0146] c. If a suitable time cannot be found in step b), expand the time window (depending on the latest completion time of the node, 8:00 AM) and scan the "Available Resource Network" until the nearest executable time point is found.

[0147] In conjunction with the above, see Figure 3 The flowchart of the big data platform task optimization method shown in the figure starts with obtaining pre-processed node operation information, which may include the collected slot usage distribution and CU usage distribution, and then performing data cleaning on the above-collected slot usage distribution and CU usage distribution. Based on the cleaned data, an available resource network is constructed to obtain new node (new task) information, scan the time window near the timing, and obtain node dependency information and define the time window (which can be 0-8:00 o'clock, which can be adjusted according to actual conditions). The latest completion time of the dependency is calculated to determine whether there are available resources. If so, the earliest time is used. If not, the time window is expanded. The latest completion time and the time window are considered at the same time to determine whether there are available resources. If so, the earliest time is used. If not, an exception is thrown.

[0148] In response to the above technical problems, this technical solution has the following beneficial effects:

[0149] 1. Resource utilization optimization: Recommend tasks to run at reasonable times to reduce resource idleness and overselling;

[0150] 2. Improved task efficiency: Reduce the average waiting time for tasks during peak periods to avoid task timeouts and failures;

[0151] 3. Enhanced system stability: Reduce OOM or interruption problems caused by resource competition.

[0152] Figure 4 A schematic diagram of the structure of a data platform task time planning device provided in an embodiment of the present disclosure includes:

[0153] at least one processor; and,

[0154] a memory communicatively connected to the at least one processor; wherein,

[0155] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0156] When a new task is received from the data platform at a specified time, the task type of the new task and the upstream dependent tasks of the new task are determined;

[0157] Based on the task type, determining resource information and slot information required for running the new task;

[0158] Get the latest completion time of the dependent task;

[0159] Based on the latest completion time, the designated time, the resource information and the slot information, resources of the corresponding time period are matched in a pre-built available resource network of each time period to determine the optimal execution time of the new task.

[0160] A non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a computer, can implement:

[0161] When a new task is received from the data platform at a specified time, the task type of the new task and the upstream dependent tasks of the new task are determined;

[0162] Based on the task type, determining resource information and slot information required for running the new task;

[0163] Get the latest completion time of the dependent task;

[0164] Based on the latest completion time, the designated time, the resource information and the slot information, resources of the corresponding time period are matched in a pre-built available resource network of each time period to determine the optimal execution time of the new task.

[0165] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the device, apparatus, and non-volatile computer storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simplified. For relevant details, refer to the descriptions of the method embodiments.

[0166] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0167] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0168] In the embodiments provided in this application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0169] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0170] In addition, the functional units in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above units may be implemented in the form of hardware or software.

[0171] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0172] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A data platform task time planning method, characterized in that: include: When a new task is received from the data platform at a specified time, the task type of the new task and the upstream dependent tasks of the new task are determined; Based on the task type, determining resource information and slot information required for running the new task; Get the latest completion time of the dependent task; Based on the latest completion time, the designated time, the resource information and the slot information, resources of the corresponding time period are matched in a pre-built available resource network of each time period to determine the optimal execution time of the new task.

2. The method according to claim 1, characterized in that Before matching resources for the corresponding time period in the pre-built available resource network for each time period and determining the optimal execution time of the new task, the method further includes: Collect the slot usage distribution and running resource usage distribution in each time period; Based on the slot usage distribution and the running resource usage distribution in each time period, an available resource network for each time period is constructed.

3. The method according to claim 1, characterized in that The matching of resources for the corresponding time period in the pre-built available resource network for each time period based on the latest completion time, the designated time, the resource information, and the slot information to determine the optimal execution time of the new task includes: Determine whether the latest completion time is less than the specified time; If so, based on the resource information and the slot information, scanning the available resource network to see whether the slot usage and the running resource usage in the time period of the specified time meet the preset requirements; If so, the specified time is used as the optimal execution time of the new task.

4. The method according to claim 3, characterized in that If the slot usage and the running resource usage in the time period of the specified time in the available resource network do not meet the preset requirements, the method further includes: Delaying the specified time based on a preset time window to obtain a delay time; Scan the available resource network to determine whether the slot usage and running resource usage in the time period of the delay time meet the preset requirements; If so, the delay time is used as the optimal execution time of the new task.

5. The method according to claim 1, wherein Before determining the optimal execution time of the new task, the method further includes matching resources for the corresponding time period in a pre-built available resource network for each time period based on the latest completion time, the designated time, the resource information, and the slot information: Based on historical execution records, predict the runtime required for the new task; The matching of resources for the corresponding time period in the pre-built available resource network for each time period based on the latest completion time, the designated time, the resource information, and the slot information to determine the optimal execution time of the new task includes: Determine whether the latest completion time is less than the specified time; If yes, determine one or more time periods occupied by the new task based on the specified time and the running time, and scan the slot usage and running resource usage of the one or more time periods in the available resource network based on the resource information and the slot information to determine whether they all meet the preset usage requirements; If so, the specified time is used as the optimal execution time of the new task.

6. The method according to claim 5, characterized in that If at least one time period of the slot usage and the running resource usage in the one or more time periods in the available resource network does not meet the preset requirements, the method further includes: Scanning the slot usage and running resource usage of each time period after the specified time in the available resource network, and obtaining an execution time period closest to the specified time in which one or more consecutive time periods all meet the preset usage requirements; A designated execution time corresponding to the execution time period is determined, and the designated execution time is used as the optimal execution time of the new task.

7. The method according to claim 5, characterized in that If at least one time period of the slot usage and the running resource usage in the one or more time periods in the available resource network does not meet the preset requirements, the method further includes: Identify specific time periods during which pre-set requirements are not met; The designated time is used as the optimal execution time of the new task, and the execution of the new task is suspended during the specific time period.

8. The method according to claim 5, characterized in that When the new task involves multiple tasks, and the slot usage and the running resource usage in the one or more time periods in the available resource network do not meet the preset requirements in at least one time period, the method further includes: Determine the priority of each new task; Identify specific time periods during which pre-set requirements are not met; Scanning the slot usage and running resource usage in the available resource network during the specific time period to determine whether both meet the preset usage requirements of the new high-priority task; If so, the specified time is used as the optimal execution time of the new task, and execution of new tasks of other priorities is suspended during the specific time period.

9. A data platform task time planning device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: When a new task is received from the data platform at a specified time, the task type of the new task and the upstream dependent tasks of the new task are determined; Based on the task type, determining resource information and slot information required for running the new task; Get the latest completion time of the dependent task; Based on the latest completion time, the designated time, the resource information and the slot information, resources of the corresponding time period are matched in a pre-built available resource network of each time period to determine the optimal execution time of the new task.

10. A non-volatile computer storage medium, characterized in that The computer-executable instructions are stored, and when the computer-executable instructions are executed by a computer, they can achieve: When a new task is received from the data platform at a specified time, the task type of the new task and the upstream dependent tasks of the new task are determined; Based on the task type, determining resource information and slot information required for running the new task; Get the latest completion time of the dependent task; Based on the latest completion time, the designated time, the resource information and the slot information, resources of the corresponding time period are matched in a pre-built available resource network of each time period to determine the optimal execution time of the new task.

Images