A big data platform system architecture and resource dynamic scheduling method of support environment

Abstract

The application relates to the technical field of big data processing, and discloses a resource dynamic scheduling method of a big data platform system architecture and a supporting environment, which comprises the following steps: acquiring task description information and performing semantic analysis; identifying the resource type range and data access mode required by workflow execution; identifying key tasks; evaluating the matching degree between each task and each resource node, and performing weighted calculation in combination with key path analysis results; solving an optimal mapping scheme of the task to the resource node; locking the computing resources of the key path task; predicting data flow and identifying resource bottlenecks; dynamically adjusting resource allocation; performing task scheduling and resource scaling operation; monitoring the execution state in real time, and performing deviation analysis and dynamic adjustment of parameters; and the application is based on a timeliness constraint guarantee mechanism of workflow key path analysis and an active prediction type elastic scaling strategy of data sensing, can accurately identify key tasks affecting the overall completion time and provide resource priority guarantee for the key tasks.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology, and more specifically, to a method for dynamic resource scheduling of a big data platform system architecture and supporting environment. Background Technology

[0002] As enterprises deepen their digital transformation, big data platforms have become a critical infrastructure supporting core business operations. Modern big data application scenarios are characterized by diversified workload types, varying data timeliness requirements, and heterogeneous computing resources, which places higher demands on the resource scheduling capabilities of big data platforms.

[0003] Current mainstream big data platforms generally use resource management frameworks for resource scheduling, which are mainly based on resource capacity and static configuration, lacking the ability to perceive and guarantee the timeliness constraints of task execution. Different types of workloads require different scheduling frameworks, lacking a unified heterogeneous workflow scheduling capability. Data location factors are rarely considered when making scheduling decisions, which may lead to a large amount of cross-node data transmission and affect execution efficiency. Existing elastic scaling solutions mainly trigger scaling operations based on resource utilization thresholds, which has a lag and may not be able to complete the expansion in time when the load increases suddenly, resulting in a decline in service quality.

[0004] Therefore, how to build a dynamic resource scheduling system that can uniformly manage heterogeneous computing resources, intelligently schedule multiple types of workloads, and accurately guarantee data timeliness constraints has become a core technical problem in the construction of big data platforms. Summary of the Invention

[0005] This invention provides a method for dynamic resource scheduling in the architecture and supporting environment of a big data platform system, which solves the technical problems in related technologies such as the lack of unified abstraction and scheduling mechanism for heterogeneous workflows, the lag in elastic scaling response, and the inability to accurately guarantee timeliness constraints.

[0006] This invention provides a method for dynamic resource scheduling in the architecture and supporting environment of a big data platform system, comprising: Obtain task description information, perform semantic parsing, and obtain a standardized workflow representation; Based on a standardized workflow representation, the resource types and data access patterns required for workflow execution are identified to obtain a global resource view; Based on standardized workflow representation and global resource view, key tasks are identified, and critical path analysis results are obtained. The matching degree between each task and each resource node is evaluated based on the standardized workflow representation and global resource view. The weighted calculation is combined with the critical path analysis results to obtain the priority-weighted matching degree matrix. Based on the priority-weighted matching degree matrix and the global resource view, the optimal mapping scheme from task to resource node is solved; based on the critical path analysis results, the computing resources of the critical path tasks are locked to obtain the task scheduling mapping scheme. Based on the task scheduling mapping scheme and critical path analysis results, data traffic is predicted and resource bottlenecks are identified; based on the identification results, resource allocation is dynamically adjusted to obtain elastic scaling decision instructions. Based on the task scheduling mapping scheme and elastic scaling decision instructions, the system performs task scheduling and resource scaling operations; it monitors the execution status in real time, performs deviation analysis and dynamic parameter adjustment, and obtains an execution deviation analysis report and optimized scheduling parameter configuration.

[0007] In a preferred embodiment, the semantic parsing includes: Based on the task declaration field and execution mode field in the task description information, a task type identification method combining rule matching and feature recognition is used to obtain task type classification labels; Based on task type classification labels, a differentiated workflow diagram construction strategy is adopted to obtain a task workflow diagram representation; Based on each task node in the task workflow graph representation, a multi-dimensional feature extraction method is used to perform feature analysis on each task node to obtain the calculated feature label vector of the task node. Based on the service level agreement parameters specified by the user in the task description information, the timeliness requirements of the task are extracted using the timeliness constraint parsing method, resulting in a set of timeliness constraint parameters for the workflow.

[0008] In a preferred embodiment, the task type classification includes batch processing, streaming computing, and hybrid tasks. Batch processing tasks are characterized by one-time execution and bounded data input, streaming computing tasks are characterized by unbounded data input and continuous execution, and hybrid tasks include both batch processing and streaming computing.

[0009] In a preferred embodiment, identifying the range of resource types and data access patterns required for workflow execution includes: Based on the computational feature label vectors of each task node in the standardized workflow representation, a resource type identification method is used to analyze the range of resource types required for workflow execution, and a set of resource requirement types is obtained. Based on the set of resource demand types and pre-configured resource management domain information, a hierarchical resource discovery method is used to scan computing resources at three levels: cloud data center, edge computing node, and terminal device. Based on the hardware configuration parameters in the resource node registration information, a multi-dimensional resource capability modeling method is used to quantify the computing power of each resource node, thereby obtaining the capability vector of the resource node.

[0010] In a preferred embodiment, the step of identifying key tasks and obtaining critical path analysis results based on standardized workflow representation and global resource view includes: Based on the computational feature label vector of each task node in the workflow graph representation and the capability vector of each resource node in the global resource view, the expected execution time of each task node on different resource nodes is calculated using an execution time estimation model. Based on the task dependencies and task baseline execution time vectors in the workflow graph representation, the critical path algorithm is used to analyze the time characteristics of the workflow and obtain the time parameters and critical path identifiers of each task node. Based on the critical path identifier and timeliness constraint parameters, a time budget allocation method is used to allocate execution time budgets to each task node on the critical path.

[0011] In a preferred embodiment, the step of evaluating the matching degree between each task and each resource node based on the standardized workflow representation and global resource view, and performing a weighted calculation in conjunction with the critical path analysis results, includes: Based on the computational feature label vector of each task node and the data scale parameter in the task description, a resource demand quantification model is used to calculate the multidimensional resource demand vector of each task node. Based on the normalized task requirement vector and the normalized resource capability vector, a weighted similarity calculation method is used to evaluate the matching degree between each task and each resource node, and the task-resource matching degree matrix is ​​obtained. Based on the critical path task list and time budget allocation scheme in the critical path analysis results, a priority weight assignment method is used to assign scheduling priority weights to each task node.

[0012] In a preferred embodiment, the optimal mapping scheme from the solution task to the resource node includes: Based on data location index and resource topology, and combined with the input data identifiers of each task node in the workflow diagram representation, a data transmission overhead calculation method is used to evaluate the data transmission cost required for each task to be executed on each candidate resource node. Based on the priority-weighted matching degree matrix and the data transmission overhead matrix, the optimization objectives are integrated into a unified objective function, resulting in the comprehensive scheduling objective function; Based on the comprehensive scheduling objective function, a heuristic optimization algorithm is used to solve for the optimal mapping scheme from tasks to resource nodes, thus obtaining the task scheduling mapping scheme.

[0013] In a preferred embodiment, predicting data traffic and identifying resource bottlenecks includes: Based on the input data source identifier of each task node in the task scheduling and mapping scheme, a traffic monitoring agent is used to continuously collect data inflow rate indicators at each data source deployment point. Based on a time series dataset of data traffic, a time series forecasting model is used to predict the future traffic of each data source. Based on the traffic prediction results and the current processing capacity of each processing stage in the task scheduling mapping scheme, a resource bottleneck prediction method is used to assess whether there are potential resource bottlenecks. Based on the bottleneck information in the resource bottleneck prediction report, the scale of resource expansion required to eliminate the bottleneck is determined using a scaling calculation method.

[0014] In a preferred embodiment, the execution of task scheduling and resource scaling operations includes: Based on the task scheduling mapping scheme, the instruction conversion method is used to convert the scheduling mapping relationship into task start instructions that can be executed by each computing node; Based on the set of elastic scaling decision instructions, resource expansion or contraction operations are performed using the resource management interface. Based on the result of the task start command, the execution status monitoring service continuously collects the task's execution status information. Based on the task execution time estimates from the time series analysis of task execution status and critical path analysis, the deviation analysis method is used to compare the actual execution indicators of the task with the expected indicators.

[0015] In a preferred embodiment, the real-time monitoring of execution status, deviation analysis, and dynamic parameter adjustment include: Based on the task execution time estimates from the task execution status time series and critical path analysis results, the actual execution indicators of the task are compared with the expected indicators to obtain an execution deviation analysis report. Based on the dynamic update triggering conditions in the execution deviation analysis report and critical path analysis results, a dynamic adjustment triggering determination method is used to evaluate whether dynamic adjustment of the scheduling strategy needs to be triggered. Based on the execution deviation analysis report, an adaptive parameter optimization method is used to adjust the parameter configuration of the scheduling algorithm according to historical execution feedback, thereby obtaining an optimized scheduling parameter configuration.

[0016] The beneficial effects of this invention are as follows: By constructing a three-layer decoupled perception and scheduling framework, this invention realizes the unified abstraction and scheduling of heterogeneous workflows such as batch processing tasks, streaming computing tasks, and machine learning inference tasks, enabling different types of workloads to flexibly share computing resources in the same resource pool, improving the overall resource utilization of the cluster, and reducing the management complexity of parallel operation and maintenance of multiple scheduling systems.

[0017] This invention, through a timeliness constraint guarantee mechanism based on workflow critical path analysis and a data-aware proactive predictive elastic scaling strategy, can accurately identify key tasks that affect the overall completion time and provide them with priority resource guarantees. At the same time, it can proactively adjust resources before resource bottlenecks actually occur, effectively ensuring the service quality of time-sensitive applications and avoiding the lag problem of traditional passive response scheduling schemes. Attached Figure Description

[0018] Figure 1 This is a flowchart of the main process of a resource dynamic scheduling method for a big data platform system architecture and supporting environment in this invention. Figure 2 This is a detailed flowchart of a resource dynamic scheduling method for a big data platform system architecture and supporting environment in this invention. Detailed Implementation

[0019] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0020] At least one embodiment of the present invention discloses a method for dynamic resource scheduling of a big data platform system architecture and supporting environment, such as... Figures 1 to 2 As shown, it includes: Step 1: Obtain task description information, perform semantic parsing, and obtain a standardized workflow representation; Step 1.1, Task type identification and classification; The system obtains user-submitted task description information. Based on the task declaration and execution mode fields in the task description information, a task type identification method combining rule matching and feature recognition is used to obtain task type classification labels. The task type identification method extracts key feature words from the task description information, including data source type identifiers, processing mode identifiers, and output target identifiers. These extracted feature words are matched with predefined task type feature templates, and the task type classification is determined based on the matching results. Task type classification includes three basic types: batch processing, streaming computing, and hybrid. Batch processing tasks are characterized by one-time execution and bounded data input; streaming computing tasks are characterized by unbounded data input and continuous execution; and hybrid tasks include both batch processing and streaming computing modes.

[0021] Step 1.2, Construction of differentiated workflow diagram; Based on the task type classification labels obtained in step 1.1, a differentiated workflow graph construction strategy is adopted to obtain the task workflow graph representation. For batch processing tasks, a standard directed acyclic graph (DAG) model is used to model the task, decomposing it into several computational sub-task nodes. Directed edges are established based on the data dependencies between sub-tasks to form a DAG structure that expresses the task execution order constraints. For streaming computation tasks, an extended streaming directed graph model is used to model the task, introducing streaming operator node types to represent continuously executed data processing units, and using data flow channel edge types to represent the data transmission relationships between operators. Streaming operator nodes have the characteristics of continuously receiving input data and continuously generating output data. For hybrid tasks, a hybrid workflow graph containing batch processing nodes and streaming operator nodes is constructed. A data interaction interface is defined between batch processing nodes and streaming operator nodes, specifying the data format and triggering conditions for transferring batch processing results to streaming processing and vice versa.

[0022] Step 1.3, Multidimensional feature extraction; Based on the task nodes in the task workflow graph representation obtained in step 1.2, a multi-dimensional feature extraction method is used to perform computational feature analysis on each task node, resulting in a computational feature label vector for the task node. The computational feature extraction method parses the computational logic description of the task node, identifies the main types of computational operations involved in the task, including numerical computation operations, data aggregation operations, data transformation operations, and model inference operations. Based on the resource consumption characteristics of the computational operation types, the demand intensity of the task in each resource dimension is evaluated. Resource dimensions include processor computing power, memory capacity, storage read / write, network transmission, and accelerator computing power. The demand intensity evaluation values ​​of each dimension are combined into a computational feature label vector, where each component represents the relative demand degree of the task in the corresponding resource dimension. The computational feature label vector uses a normalized representation, with each component ranging from zero to one; a larger value indicates a stronger demand for that resource dimension.

[0023] Step 1.4, Timeliness Constraint Analysis and Setting; Based on the service level agreement parameters specified by the user in the task description information, a timeliness constraint parsing method is used to extract the timeliness requirements of the task, resulting in a set of timeliness constraint parameters for the workflow. The timeliness constraint parsing method employs differentiated constraint expressions depending on the task type. For batch processing workflows, the timeliness constraint is expressed as the upper limit of end-to-end completion time, i.e., the maximum allowed time from the start of workflow execution to the completion of all tasks. For streaming computing workflows, the timeliness constraint is expressed as the upper limit of single-data processing delay and data freshness requirements. The upper limit of single-data processing delay specifies the maximum allowed time from data entering the system to the output of the processing result, while the data freshness requirement specifies the maximum allowed time span of the input data upon which the processing result is based. For hybrid workflows, the timeliness constraint includes both the completion time constraint of the batch processing part and the delay constraint of the streaming part. The set of timeliness constraint parameters is stored in association with the workflow graph representation, serving as an important input basis for subsequent scheduling decisions.

[0024] Step 1.5: Standardized workflow representation is generated through fusion. Based on the workflow graph representation obtained in step 1.2, the task node computational feature label vectors obtained in step 1.3, and the timeliness constraint parameter set obtained in step 1.4, a multi-layer information fusion method is used to integrate the components into a unified standardized workflow representation, resulting in a workflow description containing complete semantic information. The multi-layer information fusion method uses the workflow graph structure constructed in step 1.2 as its basic framework, embedding the computational feature label vectors extracted in step 1.3 as node attributes into the corresponding task nodes. Each task node carries its five-dimensional normalized resource requirement feature vector. Then, the timeliness constraint parameter set parsed in step 1.4 is used as a global attribute and associated with the entire workflow graph, establishing a mapping relationship between constraint parameters and workflow execution paths. The fusion process also requires establishing a consistency check mechanism between feature labels and timeliness constraints to ensure that critical task nodes with high resource requirements match strict timeliness constraints.

[0025] The standardized workflow representation generated in this step is encapsulated using a unified data structure, including the topology of the workflow graph, the computational feature label vector of each node, the data dependencies between nodes, the definition of data interaction interfaces, and global timeliness constraint parameters, forming a complete workflow description that can be directly used by subsequent scheduling decision steps.

[0026] Furthermore, since different business scenarios express timeliness constraints differently, simple time threshold constraints may not accurately represent complex business needs. A hierarchical timeliness constraint model can be used to replace simple threshold constraints, aiming to support more refined expressions of timeliness requirements and more flexible constraint satisfaction determination. Specifically, the hierarchical timeliness constraint model decomposes timeliness constraints into two levels: hard constraints and soft constraints. Hard constraints define the minimum timeliness requirement that must be met; violating hard constraints will lead to task execution failure. Soft constraints define the expected timeliness target; the degree to which soft constraints are satisfied affects the service quality score but does not cause task failure. The hierarchical constraint model also supports defining priority relationships and trade-off rules between constraints. When resources are scarce and all soft constraints cannot be satisfied simultaneously, the scheduling system can make trade-off decisions based on priority relationships.

[0027] Step 2: Based on the standardized workflow representation, identify the resource types and data access patterns required for workflow execution to obtain a global resource view; Step 2.1, Resource type identification and hierarchical discovery based on standardized workflow representation; Based on the computational feature label vectors of each task node in the standardized workflow representation output in step 1, a resource type identification method is used to analyze the range of resource types required for workflow execution, resulting in a set of resource requirement types. The resource type identification method traverses each task node in the workflow graph, extracting the demand intensity value for each dimension of its computational feature label vector. When the demand intensity for processor computing power exceeds a threshold, it is marked as requiring high-performance CPU resources; when the demand intensity for accelerator computing power is greater than zero, it is marked as requiring GPU or other dedicated accelerator resources; when the demand intensity for memory capacity is high, it is marked as requiring large memory resources; when the demand intensity for storage read / write is high, it is marked as requiring high-speed storage resources; and when the demand intensity for network transmission is high, it is marked as requiring high-bandwidth network resources. Based on the identified set of resource requirement types and pre-configured resource management domain information, a hierarchical resource discovery method is used to scan the computing resources at three levels: cloud data center, edge computing nodes, and terminal devices, obtaining the registration information of resource nodes at each level. The hierarchical resource discovery method sends a resource query request to the cloud resource management service to obtain a list of available computing nodes in the cloud data center and their basic configuration information. Then, resource probe messages are sent to each edge computing node through the edge gateway to collect the hardware configuration and current status information of the edge nodes. Finally, the computing capability information of registered terminal devices is obtained through the terminal management agent. The resource node registration information includes basic fields such as unique node identifier, node type label, hardware configuration parameters, and network address information.

[0028] Step 2.2, Multidimensional resource capability modeling; Based on the hardware configuration parameters in the resource node registration information obtained in step 2.1, a multi-dimensional resource capability modeling method is used to quantify the computing power of each resource node, resulting in a capability vector for the resource node. The multi-dimensional resource capability modeling method defines a unified resource capability dimension system, which includes five core dimensions: processor computing power, memory bandwidth capacity, storage throughput capacity, network transmission capacity, and accelerator computing power. For each resource node, the capability values ​​for each dimension are calculated based on its hardware configuration parameters. Processor computing power is calculated based on the processor model, number of cores, and clock speed, expressed in standardized computing power units. Memory bandwidth capacity is calculated based on memory type, capacity, and number of channels. Storage throughput capacity is calculated based on storage medium type and interface bandwidth. Network transmission capacity is calculated based on network interface type and nominal bandwidth. Accelerator computing power is calculated based on whether a dedicated accelerator is provided and its model; for nodes without an accelerator, this dimension has a value of zero. The capability values ​​of each dimension are combined to form the capability vector of the resource node.

[0029] Step 2.3, Network Topology Detection and Construction; Based on the network address information from the resource node registration information obtained in step 2.1, a network topology probing method is used to construct a connection graph between resource nodes, thus obtaining the resource topology structure. The network topology probing method obtains network latency information between nodes by sending probe data packets to each resource node and measuring the response time. A bandwidth testing tool is used to measure the available transmission bandwidth between nodes. Based on the node's network address and latency information, the network region and hierarchical position of the node are identified, distinguishing between cloud regions, edge regions, and terminal regions. A resource topology graph is constructed using nodes as vertices and connections between nodes as edges. The edge attributes include two indicators: network latency and available bandwidth.

[0030] Step 2.4: Data status acquisition and monitoring based on standardized workflow representation; Based on the data dependencies and data interaction interface definitions in the workflow graph of the standardized workflow representation output in step 1, and combined with the resource nodes in the resource topology obtained in step 2.3, a data status acquisition agent is used to continuously monitor the distribution status of data in the distributed environment, obtaining data location indexes and data status indicators. The data status acquisition agent determines the data range to be monitored based on the input and output data identifiers defined in the standardized workflow representation, focusing on key datasets involved in the workflow execution process. The data status acquisition agent is deployed on each resource node, responsible for monitoring data block information in local storage and reporting to the center periodically. The collected data status information includes fields such as data block identifier, data block size, node where the data block is located, data block creation time, last access time, and access frequency. For streaming data sources identified in the standardized workflow representation, the data status acquisition agent also monitors dynamic indicators such as data inflow rate, data backlog, and consumption progress. The data location index is constructed using the data block identifier as the key and the list of nodes where the data block is located as the value, supporting quick querying of the physical location of data based on the data identifier. The data status indicators are stored in time-series format, supporting querying historical trends in data status.

[0031] Step 2.5, Global resource view integration; Based on the resource node capability vectors obtained in step 2.2, the resource topology obtained in step 2.3, and the data location indexes and data status indicators obtained in step 2.4, a unified global resource view is constructed using a view integration method, resulting in a complete resource status representation that can be used for scheduling decisions. The view integration method associates and stores resource capability information, topology connection information, and data distribution information in a unified data structure. The global resource view uses resource nodes as the core entity, with each node entity associated with its capability vector, adjacent node list, local data block list, and current load status. The global resource view supports various query operations, including filtering resource nodes by capability criteria, finding neighboring nodes by topological distance, and locating the node where data is located by data identifier. The global resource view uses an incremental update mechanism to maintain data real-time performance; when resource status or data distribution changes, only the affected part is updated, rather than rebuilding the entire view.

[0032] Furthermore, since static resource capacity vectors cannot reflect the real-time load status of resource nodes, scheduling decisions may not match actual resource availability. A dynamic resource availability assessment model can be used to replace static capacity vectors, aiming to provide more accurate real-time resource status information to support scheduling decisions. Specifically, the dynamic resource availability assessment model introduces a real-time load factor on top of the static capacity vector, adjusting the resource node's capacity vector to an available capacity vector. The real-time load factor is calculated based on the current resource utilization rate of the resource node, and the available capacity for each dimension equals the static capacity multiplied by the idle ratio of the corresponding dimension. The dynamic resource availability assessment model also incorporates resource reservation information, deducting reserved but unused resources from the available capacity to avoid redundant resource allocation. The available capacity vector is updated in real-time with changes in the load of resource nodes, providing accurate resource availability information for scheduling decisions.

[0033] Step 3: Based on the standardized workflow representation and global resource view, identify key tasks and obtain critical path analysis results; Step 3.1, Task execution time estimation; Based on the computational feature label vectors of each task node in the workflow graph output from Step 1 and the capability vectors of each resource node in the global resource view output from Step 2, an execution time estimation model is used to calculate the expected execution time of each task node on different resource nodes, resulting in a task resource execution time matrix. The execution time estimation model is based on the computational complexity of the task and the processing capability of the resource nodes. The computational complexity of the task is estimated based on the data size parameter and computational operation type parameter in the task description; the larger the data size and the more complex the computational operation, the higher the computational complexity. The processing capability of the resource node is extracted from the capability vector, and the capability value of the corresponding dimension is selected as the processing capability index according to the computational feature type of the task. The expected execution time is equal to the computational complexity divided by the processing capability, multiplied by an empirical correction coefficient. The empirical correction coefficient is obtained based on historical execution data statistics and is used to correct systematic biases in the estimation model. For streaming operator nodes, the execution time estimation is the processing delay of a single data point, not the total execution time. This sub-step outputs the task execution time estimation value, including the task resource execution time matrix and the expected execution time of each task node.

[0034] Step 3.2, Determine the baseline execution time; Based on the task resource execution time matrix obtained in step 3.1, the execution time of each task node on the optimal resource node is selected as the baseline execution time of the task, resulting in the task baseline execution time vector. The selection of the optimal resource node comprehensively considers both execution time and resource availability, choosing the resource node with the shortest execution time under the premise of resource availability. Each element in the task baseline execution time vector corresponds to a task node in the workflow graph, and the element value is the baseline execution time estimate for that task.

[0035] Step 3.3, Critical Path Analysis and Identification; Based on the task dependencies in the workflow graph representation output in step 1 and the baseline execution time vector of the tasks obtained in step 3.2, the critical path algorithm is used to analyze the time characteristics of the workflow, obtaining the time parameters of each task node and the critical path identifier. The critical path algorithm performs a forward traversal, calculating the earliest start time of each task node according to the topological sort order. The earliest start time of a task is equal to the maximum of the earliest finish times of all its predecessor tasks, and the earliest finish time is equal to the earliest start time plus the baseline execution time. Then, a reverse traversal is performed, calculating the latest start time of each task node starting from the terminal node. The latest start time of a task is equal to the minimum of the latest start times of all its successor tasks minus the baseline execution time of the current task. Finally, the time margin of each task node is calculated, which is equal to the latest start time minus the earliest start time. Task nodes with a time margin of zero are identified as critical path nodes, and all critical path nodes are connected in dependency order to form the critical path. This sub-step outputs a list of critical path tasks and the time parameters of each task node, including the earliest start time, latest start time, time margin, and other time characteristic parameters.

[0036] Step 3.4, Time Budget Allocation; Based on the critical path identifier obtained in step 3.3 and the timeliness constraint parameters output in step 1, a time budget allocation method is used to allocate execution time budgets to each task node on the critical path, resulting in a time budget allocation scheme. The time budget allocation method calculates the total length of the critical path, which is the sum of the baseline execution times of all tasks on the critical path. Then, the upper limit of the end-to-end completion time in the timeliness constraints is compared with the total length of the critical path to calculate the available time margin. The time margin is allocated to each task node on the critical path according to the computational complexity weight of the tasks, allocating more time margin to tasks with higher complexity. The time budget for each critical path task equals its baseline execution time plus the allocated time margin. The time budget allocation scheme also reserves a certain proportion of contingency time margin to cope with unexpected delays during execution. The proportion of the contingency time margin is determined based on the delay fluctuation statistics in historical execution data, and its default value is set as a certain percentage of the total time budget. This sub-step outputs the time budget allocation scheme, including the time budget and contingency time margin configuration for each task node on the critical path.

[0037] Step 3.5: Dynamically update the trigger condition settings; Based on the time parameters of each task node obtained in step 3.3, a dynamic update trigger condition setting method is used to define the trigger rules for critical path recalculation, resulting in a dynamic update trigger condition set. Dynamic update trigger conditions include two categories: execution deviation trigger conditions and state change trigger conditions. Execution deviation trigger conditions stipulate that critical path recalculation is triggered when the deviation ratio between the actual execution time and the expected execution time exceeds a deviation threshold. The default value of the deviation threshold is configured according to the system's stringency regarding timeliness. State change trigger conditions stipulate that critical path recalculation is triggered when a resource node fails, a new resource node is added, or the data distribution changes significantly. The dynamic update trigger condition set is stored in association with the workflow instance, and the trigger conditions are continuously monitored during workflow execution. This sub-step outputs the dynamic update trigger condition set, including the specific parameter configurations for execution deviation trigger conditions and state change trigger conditions.

[0038] This step outputs the critical path analysis results, including a list of critical path tasks, a time budget allocation scheme, estimated task execution times, time parameters for each task node, and a dynamically updated set of triggering conditions, providing a basis for time characteristic analysis for subsequent scheduling decisions.

[0039] Furthermore, since traditional critical path algorithms assume a fixed task execution time, they cannot handle situations where execution time is uncertain. Probabilistic critical path analysis (CPR) can be used as an alternative to deterministic CPR algorithms. The aim is to more accurately identify critical tasks and allocate time budgets when execution time is uncertain. Specifically, CPR models task execution time as a probability distribution rather than a deterministic value. This probability distribution is fitted based on historical execution data, and its parameters include expected value and standard deviation. CPR employs Monte Carlo simulation techniques, randomly sampling task execution times multiple times and calculating the critical path. The frequency of each task node appearing on the critical path is statistically analyzed and used as the task's criticality index. Tasks with high criticality indices, even if not on the critical path in a single analysis, should be given higher resource priority because they are more likely to become bottlenecks in actual execution. Time budget allocation comprehensively considers both the task's criticality index and the degree of uncertainty in execution time, allocating more time margin to tasks with high uncertainty to cope with potential delays.

[0040] Step 4: Evaluate the matching degree between each task and each resource node based on the standardized workflow representation and global resource view, and perform weighted calculations based on the critical path analysis results to obtain a priority-weighted matching degree matrix; Step 4.1, Quantitative modeling of resource requirements; Based on the computational feature label vectors of each task node output in Step 1 and the data scale parameters in the task description, a resource requirement quantification model is used to calculate the multi-dimensional resource requirement vector for each task node, resulting in a set of task resource requirement vectors. The resource requirement quantification model estimates the specific resource requirements of each task across all dimensions based on the task's computational feature type and data scale. Processor computational requirements are calculated based on the task's computational intensity label and data scale; higher computational intensity and larger data scale result in greater processor requirements. Memory requirements are calculated based on the task's memory access pattern label and data scale; tasks requiring caching large amounts of intermediate data have larger memory requirements. Storage read / write requirements are estimated based on the task's data read / write volume. Network transmission requirements are estimated based on the task's input / output data volume and data distribution location. Accelerator requirements are determined based on whether the task contains accelerable computational operations. The combination of requirements across all dimensions forms the task's resource requirement vector, with vector dimensions corresponding to the dimensions of the resource capability vector in Step 2.

[0041] Step 4.2, resource capability normalization processing; Based on the capability vectors of each resource node in the global resource view output in step 2, a normalization method is used to eliminate the dimensional differences between different dimensions, resulting in a normalized resource capability vector set. The normalization method employs max-min normalization; for each resource capability dimension, the maximum and minimum values ​​of all resource nodes in that dimension are calculated, and then the capability value of each node is mapped to the interval between zero and one. The normalization calculation method is: the normalized value equals the difference between the original value and the minimum value divided by the difference between the maximum value and the minimum value. Normalization makes capability values ​​in different dimensions comparable, facilitating subsequent comprehensive matching degree calculations. Simultaneously, the task resource requirement vectors obtained in step 4.1 are processed using the same normalization method to obtain a normalized task requirement vector set.

[0042] Step 4.3, weighted similarity matching calculation and candidate resource node selection; Based on the normalized task requirement vector obtained in step 4.1 and the normalized resource capability vector obtained in step 4.2, a weighted similarity calculation method is used to evaluate the matching degree between each task and each resource node, resulting in a task-resource matching degree matrix. The weighted similarity calculation method determines the weight coefficients of each dimension according to the computational characteristics of the task: processor dimension weight is added for computationally intensive tasks, memory dimension weight is added for memory-intensive tasks, and storage dimension weight is added for I / O-intensive tasks. Then, the weighted cosine similarity between the task requirement vector and the resource capability vector is calculated; a higher similarity value indicates a better matching degree between the task and the resource. The matching degree calculation also considers whether the resource capability meets the task requirement constraint. When the capability of a resource node in a certain dimension is lower than the task requirement, a penalty reduction is applied to the matching degree. The rows of the task-resource matching degree matrix correspond to task nodes, the columns correspond to resource nodes, and the matrix elements are the matching degree scores between the corresponding tasks and resources.

[0043] Based on the task resource matching degree matrix, a candidate resource node screening method is used to determine the set of executable resource nodes for each task, resulting in a candidate resource node mapping table. The candidate resource node screening method sets a matching degree threshold and resource capability constraints, marking resource nodes that simultaneously meet the following conditions as candidate resource nodes for a task: the resource node's capabilities in all dimensions are not lower than the task's basic requirements; the matching degree score between the task and the resource node exceeds a preset matching degree threshold. The matching degree threshold is configured according to the system's scheduling quality requirements; a higher threshold results in fewer candidate resource nodes but better matching quality. The candidate resource node mapping table is stored in a structure with task identifiers as keys and the list of candidate resource nodes as values, providing a range of optional resources for subsequent scheduling decisions.

[0044] Step 4.4, assign priority weights; Based on the critical path task list and time budget allocation scheme output from step 3, a priority weight assignment method is used to assign scheduling priority weights to each task node, resulting in a task priority weight vector. The priority weight assignment method determines the priority weight based on whether the task is on the critical path and the size of the task's time margin. Tasks on the critical path are assigned higher base priority weights, while the priority weights of non-critical path tasks decrease according to their time margins; the smaller the time margin, the higher the priority. Priority weights also consider the strictness of the task's timeliness constraints; tasks with stricter timeliness constraints have higher priority. Priority weights are represented using normalization, with the sum of all task priority weights being one, facilitating weighted calculations in subsequent scheduling decisions. Each element in the task priority weight vector corresponds to the priority weight value of a task node.

[0045] Step 4.5, calculate the priority-weighted matching degree; Based on the task resource matching degree matrix obtained in step 4.3 and the task priority weight vector obtained in step 4.4, the final scheduling reference matrix is ​​generated using the priority-weighted matching degree calculation method, resulting in the priority-weighted matching degree matrix. The priority-weighted matching degree calculation method multiplies the task's priority weight by its matching degree score to obtain the priority-weighted matching degree value. The priority-weighted matching degree matrix is ​​used to guide subsequent scheduling decisions; the weighted matching degree value of high-priority tasks is amplified, making it easier for them to obtain the allocation of high-quality resources during resource contention.

[0046] Furthermore, since static weight coefficients cannot adapt to the differentiated needs of different workload scenarios, an adaptive weight learning mechanism can be used to replace fixed weight configuration. The aim is to automatically optimize weight parameters based on historical scheduling performance to improve matching accuracy. Specifically, the adaptive weight learning mechanism maintains historical versions of weight parameters and corresponding scheduling performance evaluation metrics, including task completion time deviation, resource utilization, and timeliness constraint satisfaction rate. The adaptive weight learning mechanism uses an online learning algorithm to adjust weight parameters based on recent scheduling performance feedback. When the completion time deviation of a certain type of task is large, the weight of the corresponding resource dimension for that type of task is increased to improve matching accuracy. The adjustment of weight parameters adopts a gradual update strategy, with each adjustment limited by a preset learning rate parameter to avoid drastic weight fluctuations that could lead to scheduling instability. The default value of the learning rate parameter is configured according to the system's stability requirements; a smaller learning rate is used for scenarios with high stability requirements.

[0047] Step 5: Based on the priority-weighted matching degree matrix and the global resource view, solve for the optimal mapping scheme from task to resource node; based on the critical path analysis results, lock the computing resources of the critical path tasks to obtain the task scheduling mapping scheme. Step 5.1, Data transmission overhead assessment; Based on the data location index and resource topology output in step 2, and combined with the input data identifiers of each task node in the workflow graph representation output in step 1, a data transmission cost calculation method is used to evaluate the data transmission cost required for each task to be executed on each candidate resource node, resulting in a data transmission cost matrix. The data transmission cost calculation method queries the data location index based on the task's input data identifier to obtain a list of resource nodes where the input data resides. Then, based on the network latency and bandwidth information in the resource topology, the data transmission time from the data node to the candidate execution node is calculated. The data transmission time equals the data size divided by the available bandwidth plus the network latency. When the input data is distributed across multiple nodes, the data transmission cost is taken as the maximum value of the transmission time for each data block, because data transmission can be performed in parallel, but task startup requires waiting for all data to be ready. The rows of the data transmission cost matrix correspond to task nodes, the columns correspond to resource nodes, and the matrix elements are the estimated data transmission cost values ​​when the corresponding task is executed on the corresponding resource node.

[0048] Step 5.2, Data prefetching decision generation; Based on the critical path task list and time budget allocation scheme output in step 3, and combined with the data transmission cost matrix obtained in step 5.1, a data prefetching decision method is used to determine the tasks requiring data prefetching and the prefetching parameters, resulting in a data prefetching instruction list. The data prefetching decision method filters tasks with high data transmission costs on the critical path as prefetching candidates, with the data transmission cost threshold determined as a certain proportion of the task's time budget. For each prefetching candidate task, the optimal data prefetching start time is calculated. The prefetching start time equals the task's expected start time minus the data transmission time, minus the prefetching lead time. The prefetching lead time is used to cope with fluctuations during transmission, and its default value is determined based on network stability statistics. The data prefetching decision also needs to check whether the target node's storage resources are sufficient to accommodate the prefetched data. When storage resources are insufficient, data eviction or selection of another target node needs to be triggered. The data prefetching instruction includes fields such as prefetch data identifier, source node, target node, prefetching start time, and prefetch data volume.

[0049] Step 5.3, Construction of the comprehensive scheduling objective function; Based on the priority-weighted matching degree matrix output in step 4 and the data transmission overhead matrix obtained in step 5.1, a comprehensive scheduling objective function is constructed to integrate multiple optimization objectives into a unified objective function, resulting in the comprehensive scheduling objective function. The comprehensive scheduling objective function includes three optimization objectives: maximizing resource matching degree, minimizing data transmission overhead, and load balancing. The resource matching degree objective uses the priority-weighted matching degree value output in step 4, aiming to maximize the sum of matching degree scores for all tasks. The data transmission overhead objective uses the data transmission overhead value obtained in step 5.1, aiming to minimize the sum of data transmission overhead for all tasks. The load balancing objective uses the load variance of resource nodes, aiming to minimize the load difference between resource nodes. These three objectives are integrated into the comprehensive objective function through a weighted summation. The weight coefficients are configured according to business priorities; for scenarios with high timeliness requirements, the weight of the data transmission overhead objective is increased, while for scenarios sensitive to resource costs, the weight of the load balancing objective is increased.

[0050] Step 5.4, heuristic optimization solution; Based on the comprehensive scheduling objective function obtained in step 5.3, a heuristic optimization algorithm is used to solve for the optimal mapping scheme from tasks to resource nodes, resulting in a task scheduling mapping scheme. The heuristic optimization algorithm combines a greedy strategy with local search. In the initial solution generation phase, resource nodes with the optimal comprehensive objective function value are selected sequentially for each task according to its priority from high to low, generating an initial scheduling scheme. In the local search phase, the initial scheme is optimized through two neighborhood operations: task swapping and task migration. Task swapping involves exchanging the execution nodes of two tasks, while task migration involves moving a task from its current node to another node. After each neighborhood operation, the change in the comprehensive objective function value is evaluated; operations that improve the objective function value are accepted, while operations that worsen it are rejected. The local search iterates until a preset iteration limit is reached or multiple consecutive iterations fail to improve the objective function value. Task dependency constraints and resource capacity constraints are followed during the solution process to ensure the feasibility of the generated scheduling scheme. The task scheduling mapping scheme is represented as a mapping table with task node identifiers as keys and target resource node identifiers as values.

[0051] Step 5.5, resource reservation execution; Based on the task scheduling mapping scheme obtained in step 5.4 and the critical path task list and time budget allocation scheme in the critical path analysis results output in step 3, a resource reservation execution method is used to lock the required computing resources for critical path tasks, resulting in resource reservation instructions. The resource reservation execution method iterates through the critical path task list. For each critical path task, a resource reservation record is created on the target resource node based on its resource requirement vector and time budget. The resource reservation record includes fields such as the reserved task identifier, the reserved resource node, the reserved resource amount, the reservation start time, and the reservation end time. The reservation start time is determined based on the earliest start time of the task, and the reservation end time is calculated based on the task's time budget. The resource reservation operation deducts the reserved resources from the node's available resources to prevent them from being occupied by other tasks. The resource reservation status record summarizes the reservation information of all critical path tasks for subsequent resource availability queries and reservation release management.

[0052] Furthermore, since greedy strategies may get stuck in local optima and fail to find a globally optimal scheduling scheme, simulated annealing optimization can be used to replace simple greedy strategies. The aim is to improve the global optimization effect of the scheduling scheme by probabilistically accepting inferior solutions to escape local optima. Specifically, the simulated annealing optimization algorithm introduces a temperature-controlled probabilistic acceptance mechanism during the local search process. When a neighborhood operation causes the objective function value to deteriorate, the operation is accepted with a certain probability instead of being directly rejected. The acceptance probability is related to the degree of deterioration of the objective function value and the current temperature parameter; the smaller the deterioration and the higher the temperature, the greater the acceptance probability. The temperature parameter gradually decreases during the iteration process. In the initial high-temperature stage, the algorithm has a strong exploration ability and can escape local optima. In the later low-temperature stage, the algorithm converges to the vicinity of the current optimal solution for a refined search. The initial value of the temperature parameter and the cooling rate are configured according to the problem size and solution time constraints. For large-scale problems, a higher initial temperature and a slower cooling rate are used to ensure sufficient search.

[0053] Step 6: Based on the task scheduling mapping scheme and critical path analysis results, predict data traffic and identify resource bottlenecks; dynamically adjust resource allocation based on the identification results to obtain elastic scaling decision instructions; Step 6.1, Data traffic monitoring and collection; Based on the input data source identifiers of each task node in the task scheduling mapping scheme output in step 5, a traffic monitoring agent continuously collects data inflow rate indicators at each data source deployment point to obtain a data traffic time series set. The traffic monitoring agent records the instantaneous inflow rate of the data source at a fixed sampling period. The default value of the sampling period is configured according to the data source type and monitoring accuracy requirements. Streaming data sources with high real-time requirements use a shorter sampling period. The collected traffic indicators include instantaneous inflow rate, average inflow rate within the sampling period, peak inflow rate, and data backlog. The data traffic time series is stored in a structure with the data source identifier as the key and the traffic indicator time series as the value, supporting querying historical traffic data of a specified data source within a specified time range.

[0054] Step 6.2, Time Series Flow Forecasting; Based on the time series data flow obtained in step 6.1, a time series forecasting model is used to predict the future flow of each data source, resulting in a set of flow forecasting results. The time series forecasting model employs exponential smoothing to model and predict the flow data. Exponential smoothing can capture the trend changes and periodic patterns of flow. The forecasting model smooths historical flow data and calculates the horizontal, trend, and seasonal components of the flow. Then, based on the components obtained from the decomposition, the predicted flow values ​​within the future time window are extrapolated. The length of the forecasting time window is determined according to the expected completion time of the elastic scaling operation, ensuring that the forecasting window covers the entire time period from decision-making to scaling completion. The flow forecasting results include predicted values ​​and prediction confidence intervals. The confidence intervals reflect the degree of uncertainty in the forecast and are used for risk assessment in subsequent decision-making.

[0055] Step 6.3, Resource bottleneck prediction and analysis; Based on the traffic prediction results obtained in step 6.2 and the current processing capacity of each processing stage in the task scheduling mapping scheme output in step 5, a resource bottleneck prediction method is used to assess whether there are potential resource bottlenecks, resulting in a resource bottleneck prediction report. The resource bottleneck prediction method calculates the resource demand of each processing stage under the predicted traffic based on the traffic prediction results. The resource demand equals the predicted traffic multiplied by the resource consumption coefficient per unit of data. Then, the resource demand is compared with the currently allocated resources to calculate the predicted resource utilization rate. When the predicted resource utilization rate exceeds the bottleneck threshold, the processing stage is marked as a potential bottleneck. The default value of the bottleneck threshold is configured according to the resource type and service quality requirements. The resource bottleneck prediction also incorporates the timeliness constraint parameters from the critical path analysis results output in step 3 to assess whether the bottleneck will lead to a violation of the timeliness constraint. When the predicted processing delay exceeds the upper limit of the timeliness constraint, the bottleneck is marked as urgent. The resource bottleneck prediction report includes information such as the bottleneck location, bottleneck severity, expected occurrence time, and impact scope.

[0056] Step 6.4, Calculate the scaling scale; Based on the bottleneck information in the resource bottleneck prediction report obtained in step 6.3, a scaling-up calculation method is used to determine the resource expansion scale required to eliminate the bottleneck, thus obtaining the resource expansion demand. The scaling-up calculation method calculates the expansion scale based on the resource gap at the bottleneck point and the target resource utilization rate. The resource gap equals the predicted resource demand minus the current resource supply. The target resource utilization rate is the desired resource utilization level, with its default value configured according to resource type and cost-benefit balance requirements. The expansion scale equals the resource gap divided by the unit resource processing capacity corresponding to the target resource utilization rate. The scaling-up calculation also considers the minimum granularity constraint of the expansion operation; when the calculated expansion scale is smaller than the minimum granularity, it is rounded up to the minimum granularity. Simultaneously, the expected completion time of the expansion operation is considered to ensure that the expansion can be completed before the bottleneck actually occurs. When the expected completion time is insufficient, the expansion scale is increased to establish a resource buffer in advance.

[0057] Step 6.5, Cost Optimization Decision; Based on the resource expansion requirements obtained in step 6.4 and the resource cost information in the global resource view output in step 2, a cost optimization decision-making method is used to optimize the resource cost of elastic scaling while meeting timeliness constraints, resulting in a cost-optimized scaling solution. The cost optimization decision-making method evaluates the cost-benefit ratio of different resource types and sources, prioritizing resources with higher cost-benefit ratios for expansion. For tasks on non-critical paths, lower-cost preemptive instances or edge node resources are considered; although these resources have lower stability, their costs are reduced. The cost optimization decision also considers the elastic pricing factor of resources; expanding during periods of low resource demand can yield more favorable prices. The cost-optimized scaling solution minimizes resource costs while meeting timeliness constraints.

[0058] Step 6.6, generating scaling instructions; Based on the resource expansion requirements obtained in step 6.4 and the cost-optimized scaling plan obtained in step 6.5, a scaling instruction generation method is used to generate specific elastic scaling decision instructions, resulting in a set of elastic scaling decision instructions. The scaling instruction generation method converts the scaling plan into executable resource management operation instructions. Expansion instructions include fields such as the type of resource to be expanded, the number of resources to be expanded, the target deployment location, resource configuration parameters, and execution timing. Shrinking instructions include fields such as the shrinking resource identifier, the shrinking timing, and data migration requirements. Scaling instructions also include rollback condition definitions, triggering a rollback operation to restore the original state when the scaling operation fails or the effect does not meet expectations. The set of elastic scaling decision instructions is sorted by execution timing, supporting the sequential execution of each scaling operation in chronological order.

[0059] Furthermore, since a single time series forecasting model may not accurately capture complex traffic flow patterns, an ensemble forecasting method can be used instead of a single model. The aim is to improve the accuracy and robustness of traffic flow forecasting by fusing the results of multiple forecasting models. Specifically, the ensemble forecasting method runs multiple different types of time series forecasting models simultaneously, including exponential smoothing models, autoregressive moving average models, and machine learning-based forecasting models. Each model independently forecasts traffic flow data, generating its own prediction results. The ensemble forecasting method uses a weighted average to fuse the prediction results from each model, with the weights dynamically adjusted based on the prediction accuracy of each model on historical data; models with higher prediction accuracy receive greater weight. The ensemble forecasting method also employs prediction interval fusion technology, combining the prediction confidence intervals of each model to generate a more reliable uncertainty estimate. By integrating the prediction results of multiple models, the ensemble forecasting method can reduce the prediction bias of a single model and improve the overall accuracy of traffic flow forecasting.

[0060] Step 7: Based on the task scheduling mapping scheme and elastic scaling decision instructions, execute task scheduling and resource scaling operations; monitor the execution status in real time, perform deviation analysis and dynamic parameter adjustment, and obtain an execution deviation analysis report and optimized scheduling parameter configuration; Step 7.1, scheduling instruction conversion and issuance; Based on the task scheduling mapping scheme output in step 5, an instruction conversion method is used to convert the scheduling mapping relationship into task startup instructions executable by each computing node, resulting in a set of task startup instructions. The instruction conversion method iterates through each task resource mapping entry in the scheduling mapping scheme, generating a corresponding startup instruction for each task. The task startup instruction includes fields such as task identifier, task execution code or image reference, task input data location, task output data target location, resource quota limits, and startup time constraints. Instruction conversion also needs to handle dependencies between tasks, setting startup trigger conditions for tasks with predecessor dependencies. These trigger conditions stipulate that a task can only start after all predecessor tasks have been completed. The task startup instruction is sent to the task execution agent of the target resource node via a message queue. The execution agent returns a confirmation message after receiving the instruction. Simultaneously, the data prefetch instruction output in step 5 is sent to the data management agent of the relevant node, triggering the execution of the data prefetch operation.

[0061] Step 7.2, execute elastic scaling; Based on the set of elastic scaling decision instructions output in step 6, the resource management interface is used to execute resource expansion or contraction operations, obtaining a scaling operation execution status record. The resource management interface calls the corresponding resource provider's interface to execute resource operations according to the resource type and target location in the scaling instruction. For cloud resources, the cloud service provider's resource management application programming interface is called to create or destroy computing instances. For edge resources, the edge resource management platform's interface is called to adjust resource allocation. During the scaling operation execution, the operation status is continuously monitored, recording the operation start time, current progress, and estimated completion time. When the scaling operation is completed, the global resource view in step 2 is updated, adding the newly added resources to the schedulable resource pool or removing the released resources from the resource pool. The scaling operation execution status record includes fields such as operation identifier, operation type, execution status, start time, completion time, and operation result.

[0062] Step 7.3, perform status monitoring; Based on the task startup command issuance result obtained in step 7.1, the execution status monitoring service continuously collects task execution status information to obtain a task execution status time series. The execution status monitoring service obtains task execution information through status messages periodically reported by the task execution agent. The monitored status information includes the current execution stage of the task, the amount of data processed, resource consumption, execution start time, and estimated completion time. For critical path tasks, the monitoring service uses a higher sampling frequency to promptly detect execution delay risks. The execution status monitoring service also monitors the execution progress of data prefetching operations to ensure that prefetched data is ready before task startup. The task execution status time series is stored in a structure with the task identifier as the key and the status information time series as the value, supporting queries of the task's execution status at any point in time.

[0063] Step 7.4: Perform deviation analysis; Based on the task execution status time series obtained in step 7.3 and the estimated task execution time values ​​from the critical path analysis results output in step 3, a deviation analysis method is used to compare the actual execution indicators with the expected indicators of the tasks, resulting in an execution deviation analysis report. The deviation analysis method calculates the difference and ratio between the actual and expected execution times of the tasks; the difference reflects the degree of absolute deviation, and the ratio reflects the degree of relative deviation. Deviation analysis also statistically analyzes the deviation between the actual and expected values ​​of resource consumption, identifying task types with inaccurate resource estimations. The deviation analysis method further analyzes the causes of deviations, identifying the correlation between deviations and factors such as resource competition, data skew, and network fluctuations through correlation analysis. The execution deviation analysis report includes deviation statistics for each task, deviation cause analysis, and deviation trends, providing a basis for subsequent parameter optimization.

[0064] Step 7.5, dynamically adjust the trigger determination; Based on the execution deviation analysis report obtained in step 7.4 and the dynamic update triggering conditions in the critical path analysis results output in step 3, a dynamic adjustment triggering decision method is used to evaluate whether dynamic adjustment of the scheduling strategy needs to be triggered, resulting in a dynamic adjustment triggering decision. The dynamic adjustment triggering decision method compares the execution deviation with a preset triggering threshold. When the execution time deviation of a critical path task exceeds the deviation threshold, an emergency adjustment operation is triggered. Emergency adjustment operations include adding resources to delayed tasks, triggering task preemption to release resources for high-priority tasks, or recalculating the critical path and adjusting the scheduling scheme of subsequent tasks. The dynamic adjustment triggering decision also monitors whether the critical path has changed. When a previously non-critical path task becomes a new critical path due to execution delay, the critical path recalculation process in step 3 is triggered. The dynamic adjustment triggering decision records information such as the type of adjustment, the triggering reason, and the adjustment target.

[0065] Step 7.6, adaptive parameter optimization; Based on the execution deviation analysis report obtained in step 7.4, an adaptive parameter optimization method is used to adjust the parameter configuration of the scheduling algorithm according to historical execution feedback, resulting in an optimized scheduling parameter configuration. The adaptive parameter optimization method identifies parameter configuration problems leading to systematic deviations and generates parameter adjustment suggestions. For task execution time estimation deviations, the empirical correction coefficients of the execution time estimation model in step 3.1 are adjusted to make the estimated values ​​closer to the actual values. For resource matching degree calculation deviations, the weight coefficients of each dimension in step 4.3 are adjusted to improve matching accuracy. For elastic scaling timing deviations, the bottleneck threshold in step 6.3 and the lead time parameter in step 6.4 are adjusted to optimize the timeliness of scaling decisions. Parameter adjustments adopt a gradual update strategy, with each adjustment limited to a preset adjustment step size to avoid system instability caused by drastic parameter changes. The optimized scheduling parameter configuration is persistently stored and takes effect in subsequent scheduling decisions.

[0066] Furthermore, since rule-based parameter adjustment methods struggle to handle complex interactions between parameters, reinforcement learning can be used as an alternative. The aim is to automatically discover the optimal parameter configuration combination through end-to-end learning. Specifically, reinforcement learning models scheduling parameter optimization as a Markov decision process. The state space includes current workload characteristics, resource status, and historical performance metrics; the action space includes adjustment operations for each scheduling parameter; and the reward function is designed based on scheduling performance metrics, including a weighted combination of factors such as timeliness constraint satisfaction rate, resource utilization, and task completion time. Reinforcement learning employs a policy gradient algorithm to learn the mapping policy from state to action, continuously optimizing policy parameters through interaction with the environment. To avoid the impact of the exploration process on the production system, reinforcement learning combines offline learning with online fine-tuning. Offline policy learning is performed on historical data, followed by conservative online fine-tuning in the production environment.

[0067] In one embodiment of the present invention, a specific example is provided: A large infrastructure construction company has deployed a unified big data platform that needs to simultaneously support various data processing tasks, including tunnel construction safety risk early warning, historical data analysis, and model training. The tunnel construction safety risk early warning system collects real-time monitoring data such as surrounding rock stress, groundwater level, and harmful gas concentration from hundreds of sensors deployed at the construction site. It needs to complete data processing, multi-model inference, and early warning decision output within strict time constraints.

[0068] The data obtained by the system after applying the resource dynamic scheduling method of the present invention is shown in Table 1: Table 1: Examples of data acquired by the system; The system's resource configuration examples are shown in Table 2: Table 2: Example of system resource configuration; After applying the scheduling method of this invention, the system exhibits the following advantages: by mixing real-time tasks for risk warning and training tasks for historical data batch processing, the overall resource utilization of the cluster is significantly improved; by prioritizing resources and scheduling data locally for tasks on the critical path of the warning workflow, the end-to-end latency of time-sensitive tasks is reduced; and by automatically scheduling some latency-insensitive data preprocessing tasks to lower-cost edge nodes based on task characteristics, computational costs are effectively controlled.

[0069] The big data platform resource dynamic scheduling method and system provided by this invention not only performs well in the tunnel construction risk early warning scenario of this embodiment, but is also applicable to any big data application scenario with complex workflows and timeliness requirements, such as financial risk control, e-commerce recommendation, smart cities, and industrial Internet of Things.

[0070] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A method for dynamic resource scheduling in a big data platform system architecture and supporting environment, characterized in that, include: Obtain task description information, perform semantic parsing, and obtain a standardized workflow representation; Based on a standardized workflow representation, the resource types and data access patterns required for workflow execution are identified to obtain a global resource view; Based on standardized workflow representation and global resource view, key tasks are identified, and critical path analysis results are obtained. The matching degree between each task and each resource node is evaluated based on the standardized workflow representation and global resource view. The weighted calculation is combined with the critical path analysis results to obtain the priority-weighted matching degree matrix. Based on the priority-weighted matching degree matrix and the global resource view, the optimal mapping scheme from task to resource node is solved; based on the critical path analysis results, the computing resources of the critical path tasks are locked to obtain the task scheduling mapping scheme. Based on the task scheduling mapping scheme and critical path analysis results, data traffic is predicted and resource bottlenecks are identified; based on the identification results, resource allocation is dynamically adjusted to obtain elastic scaling decision instructions. Based on the task scheduling mapping scheme and elastic scaling decision instructions, execute task scheduling and resource scaling operations; Real-time monitoring of execution status, deviation analysis and dynamic parameter adjustment, resulting in execution deviation analysis reports and optimized scheduling parameter configurations.

2. The resource dynamic scheduling method for a big data platform system architecture and supporting environment according to claim 1, characterized in that, The semantic parsing includes: Based on the task declaration field and execution mode field in the task description information, a task type identification method combining rule matching and feature recognition is used to obtain task type classification labels; Based on task type classification labels, a differentiated workflow diagram construction strategy is adopted to obtain a task workflow diagram representation; Based on each task node in the task workflow graph representation, a multi-dimensional feature extraction method is used to perform feature analysis on each task node to obtain the calculated feature label vector of the task node. Based on the service level agreement parameters specified by the user in the task description information, the timeliness requirements of the task are extracted using the timeliness constraint parsing method, resulting in a set of timeliness constraint parameters for the workflow.

3. The resource dynamic scheduling method for a big data platform system architecture and supporting environment according to claim 2, characterized in that, The task type classification includes batch processing, streaming computing, and hybrid tasks. Batch processing tasks are characterized by one-time execution and bounded data input, while streaming computing tasks are characterized by unbounded data input and continuous execution. Hybrid tasks include both batch processing and streaming computing.

4. The resource dynamic scheduling method for a big data platform system architecture and supporting environment according to claim 1, characterized in that, The range of resource types and data access modes required for the identification workflow execution include: Based on the computational feature label vectors of each task node in the standardized workflow representation, a resource type identification method is used to analyze the range of resource types required for workflow execution, and a set of resource requirement types is obtained. Based on the set of resource demand types and pre-configured resource management domain information, a hierarchical resource discovery method is used to scan computing resources at three levels: cloud data center, edge computing node, and terminal device. Based on the hardware configuration parameters in the resource node registration information, a multi-dimensional resource capability modeling method is used to quantify the computing power of each resource node, thereby obtaining the capability vector of the resource node.

5. The resource dynamic scheduling method for a big data platform system architecture and supporting environment according to claim 1, characterized in that, The process of identifying key tasks and obtaining critical path analysis results based on standardized workflow representation and global resource view includes: Based on the computational feature label vector of each task node in the workflow graph representation and the capability vector of each resource node in the global resource view, the expected execution time of each task node on different resource nodes is calculated using an execution time estimation model. Based on the task dependencies and task baseline execution time vectors in the workflow graph representation, the critical path algorithm is used to analyze the time characteristics of the workflow and obtain the time parameters and critical path identifiers of each task node. Based on the critical path identifier and timeliness constraint parameters, a time budget allocation method is used to allocate execution time budgets to each task node on the critical path.

6. The resource dynamic scheduling method for a big data platform system architecture and supporting environment according to claim 1, characterized in that, The process of evaluating the matching degree between each task and each resource node based on standardized workflow representation and global resource view, and performing weighted calculations in conjunction with critical path analysis results, includes: Based on the computational feature label vector of each task node and the data scale parameter in the task description, a resource demand quantification model is used to calculate the multidimensional resource demand vector of each task node. Based on the normalized task requirement vector and the normalized resource capability vector, a weighted similarity calculation method is used to evaluate the matching degree between each task and each resource node, and the task-resource matching degree matrix is ​​obtained. Based on the critical path task list and time budget allocation scheme in the critical path analysis results, a priority weight assignment method is used to assign scheduling priority weights to each task node.

7. The resource dynamic scheduling method for a big data platform system architecture and supporting environment according to claim 1, characterized in that, The optimal mapping scheme from the solution task to the resource node includes: Based on data location index and resource topology, and combined with the input data identifiers of each task node in the workflow diagram representation, a data transmission overhead calculation method is used to evaluate the data transmission cost required for each task to be executed on each candidate resource node. Based on the priority-weighted matching degree matrix and the data transmission overhead matrix, the optimization objectives are integrated into a unified objective function, resulting in the comprehensive scheduling objective function; Based on the comprehensive scheduling objective function, a heuristic optimization algorithm is used to solve for the optimal mapping scheme from tasks to resource nodes, thus obtaining the task scheduling mapping scheme.

8. The resource dynamic scheduling method for a big data platform system architecture and supporting environment according to claim 1, characterized in that, The predicted data flow and identification of resource bottlenecks include: Based on the input data source identifier of each task node in the task scheduling and mapping scheme, a traffic monitoring agent is used to continuously collect data inflow rate indicators at each data source deployment point. Based on a time series dataset of data traffic, a time series forecasting model is used to predict the future traffic of each data source. Based on the traffic prediction results and the current processing capacity of each processing stage in the task scheduling mapping scheme, a resource bottleneck prediction method is used to assess whether there are potential resource bottlenecks. Based on the bottleneck information in the resource bottleneck prediction report, the scale of resource expansion required to eliminate the bottleneck is determined using a scaling calculation method.

9. The resource dynamic scheduling method for a big data platform system architecture and supporting environment according to claim 1, characterized in that, The execution of task scheduling and resource scaling operations includes: Based on the task scheduling mapping scheme, the instruction conversion method is used to convert the scheduling mapping relationship into task start instructions that can be executed by each computing node; Based on the set of elastic scaling decision instructions, resource expansion or contraction operations are performed using the resource management interface. Based on the result of the task start command, the execution status monitoring service continuously collects the task's execution status information. Based on the task execution time estimates from the time series analysis of task execution status and critical path analysis, the deviation analysis method is used to compare the actual execution indicators of the task with the expected indicators.

10. The resource dynamic scheduling method for a big data platform system architecture and supporting environment according to claim 1, characterized in that, The real-time monitoring of execution status, deviation analysis, and dynamic parameter adjustment include: Based on the task execution time estimates from the task execution status time series and critical path analysis results, the actual execution indicators of the task are compared with the expected indicators to obtain an execution deviation analysis report. Based on the dynamic update triggering conditions in the execution deviation analysis report and critical path analysis results, a dynamic adjustment triggering determination method is used to evaluate whether dynamic adjustment of the scheduling strategy needs to be triggered. Based on the execution deviation analysis report, an adaptive parameter optimization method is used to adjust the parameter configuration of the scheduling algorithm according to historical execution feedback, thereby obtaining an optimized scheduling parameter configuration.

Images