High-balance serverless workflow scheduling method and system based on data flow driving of kubernetes

Abstract

The invention discloses a kubernetes-based data flow driven high-balance serverless workflow scheduling method and system, and the method comprises the steps: S1, receiving a JSON format workflow file transmitted by a user, and creating related resources; s2, on the basis of related resources, task pods are deployed in combination with node resource use conditions and task resource requests, and each task pod maintains a request queue; s3, based on the deployed task pod, selecting a root task pod with the shortest request queue length to send a task execution request; s4, based on the execution request, executing a task through a sidecar container, and based on a subsequent task scheduling strategy of data transmission time, request queue length and task execution time, selecting an optimal subsequent task pod to send request data; and S5, based on the execution condition, evaluating the workflow execution condition, and based on the automatic expansion strategy, triggering the automatic expansion of the task pod. The problems that an existing data-driven workflow scheduling scheme is too coarse in granularity, low in resource utilization efficiency, inflexible in expansion strategy and the like are solved.

Description

Technical Field

[0001] The present invention relates to the fields of container orchestration and cloud native computing, and in particular to a highly balanced serverless workflow scheduling method and system driven by data streams based on Kubernetes. Background Art

[0002] With the rapid development of cloud computing technology, the Serverless architecture has become a key solution for modern distributed systems. By abstracting computing resource management, this architecture allows developers to focus on business logic implementation without having to worry about the underlying infrastructure. The Serverless architecture offers high flexibility and automatic scalability, effectively reducing costs when handling short-duration, high-frequency requests and providing on-demand resource allocation.

[0003] Kubernetes, a widely used open-source container orchestration platform, excels in automating the deployment, scaling, and management of containerized workloads. Its powerful automated management and scalability dynamically schedules computing resources, ensuring high availability for containerized applications. Kubernetes automates tasks such as service discovery, load balancing, and container orchestration, offering significant advantages when handling large numbers of distributed applications.

[0004] However, existing serverless workflow scheduling systems face multiple challenges. Current data-driven workflow scheduling systems primarily rely on coarse-grained task partitioning, dividing and scheduling workflows into subgraphs. This makes it difficult to implement refined task-level scheduling and is unable to cope with complex task dependencies and load fluctuations. Existing scheduling methods lack effective feedback mechanisms for inter-task dependencies and runtime performance, making it impossible to flexibly adjust scheduling strategies based on real-time task status and resource requirements, resulting in underutilized resources and degraded performance.

[0005] Furthermore, current Kubernetes workflow scheduling systems often use fixed-granularity scheduling strategies. This coarse-grained scheduling approach struggles to dynamically adapt to complex dependencies between tasks, leading to resource waste and performance degradation under high-concurrency loads. The system's scheduling capabilities and auto-scaling mechanisms are also limited when handling high-concurrency requests, making it difficult to ensure efficient load balancing and real-time task execution despite high data transmission overhead.

[0006] These problems prompted us to propose a new solution based on Kubernetes technology, which aims to provide more fine-grained data-driven workflow scheduling and elastic execution mechanism to cope with workflow application scenarios with strong task correlation, high data transmission overhead and significant request load fluctuations. Summary of the Invention

[0007] The present invention provides a highly balanced serverless workflow scheduling method and system driven by data streams based on Kubernetes to solve the above-mentioned problems existing in the prior art.

[0008] In order to achieve the above object, the present invention provides the following technical solutions: A highly balanced serverless workflow scheduling method driven by data streams based on Kubernetes, including: S1: Receive the JSON format workflow file sent by the user and create related resources; S2: Deploys task pods based on relevant resources, node resource usage, and task resource requests. Each task pod maintains a request queue. S3: Based on the deployed task pods, select the root task pod with the shortest request queue length to send the task execution request; S4: Based on the execution request, the sidecar container executes the task and selects the optimal subsequent task pod to send the request data based on the subsequent task scheduling strategy based on data transmission time, request queue length, and task execution time. S5: Based on the execution status, the workflow execution status is evaluated and the automatic scaling of the task pod is triggered based on the automatic scaling policy.

[0009] The S2 step includes: S21: Set the number of replicas for each task Pod to 2; S22: Sort tasks in the workflow based on their resource requirements, deploying task pods from most to least resource-demanding. S23: When selecting a node, select a node whose remaining computing resources are greater than the task requested resources and whose remaining computing resources have the highest ratio to the total computing resources of the node; S24: Update the resource usage record of the node, including the number of allocated CPU cores, allocated memory capacity, and remaining available resources, so as to provide resource data for the scheduling of subsequent tasks.

[0010] Among them, step S4 includes: S41: The sidecar container data synchronization module receives request data from the outside and puts the request into the request queue. S42: The sidecar container log module records the data transmission time; S43: The sidecar container task scheduling module obtains and executes task requests from the request queue; S44: The sidecar container log module records and synchronizes task execution time; S45: After the task is executed, the expected waiting time is calculated based on the data transmission time and the request queue length, and the subsequent task pod with the shortest expected waiting time is selected to send the request data.

[0011] Wherein, step S45 includes: S451: Obtain the request queue lengths of the external task Pod and the internal task Pod from the state database, and determine the Pod with the smallest queue length value among the external and internal Pods respectively; S452: Obtain the average execution time of historical tasks and the data transmission time of the current task from the log database; S453: The estimated waiting time for the computing task to be dispatched to the external Pod is equal to the queue length multiplied by the average execution time plus the data transmission time; S454: Compare the external estimated waiting time with the internal Pod estimated waiting time, and select the Pod with the shorter estimated waiting time for task scheduling.

[0012] Among them, step S5 includes: S51: monitors the completion time of the task Pod request; S52: Analyze the cause of the delay, determine whether there is a computing resource bottleneck by comparing the queue time with a preset threshold, and determine whether there is a data location problem by comparing the data transmission time with a preset threshold; S53: If the queue time is too long, enter the computing resource expansion process; S54: If the data transmission time is too long, enter the data perception expansion process.

[0013] Wherein, step S53 includes: S531: Evaluate the parent task and child tasks of the current task; S532: Based on the node's resource status and data transmission overhead, a weighted score is formed by calculating the inverse relationship between the percentage of the node's remaining resources to the total resources and the data transmission time. The node with the highest score is selected for task pod expansion. S533: Expand the task Pod on the selected node to alleviate the load pressure of the set node.

[0014] Wherein, step S54 includes: S541: Analyze the data transmission between the parent and child tasks and locate the task with the longest data transmission time; S542: Compare the resource status of the current task node and the data source task node, calculate the CPU utilization and memory usage of both, and select the node whose average CPU utilization and memory usage is lower than the system preset threshold as the deployment target, where the system preset threshold is dynamically set based on historical task execution data; S543: If a data source node is selected, expand the Pod of the current task; if the current task node is selected, expand its parent task.

[0015] Among them, also include: S6: Record the idle time of each task pod. If the idle time exceeds the preset threshold continuously, the pod is destroyed. The preset threshold is dynamically calculated by the system based on the task arrival interval and task execution frequency in the workload history data.

[0016] Among them, a system of the highly balanced serverless workflow scheduling method driven by data flow based on Kubernetes includes: Gateway request receiving module, used to receive JSON format workflow files sent by users and create related resources; The central task scheduling module is used to deploy task pods based on node resource usage and task resource requests, and select the root task pod with the shortest request queue length to send task execution requests; The sidecar container data synchronization module is used to receive external request data and queue the request; The sidecar container task scheduling module is used to obtain and execute task requests from the request queue and select the optimal subsequent task pod based on the subsequent task scheduling strategy based on data transmission time, request queue length, and task execution time; The sidecar container log module is used to record data transmission time and task execution time; The auto-scaling module is used to evaluate the workflow execution and trigger the automatic scaling of task pods based on the auto-scaling policy; The central database includes a status information database and a log database, which are used to store system status information and execution logs.

[0017] Among them, an electronic device includes a memory and a processor. Compared with the prior art, the present invention has the following advantages: The present invention aims to solve the problems of existing data-driven workflow scheduling solutions, such as over-coarse granularity, low resource utilization efficiency, and inflexible expansion strategies. Traditional data-driven workflow scheduling platforms usually divide workflows into subgraphs based on data transmission overhead, and schedule and scale them in units of subgraphs, which makes it difficult to achieve refined control of individual tasks and easily leads to waste of resources. Based on the continuation of the data-driven scheduling idea, the present invention introduces a more fine-grained task-level scheduling mechanism, which can accurately identify tasks with long queuing times, and schedule tasks to the optimal node based on data transmission conditions, thereby avoiding unnecessary resource expansion while ensuring data locality. In addition, during the task expansion process, the present invention not only pays attention to the data flow, but also comprehensively analyzes the load conditions of the source node and the target node, giving priority to nodes with better performance, and realizing the unification of data-driven and load balancing. The scheduling strategy takes into account both the task queuing time and the data transmission overhead, improves the overall execution efficiency, and enhances the resource scheduling and flexible execution capabilities of the system in complex, high-concurrency workflow scenarios.

[0018] Other features and advantages of the present invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the present invention.

[0019] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 This is a flow chart of a highly balanced serverless workflow scheduling method driven by Kubernetes-based data streams in an embodiment of the present invention; Figure 2 This is the overall architecture diagram of a highly balanced serverless workflow scheduling driven by Kubernetes-based data flow in an embodiment of the present invention; Figure 3 This is a workflow diagram of the sidecar container task scheduling module in an embodiment of the present invention; Figure 4 This is a flow chart of the subsequent task scheduling strategy of the sidecar container task scheduling module of the present invention; Figure 5 This is a flow chart of the automatic expansion strategy of the present invention. DETAILED DESCRIPTION

[0021] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0022] The embodiment of the present invention provides Figure 1 As shown in the figure, a highly balanced serverless workflow scheduling method driven by data flow based on Kubernetes includes: S1: Receive the JSON format workflow file sent by the user and create related resources; S2: Deploys task pods based on relevant resources, node resource usage, and task resource requests. Each task pod maintains a request queue. S3: Based on the deployed task pods, select the root task pod with the shortest request queue length to send the task execution request; S4: Based on the execution request, the sidecar container executes the task and selects the optimal subsequent task pod to send the request data based on the subsequent task scheduling strategy based on data transmission time, request queue length, and task execution time. S5: Based on the execution status, the workflow execution status is evaluated and the automatic scaling of the task pod is triggered based on the automatic scaling policy.

[0023] The working principle and beneficial effects of the above technical solution are as follows: Step S1: Receive a JSON-formatted workflow file sent by a user and create related resources. The user sends the workflow file via the Gateway request receiving module. The workflow file, formatted in JSON, contains a detailed description of the workflow, such as the definition of each task, inter-task dependencies, and resource requirements. Upon receiving the file, the Gateway request receiving module performs preliminary verification, checking the format and data integrity. It then passes the verified workflow file to the central task scheduling module and registers the request in the status information database for subsequent tracking.

[0024] Step S2: Deploy task pods based on relevant resources, node resource usage, and task resource requests. Each task pod maintains a request queue. After receiving the workflow, the central task scheduling module analyzes each task's resource requirements and performs initial deployment based on cluster node resource usage. A pod is created for each task and the number of replicas is set (default is 2 to ensure high availability), prioritizing nodes with ample resources for task deployment. After creating the pod, the scheduling module updates the status information database, recording information such as the pod's resource status, task execution status, and request queue length. Each task pod maintains a request queue to store pending tasks. The queue determines the order in which tasks are scheduled.

[0025] Step S3: Based on the deployed task pods, the central task scheduling module selects the root task pod with the shortest request queue length and sends a task execution request. The central task scheduling module queries the status information database for the root task (a task without dependencies) of the current workflow. Based on factors such as the task pod's request queue length and node resource availability, the scheduling module selects the pod with the shortest request queue and the most abundant resources to execute the root task. After determining the execution pod, the scheduling module sends it an execution request containing task parameters, input data, and execution instructions. Upon receiving the request, the task pod begins executing the task.

[0026] Step S4: Based on the execution request, the task is executed through the sidecar container and based on the subsequent task scheduling strategy of data transmission time, request queue length and task execution time, the optimal subsequent task pod is selected to send the request data; wherein, after the task Pod is started, the sidecar container data synchronization module listens to and receives data from other tasks or external systems, and puts the task request into the request queue after synchronization is completed. The sidecar container task scheduling module obtains the task from the queue and executes it. After the task is executed, the sidecar task scheduling module comprehensively considers the data transmission time, request queue length and task execution time, and selects the optimal subsequent task Pod to send the request data. The sidecar container log module records key information such as task execution time, data transmission time and queue time, and synchronizes it to the log database to provide a basis for subsequent scheduling optimization.

[0027] Step S5: Based on the execution status, the workflow execution status is evaluated, and the automatic scaling of the task pod is triggered based on the automatic scaling policy. The automatic scaling module monitors the task execution status, paying special attention to task completion time, queue time, and data transmission time. When a performance bottleneck is discovered, the module analyzes the cause and takes appropriate measures. If the queue time is too long, it is determined to be a computing resource bottleneck. The data transmission status of the parent and child tasks is analyzed, and the optimal node is selected to add Pod replicas. If the data transmission time is too long, data-aware scaling is performed to optimize the data transmission path and deploy new Pods on nodes with sufficient resources. This automatic scaling policy takes into account data locality and load balancing to ensure balanced optimization of system performance and resource utilization.

[0028] The specific steps are as follows:

[0029] Step 1: Initialize platform components: The components include: gateway request acceptance module, central task scheduling module, sidecar container data synchronization module, sidecar container task scheduling module, sidecar container logging module, automatic scaling module and central database.

[0030] Step 2: The user sends a workflow file specified in JSON format to the Gateway request receiving module;

[0031] Step 3: The Gateway request receiving module processes the received JSON file and creates related resources, and then passes them to the central task scheduling module;

[0032] In step 4, the central task scheduling module will comprehensively consider the node resource usage and resource requests of each task in the workflow, and initially deploy the task pod. Each task pod will maintain a request queue.

[0033] Step 5: The central task scheduling module searches the status information database for the pod with the shortest request queue length among all root task pods and sends a task execution request.

[0034] Step 6: The sidecar container data synchronization module continuously receives external request data and, after receiving the request data, queues the request. The sidecar log module also records the data transmission time.

[0035] Step 7: The sidecar container task scheduling module continuously obtains and executes task requests from the request queue, and the sidecar log module records and synchronizes the task execution time.

[0036] Step 8: After the current task is executed, the sidecar task scheduling module will comprehensively consider the data transmission time saved in the log of the current task, the request queue length of the subsequent task, and the task execution time, and select the most appropriate subsequent task pod to send the request data.

[0037] Step 9: After the workflow is completed, the pod will not be destroyed, but will wait for the next task execution of the central scheduling module or the sidecar task scheduling module.

[0038] Step 10: When the workflow is completed, the auto-scaling module is triggered. The workflow and task execution status are obtained and evaluated through logs. If the current schedule cannot meet the SLO, the task pod is automatically scaled.

[0039] In another embodiment, step S2 includes: S21: Set the number of replicas for each task Pod to 2; S22: Sort tasks in the workflow based on their resource requirements, deploying task pods from most to least resource-demanding. S23: When selecting a node, select a node whose remaining computing resources are greater than the task requested resources and whose remaining computing resources have the highest ratio to the total computing resources of the node; S24: Update the resource usage record of the node, including the number of allocated CPU cores, allocated memory capacity, and remaining available resources, so as to provide resource data for the scheduling of subsequent tasks.

[0040] The working principle and beneficial effects of the above technical solution are as follows: The initial deployment of the task pod is as follows: During the initial deployment of the task Pod, the central task scheduling module will execute the deployment task based on the resource usage of each node in the cluster and the resource requests of each task in the workflow.

[0041] First, the central task scheduling module sets the number of replicas of each task Pod to 2, which means that two replicas will be started in the initial setting of each task.

[0042] The resource request of a task Pod is defined by the request sent by the user.

[0043] Next, the central task scheduling module sorts all tasks in the workflow according to resources and deploys task Pods from most to least as required.

[0044] When selecting nodes, the scheduling module selects those nodes with the most remaining resources.

[0045] Once the task Pod is successfully deployed to the node, the scheduling module will update the node's resource usage to provide accurate resource data for the scheduling of subsequent tasks.

[0046] The central task scheduling module continues to gradually deploy all task pods in the workflow according to the same rules. This scheduling mode ensures efficient and fair resource utilization by prioritizing tasks with the most resource requests to nodes with the most abundant resources.

[0047] Step S21: Set the number of replicas for each task pod to 2. The central task scheduling module sets two replicas for each task by default when initially deploying the task pod. This configuration ensures high task availability. Even if one pod fails, another pod can continue executing the task, preventing service interruptions caused by single points of failure. Each replica can receive and process the same type of task requests, thereby improving system stability and processing capacity.

[0048] Step S22: Tasks within the workflow are sorted based on their resource requirements, with task pods deployed from highest to lowest resource requirements. The central task scheduling module parses the workflow file to obtain each task's resource requirements (CPU, memory, etc.), and sorts tasks by resource requirements from highest to lowest. This sorting strategy ensures that tasks with higher resource requirements are deployed first, preventing high-demand tasks from being unscheduled later due to insufficient resources. Resource requirement sorting takes into account the computational complexity and data processing volume of tasks, enabling the system to allocate resources rationally and improve overall resource utilization efficiency.

[0049] Step S23: When selecting a node, the node with the highest remaining computing resources relative to the task's requested resources and the highest ratio of remaining computing resources to total node computing resources is selected. For each task pod, the central task scheduling module evaluates the resource status of all nodes in the cluster and selects nodes with remaining resources greater than the task's requested resources. Among these nodes, the remaining resource ratio (remaining resources / total resources) is calculated for each node, and the node with the highest ratio is selected for task deployment. This selection strategy ensures balanced resource allocation, avoids overloading some nodes while leaving others idle, and improves overall cluster resource utilization and task execution efficiency.

[0050] Step S24: Update the node's resource usage record, including the number of allocated CPU cores, allocated memory capacity, and remaining available resources, to provide resource data for the scheduling of subsequent tasks. After the task pod is deployed, the central task scheduling module updates the node resource usage record in the status information database in real time. Updates include detailed information such as the number of allocated CPU cores, allocated memory capacity, and the node's remaining available resources. This data is crucial for the scheduling of subsequent tasks, ensuring that scheduling decisions are based on the latest resource usage and preventing resource overallocation or scheduling conflicts. Resource records in the database are also used for system monitoring and resource optimization analysis, helping administrators understand system resource utilization efficiency and potential bottlenecks.

[0051] In another embodiment, step S4 includes: S41: The sidecar container data synchronization module receives request data from the outside and puts the request into the request queue. S42: The sidecar container log module records the data transmission time; S43: The sidecar container task scheduling module obtains and executes task requests from the request queue; S44: The sidecar container log module records and synchronizes task execution time; S45: After the task is executed, the expected waiting time is calculated based on the data transmission time and the request queue length, and the subsequent task pod with the shortest expected waiting time is selected to send the request data.

[0052] The working principle and beneficial effects of the above technical solution are as follows: Step S41: The sidecar container data synchronization module receives the request data sent from the outside and puts the request into the request queue; wherein, the data synchronization module in the sidecar container runs continuously, listens for and receives the request data from the outside. The received request data usually contains a unique identifier (UID) and task parameter information. After the data is received, the module marks the request status as "received", reads the parameter data required by the current task Pod from the environment variables, and determines whether the parameters required for the request are complete. If the parameters are complete, the request is added to the request queue, and the status information of the corresponding task in the status database is updated to ensure that the task is ready for scheduling and execution.

[0053] Step S42: The sidecar container log module records data transmission time. During data reception, the log module records the reception time of each parameter data for subsequent analysis of data transmission efficiency. These records include the start time of data transmission, the completion time of data reception, and the total transmission time. This time data is uploaded to the log database, providing the system with traceability and monitoring capabilities for data flow, helping to analyze potential data transmission bottlenecks and providing important information for subsequent system optimization and task scheduling decisions.

[0054] Step S43: The sidecar container task scheduling module retrieves and executes task requests from the request queue. The task scheduling module is responsible for actual task execution, sequentially retrieving completed task requests from the request queue and beginning execution. The module first parses the execution parameters and instructions in the task request and then calls the appropriate computing resources to execute the task. During task execution, the scheduling module monitors the execution status to ensure the task is progressing smoothly and updates the task status upon completion. This queue processing mechanism ensures the orderly execution of tasks, improving the system's responsiveness and processing efficiency.

[0055] Step S44: The sidecar container log module records and synchronizes task execution time. The log module records information such as the task start time, task completion time, and total execution time. This execution time data is synchronized to the log database and, together with the previously recorded data transmission time, forms a time record of the entire task lifecycle. This record is crucial for analyzing task execution efficiency, identifying performance bottlenecks, and optimizing system scheduling strategies. Based on this historical execution data, the system can calculate the average execution time of tasks, providing more accurate time estimates for subsequent scheduling decisions.

[0056] Step S45: After the task is executed, the expected waiting time is calculated based on the data transmission time and the request queue length, and the subsequent task pod with the shortest expected waiting time is selected to send the request data. After the task is executed, the sidecar task scheduling module obtains the subsequent task information from the environment variables. Combined with the running information in the status database (such as the IP address of each Pod, the request queue length, the current execution time of the task) and the historical data in the log database (such as the average execution time, the data transmission duration), the expected waiting time of each potential target Pod is calculated. The expected waiting time is equal to the queue length multiplied by the average execution time plus the data transmission time. The module selects the Pod with the shortest expected waiting time to send the result data, thereby achieving an intelligent balance between data locality and computing load, and optimizing the overall workflow execution efficiency.

[0057] Specifically, in the workflow of the Sidecar container, the system mainly consists of the data synchronization module, log collection module, task scheduling module, and state database to collaboratively complete the reception, scheduling, and execution of tasks. The specific operation process is as follows: When the Pod is started, the data synchronization module in the Sidecar container starts running immediately to continuously monitor and receive request data from the outside.

[0058] Each request data typically contains a unique identifier (UID) and corresponding task parameter information. For each received request data, the system will first mark the request status as "Received" after the data is received.

[0059] The data synchronization module then reads the parameter data required by the current task Pod from the environment variables and determines whether all the parameter data required for the current request is complete. If the parameter data is confirmed to be complete, the data synchronization module adds the request to the request queue and simultaneously updates the status information of the corresponding task in the status database to ensure that the task is ready for scheduling and execution.

[0060] While receiving data, the log collection module will record the reception time of each parameter data, thereby achieving traceability and monitoring of the entire data flow process.

[0061] In addition, the data synchronization module keeps running continuously to ensure that it can respond to and receive new external request data in real time.

[0062] The actual execution of tasks is handled by the task scheduling module in the sidecar container. This module retrieves ready-to-complete request tasks from the request queue and begins execution. After a task completes, the scheduling module first obtains subsequent task information from environment variables and, combined with the running information in the status database, determines the target Pod for the next stage of the task.

[0063] Specifically, the system queries the current running status of each task Pod in the status database, including its IP address, request queue length, current execution time of the task, and historical data transmission time.

[0064] Based on this information, the task scheduling module selects the most appropriate target Pod and sends the processing results or data to it. Once the data is sent, the system retrieves the next pending task from the request queue and updates the status of the relevant Pod in the state database, achieving a dynamic balance and efficient execution of task scheduling and resource allocation.

[0065] In another embodiment, step S45 includes: S451: Obtain the request queue lengths of the external task Pod and the internal task Pod from the state database, and determine the Pod with the smallest queue length value among the external and internal Pods respectively; S452: Obtain the average execution time of historical tasks and the data transmission time of the current task from the log database; S453: The estimated waiting time for the computing task to be dispatched to the external Pod is equal to the queue length multiplied by the average execution time plus the data transmission time; S454: Compare the external estimated waiting time with the internal Pod estimated waiting time, and select the Pod with the shorter estimated waiting time for task scheduling.

[0066] The working principle and beneficial effects of the above technical solution are as follows: Step S451: Obtain the request queue lengths of the external task Pod and the internal task Pod from the status database, and determine the Pod with the smallest queue length in the external and internal Pods respectively; wherein, the sidecar task scheduling module first distinguishes between the external task Pod (Pod deployed on different nodes) and the internal task Pod (Pod deployed on the same node). Then query the current request queue lengths of all subsequent task-related Pods from the status database. Compare the queue lengths of the external and internal Pods respectively to determine the Pod with the smallest queue length. This strategy of distinguishing between internal and external Pods takes into account the physical distance factor of data transmission, and provides basic data support for subsequent scheduling decisions based on data locality and load balancing.

[0067] Step S452: Obtain the average execution time of historical tasks and the data transmission duration of the current task from the log database. The sidecar task scheduling module obtains two types of key time data from the log database: the average execution time of historical tasks, which reflects the computational complexity and processing efficiency of the task; and the actual data transmission duration of the current task, which reflects the network status and data volume of the data transmission. The average execution time is calculated by analyzing the historical execution records of similar tasks, while the data transmission duration is extracted from the log records of the current execution cycle. These time data provide important parameters for subsequent waiting time prediction.

[0068] Step S453: Calculate the estimated waiting time for the task to be scheduled to the external Pod, which is equal to the queue length multiplied by the average execution time plus the data transmission time. The sidecar task scheduling module uses a scientific calculation method to estimate the waiting time of the task in the external Pod. The specific calculation formula is: Estimated waiting time = queue length × average execution time + data transmission time. This formula considers two key factors: the processing time of tasks already in the queue (queue length multiplied by average execution time) and the time overhead for transmitting data from the current node to the target node. This calculation method comprehensively considers computing load and network transmission factors, making scheduling decisions more accurate and efficient.

[0069] Step S454: Compare the external expected waiting time with the internal Pod's expected waiting time, and select the Pod with the shorter expected waiting time for task scheduling. The sidecar task scheduling module compares the external Pod's expected waiting time calculated in step S453 with the internal Pod's expected waiting time. According to the comparison result, if the external Pod's expected waiting time is greater than the internal Pod's expected waiting time, the internal Pod is selected for task scheduling; otherwise, the Pod with the shortest request queue length in the external Pod is selected for scheduling. This decision-making process ensures the efficiency of task scheduling, minimizes the waiting time of tasks, and thus improves the execution efficiency of the overall workflow.

[0070] Specifically, the sidecar task scheduling module selects subsequent task pods in the following way: The Sidecar task scheduling module adopts an intelligent scheduling strategy that combines computing load and data transmission evaluation, which is highly consistent with the current "data-driven" scheduling concept.

[0071] During the scheduling process, the module first obtains the shortest request queue length of the external task Pod and the internal task Pod from the status data center, and obtains the average execution time of historical tasks and the data transmission time of this task from the log data center.

[0072] Based on this information, the system can calculate the expected waiting time if the task is scheduled to an external Pod, specifically "queue length × average execution time + data transmission time".

[0073] The scheduling module then compares this external estimated wait time with the internal Pod's wait time. If the external wait time is significantly higher than the internal one, the task will be assigned to the internal task Pod first; otherwise, the task will be scheduled to the external task Pod, minimizing overall latency.

[0074] This process achieves a dynamic trade-off between nodes inside and outside the system, making scheduling more agile and adaptable to changing resource states. Compared with the traditional method of focusing only on computing load or scheduling based solely on the principle of proximity, the biggest advantage of this strategy is that it comprehensively considers two key factors: request queue length and data transmission cost. This not only reflects the workflow design idea centered on data flow efficiency, but also makes task scheduling more in line with the dynamic changes of resources in the actual operating environment. By incorporating data locality, execution pressure, and communication costs into a unified scheduling model, the system can significantly optimize the response time and overall throughput of tasks while maintaining resource utilization, truly realizing a unified scheduling mechanism that is data-driven and load-balanced.

[0075] In another embodiment, step S5 includes: S51: monitors the completion time of the task Pod request; S52: Analyze the cause of the delay, determine whether there is a computing resource bottleneck by comparing the queue time with a preset threshold, and determine whether there is a data location problem by comparing the data transmission time with a preset threshold; S53: If the queue time is too long, enter the computing resource expansion process; S54: If the data transmission time is too long, enter the data perception expansion process.

[0076] The working principle and beneficial effects of the above technical solution are as follows: Step S51: Monitoring the completion time of task pod requests. The auto-scaling module continuously monitors the completion time of each task pod request and records the execution duration of each task. This monitoring process ensures that the system can obtain real-time task execution performance data, providing basic information for subsequent performance analysis and resource adjustment.

[0077] Step S52: Analyze the cause of the delay. By comparing the queue time with a preset threshold, determine whether there is a computing resource bottleneck. By comparing the data transmission time with a preset threshold, determine whether there is a data location issue. The auto-scaling module further analyzes the cause of the delay if it detects an abnormal increase in request completion time. The system compares the queue time of the current task with a preset threshold to determine whether there is a computing resource bottleneck. It also compares the data transmission time with a preset threshold to determine whether there is a data location issue. This analysis process helps the system identify the specific source of the performance bottleneck and provides a basis for subsequent scaling strategies.

[0078] Step S53: If the queue time is too long, the system begins the resource expansion process. If the queue time exceeds a preset threshold, the system determines that the current task pod is a performance bottleneck node, and the auto-scaling module initiates the resource expansion process. This process evaluates the current task's parent and child tasks, and based on the node's resource availability and data transmission overhead, selects the most appropriate node for task pod expansion. This ensures balanced resource distribution, improving overall cluster resource utilization and task scheduling efficiency.

[0079] Step S54: If the data transmission time is too long, the system enters the data-aware expansion process. If the monitored data transmission time exceeds a preset threshold, the system will determine that there is significant data exchange between tasks and that the current task cannot be deployed on a node close to the data source. At this point, the automatic scaling module will enter the data-aware expansion process, analyzing the data transmission between parent and child tasks and locating the task with the longest data transmission time as the optimization priority. The system compares the resource status of the current task node with that of the data source task node and selects the node with more abundant resources as the deployment target, thereby optimizing the data transmission path and improving task execution efficiency.

[0080] Specifically, the automatic scaling strategy includes the following: To improve task pod scheduling efficiency, this expansion module first monitors the completion time of task pod requests to determine whether there are performance bottlenecks. If an abnormal increase in task request completion time is detected, the system further analyzes the main cause of the delay to determine whether it is a computing resource bottleneck caused by excessive queue times or a data location issue caused by long data transmission times.

[0081] If the queue time is too long, it indicates that the current node's computing resources are insufficient. The system then initiates the resource expansion process. This process evaluates the current task's parent and child tasks, and based on the node's resource availability and data transmission overhead, selects the most appropriate node for task pod expansion. This helps achieve a balanced resource distribution, alleviates the load on specific nodes, and improves overall cluster resource utilization and task scheduling efficiency.

[0082] If data transmission takes too long, it indicates that there's significant data exchange between tasks, and the current task isn't deployed on a node close to the data source. The system then enters a data-aware expansion process, analyzing the data transmission between parent and child tasks and prioritizing the task with the longest data transmission time. It then compares the resource availability of the current task node with that of the data source task node, selecting the node with the most abundant resources as the deployment target.

[0083] If the data source node is selected, the Pod of the current task is directly expanded to improve data locality; if the current task node is selected, its parent task is expanded to optimize the data path from the source.

[0084] While considering data access efficiency, this strategy still maintains awareness and utilization of node resource status, thereby achieving dual optimization of "data awareness" and "resource balance".

[0085] This expansion mechanism not only enables targeted scheduling based on bottleneck types, improving the accuracy of scheduling responses, but also achieves a dynamic balance between data locality and resource load. This reduces unnecessary data transmission overhead and effectively distributes resource pressure, creating an intelligent expansion strategy that balances performance and efficiency.

[0086] In another embodiment, step S53 includes: S531: Evaluate the parent task and child tasks of the current task; S532: Based on the node's resource status and data transmission overhead, a weighted score is formed by calculating the inverse relationship between the percentage of the node's remaining resources to the total resources and the data transmission time. The node with the highest score is selected for task pod expansion. S533: Expand the task Pod on the selected node to alleviate the load pressure of the set node.

[0087] The working principle and beneficial effects of the above technical solution are as follows: S531: Evaluate the parent and child tasks of the current task; the automatic scaling module comprehensively analyzes the relationship and data transfer between the parent and child tasks of the current task. The system extracts historical execution records from the log database and identifies the parent and child tasks that have a direct data transfer relationship with the current task. Through this analysis, the system can determine which tasks have a large amount of data exchange, providing a basis for subsequent task expansion decisions. This evaluation of parent-child task relationships is the foundation for data-driven scheduling, enabling the system to accurately schedule data flows within the workflow.

[0088] S532: Select the optimal node based on the node resource status and data transmission overhead; after determining that Pod expansion is necessary, the system will comprehensively consider the resource status of each node and the potential data transmission overhead. Specifically, the system first obtains the real-time resource usage of each node in the cluster, including indicators such as CPU utilization and memory utilization. Then, the system calculates the percentage of the node's remaining resources to the total resources, and estimates the data transmission time. Based on these two factors, the system establishes a weighted scoring mechanism: an inverse relationship between the percentage of the node's remaining resources and the data transmission time. This scoring mechanism takes into account both resource abundance and data locality, and can strike a balance between the two. Ultimately, the system selects the node with the highest score as the target node for task Pod expansion.

[0089] S533: Expand the task pod on the selected node. Once the optimal expansion node is determined, the system deploys the new task pod on that node. This expansion operation is performed through the Kubernetes API. The system creates a new pod instance with the same configuration as the original task pod and ensures that the new pod is connected to the workflow's scheduling system. This expansion strategy can effectively alleviate the load on specific nodes and achieve more balanced resource allocation. Because the expansion location selection takes data transmission overhead into account, the newly expanded pod can reduce unnecessary data transmission and improve the overall workflow execution efficiency.

[0090] In another embodiment, step S54 includes: S541: Analyze the data transmission between the parent and child tasks and locate the task with the longest data transmission time; S542: Compare the resource status of the current task node and the data source task node, calculate the CPU utilization and memory usage of both, and select the node whose average CPU utilization and memory usage is lower than the system preset threshold as the deployment target, where the system preset threshold is dynamically set based on historical task execution data; S543: If a data source node is selected, expand the Pod of the current task; if the current task node is selected, expand its parent task.

[0091] The working principle and beneficial effects of the above technical solution are as follows: S541: Analyze the data transmission between parent and child tasks; when the system detects that the data transmission time is too long, the automatic scaling module will conduct an in-depth analysis of the data transmission between the parent and child tasks. The system extracts data transmission records for related tasks from the log database, including information such as the amount of data transmitted, transmission duration, and transmission frequency. Using this data, the system can locate the task pairs with the longest data transmission time, which is often the key link leading to performance bottlenecks. This precise data flow analysis capability is the core of the system's data-driven scheduling, enabling subsequent expansion decisions to optimize data flow paths in a targeted manner.

[0092] S542: Compare the resource status of the task node and the data source node; after determining the task pair that needs to be optimized, the system will compare the resource status of the current task node and the data source task node in detail. The system obtains the CPU utilization and memory usage of these two types of nodes in real time and calculates their average value. The system presets a resource usage threshold, which is dynamically set based on historical task execution data. The system will select nodes whose average CPU utilization and memory usage are lower than this threshold as deployment targets. This dynamic threshold setting based on historical data enables the system to adapt to resource changes under different workloads, improving the accuracy and adaptability of scheduling decisions.

[0093] S543: Perform expansion operations based on the selection result; based on the analysis results of the first two steps, the system will decide on which node to perform the expansion operation. If the system selects the data source node, it means that the node has good resource conditions. The system will expand the Pod of the current task on this node, which can reduce data transmission overhead and improve data locality. If the system selects the current task node, it means that the node has more abundant resources. The system will expand the Pod of its parent task on this node, which can optimize data flow and reduce data transmission time. This flexible expansion strategy fully reflects the system's intelligent trade-off between data locality and resource balance, and can select the optimal expansion plan based on actual conditions.

[0094] In another embodiment, further comprising: S6: Record the idle time of each task pod. If the idle time exceeds the preset threshold continuously, the pod is destroyed. The preset threshold is dynamically calculated by the system based on the task arrival interval and task execution frequency in the workload history data.

[0095] The working principle and beneficial effects of the above technical solution are as follows: the automatic scaling module will continuously record and monitor the idle time of each task Pod. When the automatic scaling module records the idle time of each task pod, it will destroy the pod when the idle time exceeds the threshold. The idle state defined by the system means that the Pod's request queue is empty and there are no tasks being executed. When the system detects that the idle time of a Pod exceeds the preset threshold continuously, the Pod destruction mechanism will be triggered. This preset threshold is not fixed, but is dynamically calculated by the system based on historical workload data. Specifically, the system will analyze the task arrival interval and task execution frequency in the historical workflow to derive a reasonable Pod idle time threshold.

[0096] This dynamic threshold setting allows the system to adapt to the characteristics of different workload types. For example, for workflows with frequently arriving tasks, the system sets a longer idle time threshold to avoid the overhead of frequently creating and destroying Pods. For workflows with sparse tasks, the system sets a shorter threshold to promptly reclaim idle resources. This intelligent reduction mechanism ensures efficient utilization of system resources, avoids resource waste, and maintains the system's ability to quickly respond to workload changes.

[0097] In another embodiment, a system for a highly balanced serverless workflow scheduling method driven by data flow based on Kubernetes includes: Gateway request receiving module, used to receive JSON format workflow files sent by users and create related resources; The central task scheduling module is used to deploy task pods based on node resource usage and task resource requests, and select the root task pod with the shortest request queue length to send task execution requests; The sidecar container data synchronization module is used to receive external request data and queue the request; The sidecar container task scheduling module is used to obtain and execute task requests from the request queue and select the optimal subsequent task pod based on the subsequent task scheduling strategy based on data transmission time, request queue length, and task execution time; The sidecar container log module is used to record data transmission time and task execution time; The auto-scaling module is used to evaluate the workflow execution and trigger the automatic scaling of task pods based on the auto-scaling policy; The central database includes a status information database and a log database, which are used to store system status information and execution logs.

[0098] The working principle and beneficial effects of the above technical solution are as follows: The Gateway request receiving module is the entry point of the system and is responsible for receiving JSON format workflow files sent by users. When a user submits a workflow request, the module first performs format verification and integrity check on the JSON file to ensure that the file complies with the workflow specifications defined by the system. After the verification is passed, the module will parse the workflow file and extract key data such as the workflow's structural information, task definitions, and dependencies between tasks. Subsequently, the module will create relevant resources in the system, including creating a unique identifier for the workflow, registering workflow information in the state database, etc. Finally, the module passes the processed workflow data to the central task scheduling module to prepare for subsequent task deployment and execution.

[0099] After receiving workflow data from the Gateway Request Receiving Module, the central task scheduling module is responsible for initial task deployment based on node resource usage and task resource requests. The module first obtains the real-time resource status of each node in the cluster and then analyzes the resource requirements of each task in the workflow. During deployment, the module sorts tasks by resource requirement from highest to lowest, prioritizing tasks with high resource requirements to nodes with the most abundant resources. For each task, the module creates two Pod replicas by default to ensure high availability. After deployment, the module maintains a task status table that records the location, resource usage, and request queue length of each task Pod. When a root task needs to be executed, the central task scheduling module queries the status database, identifies the Pod with the shortest request queue length among all root task Pods, and sends the task execution request to it. This queue-length-based scheduling strategy balances the load across Pods, preventing some Pods from being overloaded while others remain idle, thereby improving overall resource utilization.

[0100] The sidecar container data synchronization module is deployed in each task Pod and is responsible for receiving request data sent from the outside. When the data synchronization module receives the request data, it will first perform data integrity verification to ensure that all necessary parameter data has been received. Then, the module will read the parameter information required by the current task Pod from the environment variables to determine whether all parameters required for the current request are complete. If the parameters are complete, the module will queue the request and update the relevant information in the status database. At the same time, the data synchronization module will record the reception time of each data packet. This time information is crucial for subsequent data transmission analysis and task scheduling optimization. The module adopts an asynchronous processing mechanism and can process data from multiple data sources at the same time to ensure that the data can be passed to the task execution module in a timely and complete manner.

[0101] The sidecar container task scheduling module is responsible for obtaining and executing task requests from the request queue. The module uses a FIFO (first-in, first-out) strategy to process tasks in the queue, ensuring that tasks are executed in the order they arrive. When executing a task, the module calls the corresponding task processing function and monitors the task's execution progress and resource usage. After the task is completed, the module needs to decide to which subsequent task Pod to send the result data. To do this, the module comprehensively considers three factors: data transmission time, request queue length, and task execution time. The module obtains the request queue length of each subsequent task Pod from the status database and historical data transmission time and task execution time from the log database. Based on this information, the module calculates the estimated waiting time required to send data to each possible subsequent task Pod and selects the Pod with the shortest waiting time as the target. This comprehensive scheduling strategy balances data locality and load balancing, improving the execution efficiency of the overall workflow.

[0102] The sidecar container log module is responsible for recording key time nodes and performance indicators during task execution. The module records data transmission time, that is, the time interval from the start of data sending to the completion of data reception; records task execution time, that is, the time interval from the start of task execution to the completion of execution; and records request queue waiting time, that is, the time interval from the task request entering the queue to the start of execution. These detailed time records provide the system with rich performance data, enabling the system to identify potential performance bottlenecks and provide a basis for automatic scaling decisions. Log data is synchronized to the central log database in real time for query and analysis by other modules. The module adopts an efficient log collection and processing mechanism to ensure that the log collection process does not have a significant impact on task execution.

[0103] The autoscaling module evaluates workflow execution and triggers autoscaling of task pods based on autoscaling policies. The module continuously monitors performance metrics for each task pod, including request completion time, queue length, and data transfer time. When a performance anomaly is detected, the module analyzes the cause to determine whether the issue is caused by excessive queueing or data transfer times. For cases where queueing times are excessive, the module executes a compute resource expansion process. This process evaluates the parent and child tasks of the current task and selects the most suitable node for pod expansion based on node resource availability and data transfer overhead. For cases where data transfer times are excessive, the module executes a data-aware expansion process. This process analyzes data transfer between parent and child tasks, compares the resource availability of related nodes, selects the optimal node for expansion, and optimizes data transfer paths. The module also automatically scales down task pods. The module records the idle time of each pod. When the idle time exceeds a preset threshold for consecutive periods, it triggers a pod destruction mechanism to reclaim idle resources. This autoscaling mechanism enables the system to dynamically adjust resource allocation based on workload changes, improving resource utilization and system responsiveness.

[0104] The central database consists of a status information database and a log database. The status information database stores real-time system status information, including resource usage of each node, the location of each task pod, request queue length, and workflow execution progress. This status information provides an important basis for task scheduling and resource allocation.

[0105] The log database stores system execution logs, including performance metrics such as task execution time, data transfer time, and request queue wait time. This log data is used for performance analysis, bottleneck identification, and automatic scaling decisions. The central database utilizes efficient data storage and query mechanisms, supporting real-time data updates and rapid data retrieval, ensuring that each functional module can obtain the required information in a timely manner.

[0106] like Figure 2As shown, the overall architecture of the system is as follows: User: Users send requests to the platform through the Gateway request receiving module. This is the entry point for users to interact with the platform. User-requested tasks are passed to the master node through the Gateway request receiving module for processing.

[0107] Master Node: The master node contains the following core modules: Gateway: Receives user requests and passes them to the master node for processing.

[0108] Central task scheduling module: responsible for scheduling and managing the root task execution process.

[0109] Automatic scaling module: Automatically adjusts system resources to optimize load and task execution efficiency based on the actual situation of task execution, such as task completion time, queue time, and data transmission time.

[0110] Central database: stores system status information, logs, and other data. All tasks and system operation status information will be recorded and synchronized for subsequent monitoring and analysis.

[0111] Worker Node: A worker node is responsible for the actual task execution and includes the following modules: Sidecar container: Each task pod is equipped with a sidecar container, which is responsible for data synchronization, task scheduling, log collection, etc. It ensures that task execution dependencies and resource allocation are optimized.

[0112] Data synchronization: Responsible for obtaining necessary data from external or other task Pods and ensuring that data is correctly transmitted between tasks.

[0113] Task scheduling: Schedule the execution of task Pods based on task dependencies, queue lengths, and system resources.

[0114] Log collection: Collect log information during task execution and upload it to the master node's database for storage.

[0115] Task container (Pod): performs actual task calculations.

[0116] Database: The system contains two databases: Status information database: stores real-time data such as the system's current operating status, task information, and resource usage.

[0117] Log database: records log data during task execution to facilitate system monitoring, error troubleshooting, and performance optimization.

[0118] The specific execution process is as follows: Accepting requests: When a user submits a workflow request, it is first received by the Gateway request receiving module. This module receives the user's workflow file, which is in JSON format and contains a detailed description of the workflow, including the definition of each task, the dependencies between tasks, the types and quantities of resources required for the tasks, and other information. Workflow files are usually generated by the user's application and sent to the Gateway request receiving module over the network. When the Gateway request receiving module receives the request, it will perform a preliminary verification of the request, including checking the format and data integrity of the workflow file to ensure that the file meets the specified structure and content requirements. After verification, the module will pass the workflow file to the central task scheduling module and register the request in the status information database for subsequent query and tracking.

[0119] Request Data Processing and Resource Creation: After receiving a workflow request, the central task scheduling module analyzes the resource requirements of each task in the request. Specifically, the scheduling module parses the task information in the workflow and extracts each task's resource requirements (such as CPU, memory, and storage). The scheduling module then performs the initial task deployment based on the resource usage of each node in the cluster.

[0120] During task deployment, the central task scheduling module allocates resources according to the following steps: It creates a pod for each task and assigns a preliminary number of replicas to each pod (the default number of replicas is 2 to ensure high availability of the task). Based on the remaining resources of the cluster nodes, nodes with ample resources are prioritized for deploying task pods. Resource scheduling policies ensure that resources are properly allocated to each node to avoid excessive congestion of node resources. After the task pod is created, the central task scheduling module updates the status information database to record the resource status of each pod, including information such as the resources occupied by the pod, the current task execution status of the pod, and the request queue length. This database provides important data support for subsequent task scheduling and resource allocation.

[0121] Task queue management and preparation for execution: Figure 3 As shown, each task Pod maintains a request queue to store tasks to be executed. The order in which tasks are scheduled is determined by the tasks in the queue. The central task scheduling module determines the execution Pod for the root task (i.e., a task that does not depend on other tasks) based on information such as the request queue length and node resource status of each Pod. When all task Pods are deployed, the central task scheduling module queries the status information database for the root task of the current request workflow. The scheduling module selects the task Pod with the shortest request queue and sends the task execution request and data to the sidecar container data synchronization module.

[0122] Root task scheduling and task execution: Figure 2 As shown in the figure, during root task scheduling, the central task scheduling module determines the most suitable task pod for executing the root task based on the task pod's resource request and current node resource usage. The scheduling module selects the pod with the shortest request queue and the most abundant resources to execute the root task. Once the task pod for executing the root task is determined, the central task scheduling module sends the data and execution request to the pod. The request contains all the information required to execute the root task, including task parameters, input data, and execution instructions. After the task pod receives the request, the task begins execution.

[0123] Task scheduling and execution: Figure 3 As shown in the figure, after a task pod starts, its sidecar container's data synchronization module takes over data transmission and synchronization. The data synchronization module listens for and receives data from other tasks or external systems in real time. When synchronization is complete, the data synchronization module places the task request in the sidecar container's request queue.

[0124] like Figure 3 As shown in the figure, the task scheduling module is responsible for obtaining tasks from its request queue and executing them. The task scheduling module in the sidecar container within each task pod continuously monitors and obtains pending tasks from the request queue. Once a task is completed, the sidecar task scheduling module updates the task execution status and executes the next task.

[0125] Subsequent task scheduling: Figure 4 As shown, when the current task is completed, the sidecar task scheduling module will decide how to schedule the subtasks of the current task according to the following scheduling strategy: Get the current shortest queue length of the external task Pod (the external task Pod is the task Pod on the same node as the current Pod) and the shortest queue length of the internal task Pod (the internal task Pod is the task Pod on the same node as the current Pod): First, get the current request queue length of the subsequent external task Pod from the status data center, and get the queue length data of the internal task Pod. Get the execution time and data transmission duration of the subsequent tasks: Then, get the execution time of the task and the data transmission duration of this time from the log data center. Calculate the expected waiting time of the external task Pod: Through calculation, the expected waiting time of the external task Pod is: queue length × execution time + data transmission duration.

[0126] Compare the waiting time of external and internal task Pods: Compare the expected waiting time of the external task Pod with the waiting time of the internal task Pod to determine which type of Pod to schedule the task to. If the expected waiting time of the external task Pod is longer than the waiting time of the internal task Pod, the task will be scheduled to the internal task Pod. If the expected waiting time of the external task Pod is shorter than the waiting time of the internal task Pod, the task will be scheduled to the Pod with the shortest external task Pod request queue. This scheduling strategy ensures that during the task scheduling process, the system can respond to load changes of external and internal task Pods in real time, maximize resource utilization, and optimize task response time.

[0127] Log Collection: The sidecar container log processing module collects and records key information such as task execution time, data transfer time, and request queue time, and synchronizes it to the log database. This data provides detailed runtime feedback for subsequent task scheduling, helping the system optimize resource allocation and task scheduling strategies.

[0128] Automatic expansion: Figure 5 As shown in the figure, the role of the auto-scaling module is to dynamically adjust the resource allocation of task pods based on performance bottlenecks that occur during request execution (such as long task request completion time, long queue time, long data transmission time, etc.) to optimize the overall performance of the system. The specific auto-scaling process is as follows: Monitor task request completion time: The auto-scaling module continuously monitors the completion time of each task pod. If the completion time is too long, it may be due to a performance bottleneck caused by long queue times or data transmission times. The auto-scaling module will initiate further analysis. Determine the bottleneck source: The system first determines whether the long pod completion time is caused by long queue time or long data transmission time. Long queue time: If the queue time is too long, it means that the current task has become a performance bottleneck, so the current task Pod must be expanded. Obtaining the data transfer overhead of the parent and child tasks: After determining that the current task needs to be scaled, the system analyzes the data transfer time of the parent and child tasks and selects the task with the longest data transfer time. This operation aims to ensure that when the task pod is scaled, data locality can be improved, thereby reducing unnecessary data transfer delays and optimizing overall system performance. Determine the resource status of the external task node and the current task node: The system then compares the resource status of the external task Pod node with the current task Pod node to ensure system load balancing. If the current task node has sufficient resources, the expansion will select the current node; if the current node has insufficient resources, the expansion will be to the external node, thereby effectively sharing the load; Long data transmission times: If data transmission times are too long, it indicates that there is a large amount of data exchange between tasks, and the current task is not deployed on a node close to the data source. In this case, the system will perform data-aware expansion, optimize the data transmission path, and select the node with the most abundant resources for data transmission task expansion.

[0129] Select the expansion node for the task pod. Based on the previous judgment, the automatic scaling module will select the node: Expand external task nodes: If the node where the data sender is located has sufficient resources, the system will choose to add a data receiving task Pod to the data sender node.

[0130] Expand the current task node: If the node where the data receiver is located is relatively sufficient, the system will choose to expand the data sending task Pod on the node where the data receiver is located.

[0131] This scaling strategy balances data locality and load balancing. By optimizing data transmission paths and dynamically scaling task pods, it reduces unnecessary data transmission latency and balances node load. It maximizes resource utilization and reduces bottlenecks while improving task execution efficiency.

[0132] In another embodiment, an electronic device includes a memory and a processor.

[0133] The working principle and beneficial effects of the above technical solution are as follows: the electronic device includes a memory and a processor, wherein the memory is used to store program instructions, and the processor is responsible for executing these instructions. Specifically, the processor will execute the steps of initializing platform components, receiving user requests, analyzing task resource requirements, scheduling task Pods, monitoring task execution, recording logs, and automatic scaling in sequence according to the program instructions in the memory. The program instructions stored in the memory include the operating logic of each module, data processing flow, resource management strategy, etc., to ensure that the electronic device can efficiently execute the high-load-balanced data-driven workflow scheduling method based on Kubernetes. In this way, the electronic device can realize dynamic management and scheduling of complex workflows, improving the overall performance and resource utilization of the system.

[0134] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the invention.

Claims

1. A highly balanced serverless workflow scheduling method driven by data flow based on Kubernetes, characterized by: include: S1: Receive the JSON format workflow file sent by the user and create related resources; S2: Deploys task pods based on relevant resources, node resource usage, and task resource requests. Each task pod maintains a request queue. S3: Based on the deployed task pods, select the root task pod with the shortest request queue length to send the task execution request; S4: Based on the execution request, the sidecar container executes the task and selects the optimal subsequent task pod to send the request data based on the subsequent task scheduling strategy based on data transmission time, request queue length, and task execution time. S5: Based on the execution status, the workflow execution status is evaluated and the automatic scaling of the task pod is triggered based on the automatic scaling policy.

2. The highly balanced serverless workflow scheduling method driven by data flow based on Kubernetes according to claim 1 is characterized in that: Step S2 includes: S21: Set the number of replicas for each task Pod to 2; S22: Sort tasks in the workflow based on their resource requirements, deploying task pods from most to least resource-demanding. S23: When selecting a node, select a node whose remaining computing resources are greater than the task requested resources and whose remaining computing resources have the highest ratio to the total computing resources of the node; S24: Update the resource usage record of the node, including the number of allocated CPU cores, allocated memory capacity, and remaining available resources, so as to provide resource data for the scheduling of subsequent tasks.

3. The highly balanced serverless workflow scheduling method driven by data flow based on Kubernetes according to claim 1 is characterized in that: The S4 step includes: S41: The sidecar container data synchronization module receives request data from the outside and puts the request into the request queue. S42: The sidecar container log module records the data transmission time; S43: The sidecar container task scheduling module obtains and executes task requests from the request queue; S44: The sidecar container log module records and synchronizes task execution time; S45: After the task is executed, the expected waiting time is calculated based on the data transmission time and the request queue length, and the subsequent task pod with the shortest expected waiting time is selected to send the request data.

4. The highly balanced serverless workflow scheduling method driven by data flow based on Kubernetes according to claim 1 is characterized in that: Step S45 includes: S451: Obtain the request queue lengths of the external task Pod and the internal task Pod from the state database, and determine the Pod with the smallest queue length value among the external and internal Pods respectively; S452: Obtain the average execution time of historical tasks and the data transmission time of the current task from the log database; S453: The estimated waiting time for the computing task to be dispatched to the external Pod is equal to the queue length multiplied by the average execution time plus the data transmission time; S454: Compare the external estimated waiting time with the internal Pod estimated waiting time, and select the Pod with the shorter estimated waiting time for task scheduling.

5. The highly balanced serverless workflow scheduling method driven by data flow based on Kubernetes according to claim 1 is characterized in that: Step S5 includes: S51: monitors the completion time of the task Pod request; S52: Analyze the cause of the delay, determine whether there is a computing resource bottleneck by comparing the queue time with a preset threshold, and determine whether there is a data location problem by comparing the data transmission time with a preset threshold; S53: If the queue time is too long, enter the computing resource expansion process; S54: If the data transmission time is too long, enter the data perception expansion process.

6. The highly balanced serverless workflow scheduling method driven by data flow based on Kubernetes according to claim 5 is characterized in that: Step S53 includes: S531: Evaluate the parent task and child tasks of the current task; S532: Based on the node's resource status and data transmission overhead, a weighted score is formed by calculating the inverse relationship between the percentage of the node's remaining resources to the total resources and the data transmission time. The node with the highest score is selected for task pod expansion. S533: Expand the task Pod on the selected node to alleviate the load pressure of the set node.

7. The highly balanced serverless workflow scheduling method driven by data flow based on Kubernetes according to claim 5 is characterized in that: Step S54 includes: S541: Analyze the data transmission between the parent and child tasks and locate the task with the longest data transmission time; S542: Compare the resource status of the current task node and the data source task node, calculate the CPU utilization and memory usage of both, and select the node whose average CPU utilization and memory usage is lower than the system preset threshold as the deployment target, where the system preset threshold is dynamically set based on historical task execution data; S543: If a data source node is selected, expand the Pod of the current task; if the current task node is selected, expand its parent task.

8. The highly balanced serverless workflow scheduling method driven by data flow based on Kubernetes according to claim 1 is characterized in that: Also includes: S6: Record the idle time of each task pod. If the idle time exceeds the preset threshold continuously, the pod is destroyed. The preset threshold is dynamically calculated by the system based on the task arrival interval and task execution frequency in the workload history data.

9. A system for a highly balanced serverless workflow scheduling method driven by data flow based on Kubernetes according to any one of claims 1 to 8, characterized in that: include: Gateway request receiving module, used to receive JSON format workflow files sent by users and create related resources; The central task scheduling module is used to deploy task pods based on node resource usage and task resource requests, and select the root task pod with the shortest request queue length to send task execution requests; The sidecar container data synchronization module is used to receive external request data and queue the request; The sidecar container task scheduling module is used to obtain and execute task requests from the request queue and select the optimal subsequent task pod based on the subsequent task scheduling strategy based on data transmission time, request queue length, and task execution time; The sidecar container log module is used to record data transmission time and task execution time; The auto-scaling module is used to evaluate the workflow execution and trigger the automatic scaling of task pods based on the auto-scaling policy; The central database includes a status information database and a log database, which are used to store system status information and execution logs.

10. An electronic device comprising a memory and a processor, characterized in that: The processor runs the program instructions stored in the memory to perform the method according to any one of claims 1 to 8.

Images