Kubernetes-based data flow driven high-balance serverless workflow scheduling method and system

Abstract

The application discloses a kind of based on kubernetes's data stream driven high equilibrium serverless workflow scheduling method and system, this workflow scheduling method includes: S1: receiving the JSON format workflow file sent by user, creates relevant resources;S2: based on relevant resources, in combination with node resource use and task resource request deployment task pod, each task pod maintains a request queue;S3: based on the root task pod of deployed task pod, the shortest request queue length selects and sends task execution request;S4: based on execution request, execute task through sidecar container and based on the subsequent task scheduling strategy of data transmission time, request queue length and task execution time, select optimal subsequent task pod and send request data;S5: based on execution, assess workflow execution, and based on automatic expansion strategy triggers the automatic scaling of task pod.Solve the problem that existing data-driven workflow scheduling scheme is too coarse, resource utilization efficiency is low and expansion strategy is not flexible etc.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of container orchestration and cloud native computing, and particularly relates to a Kubernetes-based data flow driven high-balance serverless workflow scheduling method and system. BACKGROUND

[0002] With the rapid development of cloud computing technology, Serverless architecture has become a key solution for modern distributed systems. This architecture abstracts the management of computing resources, allowing developers to focus on business logic implementation without worrying about underlying infrastructure. Serverless architecture has high flexibility and automatic scaling capabilities, effectively reducing costs when handling short-term high-frequency requests, and providing on-demand resource allocation.

[0003] Kubernetes, as a widely used open-source container orchestration platform, excels in the automatic deployment, scaling, and management of containerized workloads. Its powerful automated management and scaling capabilities dynamically schedule computing resources to ensure the high availability of containerized applications. Kubernetes can automatically handle tasks such as service discovery, load balancing, and container orchestration, providing significant advantages when handling large distributed applications.

[0004] However, existing Serverless workflow scheduling systems face several challenges. Current data-driven workflow scheduling systems mainly rely on coarse-grained task division, dividing and scheduling workflows by subgraphs, which makes it difficult to implement fine-grained task-level scheduling and cannot handle complex task dependencies and load fluctuations. Existing scheduling methods lack effective feedback mechanisms for task dependencies and runtime performance, making it impossible to adjust scheduling strategies in real-time based on task status and resource requirements, resulting in insufficient resource utilization or performance degradation.

[0005] In addition, current Kubernetes workflow scheduling systems mostly use fixed-grained scheduling strategies. This coarse-grained scheduling approach is difficult to dynamically adapt to complex dependencies between tasks, and can easily lead to resource waste or performance degradation under high-concurrency loads. When handling high-concurrency requests, the scheduling capabilities and automatic scaling mechanisms of the system are also limited, making it difficult to ensure efficient load balancing and task execution real-time performance in the case of large data transmission overhead.

[0006] These problems have prompted us to propose a new solution based on Kubernetes technology, aiming to provide more fine-grained data-driven workflow scheduling and elastic execution mechanisms to address scenarios with strong task correlation, large data transmission overhead, and significant request load fluctuations. SUMMARY

[0007] The application provides a kubernetes-based data flow driven high-balance serverless workflow scheduling method and system to solve the above problems in the prior art.

[0008] In order to achieve the above purpose, the application provides the following technical solutions.

[0009] A kubernetes-based data flow driven high-balance serverless workflow scheduling method comprises the following steps.

[0010] S1: receiving a JSON format workflow file sent by a user, and creating relevant resources;

[0011] S2: deploying task pods based on the relevant resources, combining node resource usage and task resource requests, and maintaining a request queue for each task pod;

[0012] S3: selecting a root task pod with the shortest request queue length based on the deployed task pods to send a task execution request;

[0013] S4: executing a task through a sidecar container based on the execution request, and selecting an optimal subsequent task pod to send request data based on a subsequent task scheduling strategy of data transmission time, request queue length and task execution time;

[0014] S5: evaluating workflow execution based on the execution, and triggering automatic scaling of the task pod based on an automatic scaling strategy.

[0015] The S2 step comprises the following steps.

[0016] S21: setting the number of replicas of each task pod to 2;

[0017] S22: sorting tasks according to resource requirements of tasks in the workflow, and deploying task pods from high to low resource requirements;

[0018] S23: when selecting a node, selecting a node with a remaining computing resource amount greater than a task request resource amount and a highest ratio of the remaining computing resource to total computing resources of the node;

[0019] S24: updating resource usage records of the node, including allocated CPU cores, allocated memory capacity and remaining available resource amount, so as to provide resource data for scheduling of subsequent tasks.

[0020] The S4 step comprises the following steps.

[0021] S41: a sidecar container data synchronization module receives request data sent from outside, and arranges the request into a request queue;

[0022] S42: The sidecar container log module records the data transmission time;

[0023] S43: The sidecar container task scheduling module obtains and executes the task request from the request queue;

[0024] S44: The sidecar container log module records and synchronizes the task execution time;

[0025] S45: After the task execution is completed, the request data is sent to the subsequent task pod with the shortest expected waiting time according to the expected waiting time calculated based on the data transmission time and the request queue length.

[0026] The S45 step includes:

[0027] S451: The request queue lengths of the external task pod and the internal task pod are obtained from the state database, and the pod with the smallest queue length value in the external and internal pods is determined respectively;

[0028] S452: The average execution time of the historical task and the data transmission time of the current task are obtained from the log database;

[0029] S453: The expected waiting time of the task scheduled to the external pod is calculated, which is equal to the queue length multiplied by the average execution time and then added to the data transmission time;

[0030] S454: The external expected waiting time is compared with the internal pod expected waiting time, and the pod with shorter expected waiting time is selected for task scheduling.

[0031] The S5 step includes:

[0032] S51: The completion time of the task pod request is monitored;

[0033] S52: The cause of the delay is analyzed, whether there is a computing resource bottleneck is judged by comparing the queuing time with the preset threshold, and whether there is a data location problem is judged by comparing the data transmission time with the preset threshold;

[0034] S53: If the queuing time is too long, the computing resource expansion process is entered;

[0035] S54: If the data transmission time is too long, the data perception expansion process is entered.

[0036] The S53 step includes:

[0037] S531: The parent task and the child task of the current task are evaluated;

[0038] S532: In combination with the resource status of the node and the data transmission overhead, the percentage of the remaining resources of the node to the total resources is calculated in inverse proportion to the data transmission time to form a weighted score, and the node with the highest score is selected for task Pod expansion;

[0039] S533: Expanding the task Pod on the selected node to relieve the load pressure of the set node.

[0040] Among them, the S54 step includes:

[0041] S541: Analyze the data transmission between parent and child tasks, and locate the task with the longest data transmission time;

[0042] S542: Compare the resource status of the current task node and the data source task node, calculate the CPU utilization and memory usage of the two, and select the node with an average CPU utilization and memory usage lower than the system preset threshold as the deployment target, wherein the system preset threshold is dynamically set based on historical task execution data;

[0043] S543: If the data source node is selected, expand the Pod of the current task; if the current task node is selected, expand its parent task.

[0044] Among them, it also includes:

[0045] S6: Record the idle time of each task pod, and destroy the pod when the idle time continuously exceeds the preset threshold, wherein the preset threshold is dynamically calculated by the system based on the task arrival interval and task execution frequency in the historical data of the workload.

[0046] Among them, a system of the Kubernetes-based data flow-driven high-balance serverless workflow scheduling method includes:

[0047] The Gateway request receiving module is used to receive the JSON format workflow file sent by the user and create related resources;

[0048] The central task scheduling module is used to deploy task pods in combination with node resource usage and task resource requests, and select the root task pod with the shortest request queue length to send task execution requests;

[0049] The sidecar container data synchronization module is used to receive request data sent from the outside and put the request into the request queue;

[0050] The sidecar container task scheduling module is used to obtain and execute task requests from the request queue, and select the optimal subsequent task pod based on the subsequent task scheduling strategy of data transmission time, request queue length and task execution time;

[0051] a sidecar container log module for recording data transmission time and task execution time;

[0052] an automatic scaling module for evaluating workflow execution and triggering automatic scaling of task pods based on automatic scaling strategies;

[0053] a central database including a state information database and a log database for storing system state information and execution logs.

[0054] Among them, an electronic device includes a memory and a processor. Compared with the prior art, the present application has the following advantages:

[0055] The present application aims to solve the problems of existing data-driven workflow scheduling schemes, such as coarse granularity, low resource utilization efficiency, and inflexible expansion strategy. Traditional data-driven workflow scheduling platforms usually divide workflows into subgraphs based on data transmission overhead and schedule and scale them in units of subgraphs, which is difficult to achieve fine-grained control of individual tasks and is prone to resource waste. The present application continues the data-driven scheduling idea and introduces a more fine-grained task-level scheduling mechanism, which can accurately identify tasks with long queuing time and, combined with data transmission conditions, schedule tasks to the optimal node, ensuring data locality while avoiding unnecessary resource expansion. In addition, the present application not only focuses on data flow during task expansion, but also analyzes the load conditions of source and target nodes to preferentially select nodes with better performance, realizing the unification of data-driven and load balancing. The scheduling strategy takes into account both task queuing time and data transmission overhead to improve overall execution efficiency and enhance the resource scheduling and elastic execution capability of the system in complex and high-concurrency workflow scenarios.

[0056] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application.

[0057] The technical solutions of the present application will be further described in detail below with the help of the accompanying drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0058] The accompanying drawings are used to provide a further understanding of the present application, and constitute a part of the specification, together with the embodiments of the present application, to explain the present application, and do not constitute a limitation on the present application. In the drawings:

[0059] Figure 1 A flowchart of a data flow-driven high-balance serverless workflow scheduling method based on kubernetes in an embodiment of the present application;

[0060] Figure 2A high-balance serverless workflow scheduling method based on kubernetes data flow driving in the embodiment of the application is shown in the figure, which comprises the following steps:

[0061] Figure 3 A sidecar container task scheduling module workflow chart in the embodiment of the application is shown in the figure.

[0062] Figure 4 A subsequent task scheduling strategy flow chart of the sidecar container task scheduling module in the application is shown in the figure.

[0063] Figure 5 An automatic expansion strategy flow chart in the application is shown in the figure. DETAILED DESCRIPTION

[0064] The preferred embodiments of the application are described below in combination with the drawings, and it should be understood that the preferred embodiments described herein are only used to illustrate and explain the application, and are not used to limit the application.

[0065] The embodiment of the application provides a high-balance serverless workflow scheduling method based on kubernetes data flow driving as shown in the figure, which comprises the following steps: Figure 1

[0066] S1: receiving a JSON format workflow file sent by a user, creating relevant resources;

[0067] S2: based on the relevant resources, combining the node resource usage and the task resource request to deploy a task pod, each task pod maintaining a request queue;

[0068] S3: based on the deployed task pod, selecting a root task pod with the shortest request queue length to send a task execution request;

[0069] S4: based on the execution request, executing the task through a sidecar container and selecting an optimal subsequent task pod to send a request data based on the subsequent task scheduling strategy of the data transmission time, the request queue length and the task execution time;

[0070] S5: based on the execution, evaluating the workflow execution, and triggering the automatic scaling of the task pod based on the automatic expansion strategy.

[0071] ​The working principle and beneficial effects of the above technical solution are as follows: Step S1: receiving a JSON format workflow file sent by a user, and creating related resources; wherein the user sends a workflow file through a Gateway request receiving module. The workflow file is in JSON format and contains detailed descriptions of the workflow, such as the definition of each task, the dependency relationship between tasks, resource requirements, etc. After the Gateway request receiving module receives the file, it performs preliminary verification to check the format and data integrity, and then passes the workflow file that has passed verification to the central task scheduling module and registers the request in the state information database, facilitating subsequent tracking.

[0072] Step S2: deploying task pods based on related resources and combining node resource usage and task resource requests; wherein the central task scheduling module receives the workflow, parses the resource requirements of each task, and performs initial deployment in combination with the cluster node resource usage. A pod is created for each task and the number of replicas is set (default is 2 to ensure high availability), and the task is preferentially deployed on a node with sufficient resources. After creating the pod, the scheduling module updates the state information database to record information such as pod resource status, task execution status, and request queue length. Each task pod maintains a request queue to store tasks to be executed, and the queue determines the scheduling order of the tasks.

[0073] Step S3: selecting the root task pod with the shortest request queue length to send a task execution request based on the deployed task pod; wherein the central task scheduling module queries the root task (independent task) of the current workflow from the state information database. The scheduling module selects the pod with the shortest request queue and the most abundant resources to execute the root task based on factors such as the request queue length of the task pod and the node resource situation. After determining the execution pod, it sends an execution request containing task parameters, input data, and execution instructions to it, and the task pod starts executing the task after receiving the request.

[0074] Step S4: Based on the execution request, the sidecar container executes the task and selects the optimal subsequent task pod to send the request data based on the subsequent task scheduling strategy of data transmission time, request queue length and task execution time. After the task pod is started, the sidecar container data synchronization module listens to and receives data from other tasks or external systems, and after synchronization, the task request is put into the request queue. The sidecar container task scheduling module obtains the task from the queue and executes it. After the task execution is completed, the sidecar task scheduling module considers the data transmission time, request queue length and task execution time to select the optimal subsequent task pod to send the request data. The sidecar container log module records the task execution time, data transmission time and queuing time, etc. Key information is synchronized to the log database to provide basis for subsequent scheduling optimization.

[0075] Step S5: Based on the execution, the workflow execution is evaluated, and the automatic scaling of the task pod is triggered based on the automatic scaling strategy; wherein the automatic scaling module monitors the task execution, and pays special attention to the task completion time, queuing time and data transmission time. When the performance bottleneck is found, the module will analyze the reason and take corresponding measures. If the queuing time is too long, it is determined that the computing resource is the bottleneck, the data transmission of parent and child tasks is analyzed, and the optimal node is selected to increase the Pod replica; if the data transmission time is too long, the data-aware expansion is executed to optimize the data transmission path and deploy a new Pod in the resource-rich node. This automatic scaling strategy takes into account data locality and load balancing to ensure balanced optimization of system performance and resource utilization.

[0076] The specific operation steps are as follows:

[0077] Step one: initialize platform components: various components include: gateway request receiving module, central task scheduling module, sidecar container data synchronization module, sidecar container task scheduling module, sidecar container log module, automatic scaling module and central database.

[0078] Step two, the user sends a workflow file specified by JSON format to the Gateway request receiving module;

[0079] Step three, the Gateway request receiving module processes the received JSON file and creates related resources, and then passes them to the central task scheduling module;

[0080] Step four, the central task scheduling module will comprehensively consider the node resource usage and the resource request of each task in the workflow to initially deploy the task pod, and each task pod will maintain a request queue.

[0081] Step five: the central task scheduling module will find the pod with the shortest request queue length in all root task pods in the state information database, and send a task execution request.

[0082] Step six: the sidecar container data synchronization module will continuously receive request data from the outside, and after the request data is received, the request will be put into the request queue. At the same time, the sidecar log module will record the data transmission time.

[0083] Step seven: the sidecar container task scheduling module will continuously obtain and execute task requests from the request queue, and the sidecar log module will record and synchronize the task execution time.

[0084] Step eight: after the current task execution is completed, the sidecar task scheduling module will select the most suitable subsequent task pod to send request data by comprehensively considering the data transmission time saved in the log of the current task, the request queue length of the subsequent task, and the task execution time.

[0085] Step nine: after the workflow is completed, the pod will not be destroyed, but will wait for the next task execution of the central scheduling module or the sidecar task scheduling module.

[0086] Step ten: when the workflow is completed, trigger the auto-scaling module to obtain the workflow and task execution situation through the log and perform evaluation, and if the current scheduling cannot meet the slo, trigger the automatic scaling of the task pod.

[0087] In another embodiment, the S2 step comprises:

[0088] S21: set the number of replicas of each task pod to 2;

[0089] S22: sort the tasks according to the resource requirements of the tasks in the workflow, and deploy the task pods from the most to the least resource requirements;

[0090] S23: when selecting nodes, select the node with the highest ratio of remaining computing resources to total computing resources, whose remaining computing resources are greater than the task request resource amount;

[0091] S24: update the resource usage record of the node, including the allocated CPU core number, the allocated memory capacity, and the remaining available resource amount, to provide resource data for the scheduling of subsequent tasks.

[0092] The working principle and beneficial effects of the above technical solution are as follows:

[0093] In the process of deploying task Pods initially, the central task scheduling module will execute the deployment tasks according to the resource usage of each node in the cluster and the resource requests of each task in the workflow.

[0094] First, the central task scheduling module sets the number of replicas for each task Pod to 2, which means that two replicas will be started for each task initially.

[0095] The resource requests of task Pods are defined by the user's request.

[0096] Next, the central task scheduling module sorts all tasks in the workflow according to resource requirements and deploys task Pods from more to less as required.

[0097] When selecting nodes, the scheduling module selects those nodes with the most remaining resources.

[0098] Once the task Pods are successfully deployed to the nodes, the scheduling module updates the resource usage of the nodes to provide accurate resource data for the scheduling of subsequent tasks.

[0099] The central task scheduling module will continue to deploy all task Pods in the workflow according to the same rules. This scheduling mode ensures efficient and fair use of resources by prioritizing the deployment of tasks with the most resource requests to nodes with the most resources.

[0100] Step S21: Set the number of replicas for each task Pod to 2; wherein the central task scheduling module initially deploys task Pods by default setting 2 replicas for each task. This configuration ensures high availability of tasks, so that even if one Pod fails, the other Pod can continue to execute the task, preventing service interruption caused by single-point failure. Each replica can receive and process the same type of task request, thereby improving the stability and processing capacity of the system.

[0101] Step S22: Sort tasks according to resource requirements and deploy task Pods from more to less; wherein the central task scheduling module parses the workflow file to obtain the resource requests (CPU, memory, etc.) of each task, and sorts the tasks from high to low according to resource requirements. This sorting strategy ensures that tasks with higher resource requirements are deployed first, avoiding situations where high-demand tasks cannot be scheduled due to insufficient resources later. The sorting of resource requirements takes into account the computational complexity and data processing capacity of the task, allowing the system to allocate resources reasonably and improve overall resource utilization efficiency.

[0102] Step S23: When selecting a node, select the node with the largest remaining computing resource amount and the highest ratio of remaining computing resources to total computing resources; wherein for each task Pod, the central task scheduling module evaluates the resource status of all nodes in the cluster, and filters out nodes with remaining resources greater than the task request resources. Among these nodes, the proportion of the remaining resources of each node (remaining resources / total resources) is calculated, and the node with the highest proportion is selected to deploy the task. This selection strategy ensures balanced allocation of resources, avoids the situation where some nodes are overloaded while others are idle, and improves the overall cluster resource utilization and task execution efficiency.

[0103] Step S24: Update the resource usage record of the node, including the number of allocated CPU cores, the allocated memory capacity and the amount of remaining available resources, in order to provide resource data for subsequent task scheduling; wherein after the task Pod is deployed, the central task scheduling module updates the node resource usage record in the state information database in real time. The update content includes the number of allocated CPU cores, the allocated memory capacity and the amount of remaining available resources of the node, etc. These data are crucial for subsequent task scheduling, ensuring that the scheduling decision is based on the latest resource usage, preventing excessive resource allocation or scheduling conflicts. The resource records in the database are also used for system monitoring and resource optimization analysis, helping administrators understand the system resource usage efficiency and potential bottlenecks.

[0104] In another embodiment, the S4 step includes:

[0105] S41: The sidecar container data synchronization module receives the request data sent by the outside, and puts the request into the request queue;

[0106] S42: The sidecar container log module records the data transmission time;

[0107] S43: The sidecar container task scheduling module obtains and executes the task request from the request queue;

[0108] S44: The sidecar container log module records and synchronizes the task execution time;

[0109] S45: After the task execution is completed, the expected waiting time of the subsequent task pod sending request data is selected according to the expected waiting time calculated by the data transmission time and the length of the request queue.

[0110] The working principle and beneficial effects of the above technical solution are as follows: Step S41: The sidecar container data synchronization module receives the request data sent from the outside, and puts the request into the request queue; wherein the data synchronization module in the sidecar container runs continuously, listens to and receives the request data from the outside. The received request data usually contains a unique identifier (UID) and task parameter information. After the data is received, the module marks the request state as "received", reads the parameter data required by the current task Pod from the environment variable, judges whether the required parameters of the request are complete. If the parameters are complete, the request is added to the request queue, and the state information of the corresponding task in the state database is updated to ensure that the task is ready for scheduling and execution.

[0111] Step S42: The sidecar container log module records the data transmission time; wherein the log module records the receiving time of each parameter data in the data receiving process, which is used for subsequent analysis of data transmission efficiency. These records include the data transmission start time, data receiving completion time and total transmission time consumption, etc. These time data are uploaded to the log database to provide traceability and monitoring capability for data flow of the system, help to analyze potential data transmission bottlenecks, and provide important basis for subsequent system optimization and task scheduling decision.

[0112] Step S43: The sidecar container task scheduling module obtains and executes the task request from the request queue; wherein the task scheduling module is responsible for actual task execution, which obtains the task request prepared from the request queue in sequence and starts execution. The module first parses the execution parameters and instructions in the task request, and then calls the corresponding computing resources to execute the task. During the task execution, the scheduling module monitors the execution state to ensure the normal progress of the task, and updates the task state after the execution is completed. This queue processing mechanism ensures the orderly execution of the task, improves the response ability and processing efficiency of the system.

[0113] Step S44: The sidecar container log module records and synchronizes the task execution time; wherein the log module records the task start execution time, task completion time and total execution time consumption, etc. These execution time data are synchronized to the log database, and together with the previously recorded data transmission time, they constitute the time record of the whole life cycle of the task. These records are crucial for analyzing task execution efficiency, identifying performance bottlenecks and optimizing system scheduling strategy. The system can calculate the average execution time of the task based on these historical execution data, and provide more accurate time estimation for subsequent scheduling decision.

[0114] Step S45: After the task execution is completed, the expected waiting time of the subsequent task pod sending request data is selected according to the data transmission time and the request queue length. Wherein, after the task execution is completed, the sidecar task scheduling module obtains the subsequent task information from the environment variable. In combination with the running information (such as the IP address of each Pod, the request queue length, and the current execution task time) in the state database and the historical data (such as the average execution time and the data transmission time length) in the log database, the expected waiting time of each potential target Pod is calculated. The expected waiting time is equal to the queue length multiplied by the average execution time and then added to the data transmission time. The module selects the Pod with the shortest expected waiting time to send the result data, thereby achieving intelligent balance between data locality and computing load and optimizing the overall workflow execution efficiency.

[0115] Specifically, in the workflow of the Sidecar container, the system mainly cooperates with the data synchronization module, the log collection module, the task scheduling module, and the state database to complete the reception, scheduling, and execution of tasks. The specific running process is as follows:

[0116] When the Pod is started, the data synchronization module in the Sidecar container immediately starts running, which is used to continuously listen to and receive the request data from the outside.

[0117] Each request data usually contains a unique identifier (UID) and corresponding task parameter information. For each received request data, the system will first mark the state of the request as “received” after the data is received.

[0118] Subsequently, the data synchronization module reads the parameter data required by the current task Pod from the environment variable and judges whether all the parameter data required by the current request is complete. If it is confirmed that the parameter data is complete, the data synchronization module will add the request to the request queue and synchronously update the state information of the corresponding task in the state database, so as to ensure that the task is ready for scheduling and execution.

[0119] At the same time of data reception, the log collection module records the reception time of each parameter data, thereby realizing the traceability and monitoring of the entire data flow process.

[0120] In addition, the data synchronization module keeps running to ensure that it can respond and receive new external request data in real time.

[0121] The actual execution of the task is responsible by the task scheduling module in the Sidecar container. The module will obtain the request task which is ready for execution from the request queue and start execution. After the task execution is completed, the scheduling module will first obtain the subsequent task information of the task from the environment variable and determine the target Pod of the next stage in combination with the running information in the state database.

[0122] Specifically, the system queries the current running state of each task Pod in the state database, including its IP address, request queue length, current task execution time, and historical data transmission time, etc.

[0123] Based on this information, the task scheduling module will select the most suitable target Pod and send the processing results or data to it. After the data is sent, the system will get the next task to be processed from the request queue and update the status record of the relevant Pod in the state database again to achieve dynamic balance and efficient execution of task scheduling and resource allocation.

[0124] In another embodiment, the S45 step includes:

[0125] S451: Obtain the request queue length of the external task Pod and the internal task Pod from the state database, and determine the Pod with the smallest queue length value in the external and internal Pods respectively;

[0126] S452: Obtain the average execution time of historical tasks and the data transmission time of the current task from the log database;

[0127] S453: Calculate the expected waiting time of the task scheduled to the external Pod, which is equal to the queue length multiplied by the average execution time plus the data transmission time;

[0128] S454: Compare the external expected waiting time with the internal Pod expected waiting time, and select the Pod with shorter expected waiting time for task scheduling.

[0129] The working principle and beneficial effects of the above technical solution are: step S451: Obtain the request queue length of the external task Pod and the internal task Pod from the state database, and determine the Pod with the smallest queue length value in the external and internal Pods respectively; wherein, the sidecar task scheduling module first distinguishes the external task Pod (Pod deployed on different nodes) and the internal task Pod (Pod deployed on the same node). Then query the current request queue length of all subsequent task related Pods from the state database. Compare the queue lengths of the external and internal two types of Pods respectively to determine the Pod with the smallest queue length. This strategy of distinguishing internal and external Pods takes into account the physical distance factor of data transmission, providing basic data support for subsequent scheduling decisions based on data locality and load balancing.

[0130] Step S452: Obtain the average execution time of historical tasks and the data transmission duration of the current task from the log database; wherein, the sidecar task scheduling module obtains two types of key time data from the log database: one is the average execution time of historical tasks, which reflects the computational complexity and processing efficiency of the task; the other is the actual data transmission duration of the current task, which reflects the network status and data size of data transmission. The average execution time is calculated by analyzing the historical execution records of similar tasks, while the data transmission duration is extracted from the log records of the current execution period. These time data provide important parameters for subsequent waiting time prediction.

[0131] Step S453: Calculate the estimated waiting duration of the task scheduled to the external Pod, which is equal to the queue length multiplied by the average execution time and then added to the data transmission time; wherein, the sidecar task scheduling module uses a scientific calculation method to estimate the waiting time of the task in the external Pod. The specific calculation formula is: estimated waiting time = queue length x average execution time + data transmission time. This formula considers two key factors: one is the processing time of the tasks already in the queue (queue length multiplied by average execution time), the other is the time overhead of data transmission from the current node to the target node. This calculation method comprehensively considers the computational load and network transmission factors, making the scheduling decision more accurate and efficient.

[0132] Step S454: Compare the external estimated waiting time with the internal Pod estimated waiting time, and select the Pod with shorter estimated waiting time for task scheduling. Wherein, the sidecar task scheduling module compares the external Pod estimated waiting time calculated in step S453 with the internal Pod estimated waiting time. According to the comparison result, if the external Pod estimated waiting time is greater than the internal Pod estimated waiting time, select the internal Pod for task scheduling; otherwise, select the Pod with the shortest request queue length in the external Pod for scheduling. This decision-making process ensures the efficiency of task scheduling, maximally reduces the waiting time of the task, and thus improves the execution efficiency of the overall workflow.

[0133] Specifically, the subsequent task pod selection of the sidecar task scheduling module is as follows:

[0134] The sidecar task scheduling module adopts an intelligent scheduling strategy that combines computational load and data transmission evaluation, which is highly consistent with the current "data-driven" scheduling concept.

[0135] In the scheduling process, the module first obtains the shortest request queue length of the external task Pod and the internal task Pod from the state data center, and obtains the average execution time of historical tasks and the data transmission duration of the current task from the log data center.

[0136] Based on this information, the system can calculate the expected waiting time of the task if it is scheduled to an external Pod, which is "queue length x average execution time + data transmission time".

[0137] Subsequently, the scheduling module compares this external expected waiting time with the waiting time of the internal Pod. If the external waiting time is significantly higher than the internal, the task will be preferentially assigned to the internal task Pod; otherwise, the task will be scheduled to the external task Pod, thereby minimizing the overall delay.

[0138] This process achieves a dynamic trade-off between internal and external nodes in the system, making scheduling more agile and adaptive to changing resource states. Compared to traditional methods that only focus on computational load or only schedule based on proximity, the biggest advantage of this strategy is that it considers both request queue length and data transmission cost as key factors. This not only reflects the data flow efficiency-centered workflow design idea, but also makes task scheduling more in line with the dynamic changes of resources in the actual running environment. By incorporating data locality, execution pressure, and communication cost into a unified scheduling model, the system can maintain resource utilization while significantly optimizing task response time and overall throughput, truly realizing a unified scheduling mechanism that combines data-driven and load balancing.

[0139] In another embodiment, the S5 step comprises:

[0140] S51: Monitor the completion time of task Pod requests;

[0141] S52: Analyze the causes of delay, determine whether there is a computational resource bottleneck by comparing the queuing time with a preset threshold, and determine whether there is a data location problem by comparing the data transmission time with a preset threshold;

[0142] S53: If the queuing time is too long, enter the computational resource expansion process;

[0143] S54: If the data transmission time is too long, enter the data-aware expansion process.

[0144] The working principle and beneficial effects of the above technical solution are as follows: Step S51: Monitor the completion time of task Pod requests; wherein the automatic scaling module continuously monitors the request completion time of each task Pod and records the execution time of each task. This monitoring process ensures that the system can obtain real-time performance data of task execution, providing basic information for subsequent performance analysis and resource adjustment.

[0145] Step S52: analyze the cause of the delay, determine whether there is a computing resource bottleneck by comparing the queuing time with the preset threshold, and determine whether there is a data location problem by comparing the data transmission time with the preset threshold; wherein, when the automatic scaling module monitors the abnormal increase of the request completion time, it further analyzes the cause of the delay. The system compares the queuing time of the current task with the preset threshold to determine whether there is a computing resource bottleneck; at the same time, the data transmission time is compared with the preset threshold to determine whether there is a data location problem. This analysis process helps the system to identify the specific source of the performance bottleneck, and provides a basis for subsequent expansion strategy.

[0146] Step S53: if the queuing time is too long, enter the computing resource expansion process; wherein, if the monitoring queuing time exceeds the preset threshold, the system will determine that the current task Pod is a performance bottleneck node, and the automatic scaling module will start the computing resource expansion process. This process will evaluate the parent task and child task of the current task, combine the resource status of the node and the data transmission overhead, and select the most suitable node for task Pod expansion to achieve balanced distribution of resources and improve the resource utilization and task scheduling efficiency of the overall cluster.

[0147] Step S54: if the data transmission time is too long, enter the data-aware expansion process. Wherein, if the monitoring data transmission time exceeds the preset threshold, the system will determine that there is a large data exchange between tasks, and the current task cannot be deployed on a node close to the data source. At this time, the automatic scaling module will enter the data-aware expansion process, analyze the data transmission between the parent and child tasks, and locate the task with the longest data transmission time as the optimization focus. The system will compare the resource status of the current task node and the data source task node, and select the node with more abundant resources as the deployment target to optimize the data transmission path and improve the task execution efficiency.

[0148] Specifically, the automatic expansion strategy includes: in order to improve the scheduling efficiency of the task Pod, the expansion module first judges whether there is a performance bottleneck by monitoring the completion time of the task Pod request. When detecting the abnormal increase of the task request completion time, the system will further analyze the main cause of the delay to determine whether it is caused by the long queuing time of the computing resource bottleneck or the long data transmission time of the data location problem.

[0149] If the queuing time is too long, it means that the computing resources of the current node are insufficient, at which time the system will enter the computing resource expansion process. It will evaluate the parent task and child task of the current task, combine the resource status of the node and the data transmission overhead, and select the most suitable node for task Pod expansion. This helps to achieve balanced distribution of resources, relieve the load pressure of specific nodes, and thus improve the resource utilization and task scheduling efficiency of the overall cluster.

[0150] If the data transmission time is too long, it indicates that there is a large data exchange between tasks, and the current task is not deployed on a node close to the data source. At this time, the system will enter the data-aware expansion process, analyze the data transmission between parent and child tasks, and locate the task with the longest data transmission time as the optimization focus. Then, it will compare the resource status of the current task node and the data source task node, and select the node with more abundant resources as the deployment target.

[0151] If the data source node is selected, the Pod of the current task is directly expanded to improve data locality; if the current task node is selected, the parent task is expanded to optimize the data path from the source.

[0152] This strategy considers data access efficiency while maintaining awareness and utilization of node resource status, thereby achieving dual optimization of "data awareness" and "resource balance".

[0153] This expansion mechanism not only makes targeted scheduling according to the bottleneck type, improving the accuracy of scheduling response, but also achieves dynamic balance between data locality and resource load. It not only reduces unnecessary data transmission overhead, but also effectively disperses resource pressure, making it an intelligent expansion strategy that balances performance and efficiency.

[0154] In another embodiment, the S53 step comprises:

[0155] S531: evaluating the parent task and child task of the current task;

[0156] S532: combining the resource status of the node and the data transmission overhead, forming a weighted score through the inverse relationship between the percentage of node remaining resources to total resources and data transmission time, and selecting the node with the highest score for task Pod expansion;

[0157] S533: expanding the task Pod on the selected node to relieve the load pressure of the set node.

[0158] The working principle and beneficial effects of the above technical solution are as follows: S531: evaluating the parent task and child task of the current task; the automatic scaling module comprehensively analyzes the relationship and data transmission between the parent task and child task of the current task. The system extracts historical execution records from the log database and identifies the parent task and child task that have direct data transmission relationship with the current task. Through this analysis, the system can determine which tasks have a large amount of data exchange, thereby providing a basis for subsequent task expansion decisions. This evaluation of parent and child task relationships is the basis for data-driven scheduling, enabling the system to accurately schedule based on data flow in the workflow.

[0159] S532: Select the optimal node based on node resource status and data transmission overhead; After determining the need for Pod expansion, the system will consider the resource status of each node and the potential data transmission overhead. Specifically, the system first obtains the real-time resource usage of each node in the cluster, including CPU utilization, memory usage, etc. Then, the system calculates the percentage of remaining resources to total resources, and estimates the data transmission time. Based on these two factors, the system establishes a weighted scoring mechanism: the percentage of remaining resources is inversely proportional to the data transmission time. This scoring mechanism takes into account both resource adequacy and data locality, and can strike a balance between the two. Finally, the system selects the node with the highest score as the target node for task Pod expansion.

[0160] S533: Expand the task Pod on the selected node; Once the optimal expansion node is determined, the system will deploy a new task Pod on the node. The expansion operation is completed through the API interface of Kubernetes, and the system will create a new Pod instance with the same configuration as the original task Pod, and ensure that the new Pod can access the workflow scheduling system. This expansion strategy can effectively alleviate the load pressure of specific nodes and achieve more balanced resource allocation. Since the selection of the expansion location has taken into account the data transmission overhead, the newly expanded Pod can reduce unnecessary data transmission and improve the overall workflow execution efficiency.

[0161] In another embodiment, the S54 step includes:

[0162] S541: Analyze the data transmission between parent and child tasks, and locate the task with the longest data transmission time;

[0163] S542: Compare the resource status of the current task node and the data source task node, calculate their CPU utilization and memory usage, and select the node with an average CPU utilization and memory usage lower than the system preset threshold as the deployment target, wherein the system preset threshold is dynamically set based on historical task execution data;

[0164] S543: If the data source node is selected, expand the Pod of the current task; if the current task node is selected, expand its parent task.

[0165] The working principle and beneficial effects of the above technical solution are as follows: S541: Analyzing the data transmission between parent and child tasks; when the system detects that the data transmission time is too long, the automatic scaling module will deeply analyze the data transmission between the parent and child tasks. The system extracts the data transmission records of related tasks from the log database, including transmission data volume, transmission time length, transmission frequency and other information. Through these data, the system can locate the task pair with the longest data transmission time, which is usually the key link causing the performance bottleneck. This precise data flow analysis capability is the core of the system to realize data-driven scheduling, enabling the subsequent expansion decision to optimize the data flow path.

[0166] S542: Comparing the resource status of the task node and the data source node; after determining the task pair that needs to be optimized, the system will compare the resource status of the current task node and the data source task node in detail. The system real-time acquires the CPU utilization and memory usage of these two types of nodes, and calculates their average values. The system presets a resource usage threshold, which is dynamically set based on historical task execution data. The system selects the node with an average CPU utilization and memory usage lower than the threshold as the deployment target. This dynamic threshold setting based on historical data enables the system to adapt to resource changes under different workloads, improving the accuracy and adaptability of scheduling decisions.

[0167] S543: Performing expansion operation according to the selection result; according to the analysis results of the previous two steps, the system will decide which node to perform the expansion operation. If the system selects the data source node, it means that the resource status of the node is good, and the system will expand the Pod of the current task on the node, which can reduce the data transmission overhead and improve the data locality. If the system selects the current task node, it means that the node has more abundant resources, and the system will expand the Pod of its parent task on the node, which can optimize the data flow and reduce the data transmission time. This flexible expansion strategy fully reflects the intelligent trade-off of the system between data locality and resource balance, and can select the optimal expansion scheme according to the actual situation.

[0168] In another embodiment, it further comprises:

[0169] S6: Recording the idle time of each task pod, and destroying the pod when the idle time continuously exceeds a preset threshold, wherein the preset threshold is dynamically calculated by the system according to the task arrival interval and task execution frequency in the historical workload data.

[0170] The working principle and beneficial effects of the above technical solution are: the automatic scaling module continuously records and monitors the idle time of each task Pod. When the automatic scaling module records the idle time of each task Pod, the Pod is destroyed when the idle time exceeds the threshold. The idle state defined by the system means that the request queue of the Pod is empty and no task is being executed. When the system detects that the idle time of a Pod continuously exceeds the preset threshold, the Pod destruction mechanism is triggered. This preset threshold is not fixed, but is dynamically calculated by the system based on historical workload data. Specifically, the system analyzes the task arrival interval and task execution frequency in the historical workflow to derive a reasonable Pod idle time threshold.

[0171] This dynamic threshold setting enables the system to adapt to the characteristics of different types of workloads. For example, for a workflow with frequent task arrivals, the system sets a longer idle time threshold to avoid the additional overhead of frequently creating and destroying Pods. For a sparse task workflow, the system sets a shorter threshold to promptly reclaim idle resources. This intelligent reduction mechanism ensures efficient use of system resources, avoids waste of resources, and maintains the system's ability to quickly respond to changes in workloads.

[0172] In another embodiment, a system for a Kubernetes-based data flow-driven high-balance serverless workflow scheduling method includes:

[0173] The Gateway request receiving module is configured to receive a JSON format workflow file sent by a user and create related resources.

[0174] The central task scheduling module is configured to deploy task Pods in combination with node resource usage and task resource requests, and select the root task Pod with the shortest request queue length to send a task execution request.

[0175] The sidecar container data synchronization module is configured to receive request data from the outside and put the request into the request queue.

[0176] The sidecar container task scheduling module is configured to obtain and execute task requests from the request queue, and select the optimal subsequent task Pod based on the subsequent task scheduling strategy of data transmission time, request queue length, and task execution time.

[0177] The sidecar container log module is configured to record data transmission time and task execution time.

[0178] The automatic scaling module is configured to evaluate the workflow execution and trigger automatic scaling of task Pods based on an automatic scaling strategy.

[0179] A central database, including a state information database and a log database, is configured to store system state information and execution logs.

[0180] The working principle and beneficial effects of the technical solution are as follows: the Gateway request receiving module is the entry point of the system, responsible for receiving the JSON format workflow file sent by the user. When the user submits a workflow request, the module first performs format verification and integrity check on the JSON file to ensure that the file meets the system-defined workflow specification. After verification, the module parses the workflow file, extracts key data such as workflow structure information, task definition, and task dependency relationship. Subsequently, the module creates relevant resources in the system, including creating a unique identifier for the workflow and registering workflow information in the state database. Finally, the module passes the processed workflow data to the central task scheduling module to prepare for subsequent task deployment and execution.

[0181] After the central task scheduling module receives the workflow data from the Gateway request receiving module, it is responsible for initial task deployment based on node resource usage and task resource requests. The module first obtains the real-time resource status of each node in the cluster, and then analyzes the resource requirements of each task in the workflow. During deployment, the module sorts tasks by resource requirements from high to low, and preferentially deploys tasks with high resource requirements to nodes with the most abundant resources. For each task, the module creates two Pod replicas by default to ensure high availability. After deployment, the module maintains a task state table to record the location, resource usage, and request queue length of each task Pod. When the root task needs to be executed, the central task scheduling module queries the state database to find the Pod with the shortest request queue length among all root task Pods, and sends a task execution request to it. This queue length-based scheduling strategy can balance the load of each Pod, avoiding the situation where some Pods are overloaded while others are idle, and improving overall resource utilization.

[0182] The sidecar container data synchronization module is deployed in each task Pod and is responsible for receiving request data from the outside. When the data synchronization module receives request data, it first performs data integrity verification to ensure that all necessary parameter data has been received. Then, the module reads the required parameter information from the environment variables to determine whether all parameters required by the current request are complete. If the parameters are complete, the module will put the request into the request queue and update the relevant information in the state database. At the same time, the data synchronization module records the reception time of each data packet, which is crucial for subsequent data transmission analysis and task scheduling optimization. The module uses an asynchronous processing mechanism to handle data from multiple data sources simultaneously, ensuring that data is transmitted to the task execution module in a timely and complete manner.

[0183] The sidecar container task scheduling module is responsible for obtaining and executing task requests from the request queue. The module uses a FIFO (First-In-First-Out) strategy to handle tasks in the queue, ensuring that tasks are executed in the order of arrival. When executing tasks, the module calls the corresponding task processing function and monitors the execution progress and resource usage of the task. After the task is completed, the module needs to decide which subsequent task Pod to send the result data to. To do this, the module considers three factors: data transmission time, request queue length, and task execution time. The module obtains the request queue length of each subsequent task Pod from the state database and obtains historical data transmission time and task execution time from the log database. Based on this information, the module calculates the estimated waiting time required to send data to each possible subsequent task Pod and selects the Pod with the shortest waiting time as the target. This comprehensive scheduling strategy balances data locality and load balancing, improving the execution efficiency of the overall workflow.

[0184] The sidecar container log module is responsible for recording key time nodes and performance indicators during task execution. The module records data transmission time, which is the time interval from the start of data transmission to the completion of data reception; records task execution time, which is the time interval from the start of task execution to the completion of execution; and records request queue waiting time, which is the time interval from the entry of the task request into the queue to the start of execution. These detailed time records provide rich performance data for the system, allowing the system to identify potential performance bottlenecks and provide the basis for automatic scaling decisions. Log data is synchronized in real time to the central log database for querying and analysis by other modules. The module uses an efficient log collection and processing mechanism to ensure that the log collection process does not significantly affect task execution.

[0185] The automatic scaling module is responsible for evaluating the workflow execution and triggering the automatic scaling of task pods based on the automatic scaling strategy. The module continuously monitors the performance indicators of each task Pod, including request completion time, queue length, data transmission time, etc. When detecting performance anomalies, the module analyzes the causes of the anomalies and determines whether the problem is caused by excessive queuing time or excessive data transmission time. For cases of excessive queuing time, the module executes the computing resource expansion process, evaluates the parent and child tasks of the current task, and selects the most suitable node for Pod expansion based on node resource conditions and data transmission overhead. For cases of excessive data transmission time, the module executes the data-aware expansion process, analyzes the data transmission between parent and child tasks, compares the resource conditions of related nodes, and selects the optimal node for expansion to optimize the data transmission path. In addition, the module is also responsible for the automatic reduction of task Pods. The module records the idle time of each Pod, and when the idle time exceeds the preset threshold continuously, it triggers the Pod destruction mechanism to recover idle resources. This automatic scaling mechanism enables the system to dynamically adjust resource configuration according to changes in workload, improving resource utilization and system response capability.

[0186] The central database includes two parts: the state information database and the log database. The state information database stores real-time state information of the system, including resource usage of each node, location and request queue length of each task Pod, execution progress of the workflow, etc. These state information provides important basis for task scheduling and resource allocation.

[0187] The log database stores the execution log of the system, including task execution time, data transmission time, request queue waiting time, and other performance indicators. These log data are used for performance analysis, bottleneck identification, and automatic scaling decision. The central database adopts efficient data storage and query mechanism, supports real-time data update and fast data retrieval, and ensures that each functional module can obtain the required information in time.

[0188] As shown in Figure 2 , the overall architecture diagram of the system is as follows:

[0189] User (User): Users send requests to the platform through the Gateway request receiving module. This is the entry point for user interaction with the platform, and user requests will be passed to the master node for processing through the Gateway request receiving module.

[0190] Master Node (Master Node): The master node contains the following core modules:

[0191] Gateway: Receive user requests and pass them to the master node for processing.

[0192] Central task scheduling module: responsible for scheduling and managing the process of root task execution.

[0193] Automatic Scaling Module: Automatically adjusts system resources based on actual task execution conditions such as task completion time, queue time, and data transmission time, optimizing load and task execution efficiency.

[0194] Central Database: Stores system state information, logs, and other data. All task and system running state information is recorded and synchronized for subsequent monitoring and analysis.

[0195] Worker Node: Responsible for actual task execution, containing the following modules:

[0196] Sidecar Container: Each task Pod is equipped with a Sidecar container, which is responsible for data synchronization, task scheduling, and log collection. It ensures the optimization of task execution dependencies and resource allocation.

[0197] Data Synchronization: Responsible for obtaining necessary data from external or other task Pods and ensuring correct data transmission between tasks.

[0198] Task Scheduling: Schedules task Pod execution based on task dependencies, queue length, and system resource conditions.

[0199] Log Collection: Collects log information during task execution and uploads it to the master node's database for storage.

[0200] Task Container (Pod): Executes actual task computation.

[0201] Database: The system contains two databases:

[0202] State Information Database: Stores real-time data such as the current running state of the system, task information, and resource usage.

[0203] Log Database: Records log data during task execution, facilitating system monitoring, error troubleshooting, and performance optimization.

[0204] The specific execution process is shown below:

[0205] Request receiving: When a user submits a workflow request, the first step is to pass it through the Gateway request receiving module. This module receives the user's workflow file, which is in JSON format and contains detailed descriptions of the workflow, including the definition of each task, the dependency relationship between tasks, the required resource types and quantities, and other information. The workflow file is usually generated by the user's application and sent to the Gateway request receiving module through the network. When the Gateway request receiving module receives the request, it will perform preliminary verification, including checking the format of the workflow file, data integrity, etc., to ensure that the file meets the specified structure and content requirements. After verification, the module will pass the workflow file to the central task scheduling module and register the request in the state information database for subsequent query and tracking.

[0206] Request data processing and resource creation: After the central task scheduling module receives the workflow request, it will analyze the resource requirements of each task in the request. Specifically, the scheduling module will parse the task information in the workflow and extract the resource requests for each task (such as CPU, memory, storage, etc.). Then, the scheduling module will combine the resource usage of each node in the cluster to perform initial deployment of the tasks.

[0207] During task deployment, the central task scheduling module will allocate resources according to the following steps: create a Pod for each task and assign an initial number of replicas to each Pod (the default number of replicas is 2 to ensure high availability of the task). According to the remaining resources of the cluster nodes, the resource scheduling strategy will prioritize deploying task Pods on nodes with sufficient resources. The resource scheduling strategy will ensure that the resources of each node are allocated reasonably to avoid excessive congestion of node resources. After the task Pod is created, the central task scheduling module will update the state information database to record the resource status of each Pod, including the resources occupied by the Pod, the current task execution status of the Pod, the request queue length, and other information. This database provides important data support for subsequent task scheduling and resource allocation.

[0208] Task queue management and preparation for execution: As shown in Figure 3 , each task Pod maintains a request queue to store tasks to be executed. The scheduling order of tasks is determined by the tasks in the queue. The central task scheduling module will determine the execution Pod of the root task (i.e., the task that does not depend on other tasks) based on the request queue length of each Pod, node resource conditions, and other information. When all task Pods are deployed, the central task scheduling module will query the root task of the current request workflow from the state information database. The scheduling module will select the task Pod with the shortest request queue and send the task execution request and data to the sidecar container data synchronization module.

[0209] Root task scheduling and task execution: As shown in Figure 2 , in the root task scheduling process, the central task scheduling module determines the most suitable task Pod to execute the root task based on the task Pod's resource request and current node resource usage. The scheduling module selects the Pod with the shortest request queue and the most abundant resources to execute the root task. Once the task Pod to execute the root task is determined, the central task scheduling module sends data and execution requests to the Pod. The request contains all the information required for the execution of the root task, including task parameters, input data, execution instructions, etc. After the task Pod receives the request, the task will begin execution.

[0210] Task scheduling and execution: As shown in Figure 3 , after the task Pod starts, its sidecar container's data synchronization module takes over the task of data transmission and synchronization. The data synchronization module listens and receives data from other tasks or external systems in real time. When synchronization is complete, the data synchronization module places the task request in the sidecar container's request queue.

[0211] As shown in Figure 3 , the task scheduling module is responsible for obtaining tasks from its request queue and executing them. The sidecar container task scheduling module within each task Pod continuously listens and obtains tasks to be executed from the request queue. Once the task execution is complete, the sidecar task scheduling module updates the task execution status and executes the next task.

[0212] Subsequent task scheduling: As shown in Figure 4 , when the current task is executed, the sidecar task scheduling module decides how to schedule the child tasks of the current task according to the following scheduling strategy: Obtain the current shortest queue length of the external task Pod (external task Pod refers to the task Pod in the same node as the current Pod) and the shortest queue length of the internal task Pod (internal task Pod refers to the task Pod in the same node as the current Pod): First, obtain the current request queue length of the subsequent external task Pod from the state data center, and obtain the queue length data of the internal task Pod. Obtain the execution time and data transmission duration of the subsequent task: Then, obtain the execution time and data transmission duration of the task from the log data center. Calculate the expected waiting time of the external task Pod: Through calculation, the expected waiting time of the external task Pod is: queue length × execution time + data transmission duration.

[0213] Compare the waiting time of external and internal task Pods: Compare the estimated waiting time of external task Pods with the waiting time of internal task Pods to decide which type of Pod to schedule the task to. If the estimated waiting time of external task Pods is greater than the waiting time of internal task Pods, the task will be scheduled to internal task Pods. If the estimated waiting time of external task Pods is less than the waiting time of internal task Pods, the task will be scheduled to the Pod with the shortest request queue. This scheduling strategy ensures that the system can respond to changes in the load of external and internal task Pods in real-time during task scheduling, maximizing resource utilization efficiency and optimizing task response time.

[0214] Log collection: The sidecar container log processing module collects and records key information such as task execution time, data transmission time, and request queuing time, and synchronizes it to the log database. These data provide detailed runtime feedback for subsequent task scheduling, helping the system optimize resource allocation and task scheduling strategies.

[0215] Automatic scaling: As shown in Figure 5 , the role of the automatic scaling module is to dynamically adjust the resource configuration of task Pods based on performance bottlenecks that occur during request execution, such as excessive task request completion time, excessive queuing time, and excessive data transmission time, to optimize the overall performance of the system. The specific automatic scaling process is as follows:

[0216] Monitor task request completion time: The automatic scaling module continuously monitors the completion time of each task Pod. If the completion time is too long, it may be due to a performance bottleneck caused by excessive queuing time or excessive data transmission time, and the automatic scaling module will begin further analysis;

[0217] Determine the source of the bottleneck: The system first determines whether the excessive task Pod completion time is caused by excessive queuing time or excessive data transmission time;

[0218] Excessive queuing time: If excessive queuing time is found, it means that the current task has become a performance bottleneck, so the current task Pod must be expanded;

[0219] Obtain the data transmission overhead of parent and child tasks: After determining that the current task needs to be expanded, the system analyzes the data transmission time of parent and child tasks and selects the task with the longest time. The purpose of this operation is to ensure that when expanding the task Pod, data locality can be improved, thereby reducing unnecessary data transmission delay and optimizing the overall performance of the system;

[0220] Judging the resource situation of the external task node and the current task node: the system then compares the resource situation of the external task Pod node and the current task Pod node to ensure load balancing of the system. If the current task node has sufficient resources, the expansion will select the current node; if the current node is insufficient, the expansion will be to the external node, thereby effectively sharing the load;

[0221] Data transmission time is too long: if the data transmission time is too long, it means that there is a large data exchange between tasks, and the current task is not deployed on the node close to the data source. At this time, the system will perform data-aware expansion to optimize the data transmission path and select the node with the most abundant resources for data transmission task expansion.

[0222] Select the expansion node of the task Pod: based on the previous judgment, the automatic scaling module will select the node:

[0223] Expand the external task node: if the node where the data sender is located has relatively abundant resources, the system will select to increase the data receiving task Pod on the data sender node.

[0224] Expand the current task node: if the node where the data receiver is located is relatively abundant, the system will select to expand the data sending task Pod on the node where the data receiver is located.

[0225] This expansion strategy takes into account both data locality and load balancing, optimizing data transmission paths and dynamically expanding task Pods to reduce unnecessary data transmission delays and balance node loads. It ensures that task execution efficiency is improved while maximizing resource utilization and reducing bottleneck problems.

[0226] In another embodiment, an electronic device includes a memory and a processor.

[0227] The working principle and benefits of the above technical solution are as follows: the electronic device includes a memory and a processor, wherein the memory is used to store program instructions, and the processor is responsible for executing these instructions. Specifically, the processor will execute each step of initializing platform components, receiving user requests, analyzing task resource requirements, scheduling task Pods, monitoring task execution, recording logs, and automatically scaling, according to the program instructions in the memory. The program instructions stored in the memory include operation logic for each module, data processing flow, resource management strategy, etc., ensuring that the electronic device can efficiently execute the high-load balancing data-driven workflow scheduling method based on Kubernetes. In this way, the electronic device can dynamically manage and schedule complex workflows, improving the overall performance and resource utilization of the system.

[0228] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application.

Claims

1. A Kubernetes-based data flow driven high-balance serverless workflow scheduling method, characterized in that, Comprise: S1: receiving the JSON format workflow file sent by the user, creating relevant resources; S2: based on the relevant resources, combining the node resource usage and task resource request to deploy task pod, each task pod maintains a request queue; S3: based on the deployed task pod, the root task pod with the shortest request queue length is selected to send a task execution request; S4: based on the execution request, the sidecar container executes the task and selects the optimal subsequent task pod to send the request data based on the subsequent task scheduling strategy of data transmission time, request queue length and task execution time; S5: based on the execution, evaluate the workflow execution, and trigger the automatic scaling of the task pod based on the automatic scaling strategy; S5 step includes: S51: monitor the completion time of the task Pod request; S52: analyze the cause of the delay, judge whether there is a calculation resource bottleneck by comparing the queuing time with the preset threshold, and judge whether there is a data location problem by comparing the data transmission time with the preset threshold; S53: if the queuing time is too long, enter the calculation resource expansion process; S54: if the data transmission time is too long, enter the data sensing expansion process; S53 step includes: S531: evaluate the parent task and child task of the current task; S532: combine the resource status of the node and the data transmission cost, form a weighted score through the inverse relationship of the percentage of the remaining resources of the node to the total resources and the data transmission time, and select the node with the highest score to expand the task Pod; S533: expand the task Pod on the selected node to relieve the load pressure of the set node; S54 step includes: S541: analyze the data transmission between the parent and child tasks, and locate the task with the longest data transmission time; S542: compare the resource status of the current task node and the data source task node, calculate the CPU utilization and memory usage of the two, and select the node with the average value of CPU utilization and memory usage lower than the system preset threshold as the deployment target, wherein the system preset threshold is dynamically set based on historical task execution data; S543: if the data source node is selected, expand the Pod of the current task; if the current task node is selected, expand its parent task.

2. The Kubernetes-based dataflow-driven high-balance serverless workflow scheduling method according to claim 1, wherein, S2 step includes: S21: set the number of replicas of each task Pod to 2; S22: sort the tasks according to their resource requirements in the workflow, from the most to the least; S23: when selecting a node, select the node with the largest remaining computing resources and the highest ratio of remaining computing resources to total computing resources; S24: update the resource usage record of the node, including the allocated CPU core number, the allocated memory capacity and the remaining available resource amount, to provide resource data for the scheduling of subsequent tasks. 3.The Kubernetes-based dataflow-driven high-balance serverless workflow scheduling method of claim 1, wherein, S4 step includes: S41: the sidecar container data synchronization module receives the request data sent by the outside, and puts the request into the request queue; S42: the sidecar container log module records the data transmission time; S43: the sidecar container task scheduling module obtains and executes the task request from the request queue; S44: the sidecar container log module records and synchronizes the task execution time; S45: after the task execution is completed, the request data is sent by the subsequent task pod with the shortest expected waiting time according to the expected waiting time calculated based on the data transmission time and the request queue length.

4. The Kubernetes-based dataflow-driven high-balance serverless workflow scheduling method of claim 1, wherein, S45 step includes: S451: obtain the request queue length of the external task pod and the internal task pod from the state database, and determine the pod with the minimum queue length value in the external and internal pods respectively; S452: obtain the average execution time of the historical task and the data transmission time of the current task from the log database; S453: calculate the expected waiting time of the task scheduled to the external pod, which is equal to the queue length multiplied by the average execution time and then added to the data transmission time; S454: compare the external expected waiting time with the internal pod expected waiting time, and select the pod with shorter expected waiting time for task scheduling. 5.The Kubernetes-based dataflow-driven high-balance serverless workflow scheduling method of claim 1, wherein, Also includes: S6: record the idle time of each task pod, and when the idle time continuously exceeds the preset threshold, destroy the pod, wherein the preset threshold is dynamically calculated by the system according to the task arrival interval and the task execution frequency in the historical data of the workload.

6. A system for a kubernetes-based dataflow-driven high-balance serverless workflow scheduling method according to any one of claims 1-5, characterized in that, Includes: Gateway request receiving module, used for receiving the JSON format workflow file sent by the user and creating related resources; Central task scheduling module, used for deploying task pods in combination with node resource usage and task resource requests, and selecting the root task pod with the shortest request queue length to send task execution request; sidecar container data synchronization module, used for receiving the request data sent by the external and putting the request into the request queue; sidecar container task scheduling module, used for obtaining and executing the task request from the request queue, and selecting the optimal subsequent task pod based on the subsequent task scheduling strategy of data transmission time, request queue length and task execution time; sidecar container log module, used for recording the data transmission time and the task execution time; Automatic scaling module, used for evaluating the workflow execution and triggering the automatic scaling of the task pod based on the automatic scaling strategy; Central database, including state information database and log database, used for storing system state information and execution log.

7. An electronic device, comprising a memory and a processor, characterized in that: The processor runs the program instructions stored in the memory to execute the method of any one of claims 1 to 5.

Images