Automatic scaling flow resource scheduling method and device

Abstract

The invention provides an automatic scaling flow resource scheduling method and device, and relates to the technical field of flow application data processing. The method comprises the following steps: running a stream application program based on a preset scheduling strategy; in the running process of a stream application program, a data detector is used for collecting information, and stream application running data is obtained; based on a preset calculation node threshold value, rescheduling verification is carried out according to the stream application operation data, and a verification result is obtained; when the verification result is that rescheduling is needed, sub-graph division is carried out on the stream application program graph by taking total cut edge weight minimization and sub-graph internal total weight equalization as targets, and stream application program sub-graphs are obtained; based on the overload threshold value and the underload threshold value, scheduling resource scaling is carried out on the streaming application program sub-graph, and an optimized program sub-graph is obtained; and resource scheduling is carried out according to the optimization program sub-graph, and an optimization resource scheduling scheme is obtained. The invention relates to an intelligent and automatic scaling high-efficiency stream application resource scheduling method.

Description

Technical Field

[0001] The present invention relates to the technical field of stream application data processing, and in particular to an automatic scaling stream resource scheduling method and device. Background Art

[0002] Resource scalability is a core advantage of stream computing systems. As the data flow rate changes dynamically, resource utilization fluctuates significantly, resulting in frequent resource surplus or scarcity issues in fluctuating data flow scenarios. Although current research has begun to address the challenge of runtime resource shortage, it is still unable to cope with variable data flows and lacks a comprehensive and effective solution.

[0003] In actual application scenarios, data flow rates are often non-uniform and fluctuate over time, and are also affected by a variety of external factors. Faced with the dynamic changes in data flow rates, fixed scheduling strategies often find it difficult to effectively cope with fluctuations in resource demand. When the data flow rate surges, the load on the computing nodes will rise sharply, exceeding their processing capacity, which will not only significantly increase latency, but may also cause the risk of system crashes. When the data flow rate remains low, the computing resources that have been dynamically allocated to tasks cannot be effectively released, resulting in a large amount of computing resources being idle and wasted.

[0004] In the existing scheduling strategies, the adverse effects of data flow fluctuations have been successfully alleviated. Resource allocation strategies based on greedy algorithms and genetic algorithms are all aimed at improving the efficiency and effectiveness of stream application execution. In addition, a heuristic algorithm ATSOOH can flexibly reconfigure stream applications based on the real-time fluctuations of data flows and the current availability of computing resources to achieve dynamic optimization of resources. However, when resource utilization is no longer the main constraint on system performance, it becomes particularly important to explore optimization methods in other dimensions.

[0005] Abstracting streaming applications into directed acyclic graphs and making full use of the dependencies between tasks opens up an effective way to further optimize system performance. For the critical path of the directed acyclic graph, a dynamic programming algorithm is used for scheduling decisions. Its core purpose is to effectively reduce communication overhead and improve overall system efficiency. Based on the task division at the subgraph level and the communication volume between schedules, the directed acyclic graph is divided into multiple subgraphs to achieve more sophisticated resource management and task scheduling. However, the task deployment strategy within the computing node has a direct impact on the communication overhead; the communication overhead between tasks within a process is significantly different from the communication overhead between tasks between processes. This difference constitutes a key factor that cannot be ignored in the task scheduling process, which is directly related to the efficient use of system resources and the optimization of overall performance. Therefore, when designing a scheduling strategy, the specific method of task deployment must be carefully considered.

[0006] In the prior art, there is a lack of an efficient streaming application resource scheduling method with intelligent automatic scaling. Summary of the invention

[0007] In order to solve the technical problem in the prior art that resource utilization fluctuations lead to frequent resource surplus or scarcity in fluctuating data flow scenarios, an embodiment of the present invention provides an automatic scaling stream resource scheduling method and device. The technical solution is as follows:

[0008] On the one hand, a method for automatically scaling stream resource scheduling is provided, the method being implemented by a stream resource scheduling device, the method comprising:

[0009] Based on a preset scheduling strategy, run a stream application; during the running of the stream application, use a data detector to collect information to obtain stream application running data; save the stream application running data to a database; and construct a stream application graph according to the database;

[0010] Based on a preset computing node threshold, rescheduling verification is performed according to the stream application running data to obtain a verification result;

[0011] When the verification result indicates that rescheduling is required, the subgraph partitioning algorithm is used to partition the stream application graph, with the goal of minimizing the total cut edge weight and balancing the total weight within the subgraph, to obtain a stream application subgraph;

[0012] Based on the overload threshold and the underload threshold, scheduling resource scaling is performed on the stream application subgraph to obtain an optimized program subgraph;

[0013] Based on the stream application running data, resource scheduling is performed according to the optimization program subgraph to obtain an optimized resource scheduling solution.

[0014] On the other hand, a device for automatically scaling stream resource scheduling is provided, which is applied to a method for automatically scaling stream resource scheduling, and the device includes:

[0015] A data acquisition module is used to run a stream application based on a preset scheduling strategy; during the running of the stream application, use a data detector to collect information and obtain stream application running data; save the stream application running data to a database; and construct a stream application graph based on the database;

[0016] A rescheduling verification module is used to perform a rescheduling verification based on a preset computing node threshold and the stream application running data to obtain a verification result;

[0017] A subgraph partitioning module is used for, when the verification result indicates that rescheduling is required, minimizing the total cut edge weight and balancing the total weight within the subgraph, and using a subgraph partitioning algorithm to partition the subgraph according to the stream application graph to obtain a stream application subgraph;

[0018] A resource scaling module, configured to perform scheduling resource scaling on the stream application subgraph based on an overload threshold and an underload threshold to obtain an optimized program subgraph;

[0019] The resource scheduling module is used to perform resource scheduling based on the stream application running data and the optimization program subgraph to obtain an optimized resource scheduling solution.

[0020] On the other hand, a stream resource scheduling device is provided, comprising: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned automatic scaling stream resource scheduling methods is implemented.

[0021] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned automatic scaling flow resource scheduling methods.

[0022] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0023] The present invention proposes an automatic scaling stream resource scheduling method, which achieves balanced minimization of communication overhead through a heuristic subgraph partitioning algorithm, and designs a matching resource scaling algorithm to flexibly respond to data flow fluctuations. Based on the task scheduling algorithm, through the thread-level task deployment strategy, the communication overhead is significantly reduced while effectively controlling the computing resource utilization. The present invention achieves excellent performance in a fluctuating data flow environment, ensuring the dual improvement of efficient resource utilization and low-latency processing capabilities. The present invention is an intelligent automatic scaling and efficient stream application resource scheduling method. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 This is a flow chart of an automatic scaling stream resource scheduling method provided by an embodiment of the present invention;

[0026] Figure 2It is a block diagram of an automatic scaling stream resource scheduling device provided by an embodiment of the present invention;

[0027] Figure 3 It is a structural diagram of a flow resource scheduling device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0029] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0030] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0031] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0032] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0033] The embodiment of the present invention provides an automatic scaling stream resource scheduling method, which can be implemented by a stream resource scheduling device, which can be a terminal or a server. Figure 1 The process flow of the method for automatically scaling stream resource scheduling shown in the flowchart may include the following steps:

[0034] S1. Run the stream application based on the preset scheduling strategy; during the running of the stream application, use the data detector to collect information and obtain the stream application running data; save the stream application running data to the database; and build a stream application graph based on the database.

[0035] In a feasible implementation, the present invention proposes an intelligent fine-grained stream application scheduling method and automatic scaling system Ra-Stream. Ra-Stream can intelligently expand or shrink computing resources according to the real-time situation of the current data flow rate, and achieve flexible adaptation to fluctuating data flows.

[0036] The streaming application running data includes the communication volume between tasks in the streaming application, the resource usage of computing nodes in the computing cluster, the resource requirements for executing tasks, and the current data flow rate.

[0037] In a feasible implementation, before rescheduling, the default scheduling strategy of the Storm framework is used to schedule the stream application to the system. A data detector is used to continuously collect information, and the collected information is saved in a database.

[0038] S2. Based on the preset computing node threshold, rescheduling verification is performed according to the stream application running data to obtain the verification result.

[0039] In a feasible implementation, it is determined whether rescheduling needs to be triggered based on the resource usage of computing nodes in the computing cluster in the stream application running data to adapt to the current data flow rate. The condition for triggering rescheduling is that the resource utilization of at least one computing node exceeds the overloaded computing node threshold set by the user.

[0040] When the verification result shows that rescheduling is required, proceed to the subsequent steps; when the verification result shows that rescheduling is not required, continue to collect stream application running data.

[0041] S3. When the verification result shows that rescheduling is required, the subgraph partitioning algorithm is used to partition the stream application graph, with the goal of minimizing the total edge weight and balancing the total weight within the subgraph, to obtain the stream application subgraph.

[0042] Optionally, when the verification result indicates that rescheduling is required, the subgraph partitioning algorithm is used to partition the stream application graph, with the goal of minimizing the total cut edge weight and balancing the total weight within the subgraph, to obtain the stream application subgraph, including:

[0043] When the verification result indicates that rescheduling is required, the partition cost ratio is constructed with the goal of minimizing the total cut edge weight;

[0044] The internal weight variance of the subgraph is constructed with the goal of balancing the total weight within the subgraph;

[0045] The objective function is constructed by weighted summation based on the partition cost ratio and the weight variance within the subgraph;

[0046] Based on the objective function, the subgraph is partitioned according to the stream application graph through a subgraph partitioning algorithm to obtain the stream application subgraph.

[0047] In a feasible implementation, the subgraph partitioning problem in the present invention can be summarized as submitting a stream application G = {V(G), E(G)} to a computing cluster consisting of n available computing nodes cn. When the stream application G needs to run on k computing nodes that do not exceed the total number of cluster nodes n, it is necessary to accurately partition G into k subgraphs G sub The core goal of subgraph partitioning is to minimize the total edge weight W connecting different subgraphs. cut (G) and maximize the weights belonging to the same subgraph. However, if we only pursue minimization of the total cut edge weight and ignore other factors, the partitioning scheme may be unbalanced in task allocation. For example, in an unbalanced subgraph partition, one subgraph may contain only two tasks, while the other may carry eight tasks.

[0048] In order to optimize the partitioning scheme and achieve a more balanced subgraph distribution, the present invention aims to minimize W while almost equally distributing the internal weights of each subgraph. cut (G). To this end, the present invention adopts W cut The ratio r of (G) to the overall total weight, i.e., the partition cost ratio, is used as a reference indicator to achieve the goal of minimizing the sum of the cut edge weights of the subgraph partition. Among them, the overall total weight is W cut (G) and the total internal weight W of all subgraphs in (G sub ). r is calculated as follows:

[0049]

[0050] In addition, in order to achieve a relatively balanced weight within each subgraph, the variance of the weight within each subgraph is used w As a reference indicator, σ W The calculation method of is as follows (2):

[0051]

[0052] in, is the average value of the internal weights of all subgraphs.

[0053] To minimize W cut (G) and equalization W in (G sub ), the objective function f(x) is used as the reference indicator in the subgraph partitioning process, and the calculation method of f(x) is as follows:

[0054] f(x)=α·r+(1-α)·σ w (3);

[0055] Among them, α is a comprehensive weighting factor defined by the user and 0<α<1, which combines r and σ w These two key indicators. By intervening in the subgraph partitioning process using the objective function, Ra-Stream can partition a streaming application G into multiple subgraphs.

[0056] Among them, the subgraph partitioning algorithm is constructed based on the simulated annealing algorithm; the subgraph partitioning algorithm is used to minimize the communication volume between subgraphs of streaming applications and reduce the communication overhead between computing nodes.

[0057] In a feasible implementation, the core pursuit of subgraph partitioning in the present invention is to accurately cut the streaming application into multiple subgraphs that are both balanced and highly communication-intensive. Driven by this goal, a new subgraph partitioning algorithm is proposed based on the global optimization capability of the simulated annealing algorithm. The essence of subgraph partitioning is that it aims to achieve two core goals simultaneously: one is to reduce the communication traffic between subgraphs as much as possible to reduce the communication overhead between computing nodes; the other is to maximize the balance and increase the communication volume within each subgraph to avoid unbalanced subgraph partitioning results.

[0058] The communication-intensive and balanced subgraph partitioning algorithm is as follows. In the subgraph partitioning algorithm, the input is the flow application graph, the number of subgraphs to be partitioned (initially the maximum number of computing nodes in the cluster, and then intelligently determined based on resource usage), the amount of communication between tasks in the flow application, the maximum number of iterations, the initial temperature, the final temperature, and the cooling rate. Among them, the maximum number of iterations, the initial temperature, the final temperature, and the cooling rate can be flexibly set according to the complexity of the subgraph to ensure that the result of the subgraph partitioning converges to the global optimal solution, that is, the communication-intensive and balanced subgraph partitioning result.

[0059]

[0060] S4. Based on the overload threshold and the underload threshold, the streaming application subgraph is scheduled and scaled to obtain an optimized program subgraph.

[0061] Optionally, scheduling resource scaling is performed on the stream application subgraph based on the overload threshold and the underload threshold to obtain an optimized program subgraph, including:

[0062] Calculate the total resource requirements based on the stream application subgraph to obtain the resource requirements of each subgraph task;

[0063] According to the stream application subgraph, map through the computing nodes to obtain the mapping resource requirements of each subgraph

[0064] Calculate the resource utilization of each computing node corresponding to each subgraph based on the resource requirements of each subgraph task and the resource requirements of each subgraph mapping;

[0065] Based on the overload threshold and the resource utilization rate of each computing node, the overloaded subgraph in the stream application subgraph is expanded to obtain the enlarged program subgraph;

[0066] Based on the underload threshold and the resource utilization rate of each computing node, the idle subgraphs in the stream application subgraph are shrunk to obtain the shrunk program subgraph.

[0067] An optimized program subgraph is obtained according to the enlarged program subgraph and the shrunk program subgraph.

[0068] In a feasible implementation, in actual application scenarios, computing resources are not in unlimited supply, so resource scaling must be carefully considered in the scheduling process. The resource scaling mechanism flexibly configures computing resources based on the current data flow rate, aiming to prevent performance degradation caused by resource scarcity and minimize resource waste. Before deploying streaming applications to a computing cluster, the key is to ensure that each computing node is not overloaded and to reduce idle and wasted resources within these nodes.

[0069] Therefore, after the subgraph division phase is completed, the specific resource requirements of each subgraph need to be considered to ensure that resource overload is avoided during the computing node scheduling process. By comprehensively summarizing the resource requirements of all tasks in the subgraph, the total resource requirements of each subgraph can be accurately obtained. The total resource requirements of each subgraph are calculated as follows (4):

[0070]

[0071] in, and Sub-graphs The CPU resource requirements and memory resource requirements at time t, for The task set, and They are Medium Task v ik CPU resource requirements and memory resource requirements at time t.

[0072] Calculated and After that, each subgraph is mapped to a computing node. The available CPU resources and memory resources of the computing node mapped by the subgraph must be greater than the CPU resource requirements and memory resource requirements of the subgraph. Resource scaling covers two major components: resource contraction and resource expansion. As the data flow rate gradually decreases, the computing resources required to run the streaming application also decrease accordingly, so that the system performance can be achieved with fewer computing nodes. Conversely, when the data flow rate shows an upward trend, the demand for computing resources by the streaming application also increases. At this time, more computing nodes need to be deployed to ensure that the system performance is maintained at the optimal state.

[0073] In order to ensure efficient resource utilization and prevent node overload, the overload threshold Tover and underload threshold Tunder are introduced. By limiting the thresholds, Ra-Stream can ensure that after the subgraph is mapped to the computing node, the resource utilization of the computing node is between the two thresholds, ensuring that the system can intelligently adapt to fluctuating data flows. under With overload threshold T over It can be customized according to actual needs. under When the resource utilization of the node exceeds T, it means that the resource utilization of the node is insufficient, and there may be idle or wasted resources. On the contrary, if the resource utilization of the node exceeds T over , it indicates that the node is overloaded with tasks, which may have an adverse effect on the overall performance of the system. The resource scaling algorithm used in the present invention is as follows.

[0074]

[0075] S5. Based on the stream application running data, resource scheduling is performed according to the optimization program subgraph to obtain an optimized resource scheduling solution.

[0076] Optionally, based on the stream application running data, resource scheduling is performed according to the optimization program subgraph to obtain an optimized resource scheduling solution, including:

[0077] Based on the decision factor and fitness factor, according to the running data of the stream application, the coarse-grained resource scheduling optimization is performed on the optimization program subgraph to obtain a preliminary optimized resource scheduling solution;

[0078] Based on the communication density and the running data of the streaming application, the preliminary optimized resource scheduling scheme is optimized in fine-grained manner to obtain the optimized resource scheduling scheme.

[0079] In a feasible implementation, after the stream application is divided and optimized into optimized program subgraphs, the task scheduling problem is transformed into: how to reasonably allocate these subgraphs to the existing available computing nodes. A computing node has the ability to run multiple subgraphs at the same time, but each subgraph can only be uniquely assigned to one computing node. Scheduling communication-intensive subgraphs to computing nodes can effectively convert inter-node communication into intra-node communication, significantly reducing communication overhead. However, within the same subgraph, the communication traffic between tasks is not evenly distributed, which leads to the existence of communication-intensive task pairs. In order to meet this challenge, the present invention designs a scheduling algorithm consisting of two parts: subgraph level scheduling (coarse-grained) and thread level scheduling (fine-grained).

[0080] Coarse-grained scheduling refers to the reasonable allocation of the divided subgraphs to each computing node. During the allocation process, the resource status of the computing nodes and the resource requirements of the subgraphs are comprehensively considered to achieve efficient resource utilization. When a subgraph needs to find a suitable computing node to meet its resource requirements, it will face multiple candidate nodes. Among these options, Ra-Stream will find the computing node with the highest fitness factor as the optimal solution.

[0081] In order to clearly represent the mapping relationship between subgraphs and computing nodes, the present invention introduces the decision factor The mathematical expression of is as follows (5):

[0082]

[0083] in, and They are computing nodes cn i The CPU resources and memory resources available at time t.

[0084] Given that there may be multiple computing nodes that can satisfy the subgraph The resource requirements of this invention introduce the fitness factor To evaluate the scheduling To calculate the fitness of the node. Based on this evaluation criterion, Will be dispatched The compute node with the largest value. The mathematical expression of is as follows (6):

[0085]

[0086] in, for Dispatch to cn i After, cn i The remaining amount of resources. The calculation process is as follows (7):

[0087]

[0088] Among them, β is the weight of combining CPU resources and memory resources and 0<β<1.

[0089] Fine-grained scheduling refers to scheduling tasks within a node. The key to this step is to identify the communication-intensive task pairs in the subgraph and assign them to the same process to minimize the communication overhead between tasks. The task scheduling algorithm analyzes the communication pattern between tasks in the subgraph and finds those task pairs with large communication volume. When assigning tasks to processes, these communication-intensive task pairs will be placed in the same process first. Since the communication volume between these task pairs is large, putting them in the same process can significantly reduce the overhead of inter-process communication and improve the overall performance of the system. The task scheduling algorithm is shown below.

[0090] In summary, the scheduling from coarse-grained to fine-grained not only realizes the reasonable allocation of subgraphs on computing nodes, but also optimizes the deployment of tasks within the nodes, effectively reducing communication overhead and improving the overall performance of the system.

[0091]

[0092] In a feasible implementation, the Ra-Stream system is integrated on the Apache Storm platform using the method proposed in the present invention, and a comprehensive and in-depth evaluation of key performance indicators such as system latency, maximum throughput, and resource utilization is performed in a real-world data stream scenario with fluctuating characteristics. Ra-Stream can reduce system latency by about 47.45%, increase maximum throughput by about 60.55%, and save about 46.25% of resource utilization.

[0093] The present invention proposes an automatic scaling stream resource scheduling method, which achieves balanced minimization of communication overhead through a heuristic subgraph partitioning algorithm, and designs a matching resource scaling algorithm to flexibly respond to data flow fluctuations. Based on the task scheduling algorithm, through the thread-level task deployment strategy, the communication overhead is significantly reduced while effectively controlling the computing resource utilization. The present invention achieves excellent performance in a fluctuating data flow environment, ensuring the dual improvement of efficient resource utilization and low-latency processing capabilities. The present invention is an intelligent automatic scaling and efficient stream application resource scheduling method.

[0094] Figure 2 1 is a block diagram of an automatic scaling stream resource scheduling device according to an exemplary embodiment, wherein the device is used in an automatic scaling stream resource scheduling method. Figure 2The device includes a data acquisition module 210, a rescheduling verification module 220, a subgraph partitioning module 230, a resource scaling module 240 and a resource scheduling module 250. Among them:

[0095] The data acquisition module 210 is used to run the stream application based on the preset scheduling strategy; during the running of the stream application, use the data detector to collect information and obtain the stream application running data; save the stream application running data to the database; and construct the stream application graph according to the database;

[0096] The rescheduling verification module 220 is used to perform rescheduling verification based on the preset computing node threshold and the stream application running data to obtain a verification result;

[0097] The subgraph partitioning module 230 is used to partition the subgraph using a subgraph partitioning algorithm according to the stream application graph to obtain a stream application subgraph, with the goal of minimizing the total cut edge weight and balancing the total weight within the subgraph when the verification result indicates that rescheduling is required;

[0098] A resource scaling module 240 is used to perform scheduling resource scaling on the stream application subgraph based on an overload threshold and an underload threshold to obtain an optimized program subgraph;

[0099] The resource scheduling module 250 is used to perform resource scheduling based on the stream application running data and the optimized program subgraph to obtain an optimized resource scheduling solution.

[0100] The streaming application running data includes the communication volume between tasks in the streaming application, the resource usage of computing nodes in the computing cluster, the resource requirements for executing tasks, and the current data flow rate.

[0101] Optionally, the subgraph partitioning module 230 is further configured to:

[0102] When the verification result indicates that rescheduling is required, the partition cost ratio is constructed with the goal of minimizing the total cut edge weight;

[0103] The internal weight variance of the subgraph is constructed with the goal of balancing the total weight within the subgraph;

[0104] The objective function is constructed by weighted summation based on the partition cost ratio and the weight variance within the subgraph;

[0105] Based on the objective function, the subgraph is partitioned according to the stream application graph through a subgraph partitioning algorithm to obtain the stream application subgraph.

[0106] Among them, the subgraph partitioning algorithm is constructed based on the simulated annealing algorithm; the subgraph partitioning algorithm is used to minimize the communication volume between subgraphs of streaming applications and reduce the communication overhead between computing nodes.

[0107] Optionally, the resource scaling module 240 is further configured to:

[0108] Calculate the total resource requirements based on the stream application subgraph to obtain the resource requirements of each subgraph task;

[0109] According to the stream application subgraph, map through the computing nodes to obtain the mapping resource requirements of each subgraph

[0110] Calculate the resource utilization of each computing node corresponding to each subgraph based on the resource requirements of each subgraph task and the resource requirements of each subgraph mapping;

[0111] Based on the overload threshold and the resource utilization rate of each computing node, the overloaded subgraph in the stream application subgraph is expanded to obtain the enlarged program subgraph;

[0112] Based on the underload threshold and the resource utilization rate of each computing node, the idle subgraphs in the stream application subgraph are shrunk to obtain the shrunk program subgraph.

[0113] An optimized program subgraph is obtained according to the enlarged program subgraph and the shrunk program subgraph.

[0114] Optionally, the resource scheduling module 250 is further configured to:

[0115] Based on the decision factor and fitness factor, according to the running data of the stream application, the coarse-grained resource scheduling optimization is performed on the optimization program subgraph to obtain a preliminary optimized resource scheduling solution;

[0116] Based on the communication density and the running data of the streaming application, the preliminary optimized resource scheduling scheme is optimized in fine-grained resource scheduling to obtain the optimized resource scheduling scheme.

[0117] The present invention proposes an automatic scaling stream resource scheduling method, which achieves balanced minimization of communication overhead through a heuristic subgraph partitioning algorithm, and designs a matching resource scaling algorithm to flexibly respond to data flow fluctuations. Based on the task scheduling algorithm, through the thread-level task deployment strategy, the communication overhead is significantly reduced while effectively controlling the computing resource utilization. The present invention achieves excellent performance in a fluctuating data flow environment, ensuring the dual improvement of efficient resource utilization and low-latency processing capabilities. The present invention is an intelligent automatic scaling and efficient stream application resource scheduling method.

[0118] Figure 3 is a structural diagram of a flow resource scheduling device provided by an embodiment of the present invention, such as Figure 3 As shown, the stream resource scheduling device may include the above Figure 2 Optionally, the stream resource scheduling device 310 may include a first processor 2001 .

[0119] Optionally, the flow resource scheduling device 310 may further include a memory 2002 and a transceiver 2003 .

[0120] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0121] Combine the following Figure 3 The components of the stream resource scheduling device 310 are introduced in detail:

[0122] The first processor 2001 is the control center of the stream resource scheduling device 310, and may be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or may be application specific integrated circuits (ASICs), or may be configured to implement one or more integrated circuits of the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0123] Optionally, the first processor 2001 may execute various functions of the stream resource scheduling device 310 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0124] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 3 CPU0 and CPU1 are shown in FIG.

[0125] In a specific implementation, as an embodiment, the flow resource scheduling device 310 may also include multiple processors, such as Figure 3 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0126] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0127] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Figure 3 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0128] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0129] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 3 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0130] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently and communicate with the first processor 2001 through the interface circuit ( Figure 3 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0131] It should be noted that Figure 3 The structure of the flow resource scheduling device 310 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than those shown in the figure, or combine certain components, or arrange the components differently.

[0132] In addition, the technical effects of the stream resource scheduling device 310 can refer to the technical effects of the automatic scaling stream resource scheduling method described in the above method embodiment, which will not be repeated here.

[0133] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0134] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0135] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0136] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0137] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0138] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0139] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0140] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0141] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0142] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0143] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0144] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., various media that can store program codes.

[0145] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An automatic scaling stream resource scheduling method, characterized in that: The method comprises: Based on a preset scheduling strategy, run a stream application; during the running of the stream application, use a data detector to collect information to obtain stream application running data; save the stream application running data to a database; and construct a stream application graph according to the database; Based on a preset computing node threshold, rescheduling verification is performed according to the stream application running data to obtain a verification result; When the verification result indicates that rescheduling is required, the subgraph partitioning algorithm is used to partition the stream application graph, with the goal of minimizing the total cut edge weight and balancing the total weight within the subgraph, to obtain a stream application subgraph; Based on the overload threshold and the underload threshold, scheduling resource scaling is performed on the stream application subgraph to obtain an optimized program subgraph; Based on the stream application running data, resource scheduling is performed according to the optimization program subgraph to obtain an optimized resource scheduling solution.

2. The method for automatically scaling stream resource scheduling according to claim 1, characterized in that: The stream application running data includes the communication volume between tasks in the stream application program, the resource usage of computing nodes in the computing cluster, the resource requirements for executing tasks and the current data flow rate.

3. The method for automatically scaling stream resource scheduling according to claim 1, characterized in that: When the verification result indicates that rescheduling is required, minimizing the total cut edge weight and balancing the total weight within the subgraph are taken as the goals, and according to the stream application graph, a subgraph partitioning algorithm is used to perform subgraph partitioning to obtain a stream application subgraph, including: When the verification result indicates that rescheduling is required, a partitioning cost ratio is constructed with the goal of minimizing the total cut edge weight; The internal weight variance of the subgraph is constructed with the goal of balancing the total weight within the subgraph; Constructing an objective function by weighted summation according to the partition cost ratio and the weight variance within the subgraph; Based on the objective function and according to the stream application graph, subgraph partitioning is performed by a subgraph partitioning algorithm to obtain a stream application subgraph.

4. The method for automatically scaling stream resource scheduling according to claim 3, characterized in that: The subgraph partitioning algorithm is constructed based on the simulated annealing algorithm; the subgraph partitioning algorithm is used to minimize the communication volume between stream application subgraphs and reduce the communication overhead between computing nodes.

5. The method for automatically scaling stream resource scheduling according to claim 1, characterized in that: The step of performing scheduling resource scaling on the stream application subgraph based on the overload threshold and the underload threshold to obtain an optimized program subgraph includes: Calculate the total resource requirement according to the stream application subgraph to obtain the resource requirement of each subgraph task; According to the stream application subgraph, mapping is performed through computing nodes to obtain the mapping resource requirements of each subgraph Calculate the resource utilization of each computing node corresponding to each subgraph according to the resource requirements of each subgraph task and the resource requirements of each subgraph mapping; Based on the overload threshold, and according to the resource utilization rate of each computing node, resources are expanded on the overloaded subgraph in the stream application subgraph to obtain an enlarged program subgraph; Based on the underload threshold, and according to the resource utilization rate of each computing node, shrinking the resources of the idle subgraphs in the stream application subgraph to obtain a shrunken program subgraph; An optimized program subgraph is obtained according to the enlarged program subgraph and the shrunk program subgraph.

6. The method for automatically scaling stream resource scheduling according to claim 1, characterized in that: The step of performing resource scheduling based on the stream application running data and according to the optimization program subgraph to obtain an optimized resource scheduling solution includes: Based on the decision factor and the fitness factor, and according to the stream application running data, the optimization program subgraph is optimized in a coarse-grained resource scheduling manner to obtain a preliminary optimized resource scheduling solution; Based on the communication density and according to the flow application running data, the preliminary optimized resource scheduling scheme is optimized in fine-grained resource scheduling to obtain an optimized resource scheduling scheme.

7. An automatic scaling stream resource scheduling device, the automatic scaling stream resource scheduling device is used to implement the automatic scaling stream resource scheduling method according to any one of claims 1 to 6, characterized in that: The device comprises: A data acquisition module is used to run a stream application based on a preset scheduling strategy; during the running of the stream application, use a data detector to collect information and obtain stream application running data; save the stream application running data to a database; and construct a stream application graph based on the database; A rescheduling verification module is used to perform a rescheduling verification based on a preset computing node threshold and the stream application running data to obtain a verification result; A subgraph partitioning module is used for, when the verification result indicates that rescheduling is required, minimizing the total cut edge weight and balancing the total weight within the subgraph, and performing subgraph partitioning according to the stream application graph using a subgraph partitioning algorithm to obtain a stream application subgraph; A resource scaling module, configured to perform scheduling resource scaling on the stream application subgraph based on an overload threshold and an underload threshold to obtain an optimized program subgraph; The resource scheduling module is used to perform resource scheduling based on the stream application running data and the optimization program subgraph to obtain an optimized resource scheduling solution.

8. The method for automatically scaling stream resource scheduling according to claim 7, characterized in that: The resource scheduling module is further used to: Based on the decision factor and the fitness factor, and according to the stream application running data, the optimization program subgraph is optimized in a coarse-grained resource scheduling manner to obtain a preliminary optimized resource scheduling solution; Based on the communication density and according to the flow application running data, the preliminary optimized resource scheduling scheme is optimized in fine-grained resource scheduling to obtain an optimized resource scheduling scheme.

9. A stream resource scheduling device, characterized in that: The stream resource scheduling device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 6.

Images