Dynamic load-oriented data processing task resource optimization method, device and system

Abstract

The invention discloses a dynamic load-oriented data processing task resource optimization method, device and system, and the method comprises the steps: constructing an optimization problem taking the total resource overhead of a system as an optimization target through continuously obtaining network time delay, the flow scale of a data processing task, the distribution of the data processing task in a network topology, and the data size of each client, and carrying out the optimization of the data processing task. Routing paths, aggregation positions and computing power resource allocation are continuously adjusted by solving corresponding optimization problems. According to the method, based on a dynamic load environment, resource configuration of a data processing task is optimized by adopting an intra-network aggregation technology, communication delay is reduced, and the overall performance of a system is improved. According to the dynamic load resource optimization system, the flexibility of the programmable switch is utilized, resources can be dynamically allocated and network flows can be adjusted according to changes of network conditions and data flows, and the utilization efficiency of network resources and the performance of data processing tasks are greatly improved.

Description

Technical Field

[0001] The present invention relates to the fields of data center network and edge computing technology, and specifically to a method, device and system for optimizing data processing task resources for dynamic loads. Background Art

[0002] Federated learning is a special data processing task that processes data in parallel on multiple terminal devices without centralizing or sharing the original data. This approach is particularly useful when processing sensitive data, such as in medical, financial, and personal device applications. However, a key feature of the federated learning environment is its dynamic nature: the number of participating devices, data distribution of devices, network conditions, and computing resources may change over time and location.

[0003] In the application of federated learning, since a large amount of data is transmitted across the network, how to effectively manage and reduce communication costs has become an important issue. The present invention proposes to optimize data flow through in-network aggregation: in the process of data from the source to the server, by pre-aggregating data at certain nodes of the network, the amount of data to be transmitted is reduced and the transmission efficiency is improved.

[0004] However, since the state of the network will continue to change over time, dynamic strategies are needed to cope with dynamic loads. Specifically, the choice of routing and data aggregation locations must be able to flexibly respond to changes in network conditions to ensure the efficiency of data processing and transmission. In addition, the reasonable allocation of client computing resources based on these dynamic changes is the key to ensuring the efficiency and effectiveness of federated learning. This requires the system to have not only efficient real-time monitoring capabilities, but also the ability to respond and adjust quickly to adapt to the continuous changes in the network environment and device status. Summary of the invention

[0005] The purpose of the present invention is to provide a method, device and system for optimizing data processing task resources under dynamic loads, which uses network aggregation technology to optimize the resource allocation of data processing tasks in a dynamic load environment, reduce communication delays and improve overall system performance.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] The method for optimizing data processing task resources for dynamic loads comprises the following steps:

[0008] Continuously monitor network latency, the traffic size of data processing tasks, their distribution in the network topology, and the amount of data at each node;

[0009] Based on the detected network delay, the traffic scale of the data processing task and its distribution in the network topology, and the amount of data at each node, a total system resource cost optimization target is constructed and constraints of the total system resource cost optimization target are determined; the total system resource cost optimization target includes path optimization items, nonlinear items, and integer programming items;

[0010] The total resource cost optimization objective is solved by optimizing the path optimization term, nonlinear term and integer programming term of the total resource cost optimization objective function of the system, and the solution result of the total resource cost optimization objective is the resource decision; the resource decision includes the aggregation position and data flow path in the network and the computing resources allocated to each node;

[0011] The resource decision is deployed in an actual system, and the total resource overhead after the resource decision is deployed is observed to guide the next time slot decision.

[0012] Resource optimization strategies for data processing tasks with dynamic loads In each iteration, data is first collected from individual client devices. In the context of federated learning, this data is usually kept locally, and only model updates or specific parameters are prepared for transmission, which can ensure data privacy and improve efficiency. Given the variability of network conditions, the best path for data transmission needs to be determined dynamically. This involves real-time analysis of network traffic, latency, and bandwidth availability to effectively route data and avoid congestion. As data is transmitted over the network, it is aggregated at strategic nodes (usually network switches with programmable capabilities). This aggregation reduces the amount of data that needs to be sent to the central server, thereby reducing bandwidth usage and improving overall system performance. Computing resources are dynamically allocated based on current network conditions and the computational load at each point in the system. This step is critical for balancing the load between clients and servers, ensuring that no single node is overloaded, which could degrade the performance of the entire system. The aggregated updates arrive at the central server, where they are merged into a global model. In this process, the server integrates the learning results of different devices, updates the global model, and then distributes it back to the individual clients for the next round of learning.

[0013] According to the above technical solution, the total resource cost optimization target of the system is:

[0014]

[0015] The constraints of the system total resource cost optimization target include:

[0016] 1.1) Aggregation operations can only be performed on selected paths:

[0017]

[0018] 1.2) Each client can only choose one path to send data:

[0019]

[0020] 1.3) Limitation of the domain of decision making:

[0021]

[0022] In the formula, x t Indicates the allocation of local resources; k L (x t ) represents the number of local iterations, κ G (x t ) represents the number of global iterations; a t Yes, local training unit costs; represents the data volume of the i-th client; ψ(S t ) is a function relative to the variable z, representing the received traffic; w i,p,t Indicates whether client i chooses path p; d p,t is the delay of path p at time slot t; z j,t Indicates whether to aggregate on j network devices; m p,j Indicates whether network node j is on path p; represents the set of all time slots in the system; c represents the resource overhead brought to the system by unit traffic; Represents the collection of all clients; represents the set of paths that client i can choose; S t represents the set of network devices selected at time t, Represents the set of network devices j.

[0023] According to the above technical solution, the specific execution steps of solving the total resource cost optimization target include:

[0024] Since multiple decision variables in the total resource cost optimization objective are coupled to each other, path optimization items are extracted, the path optimization item optimization objectives and their corresponding path optimization item constraints are constructed, the path optimization item optimization objectives are solved, and the data flow path in the network is determined;

[0025] Since there are nonlinear items in the total resource cost optimization target, in order to optimize the nonlinear items, the nonlinear items are extracted to construct the nonlinear item optimization target and its corresponding nonlinear item constraint conditions, and the nonlinear item optimization target is solved to determine the aggregation position in the network;

[0026] Constructing the integer programming item real number threshold optimization target and its corresponding integer programming item real number threshold constraint condition, relaxing the integer programming item to the real number threshold, and coupling the integer programming item real number threshold optimization target in the real number threshold to the single time slot to construct the integer programming item single time slot optimization target and its corresponding integer programming item single time slot optimization constraint condition;

[0027] Based on the determined aggregation locations and data flow paths in the network, an online learning method is used to solve the single-slot optimization objective of integer programming items and calculate the computing resources allocated to each node for the data processing task.

[0028] The solver and online learning method can be used to solve the optimization objectives of path optimization items, nonlinear items and real threshold optimization objectives of integer programming items.

[0029] According to the above technical solution, the path optimization objective function is:

[0030]

[0031] The path optimization constraints include:

[0032] 2.1) Each client can only choose one path to send data:

[0033]

[0034] 2.2) Limitation of the domain of decision making:

[0035]

[0036] In the formula, w i,p,t Indicates whether client i chooses path p; d p,t is the delay of path p at time slot t; Represents the collection of all clients; represents the set of paths that client i can choose; Represents the set of all time slots.

[0037] According to the above technical solution, the nonlinear term optimization objective is:

[0038] maxψ(S t );

[0039] The nonlinear term constraints are:

[0040] 3.1) Constraints on the upper limit of aggregation: ∑ j z j,t = l;

[0041] 3.2) Limitation of the domain of decision making:

[0042] In the formula, ψ(S t ) is a function relative to the variable z, which represents the received traffic; z j,t Indicates whether to aggregate on j network devices; l indicates the number of network devices that can be selected; represents the set of network devices j; Represents the set of all time slots.

[0043] According to the above technical solution, the real number threshold optimization target of the integer programming item is:

[0044]

[0045] The real threshold constraint condition of the integer programming item is:

[0046] 4.1) Limitation of the domain of decision making:

[0047] In the formula, κ L (x t ) represents the number of local iterations; κ G (x t ) represents the number of global iterations; a t Yes, local training unit costs; represents the data volume of the i-th client; x t Indicates the allocation amount of local resources; Represents the set of all time slots.

[0048] According to the above technical solution, the single time slot optimization objective of the integer programming item is:

[0049]

[0050] The single-slot optimization constraints of the integer programming item are:

[0051] 5.1) Limitation of the domain of decision making:

[0052] In the formula, is a vector of decision variables, and its domain is in the corresponding real number domain superior; represents the resource decision for time slot t; function represents the gradient of the function f; α represents the step size of the regularization term, which is used to control the degree of deviation of the solutions of adjacent time slots.

[0053] In another embodiment, a resource optimization device for data processing tasks includes:

[0054] Dynamic load acquisition module, used to obtain network latency, traffic size of data processing tasks and their distribution in the network topology, and data volume of each client;

[0055] The resource decision calculation module is used to construct an optimization problem with the total system resource cost as the optimization target and its corresponding constraints based on the data obtained by the dynamic load acquisition module, and solve the optimization problem with the total system resource cost as the optimization target, and the solution result is the resource decision;

[0056] The resource decision-making and delivery module distributes local client resources according to the aggregation location and data flow path in the network.

[0057] According to the above technical solution, the resource decision calculation module is constructed with the total system resource overhead as the optimization target:

[0058]

[0059] The constraint corresponding to the optimization target of the total system resource cost is:

[0060] 6.1) Aggregation operations can only be performed on selected paths:

[0061]

[0062] 6.2) Each client can only choose one path to send data:

[0063]

[0064] 6.3) Limitation of the domain of decision making:

[0065]

[0066] In the formula, x t Indicates the allocation of local resources; k L (x t ) represents the number of local iterations, k G (x t ) represents the number of global iterations; a t Yes, local training unit costs; represents the data volume of the i-th client; ψ(S t ) is a function relative to the variable z, representing the received traffic; w i,p,t Indicates whether client i chooses path p; d p,t is the delay of path p at time slot t; z j,t Indicates whether to aggregate on j network devices; m p,j Indicates whether network node j is on path p; represents the set of all time slots in the system; c represents the resource overhead brought to the system by unit traffic; Represents the collection of all clients; represents the set of paths that client i can choose; S t represents the set of network devices selected at time t, Represents the set of network devices j.

[0067] It also includes an embodiment, a resource optimization system for data processing tasks oriented to dynamic loads, including several clients and several groups of data streams generated by them, servers, several network node machines, and scheduling nodes. The clients and servers require network node interconnection, and the origin and destination of the several groups of data streams are the clients and servers respectively, and the above-mentioned resource optimization method for data processing tasks oriented to dynamic loads is executed.

[0068] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: the present invention proposes a data processing task resource optimization strategy for dynamic loads, continuously obtains network latency, the traffic scale of data processing tasks and their distribution in the network topology, and the data volume of each client, constructs an optimization problem with the total system resource overhead as the optimization target, and continuously adjusts the routing path, aggregation location, and computing resource allocation by solving the corresponding optimization problem.

[0069] Based on the current dynamic environment data, the present invention constructs and solves an optimization problem aimed at minimizing total resource consumption, and its solution guides the configuration decisions of network and computing resources. In a dynamic load environment, the intra-network aggregation technology is used to optimize the resource configuration of data processing tasks, reduce communication delays and improve the overall performance of the system. The deployment method and system can effectively reduce the resource consumption of data processing tasks. Compared with the deployment scheme of traditional technologies, the scheduling strategy proposed in the present invention can respond to network changes and differences in task distribution in real time, thereby improving the efficiency of task processing.

[0070] The dynamic load resource optimization system of the present invention utilizes the flexibility of programmable switches and can dynamically allocate resources and adjust network flows according to changes in network conditions and data traffic, thereby greatly improving the efficiency of network resource utilization and the performance of data processing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0072] Figure 1 is a schematic diagram of a data processing task architecture in an embodiment;

[0073] Figure 2It is a communication time statistics diagram of the embodiment. DETAILED DESCRIPTION

[0074] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] Example 1

[0076] The routing and in-band computing protocols are designed, where the SR field represents the source routing protocol, the port field stores the corresponding forwarding port, and the BOS field is used for control functions; the AGG field is used for in-band computing functions, storing the actual gradient and aggregating the corresponding gradient on the specified switch. Specifically, the Header field stores the header information, including data flow and data packet information, and the Payload stores the actual gradient.

[0077] The scheduling node in the system will periodically collect the current dynamic load status, calculate the appropriate path, aggregation location, and computing resource allocation, thereby minimizing the total system resource overhead. The process is as follows:

[0078] S1, continuously monitor network latency, the traffic scale of data processing tasks, their distribution in the network topology, and the amount of data at each node;

[0079] S2. Based on the detected network delay, the traffic scale of the data processing task and its distribution in the network topology, and the amount of data at each node, a total system resource cost optimization target is constructed and constraints of the total system resource cost optimization target are determined; the total system resource cost optimization target includes path optimization items, nonlinear items, and integer programming items;

[0080] Specifically, the overall goal of scheduling nodes is to minimize the resource overhead of data processing tasks (i.e., maximize efficiency) under a dynamic load environment. The optimization problem established is as follows:

[0081]

[0082] The constraints of the total system resource cost optimization goal include:

[0083] 1.1) Aggregation operations can only be performed on selected paths:

[0084]

[0085] 1.2) Each client can only choose one path to send data:

[0086]

[0087] 1.3) Limitation of the domain of decision making:

[0088]

[0089] In the formula, x t represents the allocation of local resources; κ L (x t ) represents the number of local iterations, κ G (x t ) represents the number of global iterations; a t Yes, local training unit costs; represents the data volume of the i-th client; ψ(S t ) is a function relative to the variable z, representing the received traffic; w i,p,t Indicates whether client i chooses path p; d p,t is the delay of path p at time slot t; z j,t Indicates whether to aggregate on j network devices; m p,j Indicates whether network node j is on path p; represents the set of all time slots in the system; c represents the resource overhead brought to the system by unit traffic; Represents the collection of all clients; represents the set of paths that client i can choose; S t represents the set of network devices selected at time t, Represents the set of network devices j.

[0090] S3. The total resource cost optimization target is solved by optimizing the path optimization term, nonlinear term and integer programming term of the system total resource cost optimization objective function. The solution result of the total resource cost optimization target is the resource decision; the resource decision includes the aggregation position and data flow path in the network and the computing resources allocated to each node.

[0091] Specifically, the specific execution steps for solving the total resource cost optimization target include:

[0092] S301, constructing a path optimization item optimization target and its corresponding path optimization item constraint condition, solving the path optimization item optimization target, and determining a data flow path in the network;

[0093] Path optimization objective function:

[0094]

[0095] The path optimization constraints include:

[0096] 2.1) Each client can only choose one path to send data:

[0097]

[0098] 2.2) Limitation of the domain of decision making:

[0099]

[0100] In the formula, w i,p,t Indicates whether client i chooses path p; d p,t is the delay of path p at time slot t; Represents the collection of all clients; represents the set of paths that client i can choose; Represents the set of all time slots.

[0101] S302, constructing a nonlinear item optimization target and its corresponding nonlinear item constraint condition, solving the nonlinear item optimization target, and determining the aggregation position in the network;

[0102] Nonlinear term optimization objective:

[0103] maxψ(S t );

[0104] The nonlinear term constraints are:

[0105] 3.1) Constraints on the upper limit of aggregation: ∑ j z j,t = l;

[0106] 3.2) Limitation of the domain of decision making:

[0107] In the formula, ψ(S t ) is a function of the variable z, which represents the received traffic; z j,t Indicates whether to aggregate on j network devices; l indicates the number of network devices that can be selected; Represents the collection of all clients; Represents the set of all time slots.

[0108] S303, constructing the integer programming item real number threshold optimization target and its corresponding integer programming item real number threshold constraint condition to relax the integer programming item to the real number threshold, and coupling the integer programming item real number threshold optimization target in the real number threshold to the single time slot to construct the integer programming item single time slot optimization target and its corresponding integer programming item single time slot optimization constraint condition;

[0109] Integer programming term real threshold optimization objective:

[0110]

[0111] The real number threshold constraint condition of the integer programming item is:

[0112] 4.1) Limitation of the domain of decision making:

[0113] In the formula, κ L (x t ) represents the number of local iterations; κ G (x t ) represents the number of global iterations; a t Yes, local training unit costs; represents the data volume of the i-th client; x t Indicates the allocation amount of local resources; Represents the set of all time slots.

[0114] Integer programming item single time slot optimization objective:

[0115]

[0116] The single-slot optimization constraints of the integer programming item are:

[0117] 5.1) Limitation of the domain of decision making:

[0118] In the formula, is a vector of decision variables, and its domain is in the corresponding real number domain superior; represents the resource decision for time slot t; function represents the gradient of the function f; α represents the step size of the regularization term, which is used to control the degree of deviation of the solutions of adjacent time slots.

[0119] S304: Based on the determined aggregation position and data flow path in the network, an online learning method is used to solve the single-slot optimization objective of the integer programming item, and the computing resources allocated to each node are calculated for the data processing task.

[0120] S4. Deploy the resource decision in an actual system, and observe the total resource overhead after the resource decision is deployed to guide the next time slot decision.

[0121] Federated learning training was carried out based on the routing and in-band computing protocol constructed by the present invention. The size of the traffic sent by six clients was 5KB. The data streams from multiple clients were aggregated in the network node. The specific aggregation strategy was in accordance with the method mentioned above. After the aggregation was completed, the added result was sent to the server. The completion time of the data stream executing the method proposed in the present invention was counted. Figure 2The completion time of the data processing tasks for the first 30 iterations is recorded. From the figure, it can be concluded that the method proposed in the present invention can shorten the completion time of the task and maximize the efficiency of the data processing task.

[0122] Example 2

[0123] A resource optimization device for data processing tasks, comprising:

[0124] The dynamic load acquisition module is used to obtain network latency, the traffic scale of data processing tasks and their distribution in the network topology, and the data volume of each client.

[0125] The resource decision calculation module is used to construct an optimization problem with the total system resource expenditure as the optimization target and its corresponding constraints based on the data obtained by the dynamic load acquisition module, and solve the optimization problem with the total system resource expenditure as the optimization target. The solution result is the resource decision; the optimization problem with the total system resource expenditure as the optimization target constructed by the resource decision calculation module is:

[0126]

[0127] The constraint corresponding to the optimization target of the total system resource cost is:

[0128] 6.1) Aggregation operations can only be performed on selected paths:

[0129]

[0130] 6.2) Each client can only choose one path to send data:

[0131]

[0132] 6.3) Limitation of the domain of decision making:

[0133]

[0134] In the formula, x t represents the allocation of local resources; κ L (x t ) represents the number of local iterations, κ G (x t ) represents the number of global iterations; a t Yes, local training unit costs; represents the data volume of the i-th client; ψ(S t ) is a function relative to the variable z, representing the received traffic; w i,p,t Indicates whether client i chooses path p; d p,t is the delay of path p at time slot t; z j,tIndicates whether to aggregate on j network devices; m p,j Indicates whether network node j is on path p; represents the set of all time slots in the system; c represents the resource overhead brought to the system by unit traffic; Represents the collection of all clients; represents the set of paths that client i can choose; S t represents the set of network devices selected at time t, Represents the set of network devices j.

[0135] The resource decision-making and delivery module distributes local client resources according to the aggregation location and data flow path in the network.

[0136] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0137] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for optimizing data processing task resources for dynamic loads, characterized in that: The steps include: Continuously monitor network latency, the traffic size of data processing tasks, their distribution in the network topology, and the amount of data at each node; Based on the detected network delay, the traffic scale of the data processing task and its distribution in the network topology, and the amount of data at each node, a total system resource cost optimization target is constructed and constraints of the total system resource cost optimization target are determined; the total system resource cost optimization target includes path optimization items, nonlinear items, and integer programming items; The total resource cost optimization objective is solved by optimizing the path optimization term, nonlinear term and integer programming term of the total resource cost optimization objective function of the system, and the solution result of the total resource cost optimization objective is the resource decision; the resource decision includes the aggregation position and data flow path in the network and the computing resources allocated to each node; The resource decision is deployed in an actual system, and the total resource overhead after the resource decision is deployed is observed to guide the next time slot decision.

2. The method for optimizing data processing task resources for dynamic loads according to claim 1, characterized in that: The total system resource cost optimization goal is: The constraints of the system total resource cost optimization target include: 1.1) Aggregation operations can only be performed on selected paths: 1.2) Each client can only choose one path to send data: 1.3) Limitation of the domain of decision making: In the formula, x t represents the allocation of local resources; κ L (x t ) represents the number of local iterations, κ G (x t ) represents the number of global iterations; a t Yes, local training unit costs; represents the data volume of the i-th client; ψ(S t ) is a function relative to the variable z, representing the received traffic; w i,p,t Indicates whether client i chooses path p; d p,t is the delay of path p at time slot t; z j,t Indicates whether to aggregate on j network devices; m p,j Indicates whether network device j is on path p; represents the set of all time slots in the system; c represents the resource overhead brought to the system by unit traffic; Represents the collection of all clients. represents the set of paths that client i can choose, S t represents the set of network devices selected at time t, Represents the set of network devices j.

3. The method for optimizing data processing task resources for dynamic load according to claim 1, characterized in that: The specific execution steps of solving the total resource cost optimization target include: Construct the optimization target of the path optimization item and its corresponding constraint condition of the path optimization item, solve the optimization target of the path optimization item, and determine the data flow path in the network; Construct nonlinear item optimization objectives and their corresponding nonlinear item constraints, solve the nonlinear item optimization objectives, and determine the aggregation positions in the network; Constructing the integer programming item real number threshold optimization target and its corresponding integer programming item real number threshold constraint condition, relaxing the integer programming item to the real number threshold, and coupling the integer programming item real number threshold optimization target in the real number threshold to the single time slot to construct the integer programming item single time slot optimization target and its corresponding integer programming item single time slot optimization constraint condition; Based on the determined aggregation locations and data flow paths in the network, an online learning method is used to solve the single-slot optimization objective of integer programming items and calculate the computing resources allocated to each node for the data processing task.

4. The method for optimizing data processing task resources for dynamic load according to claim 3, characterized in that: The path optimization objective function is: The path optimization constraints include: 2.1) Each client can only choose one path to send data: 2.2) Limitation of the domain of decision making: In the formula, w i,p,t Indicates whether client i chooses path p; d p,t is the delay of path p at time slot t; Represents the collection of all clients. represents the set of paths that client i can choose, Represents the total set of time slots in the system.

5. The method for optimizing data processing task resources for dynamic load according to claim 3, characterized in that: The nonlinear term optimization objective is: maxψ(S t ); The nonlinear term constraints are: 3.1) Constraints on the upper limit of aggregation: ∑ j z j,t = l; 3.2) Limitation of the domain of decision making: In the formula, ψ(S t ) is a function relative to the variable z, representing the received traffic; z j,t Indicates whether to aggregate on j network devices; l indicates the number of network devices that can be selected; represents the set of network devices j; Represents the total set of time slots in the system.

6. The method for optimizing data processing task resources for dynamic load according to claim 3, characterized in that: The real-valued threshold optimization objective of the integer programming term is: The real number threshold constraint condition of the integer programming item is: 4.1) Limitation of the domain of decision making: In the formula, κ L (x t ) represents the number of local iterations; κ G (x t ) represents the number of global iterations; a t Yes, local training unit costs; represents the data volume of the i-th client; x t Indicates the allocation amount of local resources; Represents the total set of time slots in the system.

7. The method for optimizing data processing task resources for dynamic load according to claim 6, characterized in that: The single-slot optimization objective of the integer programming item is: The single-slot optimization constraints of the integer programming item are: 5.1) Limitation of the domain of decision making: In the formula, is a vector of decision variables, and its domain is in the corresponding real number domain superior; represents the resource decision for time slot t; function represents the gradient of the function f; α represents the step size of the regularization term, which is used to control the degree of deviation of the solutions of adjacent time slots.

8. A resource optimization device for data processing tasks, characterized in that: include: Dynamic load acquisition module, used to obtain network latency, traffic size of data processing tasks and their distribution in the network topology, and data volume of each client; The resource decision calculation module is used to construct an optimization problem with the total system resource cost as the optimization target and its corresponding constraints based on the data obtained by the dynamic load acquisition module, and solve the optimization problem with the total system resource cost as the optimization target, and the solution result is the resource decision; The resource decision-making module distributes the resources of the local client according to the aggregation location and data flow path in the network.

9. The data processing task resource optimization device for dynamic load according to claim 8, characterized in that: The resource decision calculation module is constructed with the total system resource cost as the optimization target: The constraint condition corresponding to the optimization target of the total system resource cost is: 6.1) Aggregation operations can only be performed on selected paths: 6.2) Each client can only choose one path to send data: 6.3) Limitation of the domain of decision making: In the formula, x t Indicates the allocation of local resources; k L (x t ) represents the number of local iterations, k G (x t ) represents the number of global iterations; a t Yes, local training unit costs; represents the data volume of the i-th client; ψ(S t ) is a function relative to the variable z, representing the received traffic; w i,p,t Indicates whether client i selects path p; d p,t is the delay of path p at time slot t; z j,t Indicates whether to aggregate on j network devices; m p,j Indicates whether network node j is on path p; represents the set of all time slots in the system; c represents the resource overhead brought to the system by unit traffic; Represents the collection of all clients, represents the set of paths that client i can choose, S t represents the set of network devices selected at time t, Represents the set of network devices j.

10. A resource optimization system for data processing tasks with dynamic loads, comprising a plurality of clients and a plurality of groups of data streams generated by them, a server, a plurality of network nodes, and a scheduling node, wherein the clients and servers need to be interconnected by network nodes, and the plurality of groups of data streams originate from and are destined for the clients and the servers, respectively, and characterized in that: Execute the resource optimization method as described in any one of claims 1 to 7.

Images