Method for multi-edge cooperative load balancing based on two-stage decision in wireless metropolitan area network

Abstract

The application aims to provide a two-stage decision-based multi-edge collaborative load balancing method in a wireless metropolitan area network. First, a centralized decision of load balancing is performed based on global information, wherein a prediction model based on a deep neural network (DNN) is designed to evaluate the task scheduling range between adjacent edges. Next, a decentralized decision of load balancing is performed based on local information, wherein an adjustment operation Q value prediction model based on a deep Q network (DQN) is designed to evaluate the load balancing scheme between edges. Finally, a target load balancing scheme is obtained through feedback control. Simulation experiments prove that TDB-EC can well adapt to a new environment and achieve multi-edge load balancing in seconds. In addition, TDB-EC is superior to three classical methods and achieves approximate optimal performance.

Description

Technical Field

[0001] This invention relates to the field of mobile edge computing technology, specifically to a multi-edge collaborative load balancing method based on two-stage decision-making in wireless metropolitan area networks. Background Technology

[0002] With the rapid development of mobile communication technology and the Internet of Things (IoT), mobile devices (such as smartphones, vehicles, and drones) are exhibiting greater resource demands to ensure high performance in handling computationally intensive tasks within Wireless Metropolitan Area Network (WMANs) environments. However, mobile devices are constrained by their size, resulting in significant limitations in computing power and storage capacity. Offloading computationally intensive tasks from mobile devices to remote cloud servers can alleviate these resource limitations to some extent.

[0003] However, the geographical distance between mobile devices and remote cloud servers leads to significant response latency. This problem can be effectively mitigated by introducing mobile edge computing, which offloads computing resources from the core network to the network edge. Therefore, offloading computationally intensive tasks from mobile devices to the nearby edge can reduce response latency in wireless metropolitan area network environments.

[0004] In edge environments, edge servers deployed at wireless base stations can provide nearby mobile devices with low-latency access and powerful computing capabilities, thereby improving mobile device performance. Based on this, the edge can be considered an important supplementary means to improve the Quality of Service (QoS) in wireless metropolitan area network (WMAN) environments. Typically, tasks from mobile devices tend to be offloaded to their nearest edge for execution, but this may not be optimal due to fluctuating loads at different edges within WMANs. For example, when many mobile devices (such as smartphones, vehicles, and drones) are simultaneously operating within the edge's coverage area, it can lead to a sharp increase in edge load and response latency. To address this issue and improve user QoS, multi-edge collaboration mechanisms are a viable solution.

[0005] Currently, methods for solving load balancing problems mainly employ centralized or decentralized decision-making models. One approach based on centralized decision-making utilizes heuristics or machine learning (ML) algorithms. However, collecting real-time information from all edges in a WMANs environment with a large number of widely distributed edges can lead to excessive decision-making time and communication overhead. Heuristic algorithms may consume significant search time to find the target load balancing solution, making it difficult to meet the real-time service demands of mobile applications. ML algorithms cannot build accurate load balancing decision models when data support is insufficient. Another approach based on decentralized decision-making offers better scalability compared to centralized approaches. However, they may only achieve local optima rather than global optima because they only use local information in the load balancing decision-making process. Therefore, there is a need to find efficient multi-edge collaborative load balancing methods to achieve load balancing among multiple edges in dynamically changing WMANs environments. Summary of the Invention

[0006] In a wireless metropolitan area network (MAN) environment, the load on each edge is dynamically changing, necessitating task scheduling to achieve load balancing between edges. This invention utilizes a two-phase decision-based multi-edge collaborative load balancing method for MANs that can complete the load balancing task scheduling solution process within seconds, generating an efficient load balancing scheme that meets the real-time requirements of load balancing across multiple edges.

[0007] Existing multi-edge collaborative load balancing solutions employ centralized decision-making for load balancing, which leads to excessive decision-making time and communication overhead in WMANs with a large number of widely distributed edges. This invention proposes a novel two-stage decision-making method (TDB-EC) for multi-edge collaborative load balancing:

[0008] First, a centralized decision-making process for load balancing is performed based on global information, where a prediction model based on a deep neural network (DNN) is designed to evaluate the task scheduling range between adjacent edges. Next, a decentralized decision-making process for load balancing is performed based on local information, where a prediction model based on a deep Q-network (DQN) for adjusting the Q-value is designed to evaluate the load balancing scheme between edges. Finally, the target load balancing scheme is obtained through feedback control. Simulation experiments demonstrate that TDB-EC can adapt well to the new environment and achieve multi-edge load balancing within seconds. Furthermore, TDB-EC outperforms three classical methods and achieves near-optimal performance.

[0009] The specific technical solution adopted by this invention to solve its technical problem is as follows:

[0010] A multi-edge collaborative load balancing method based on two-stage decision-making in a wireless metropolitan area network is proposed. The method performs centralized load balancing decision-making based on global information and evaluates the task scheduling range between adjacent edges through a prediction model based on a deep neural network. The method also performs decentralized load balancing decision-making based on local information and evaluates the load balancing scheme between edges through a Q-value prediction model based on a deep Q-network. Finally, the target load balancing scheme is obtained through feedback control.

[0011] Furthermore, the average response time reaching the task at all edges is used as the evaluation metric, with the goal of minimizing the maximum average response time T as much as possible. max .

[0012] Furthermore, based on global information, a DNN-based load balancing scheme evaluation model is used for centralized decision-making to evaluate the task scheduling interval between each pair of adjacent edges.

[0013] Edge nodes use a DQN-based load balancing adjustment operation Q-value prediction model to make distributed decisions based on their own local information. Each edge independently executes load balancing task scheduling between itself and its neighboring nodes under the task scheduling interval constraint.

[0014] In the multi-edge collaborative load balancing method based on two-stage decision-making, global centralized decision-making and single-node decentralized decision-making are performed in parallel and independently, and a feedback control mechanism is used to gradually obtain a suitable global load balancing scheme.

[0015] Furthermore, the specific execution steps of the decision-making algorithm include: calling Q_predModel to calculate the Q-values ​​of all actions and storing them in the set Q_values; if an action is selected... Subsequent local load balancing scheme Is it illegal or related to workload? Not in the task scheduling range If the action is invalid, it is considered an illegal action, and the corresponding Q value is marked as I. Then, a predefined threshold T is used to determine whether the target load balancing solution has been found. If the Q values ​​of all valid actions except those marked with I are less than or equal to the threshold T, the target solution is considered to have been reached, and no further adjustment operation is required. Otherwise, the adjustment operation with the smallest Q value will be selected. Finally, the obtained adjustment operation a is returned.

[0016] Compared to existing technologies, this invention and its preferred embodiment propose the TDB-EC method to solve the multi-edge collaborative load balancing problem in WMAN. First, to evaluate the task scheduling interval between adjacent edges, a DNN-based load balancing scheme evaluation model is designed for centralized decision-making based on global information. Second, a DQN-based adjustment operation Q-value prediction model is designed for decentralized decision-making based on local information, where each edge performs load balancing between adjacent edges within the task scheduling interval. Finally, a feedback control mechanism is used to search for the target load balancing scheme. Based on real-world base station distribution maps, simulation experiments demonstrate that TDB-EC has good adaptability to multi-edge load balancing scenarios in different wireless metropolitan area networks. Specifically, TDB-EC can achieve optimal / near-optimal results within seconds, outperforming RF-CLB, ML-based, and Rule-based methods by 3.69%–4.61%, 4.01%–6.34%, and 14.46%–16.72%, respectively. Furthermore, the model confidence of TDB-EC consistently exceeds 95%, and the average accuracy of selecting adjustment operations reaches 92.5%. Attached Figure Description

[0017] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0018] Figure 1 This is a schematic diagram of a multi-edge collaborative load balancing model in a wireless metropolitan area network established according to an embodiment of the present invention;

[0019] Figure 2 This is an overview diagram of the method according to an embodiment of the present invention;

[0020] Figure 3 This is an example diagram of the topology graph containing 5 edges in an embodiment of the present invention;

[0021] Figure 4 This is a diagram of the DQN architecture according to an embodiment of the present invention;

[0022] Figure 5 This is a schematic diagram of an experimental scenario according to an embodiment of the present invention;

[0023] Figure 6 This is a comparison chart of the maximum average response time of TDB-EC and the ideal solution under different scenarios according to embodiments of the present invention;

[0024] Figure 7 This is a schematic diagram illustrating the confidence level of the load balancing scheme evaluation model in different scenarios according to embodiments of the present invention;

[0025] Figure 8 This is a schematic diagram illustrating the accuracy of operational decisions when the distance from the ideal solution is different with different number of adjustments according to an embodiment of the present invention;

[0026] Figure 9 This is a schematic diagram comparing TDB-EC with three benchmark methods from the perspective of maximum average response time in an embodiment of the present invention;

[0027] Figure 10 This is a schematic diagram comparing the average response time of all tasks of the TDB-EC embodiment of the present invention with three benchmark methods. Detailed Implementation

[0028] To make the features and advantages of this patent more apparent and understandable, specific embodiments are provided below for detailed explanation:

[0029] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0030] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0031] The following is a detailed description of the embodiments of the present invention:

[0032] 1. Problem Definition

[0033] like Figure 1 As shown, in the model constructed in this embodiment, a group of edges are placed on wireless access points in a WMANs environment, and the edges can interact with each other via a wireless network. This embodiment assumes that applications on mobile devices can be dynamically divided into independent, offloadable tasks that can be processed at any edge. Due to user mobility, the distribution of mobile devices is highly variable, leading to uneven load at the edges. High load results in increased response latency for task execution, while low load leads to significant resource waste. Therefore, collaborative load balancing of multiple edges in WMAN is required to reduce task response latency and improve edge resource utilization.

[0034] This embodiment assumes that N edges are deployed in a WMAN, denoted as E = {e1, e2, ..., e...}. N} represents. Where, e i Let represent the i-th edge, and each edge e iIt is only connected to c neighboring edges (the two edges that will be interconnected via a wireless network are called adjacent edges). The service rate of an edge is defined as the set V = {v1, v2, ..., v...} N}, where v i Represents edge e i The service rate, i.e., the edge e i The amount of work that can be completed within a unit of time.

[0035] Then, in WMAN, the unit task transmission delay D between each pair of adjacent edges is defined as follows:

[0036]

[0037] Where, d i,j Represents two edges (i.e., e) i and e j The unit task transmission delay between (i, j). Specifically, if i = j, then d i,j =0. When edge e i No and edge e j If they are interconnected, then d i,j =∞.

[0038] Next, the task arrival rate at the edge is defined as the set λ = {λ1, λ2, ..., λ...} N}, λ i Represents edge e i The task arrival rate, i.e., the rate at which tasks are unloaded to the edge per unit of time. i The workload. Specifically, these offloading tasks are called arrival tasks. An edge's arrival task can be processed locally or scheduled to its adjacent edge to ensure load balancing between edges.

[0039] Meanwhile, the global load balancing scheme F for edge set E mat The definition is as follows:

[0040]

[0041] In this scenario, only one edge's arrival task can be scheduled, while tasks received at adjacent edges cannot be scheduled. When i = j, f i,j e represents the edge of a unit of time i The number of tasks that arrive and are executed on this node. When i ≠ j, f i,j representing the edge e per unit time i The arrival task is scheduled to the adjacent edge e j The amount of work processed. Furthermore, if edge e i and edge e j If there is no network connection, then f i,j =0.

[0042] Furthermore, the load at the edge is defined as the set W = {w1, w2, ..., w...} N}. w i >0 represents the edge e i The total number of tasks executed per unit of time, including edge e i The arrival task consists of two parts: the amount of task processed on itself and the amount of task from adjacent edges. In the initial state, w i equals edge e i The arrival of the mission λ i Then, the arriving task can be processed locally or routed to an adjacent edge. Therefore, edge e i The average response time of the task should The definition is as follows:

[0043]

[0044] Among them, t i,j Represents the time from the edge e i Schedule tasks to the edge e j The average response time of a task is t, which includes both execution and transmission time. Therefore, t i,j The definition is as follows:

[0045] t i,j =T a (l j )+d i,j

[0046] Among them, T a (l j ) represents at the edge e j load rate At that time, the average execution time of the tasks on it. T a (l j This is related to the specific edge node configuration.

[0047] Furthermore, the average response time of the maximum arrival time of the task across all edges is used as an evaluation metric to better assess the load balancing effect, defined as follows:

[0048]

[0049] Finally, the objective function of this scheme is defined as:

[0050] min(T max )

[0051] When using different global load balancing schemes F mat At that time, the maximum average response time T maxThis may change. To achieve better multi-edge collaborative load balancing in WMANs, this embodiment is designed to minimize the maximum average response time T. max .

[0052] 2. A Multi-Edge Collaborative Load Balancing Method Based on Two-Phase Decision Making

[0053] Next, based on the constructed model and objective function, we introduce the proposed multi-edge cooperative load balancing method (TDB-EC) for wireless metropolitan area networks based on two-stage decision-making. This method combines centralized and decentralized decision-making approaches, such as... Figure 2 As shown.

[0054] First, based on global information, a DNN-based load balancing scheme evaluation model (P_predModel) is used for centralized decision-making to evaluate the task scheduling interval between each pair of adjacent edges (see Section 2.1 for details). Algorithm 1 describes the process of global centralized decision-making, with the following specific steps: First, update the current global information, including the global load balancing scheme F of the edge set. cur and the response time T of each edge task r (Line 5); Next, based on global information, use a DNN-based load balancing scheme evaluation model to calculate the set of task scheduling scheme evaluation values ​​F' between each pair of adjacent edges under the current system state (Lines 6-7); Then, calculate the set of task scheduling intervals SR between each pair of adjacent edges (Line 9); Finally, repeat the above process and continuously update SR.

[0055]

[0056]

[0057] Next, based on their local information, edge nodes use a distributed decision-making process based on a DQN-based load balancing adjustment operation Q-value prediction model (Q_predModel). Each edge independently executes load balancing task scheduling between itself and its neighboring nodes under the aforementioned task scheduling interval constraints (see Section 2.2 for details). Algorithm 2 describes the edge e i The process of decentralized decision-making involves the following steps: First, update the current local information, including the local load balancing scheme. Load factor L of itself and adjacent edges i and the local task scheduling interval set SR i (Line 6) Because global decisions and other peripheral decisions are also being made simultaneously, L i and SR i It is constantly changing, therefore, e i Reacquire data before each decision. Li and SR i , where SR i is obtained from the result of global centralized decision-making; secondly, λ i , L i and SR i are used as inputs to call the runtime decision-making algorithm (Algorithm 4) to calculate the next load balancing adjustment operation a of edge e i . If the a obtained by Algorithm 4 is empty, it means that edge e i has found the target local load balancing scheme in the local system state (represented by ), and no adjustment is made at this time (lines 8-9), otherwise the adjustment operation a obtained by executing Algorithm 4 is executed (line 11); finally, the above process is looped to continuously perform load balancing scheduling operations between adjacent nodes.

[0058]

[0059]

[0060] In the multi-edge collaborative load balancing method based on two-stage decision-making, global centralized decision-making and single-node decentralized decision-making are carried out in parallel and independently, and a feedback control mechanism is used to gradually obtain a suitable global load balancing scheme.

[0061] 2.1. DNN-based Global Load Balancing Scheme Evaluation Model

[0062] Based on global information, in this embodiment, a DNN-based load balancing scheme evaluation model is used for centralized decision-making to evaluate the task scheduling interval between each pair of adjacent edges. Among them, the input of the DNN-based load balancing scheme evaluation model is the task arrival rate λ of the edge set, the current global load balancing scheme F cur and the current task response time T r , and the output is the task scheduling scheme for each pair of adjacent edges.

[0063] In this embodiment, an undirected topology graph G=(E,Ψ) is used to represent the connection relationship of N edges in a wireless metropolitan area network. Among them, the edge set E={e1,e2,…,e N} is the vertex set of the topology graph, and the set Ψ={ψ1,ψ2,…ψ m} represents the edge set of the topology graph (including m edges). The k-th edge ψ k in the topology graph represents the network connection between edge e i and edge e j . At the same time, it is stipulated that for each pair of adjacent edges e i and e j (i<j), edge e iThe task scheduling direction is the positive direction, that is, tasks are from edge e i scheduled to edge e j (i < j), the direction is the positive direction; conversely, tasks are from edge e j scheduled to edge e i (i < j), the direction is the negative direction. Thus, the global load scheme F mat can be transformed into F = {f1, f2, …, f m}, and the global evaluation model of the load balancing scheme is shown in Table 2.1.

[0064] Table 2.1 Input and Output of the Global Evaluation Model of the Load Balancing Scheme

[0065] scheduled to edge e

[0066] Based on historical operation data, use DNN to train the global load balancing scheme evaluation model. Since the service rates v i of each edge are different, and there is λ i < v i and f i,j < min(v i , v j ). Therefore, the amounts of arriving tasks λ and task scheduling amounts f of different edges vary greatly. Thus, normalize the amounts of arriving tasks λ and task scheduling amounts f as follows:

[0067] λ′ i = λ i / v i

[0068] f′ k = f k / min(v i , v j )

[0069] Among them, λ' i represents the amount of arriving tasks of edge e i after normalization. f k represents the task scheduling amount between edge e i and edge e j , and f' k represents the task scheduling amount after normalization.

[0070] Figure 3 is a topological graph instance containing 5 edge nodes. In this scenario, the task arrival rates λ of the edge set are (6.8, 14.5, 4.6, 13.2, 9.1), and the current load balancing scheme F cur= (1.62, 0, 1.12, -3.18, -2.7), target load balancing scheme F = (0.81, 1.62, 0, -3.18, -3.3). Using the normalization formula, λ and F in the example are normalized respectively, and the results are shown in Table 2.2, where λ' i The value of f' is in the interval (0,1). k The value is in the interval (-1, 1).

[0071] Table 2.2 provides a simple example to illustrate normalization.

[0072]

[0073] The trained load balancing scheme evaluation model can evaluate the edge task arrival rate λ' and the current load balancing scheme F'. cur and the current task response time T r The target load balancing scheme F' is predicted. However, due to the complex and variable nature of edge load conditions, the load balancing scheme evaluation model cannot accurately predict the task scheduling scheme between each pair of edges. Therefore, this embodiment uses a confidence model to further describe the value range of the task scheduling scheme, as shown below:

[0074]

[0075] Where C represents the confidence level, ε represents the model error, and f' k This represents the predicted value of the task scheduling scheme. This represents the actual value of the task scheduling scheme, hence we have, There is a C% probability that the value falls within the interval [f'] k -ε,f' k +ε] within.

[0076] Based on the confidence model and normalization process described above, the reasonable range of values ​​for task scheduling between each pair of adjacent edges is represented by the set SR = {sr1, sr2, ..., sr...} m}, which is the output of Algorithm 1. Specifically, edge e i and edge e j The task scheduling interval is calculated as follows:

[0077] sr k =[(f′ k -ε)·min(v i ,v j ),(f′ k +e)·min(v i ,v j )]

[0078] 2.2 Load Balancing Adjustment Operation Q-value Prediction Model Based on DQN

[0079] An edge (such as e) i The collected local information includes the edge e i Task arrival rate (using λ) i (Indicates), edge e i and the task load rate of adjacent edges (using L) i (Indicates), edge e i The current local load balancing scheme (using (represented by) and the corresponding average response time to the task (in terms of) (representation) and edge e i Optional local load balancing schemes (using) (represented by) and the corresponding average response time to the task (in terms of) (This is indicated by the data collected locally). Each edge independently performs load balancing scheduling operations between adjacent edges based on the collected local information.

[0080] edge e i A set is defined as a network connected to c adjacent edges. in, Indicates the edge e i The j-th connected edge. In particular, a task arriving at an edge can only be processed locally or scheduled to an adjacent edge. ,in and Representing edge e respectively i and edge The load rate. in and They represent the edges e respectively i and edge The amount of tasks executed and received. Based on... After task scheduling, the corresponding... In the runtime environment, with λ i and L i The changes may lead to various options for local load balancing solutions (such as...) Of all the options, edge e i The target local load balancing scheme is to have the minimum The solution, using In addition, this embodiment uses sets. Indicates edge e i The set of task scheduling intervals between adjacent edges, where Indicates edge e i With adjacent edges The task scheduling interval.

[0081] edge e i The historical dataset consists of operational data under different system states, which includes λ. i , L i and As shown in Table 2.3, the DQN algorithm evaluates the Q-values ​​of all its adjustment operations under different system states based on the historical dataset of the edge. Specifically, the system states of the edge are... The adjustment operation mainly involves increasing / decreasing the value from the edge e. i The amount of tasks scheduled to adjacent edges in the arrival task. When the target solution is found. Then the corresponding reward will be generated.

[0082] Table 2.3 Edge e i Local information in the runtime environment

[0083]

[0084] As an automated decision-making method, Deep Reinforcement Learning (DRL) has demonstrated great potential to effectively handle load balancing problems by interacting with complex and dynamic multi-edge wireless metropolitan area network environments without prior knowledge. DQN combines Q-learning with deep learning techniques, replacing the Q-value table in Q-learning with deep neural networks (DNNs). Therefore, DQN does not need to search through a large Q-value table because it incorporates the Q-values ​​into the DNN. By inputting a state, the Q-values ​​of all corresponding actions can be obtained.

[0085] like Figure 4 As shown, the DQN agent first receives the state s of the runtime edge environment and selects action a using a greedy policy. Next, the DQN agent receives the corresponding reward r and moves to the next state s'. In each step, (s, a, r, s') is stored in an experience pool with a fixed capacity. If the threshold capacity is reached, the DNN's parameters ω are updated, and the loss function is defined as follows:

[0086] Loss=(r+γmaxQ(s',a';ω')-Q(s,a;ω)) 2

[0087] Where γ is the discount factor, ω is the weight of the DNN, Q(s,a;ω) is the current Q value, and maxQ(s',a';ω') is the maximum Q value for choosing action a' in the next state s'.

[0088] Typically, the DRL problem can be formulated as a Markov decision process (MDP) with the goal of maximizing long-run returns. Specifically, an MDP can be described as a quadruple.<S,A,P,R> Where S, A, P, and R represent the state space, action space, state transition function, and reward function, respectively. Based on the problem definition of multi-edge collaborative load balancing in WMAN, they are defined as follows:

[0089] State space: edge e i The state space is represented as S i ,in This represents a potential state. Specifically, Defined as a triple If using the scheme If the load on an edge exceeds its adjustable range (i.e., the load factor of the edge is less than 0 or greater than 1), then... It is considered an invalid state. Therefore, This is an illegal scheme.

[0090] Action space: edge e i Action space is defined as One action is to increase or decrease from the edge e. i The adjustment operation for the amount of tasks scheduled to the adjacent edge in the arrival task. Here, the amount of tasks scheduled each time is a fixed value δ. For example, Is it adding edge e? i Dispatch to the edge The workload.

[0091] State transition function: The state transition function is defined as P(s,a), which returns the next state after executing a. For example, when in state... implement The next state can be observed. Among them, edge e i The load at the edge decreases by δ. The load at that location increases by δ.

[0092] Reward function: To guide the DQN agent to find the target local load balancing solution. The reward function is defined as:

[0093]

[0094] If in state Execute action a to find DQN agents can receive a reward of 10. If in status... If action 'a' results in an illegal outcome, the reward will be set to -1 as a penalty. Otherwise, the reward remains 0.

[0095] DQN determines a new Q-value based on the Q-value estimated by the DNN and backpropagation of the reward, and then feeds the Q-value into the loss function to update the parameters of the DNN. However, it is difficult to directly train a load balancing adjustment operation Q-value prediction model through the above process due to the following problems.

[0096] (1) When the target state is reached, the theoretical Q-value for all load balancing adjustments is 0. However, as the target state approaches, these Q-values ​​become too high to accurately predict the Q-values ​​of the load balancing adjustments. Table 2.4 provides an example to illustrate the above problem, with edge e i With the edge and edge Connected, state s7 is the target state. In state s7, The predicted Q-value for the operation is 2.9670 (the theoretical value should be 0), which will affect the setting of the stopping condition during runtime decision-making. As shown in Table 2.4, when in the target state, the Q-values ​​of all operations are less than 3. Therefore, in this embodiment, the stopping condition is set to consider the target state reached when all operation Q-values ​​are less than 3. However, when in states s2, s3, and s4, the Q-values ​​corresponding to all adjustment operations are also less than 3, and in this case, the target state cannot be reached.

[0097] (2) When a certain adjustment operation is performed, the local load balancing scheme is affected. When a solution becomes illegal, the corresponding Q-value abruptly becomes negative (theoretically it should be -1), which affects the Q-value prediction of the opposite operation on the same adjacent edge. For example, in state s1, due to the execution The operation will cause the edge The load rate exceeds the schedulable range, at this time The predicted Q value is negative (the theoretical value should be -1), which also affects... The predicted Q-value of the operation. Compared to state s1 and state s2. The difference between the predicted Q-values ​​was found in this embodiment. The larger difference indicates that... The predicted Q-value of the operation is lowered in state s1, which further affects the runtime decision of the adjustment operation. As shown in Table 2.4, the theoretically selected operation in state s1 is... But in this state The predicted Q value is affected by The impact was lower than normal (Q-value predicted to be 0.4162), but it was actually selected. (The predicted Q value is 0.4675). At this point, the target state cannot be reached.

[0098] Table 2.4 Example of Q-value table for unprocessed load balancing adjustment operation

[0099]

[0100]

[0101] To address these issues, the target Q-value (denoted by Q_target) is processed as follows during DQN training:

[0102]

[0103] if This load balancing adjustment operation will be considered illegal, the corresponding Q_target will be set to 1, and it will not be stored in the DQN experience pool. hour, If found, Q_target = 0. Otherwise, Therefore, when approaching As time goes on, Q_target will gradually decrease, as shown in the table below.

[0104] The Q-value processing results in Tables 2.5 and 2.4

[0105]

[0106]

[0107] As shown in the table above, after processing the Q-value calculation during the DQN algorithm training process, the runtime decision-making can reach the target state. On one hand, the Q-value of the adjustment operation when in the target state differs significantly from that of other states. In this case, the stopping condition is set to a Q-value less than 0.15. When in state s1, the runtime decision-making can gradually reach the target state s7. On the other hand, when in state s1... The Q-value prediction of the operation is no longer affected The impact of the operation. Specifically, at s1, the operation is adjusted. and The predicted Q values ​​are 0.7581 and 0.8012, respectively, indicating that the correct adjustment operation can be selected.

[0108] Algorithm 3 demonstrates the training algorithm for a DQN-based load balancing adjustment operation Q-value prediction model. First, the DQN parameters are randomly initialized (lines 6-8), then the training process begins (line 9). In each round, the current state s is initialized based on the dataset. cur and target state sobj (Lines 10-11). When s cur ≠s obj (Line 12) Randomly select an action a with probability ∈, otherwise select action a with argminQ(s,a) (Line 13). Then, after executing action a, calculate the reward r according to the reward function and obtain the next state s' (Lines 14-16). If after executing action a, If so, skip this round of training (lines 17-18). Otherwise, (s,a,r,s') will be stored in the experience pool (line 20). Then, m samples are drawn from the experience pool in a batch and Q_tar get is calculated. j (Line 21). Then, update the network weights ω using the loss function (Line 22). In every K iterations, update ω' using ω (Line 23), after which the state s... cur A state transition occurs between state s' and state s' (line 24). Eventually, the algorithm will continue to train until it converges.

[0109]

[0110]

[0111]

[0112] 2.3 Runtime Decisions for Load Balancing Adjustment Operations

[0113] In this section, this embodiment proposes a novel runtime decision-making algorithm to obtain the target load balancing scheme between each edge and its adjacent edges. Algorithm 4 illustrates the main process of runtime decision-making. First, Q_predModel is called to calculate the Q-values ​​of all actions (i.e., load balancing adjustment operations) and store them in the set Q_values ​​(line 9). If an action is selected... Subsequent local load balancing scheme Is it illegal or related to workload? Not in the task scheduling range If the action is invalid, it is considered illegal, and the corresponding Q value is marked as I (lines 10-15). Then, a predefined threshold T is used to determine whether the target load balancing solution has been found. This is because there is a certain error in the Q value prediction of the load balancing adjustment operation. If the Q values ​​of all valid actions are less than or equal to the threshold T (except those marked with I), the target solution is considered to have been reached, and therefore no further adjustment operation is needed (lines 16-17); otherwise, the adjustment operation with the smallest Q value is selected (lines 18-20). Finally, the obtained adjustment operation a is returned (line 22).

[0114]

[0115]

[0116] It should be noted that each edge independently performs the load balancing task scheduling between adjacent edges until the corresponding decision algorithm terminates.

[0117] 3 Experimental Simulation and Results

[0118] In this section, the present embodiment first introduces the experimental settings. Next, the proposed method is analyzed and evaluated around the following research questions (RQs):

[0119] RQ1: Can TDB-EC achieve adaptive load balancing in different environments?

[0120] RQ2: What are the performances of P_predModel and Q_predModel?

[0121] RQ3: How much has the performance of TDB-EC improved compared with classical methods?

[0122] 3.1 Experimental Settings

[0123] First, five different scenarios were designed based on the base station distribution dataset and the latitude and longitude information of a certain place for simulation experiments, as Figure 5 shown. Among them, each edge e i is represented by a binary tuple (λ i , v i ). The number of edges N in each scenario is set to 15, and the number of adjacent edges connected to one edge satisfies 0 < c ≤ 3. The task arrival rate λ i of edge e i and the service rate v i respectively follow the normal distributions N(10,4) and N(15,6). According to the distance between base stations, the unit task transmission delay D is mapped to the interval [0.1, 0.2]. At edge e i , the fixed task amount δ i increased or decreased by the adjustment operation is set to 3% × v i .

[0124] In addition, based on queuing theory, the present embodiment simulates the average task execution time on different edges, which is defined as follows:

[0125]

[0126] Next, by simulating different scenarios, a historical dataset is collected. Among them, for global load balancing, the dataset includes λ, F cur , Tr and F obj For local load balancing, the dataset contains λ. i L i , and The dataset was randomly divided into a training set (75%) and a test set (25%). In P_predModel, a four-layer fully connected structure was used, with a hidden layer node ratio of 128:64. For Q_predModel, its DNN structure was the same as P_predModel, with a hidden layer node ratio of 512:256. The DQN algorithm was implemented using the TensorFlow 2.0.0 framework. The learning rate, discount factor γ, exploration probability ∈ R, number of rounds, and experience pool size were set to 0.001, 0.8, 0.1, 500, and 2000, respectively. Furthermore, the threshold T for the runtime decision algorithm was set to 0.15. Therefore, once the Q-value of all effective load balancing adjustments is less than or equal to the threshold T, no further operations are performed.

[0127] 3.2 RQ1: Effectiveness Assessment of TDB-EC

[0128] Table 3.1 Comparison of load rates at each edge before and after load balancing using the TDB-EC solution and the ideal solution in different scenarios.

[0129]

[0130]

[0131] First, this embodiment evaluates the effectiveness of TDB-EC in five different scenarios. More specifically, this embodiment compares the load rates of each edge node after load balancing task scheduling using the TDB-EC-derived scheme and the ideal load balancing scheme, respectively. Specifically, this embodiment evaluates the edge node load rates under different states using a simulation model of the edge node. i Average execution time T of the task a (l i It searches for ideal load balancing solutions for different scenarios in the solution space based on heuristic algorithms. However, due to its extreme complexity, it is difficult to find an ideal solution in practice.

[0132] As shown in Table 3.1, TDB-EC can obtain load balancing solutions that are quite similar to the ideal solution in different scenarios, with relatively small differences in load rates between different edges. Taking the first scenario as an example, after load balancing using the solution obtained by TDB-EC and the ideal solution respectively, the load rates of edge e6 and edge e... 15 The load rates of all components are equal. Furthermore, the differences in load rates at other edge points are less than 6%. For example... Figure 6As shown, this embodiment analyzes and compares the load balancing performance of the two schemes from the perspective of the maximum average response time of the task. TDB-EC can achieve performance close to the ideal scheme, with a difference of about 5% to 6%. The results show that the proposed method achieves near-optimal multi-edge load balancing performance (in terms of T...). max (As an evaluation metric), it can well meet the requirements of multi-edge load balancing in WMAN with different network topology configurations and various workloads.

[0133] This embodiment uses edge e5 (connected to edges e4 and e6 respectively) in scenario one as an example to illustrate the load balancing task scheduling process of TDB-EC. The fixed amount of tasks increased or decreased by δ5 = 0.35 (v5 × 3%) for each adjustment operation. Upper boundary and The upper bounds represent the maximum number of tasks scheduled to be processed on adjacent edges e4 and e6 among the tasks arriving at edge e5. As shown in Table 3.2, the initial state s1 indicates that λ5 is processed entirely on edge e5. and The percentages are 97%, 69%, and 77%, respectively. At this point, the calculation formula for the task scheduling interval can be used to obtain... and The values ​​range from [0, 1.16] to [0, 1.79], respectively. Since execution occurs in the initial state s1... and The operation caused the load rate of adjacent edges to exceed the schedulable range, so the operation was adjusted. and All are illegal operations I. Therefore, execution is performed in state s1. The operation (an adjustment operation with the minimum predicted Q value and not equal to I) is performed and the new load rate for each edge is obtained, thus entering the next state s2. For each step, a valid load balancing adjustment operation with the minimum predicted Q value is performed. Additionally, in state s9, the calculation is... The value range is [0, 2.14]. If the adjustment operation continues in state s9... The workload It becomes 2.45. This will exceed the upper bound of the corresponding value range, so the adjustment operation... It is marked as I. Therefore, in state s9, the adjustment operation is selected. In state s 12When the predicted Q-values ​​of all legitimate load balancing adjustment operations are less than the set threshold of 0.15, it indicates that the target state has been reached and no further adjustment operations are needed. It's worth noting that edge e4 and edge e6 also make independent decisions, which affects the load balancing task scheduling process of edge e5. For example, in state s1, edge e5 chooses to execute an operation... (That is, the workload is scheduled from edge e5 to edge e4), and in the next state s2, the load rate of edge e6 also changes.

[0134] Table 3.2 Load balancing process of this embodiment (edge ​​e5 of scenario one) )

[0135]

[0136]

[0137] 3.3 RQ2: Performance of P_predModel and Q_predModel

[0138] 3.3.1 Evaluation Model of DNN-based Load Balancing Scheme

[0139] First, this embodiment evaluates the accuracy of the DNN-based load balancing scheme based on the model's tolerance error ε and confidence level C. The error ε represents the predicted value f' of the task scheduling scheme. k Actual values ​​of task scheduling scheme The absolute value of the difference, and the confidence level C represents the predicted value f' within the error range ε. k The proportion of quantities.

[0140] like Figure 7 As shown, when the tolerance error of the load balancing scheme evaluation model is set to 0.15, the model's confidence level can reach over 95%. Specifically, the confidence levels of the load balancing scheme evaluation model in five different edge scenarios are 96.6%, 96.2%, 96%, 95.3%, and 95.7%, respectively. Therefore, the task scheduling interval sr obtained by centralized decision-making... k =[(f' k -0.15)·min(v i ,v j ),(f' k +0.15)·min(v i ,v j This can effectively guide decentralized decision-making at each edge node.

[0141] 3.3.2 Load Balancing Adjustment Operation Q-value Prediction Model Based on DQN

[0142] Action Accuracy (AAR) is defined as a measure of the correctness of management actions during the decision-making process, as shown below.

[0143]

[0144] in, This represents the total number of adjustment operations performed. This represents the correct adjustment operations taken. Adjustments are considered correct if they bring the current load balancing scheme closer to the target load balancing scheme.

[0145] Next, the accuracy of TDB-EC's actions in the decision-making process is studied based on the number of synchronization differences from the ideal load balancing scheme. For example... Figure 8 As shown, the Average Reduction (AAR) decreases as the current load balancing scheme approaches the ideal scheme. For example, when the current load balancing scheme is more than 10 steps away from the ideal scheme, the AAR exceeds 95%. When the current load balancing scheme is closer to the ideal scheme (i.e., 5 steps away), the AAR remains around 90%. On average, TDB-EC selects adjustment operations with an accuracy of 92.5%. Therefore, TDB-EC can always make adjustment operation decisions with a high accuracy rate with the support of Q_predModel. Only when the current load balancing scheme is less than 5 steps away from the ideal scheme does the AAR decrease significantly, but it still exceeds 85%, at which point the current load balancing scheme is already quite close to the ideal scheme. Therefore, the load balancing scheme obtained by TDB-EC can usually meet the system's correctness requirements.

[0146] 3.4 RQ3: Performance Improvements of TDB-EC Compared to Classical Methods

[0147] In this section, this embodiment compares and analyzes TDB-EC with three baseline methods: RF-CLB, ML-based, and Rule-based. The RF-CLB method employs a decentralized decision-making approach, combining reinforcement learning (Q-learning) and machine learning for load-balanced task scheduling among adjacent nodes. The state space, action space, and reward function settings of this method are consistent with TDB-EC. Compared to RF-CLB, TDB-EC combines centralized and decentralized decision-making, with the results of centralized decision-making guiding independent scheduling decisions at the edges. The ML-based method, based on predictions of the average execution time of tasks at different load rates at the edges, utilizes particle swarm optimization and genetic algorithms (PSO-GA) to search for a global load balancing scheme. The Rule-based method also employs a decentralized decision-making approach. This method, based on each edge e... i Current load rate iSelect the appropriate load balancing adjustment operation to execute load balancing task scheduling between adjacent nodes. For edge nodes... i The scheduling operation selection rules are as follows:

[0148] If l i ≥70%, increase from the edge e i The amount of tasks scheduled to the adjacent edge with the lowest load rate;

[0149] If 30% ≤ l i If the percentage is less than 70%, no adjustments will be made.

[0150] If l i <30%, reduce from the edge e i The amount of tasks scheduled to the adjacent edge with the highest load rate.

[0151] First, this embodiment uses four methods—TDB-EC, RF-CLB, ML-based, and Rule-based—to obtain load balancing schemes in five different edge scenarios. Experimental results are as follows: Figure 9 As shown, TDB-EC achieved the best results in all five scenarios. Specifically, the maximum average response time of the load balancing scheme obtained by TDB-EC was reduced by 3.69% to 4.61% compared to the RF-CLB method. This is because the centralized decision-making approach considers global information and evaluates the task scheduling interval between each pair of adjacent edges from a global perspective to guide the independent scheduling decisions of the edges. Compared to a purely decentralized decision-making approach, this improves the load balancing effect to a certain extent. The maximum average response time of the load balancing scheme obtained by TDB-EC was reduced by 4.01% to 6.34% compared to the ML-based method. In this embodiment, under the premise of allowing an error of 15%, the accuracy of the task average execution time prediction model used by the ML-based method was evaluated, and the accuracy of the model was only 88.2%. This is because the training of the prediction model requires a large amount of data to predict the average task execution time of the edges under different load rates. However, without sufficient training data, the inaccurate prediction of the model may lead to low load balancing efficiency. The maximum average response time of the load balancing solution obtained by TDB-EC is 14.46% to 16.72% lower than that of the rule-based method. This is because the rule-based method includes edge load adjustment rules customized by experts. However, the same adjustment rules cannot be used in different load balancing scenarios.

[0152] Next, this embodiment compares and analyzes the different methods used based on the average response time of all arriving tasks on each edge before and after load balancing. The calculation method is as follows:

[0153]

[0154] Experimental results are as follows Figure 10 As shown, compared to the average response time of all arriving tasks on each edge before load balancing, the average response time of all arriving tasks on each edge is reduced by an average of 43.0% after load balancing using the TDB-EC method, achieving the most significant effect. Specifically, TDB-EC achieves the most significant effect by reducing the average response time of all arriving tasks on each edge by 43.0% after load balancing task scheduling in five different scenarios. ave The T values ​​obtained after load balancing task scheduling are compared with those obtained by RF-CLB, ML-based, and Rule-based methods, respectively. ave On average, these figures decreased by 2.2%, 4.8%, and 12.9%.

[0155] In summary, the TDB-EC method efficiently achieves multi-edge collaborative load balancing (using T... max While serving as an evaluation metric, it can also effectively reduce the average response time T of all arriving tasks. ave .

[0156] Finally, this embodiment analyzes and evaluates the average time cost of four methods—TDB-EC, RF-CLB, ML-based, and Rule-based—for finding load balancing solutions. Table 3.3 shows the time cost of these four methods for finding the target load balancing solution in five different scenarios. The results show that the Rule-based, RF-CLB, and TDB-EC methods have average time costs of 0.104 seconds, 1.42 seconds, and 3.71 seconds, respectively. Therefore, from a system management perspective, these three methods can well meet the requirements. In contrast, the ML-based method has an average time cost of 239 seconds, making it difficult to adapt to multi-edge wireless metropolitan area network environments with dynamic load changes.

[0157] Table 3.3 Average time (s) to find a load balancing solution using different methods

[0158] TDB-EC RF-CLB ML-based Rule-based Scene 1 3.88 1.57 238 0.103 Scene 2 3.73 1.42 241 0.107 Scene 3 3.81 1.26 242 0.102 Scene 4 3.53 1.46 240 0.106 Scene 5 3.61 1.38 233 0.101

[0159] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

[0160] This patent is not limited to the above-described preferred embodiments. Anyone can derive other forms of multi-edge collaborative load balancing methods based on two-stage decision-making in wireless metropolitan area networks under the guidance of this patent. All equivalent changes and modifications made within the scope of this patent application shall fall within the scope of this patent.

Claims

1. A multi-edge collaborative load balancing method based on two-stage decision-making in a wireless metropolitan area network, characterized in that: Centralized load balancing decisions are made based on global information, and the task scheduling range between adjacent edges is evaluated using a prediction model based on a deep neural network. Decentralized load balancing decisions are made based on local information, and the load balancing scheme between edges is evaluated using a Q-value prediction model based on a deep Q-network. Finally, the target load balancing scheme is obtained through feedback control. Based on global information, a DNN-based load balancing scheme evaluation model is used for centralized decision-making to evaluate the task scheduling interval between each pair of adjacent edges. The specific steps are as follows: First, update the current global information, including the global load balancing scheme for the edge set. and the response time of each edge task; Secondly, based on global information, a DNN-based load balancing scheme evaluation model is used to calculate the set of task scheduling scheme evaluation values ​​between each pair of adjacent edges under the current system state. Then, calculate the set of task scheduling intervals between each pair of adjacent edges. Repeat the above process to continuously update. ; edge Based on its own local information, a distributed decision-making process is performed using a DQN-based load balancing adjustment operation Q-value prediction model. Each edge independently executes load balancing task scheduling between itself and its neighboring nodes under the task scheduling interval constraint. The specific steps are as follows: First, update the current local information, including the local load balancing scheme. Load rate of itself and adjacent edges and local task scheduling interval set Because global decisions and other peripheral decisions are made simultaneously, and It is constantly changing, therefore, the edge Reacquire data before each decision. and ,in, Obtained from the results of centralized global decision-making; secondly, , , and As input, the runtime decision algorithm is invoked to calculate the edge. The next load balancing adjustment operation 'a' will be performed. If the decision algorithm obtains 'a' as empty, it indicates that the edge is not properly balanced. A target local load balancing scheme has been found for the local system state. This indicates that no further adjustments will be made at this point; otherwise, the adjustment operation 'a' obtained from the decision algorithm will be executed, and the above process will be repeated cyclically to continuously perform load balancing scheduling operations between adjacent nodes; where, Unloading to the edge per unit time The workload; In the multi-edge collaborative load balancing method based on two-stage decision-making, global centralized decision-making and single-node decentralized decision-making are performed in parallel and independently, and a feedback control mechanism is used to gradually obtain a suitable global load balancing scheme.

2. The multi-edge collaborative load balancing method based on two-stage decision-making in wireless metropolitan area networks according to claim 1, characterized in that: The evaluation metric is the maximum average response time reaching the task across all edges, with the goal of minimizing the maximum average response time as much as possible. .

3. The multi-edge collaborative load balancing method based on two-stage decision-making in wireless metropolitan area networks according to claim 1, characterized in that: The specific execution steps of the decision algorithm include: calling the DQN-based load balancing adjustment operation Q-value prediction model Q_predModel to calculate the Q-values ​​of all actions and storing them in the set Q_values; if an action is selected... Subsequent local load balancing scheme Is it illegal or related to workload? Not in the task scheduling range If the action is invalid, it is considered an illegal action, and the corresponding Q value is marked as I. Then, a predefined threshold T is used to determine whether the target load balancing solution has been found. If the Q values ​​of all valid actions except those marked with I are less than or equal to the threshold T, the target solution is considered to have been reached, and no further adjustment operation is required. Otherwise, the adjustment operation with the smallest Q value will be selected. Finally, the obtained adjustment operation a is returned.

4. The multi-edge collaborative load balancing method based on two-stage decision-making in wireless metropolitan area networks according to claim 1, characterized in that: The input to the DNN-based global load balancing scheme evaluation model is the task arrival rate of the edge set. Current global load balancing scheme and current task response time The output is the task scheduling scheme for each pair of adjacent edges; Using undirected topological graphs Indicating wireless metropolitan area network The connection relationships of each edge; where the edge set For the vertex set of a topological graph, set Represents the edge set of a topological graph, containing Edge; the first edge in the topological graph Strip edge Indicates edge With the edge The network connection between them is specified in each pair of adjacent edges. and Middle, edge The task scheduling direction is positive, meaning tasks originate from the edge. Dispatch to the edge The direction is positive; conversely, the task starts from the edge. Dispatch to the edge The direction is negative, thus affecting the global load balancing scheme. Convert to ; Based on historical operational data, a global load balancing scheme evaluation model is trained using a DNN: First, regarding the amount of work completed. Task scheduling volume Normalization is performed as follows: in, Represents the normalized edge The mission was completed; Indicates edge and edge Task scheduling volume between This represents the normalized task scheduling amount. Represents the edge Service speed; The trained load balancing scheme evaluation model is based on the edge task arrival rate. Current load balancing scheme and current task response time For the target load balancing scheme Predictions are made, and a confidence model is used to further describe the value range of the task scheduling scheme, as shown below: in, Indicates the confidence level. This represents the error of the model. This represents the predicted value of the task scheduling scheme. This represents the actual value of the task scheduling scheme. have The probability falls within the interval Inside; Based on the confidence model and normalization process described above, the reasonable range of values ​​for task scheduling between each pair of adjacent edges is represented as a set. ,edge and edge The task scheduling interval is calculated as follows: 。 5. The multi-edge collaborative load balancing method based on two-stage decision-making in a wireless metropolitan area network according to claim 1, characterized in that: In the DQN-based load balancing adjustment operation Q-value prediction model: An edge The collected local information includes edges Task arrival rate It means the edge and the task load rate of adjacent edges It means the edge The current local load balancing scheme uses The average response time for the corresponding task is represented by... Representation and Edge Optional local load balancing solutions The average response time for the corresponding task is represented by... This means that, based on the collected local information, each edge independently performs load balancing scheduling operations between adjacent edges; edge and A set of adjacent edge connections is defined as follows: ;in, Indicates the edge The connected first Each edge has a task that arrives at a single edge, which can only be processed locally or dispatched to an adjacent edge. ,in and Representing the edges and edge Load rate; ,in and They represent the edges respectively and edge The amount of tasks executed in the arrival task; based on After task scheduling, calculate the corresponding Of all the options for the operating environment, edge computing is the best choice. The target local load balancing scheme is to have the minimum The solution, using Representation; in addition, using sets Indicates edge The set of task scheduling intervals between adjacent edges, where Indicates edge With adjacent edges The task scheduling interval; edge The historical dataset consists of operational data under different system states, which includes... and The DQN algorithm evaluates the Q-values ​​of all adjustment operations under different system states based on the historical dataset of the edge; specifically, the system states of the edge are... Adjustment operation to increase / decrease from the edge The amount of tasks scheduled to the adjacent edge in the arrival task; when the target solution is found. Then the corresponding reward will be generated; The DQN agent first receives the status of the runtime edge environment. And select actions through a greedy strategy. Next, the DQN agent received the corresponding reward. and reach the next state. In each step, The parameters of the DNN are stored in an experience pool with a fixed capacity; if the threshold capacity is reached, the parameters of the DNN are adjusted. It will be updated, and the loss function is defined as follows: Where γ is the discount factor. These are the weights of the DNN. It is the current Q value. Then it is in the next state Select Action The maximum Q value; Based on the problem definition of multi-edge collaborative load balancing in WMAN, it is described as follows: State space: edge The state space is represented as ,in Represents a potential state; Defined as a triple If using the scheme This causes the load at an edge to exceed its adjustable range, then It is considered an invalid state; therefore, This is an illegal scheme; Action Space: Edge Action space is defined as One action is to increase or decrease from the edge. The adjustment operation for the amount of tasks scheduled to the adjacent edge in the arrival task; wherein, the amount of tasks scheduled each time is a fixed value. ; Is it adding edges? Dispatch to the edge The workload; State transition function: The state transition function is defined as follows: It returns the execution The next state after; Reward function: To guide the DQN agent to find the target local load balancing solution. The reward function is defined as: If in state Execute action a to find DQN agents can receive a reward of 10; if in status If action 'a' results in an illegal outcome, the reward will be set to -1 as a penalty; otherwise, the reward remains 0. DQN determines a new Q value based on the Q value estimated by the DNN and the backpropagation of the reward, and then feeds the Q value into the loss function to update the parameters of the DNN. During DQN training, the target Q-value is processed as follows: if This load balancing adjustment operation will be considered an illegal operation, and the corresponding target Q value will be... It is set to 1 and will not be stored in the DQN experience pool; when hour, Found In other cases, ,in, Indicates edge e i The current local load balancing scheme.

6. The multi-edge collaborative load balancing method based on two-stage decision-making in a wireless metropolitan area network according to claim 5, characterized in that: The training process of the DQN-based load balancing adjustment operation Q-value prediction model is as follows: First, the parameters of DQN are randomly initialized, and then the training process begins; in each round, the current state is initialized according to the dataset. and target state ;when ,by The probability is used to randomly select action 'a', otherwise... Choose action a; then, after executing action a, calculate the reward r according to the reward function and obtain the next state. ; If after performing action a, If yes, then skip this round of training; otherwise, The samples will be stored in the experience pool; then, m samples will be drawn from the experience pool in a batch and calculated. Then, the network weights are updated using the loss function. In every K iterations, use renew After that, the status and state A state transition occurs between them; through continuous training, it eventually converges.

Images