Power communication network routing planning method based on reinforcement learning

Abstract

The invention discloses an electric power communication network route planning method based on reinforcement learning, relates to the technical field of electric power communication networks, and solves the problem that load and risk balance of electric power communication network route planning cannot be accurately realized due to the fact that an existing electric power communication network route planning method is single in consideration factor. The method comprises the following steps: constructing a power communication network topology model G and a power communication network service model H; constructing a load balancing and risk balancing optimization model of the power communication network; constructing a PDQN network model, and performing path selection in the Pareto optimal solution set; and training the PDQN network model, and finally outputting a routing decision of a service demand. According to the method, the limitation that a traditional method mainly pays attention to a single target is avoided, the network load balance degree and the risk balance degree can be effectively reduced, the service route planning requirement of the power communication network is met, and the overall performance and reliability of the network are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of electric power communication networks, and in particular to a method for planning routing of electric power communication networks based on reinforcement learning. Background Art

[0002] With the rapid development of smart grids, the power communication network, as the "nerve center" of the power system, undertakes key tasks such as real-time monitoring, data collection, and remote control. The modern power system is developing towards informatization, automation, and intelligence, which puts higher requirements on the reliability, real-time and security of the communication network. The power communication network needs to support the access of massive terminal devices and ensure the reliable transmission of various business data.

[0003] In the power Internet of Things scenario, a large number of terminal devices need to transmit the collected data to the control center for processing. These terminal devices are widely distributed, which puts higher requirements on the dynamic resource scheduling of the network.

[0004] However, in actual operation, the power communication network faces many challenges: first, the uneven distribution of network load may cause congestion in some links, affecting the communication quality; second, once some key links fail, it will seriously affect the reliability of the network; third, the unreasonable distribution of network security risks may reduce the overall security of the system. These problems may be more prominent in critical periods of the power system, such as peak loads or extreme weather conditions.

[0005] Traditional routing planning algorithms often only focus on optimizing a single objective, such as the shortest path or minimum delay, and are unable to meet the diverse business needs of power communication networks. How to achieve load balancing and risk dispersion while ensuring efficient network operation has become a key issue that needs to be solved urgently.

[0006] In summary, the method of the present invention is dedicated to solving the routing planning problem in the power communication network, and proposes a routing planning method that can achieve load and risk balancing. Summary of the invention

[0007] The present invention provides a power communication network routing planning method based on reinforcement learning to solve the problem that the existing power communication network routing planning method considers only a single factor, resulting in the inability to accurately achieve load and risk balance in the power communication network routing planning.

[0008] A power communication network routing planning method based on reinforcement learning is implemented by the following steps:

[0009] Step 1: construct a power communication network topology model G and a power communication network service model H;

[0010] Step 2: Construct a load balancing and risk balancing optimization model for the power communication network;

[0011] Step 3: Based on the power communication network topology model G and power communication network business model H constructed in step 1 and the power communication network load balancing and risk balancing optimization model constructed in step 2, a PDQN network model is constructed, and the state space S and action space A are defined. At each decision moment, the system selects a routing action from the action space A according to the current state to implement a load balancing and risk balancing routing strategy. Finally, a path is selected in the Pareto optimal solution set.

[0012] Step 4: Train the PDQN network model and finally output routing decisions based on business requirements.

[0013] Beneficial effects of the present invention: The routing planning method described in the present invention fully considers the characteristics of the power communication network, including the importance and weight factors of different services; through interaction with the environment and experience accumulation, it can automatically learn the optimal routing strategy and has strong adaptive learning ability; secondly, through reasonable design of the reward function, it can simultaneously consider multiple optimization goals such as load balancing and risk balancing, avoiding the limitation of traditional methods that mainly focus on a single goal, and can effectively reduce the network load balance and risk balance, meet the power communication network service routing planning needs, and aim to improve the overall performance and reliability of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is the communication subnet topology diagram;

[0015] Figure 2 It is a flow chart of the method for routing planning of a power communication network based on reinforcement learning according to the present invention;

[0016] Figure 3 This is a flow chart of the training process of the PDQN network model;

[0017] Figure 4 A comparison diagram of load balancing results in the power communication network routing planning method based on reinforcement learning according to the present invention;

[0018] Figure 5 A comparison diagram of risk balance results in the power communication network routing planning method based on reinforcement learning according to the present invention;

[0019] Figure 6 This is a comparison chart of service blocking rate results in the power communication network routing planning method based on reinforcement learning described in the present invention. DETAILED DESCRIPTION

[0020] Combination Figures 1 to 6The present embodiment is described as follows: a method for routing planning of a power communication network based on reinforcement learning, which is implemented by the following steps:

[0021] Step 1: Construct an urban power communication network model, including a power communication network topology model and a power communication network business model.

[0022] Step 1: Obtain the topology and link characteristic information of the power communication network from the communication station data and optical cable link data of a certain urban area power communication network, and construct a power communication network topology model G = (V, E, W), where:

[0023] V={v i |i=1,2,...,N} represents the node set of the communication station, v i is the communication site in the network, and N is the number of communication sites, including 500kV substation and 220Kv substation.

[0024] E={e ij |v i ,v j ∈V,i≠j,e ij =e ji} represents the set of optical cable links between communication stations, e ij Indicates that from v i to v j Is there a link between them? If v i to v j If there is a link between them, then e ij =1, otherwise e ij =0.

[0025] W is the attribute set of the edges in E, where each edge contains two attributes: bandwidth capacity B and physical length D. Where B = {b ij |e ij ∈E} represents edge e ij The bandwidth capacity of the corresponding link; D = {d ij |e ij ∈E} represents edge e ij The physical length of the corresponding link.

[0026] Step 12: Based on the service requirements and service characteristics of the urban power communication network, construct the power communication network service model H = (T, O, F, I, C, Q); where:

[0027] T={t k |k=1,2,...,M} is the set of business requirements, where t k represents the kth communication service demand, M is the total number of service demands, and for any service t k ∈T.

[0028] O={vko |v ko ∈V} represents the source node set, where v ko is the source node of the kth communication service demand;

[0029] F={v kf |v kf ∈V} represents the destination node set, where v kf is the destination node of the kth communication service demand;

[0030] I={i k |i k ∈T,0 k <1} represents the importance set of the business, i k is the business importance of the kth communication business requirement;

[0031] C={c k |c k ∈T} represents the bandwidth requirement set of the service, c k The bandwidth capacity required for the kth communication service demand;

[0032] Q = {q k |q k ∈T} represents the weight of the business, q k is the weight of the kth communication service demand, which is the inverse of the service proportion.

[0033] In this implementation, the importance of the service is determined according to the service's requirements for transmission and its impact on the power grid, and the services are divided into five categories according to the carrying mode, namely: relay protection, safety and stability system, dispatch telephone, information management and office automation. The service importance i, service bandwidth requirement c and service weight q of each type of service are specifically defined, as shown in Table 1. Table 1 shows the service attributes of the power communication network.

[0034] Table 1

[0035]

[0036] Step 2: Construct a load balancing and risk balancing optimization model for the power communication network;

[0037] In this implementation, in order to achieve the load balancing and risk balancing goals of the urban power communication network routing planning, the load balancing degree L and the risk balancing degree R are defined.

[0038] The process of calculating the load balancing degree L is:

[0039] First, it is known that Bij′ and Bij are the ij The used capacity and total capacity on link e ij The bandwidth usage L ij Expressed as:​

[0040]

[0041] Then, according to L ij The average bandwidth occupancy rate of the entire network link L is calculated by the number of communication sites N avg :

[0042]

[0043] Finally, according to L avg Calculate the load balancing degree L of the power communication network:

[0044]

[0045] The specific process of calculating the risk balance R is:

[0046] First, it is known that i k Link e ij The importance of the k-th business, q k Link e ij The weight of the kth business, K ij is the set of all services passing through the link, then link e ij The risk value R ij It is expressed as:

[0047]

[0048] Then, according to R ij Calculate the average risk R of the entire network link based on the number of communication sites N avg :

[0049]

[0050] Finally, according to R avg Calculate the risk balance R of the power communication network:

[0051]

[0052] Step 3: Establish a reinforcement learning-based urban power communication network routing planning model (PDQN network model);

[0053] As the nerve center of the power system, the routing planning of the power communication network needs to consider the two goals of load balancing and risk balancing in the network. Due to the dynamic changes in the network state, the routing planning problem of the power communication network is modeled as a multi-objective Markov decision process (MOMDP). The state space S represents the state of each node in the network; the action space A includes the set of all feasible routing paths from the source node to the target node. To solve this MOMDP problem, based on the power communication network environment, the Pareto method is used to expand the deep Q network (DQN) to form a PDQN network model. At each decision moment, the system selects a routing action from the action space A according to the current state to achieve a routing strategy of load balancing and risk balancing. The specific implementation process is:

[0054] Step 3-1, construct the initial environment of the PDQN network model;

[0055] According to the power communication network topology model G and the power communication network business model H, the initial environment of the PDQN network model is constructed. The initial environment includes: state space S, action space A, and reward function PR.

[0056] First: define the state space S: the system obtains environmental information based on the selected action and uses the bandwidth occupancy rate L of the link in the network ij and risk value R ij Define the state space S, which consists of the following three parts:

[0057] Define the link load state vector WL = [wl ij ], where wl ij Indicates link e ij The bandwidth utilization rate of link e ij Current usage of bandwidth resources.

[0058] Define the link risk state vector as WR = [wr ij ], where wr ij Indicates link e ij The risk value of link e ij Current risk situation.

[0059] Node position encoding P: defined as a one-hot vector NP, as shown in the formula.

[0060] NP=[np n |np n ∈{0,1},n=1,2,...,N]

[0061] In the routing decision process, for the business needs being processed, k The source node v ko and destination node vkf The element at the position is 1, and the elements corresponding to other nodes are 0.

[0062] Then, define the action space A:

[0063] After the system obtains the state space S, it will generate corresponding routing decision actions. The action space A corresponds to the set of all feasible paths from the source node to the destination node for the communication service demand being processed in the routing decision process. k The action space A(k) consists of the source node v ko To the destination node v kf All simple paths p for business requirements t k , the action space A(k) is defined as:

[0064] A(k)={p|p=(v ko ,v 1 ,...v kf )∈P(v ko ,v kf )}

[0065] Finally, the reward function PR: After taking action a in state s, the environment enters the next state s' and returns a reward value. The reward function PR is expressed as follows:

[0066] PR=W1×L_R+W2×R_R

[0067] In the formula, L_R and R_R are defined as the reciprocals of the load balance degree L and the risk balance degree R. W1 and W2 are the weight coefficients of the load balance degree and the risk balance degree, respectively, and both are 0.5.

[0068] Step 3-2, PDQN network model design: In order to achieve the joint balancing goal of load and risk, a PDQN network model with an independent output layer is designed.

[0069] Step 3-2-1, according to the maximum load balancing Q value maxQL(s',a') of all possible actions a' in the next state s' and whether the terminal state done is reached, define the load balancing Q value QL(s,a) of state s executing action a, which can be expressed as follows:

[0070] QL(s,a)=L_R+γmaxQL(s',a')(1-done)

[0071] Where γ is a discount factor, which is used to weigh the importance of immediate rewards and future rewards, and its value range is [0,1].

[0072] Step 3-2-2, according to the maximum risk equilibrium Q value maxQR(s',a') of all possible actions a' in the next state s' and whether the terminal state done is reached, define the risk equilibrium Q value QR(s,a) of state s executing action a, which can be expressed as follows:

[0073] QR(s,a)=R_R+γmaxQR(s',a')(1-done)

[0074] Step 3-3, action selection strategy: Based on the PDQN environment, the current state s is known, and the current business demand (v ko ,v kf ,c k ) to select an action.

[0075] Step 3-3-1, path feasibility test; for each business requirement (v ko ,v kf ,c k ) on each link (u,v) on all simple paths, perform path feasibility check:

[0076] b ij ' <c k

[0077] Among them, b ij ' indicates link e ij remaining capacity.

[0078] Step 3-3-2, load and risk balance calculation: In the current state, for each feasible action a, calculate the load balance L(s,a) and risk balance R(s,a) after allocating bandwidth.

[0079] Step 3-3-3, Pareto front construction; define the solution vector S_a of action a, as shown in the formula.

[0080] S_a=(L(s,a),R(s,a))

[0081] For any action a1, a2∈A in the current state s, if L(s, a1) ≥ L(s, a2) and R(s, a1) ≥ R(s, a2) are satisfied, and at least one inequality holds, then the value of action a1 is said to dominate the value of action a2.

[0082] Step 3-3-4: Define the Pareto optimal solution set as the set of all non-dominated values ​​in the current state s:

[0083]

[0084] The ε-greedy strategy is expressed as shown in Eq.

[0085]

[0086] Where: As is the action selected in state s; argmaxQ(s,a) is the action a corresponding to the maximum Q(s,a value in state s; ε is the exploration probability, with an initial value of ε 0 During the training process, as the number of training rounds es increases, the decay factor δ gradually decreases, that is, ε es =ε 0 ×δ es .

[0087] Step 3-3-5: Select a path in the Pareto optimal solution set; the specific process is as follows:

[0088] First, the exploration phase (probability ε); randomly select a path p from the non-dominated value set F(s) under the current state s, as shown in the formula.

[0089] p = random(F(s))

[0090] Then, using the stage (probability 1-ε); in the non-dominated value set F(s) under the current state s, the weighted load balance Q value QL and the risk balance Q value QR define a dual-objective Q value Q(s,a), and select the path with the maximum weighted sum of Q(s,a), which is expressed as follows:

[0091] Q(s,a)=W1×QL+W2×QR

[0092] Among them, W1 and W2 are the weight coefficients of QL and QR respectively.

[0093] Step 4: Model the joint balanced routing planning problem of load and risk in the power communication network as a multi-objective Markov decision process, and train the PDQN network model in step 3, that is, solve the network model described in step 3. The flow chart is as follows: Figure 3 As shown, the specific training process is:

[0094] Step S1, initializing network parameters;

[0095] Initialize the PDQN network parameters, including learning rate learning_rate, discount factor gamma, initial exploration rate epsilon_start, minimum exploration rate epsilon_end, exploration rate decay epsilon_decay, experience replay pool size memory_size, batch size batch_size, target network update frequency target_update, load balancing weight w1, and risk balancing weight w2.

[0096] Step S2: input network topology and service sequence to be processed;

[0097] Step S3, determine whether the maximum number of rounds has been reached, if yes, execute step S8, otherwise, execute step S4;

[0098] Step S4, resetting the environment to obtain the PDQN initial environment;

[0099] Step S5: perform the following operations for each business requirement;

[0100] Get the current network status and service request;

[0101] Find all feasible paths and construct the Pareto frontier;

[0102] Select actions based on the ε-greedy strategy;

[0103] Step S6, interact with the environment to obtain the next state and reward value;

[0104] Step S7: storing the experience in the replay pool, and performing training when there are enough samples in the replay pool, including: sampling from the experience replay pool, calculating the target Q value, and updating the network parameters;

[0105] Step S8: Use the trained network to output the routing decision of the service.

[0106] In this embodiment, the performance of the proposed method is evaluated by simulation analysis and compared with the performance of the benchmark algorithm. The model of the present invention is to reduce the load balance and risk balance to optimize the target and perform routing planning based on the urban power communication network service.

[0107] The method proposed in this implementation mode is simulated and verified in a power communication network subnet in a certain city of a certain province. The network topology is as follows: Figure 1 shown.

[0108] like Figures 4 to 6 As shown, Figures 4 to 6 The comparison of the result data based on different service routing algorithms shows the performance comparison results of different algorithms in three aspects: load balance, risk balance, and service blocking rate. Through data comparison, it can be found that the PDQN method of this implementation method combined with the improved load and risk balance indicators can have the advantages of reducing the load balance and risk balance of the power communication network, and the effect is more obvious than other methods.

[0109] Figures 4 to 6 A comparison of routing planning results under different business requirements is given among three methods: KSP (K shortest path), LRJB (shortest path with load and risk as weights), and PDQN.

[0110] from Figure 4 and Figure 5The results show that under two different groups of business numbers, the PDQN method performs well in terms of load balance and risk balance. For the first group of business, the load balance and risk balance values ​​of the PDQN method are 28% and 32.63% lower than those of LRJB, and 43.75% and 47.01% lower than those of KSP. The PDQN method can better adapt to changes in network status and obtain routing decisions with better load balance and risk balance.

[0111] from Figure 6 The service blocking rate results of the three methods show that when the number of services is 400, the three methods can handle service needs well and no service blocking occurs. When the number of services increases to 500, the service blocking rates of the PDQN and LRJB methods are both 2%, which is lower than the 3% of the KSP method. Combined with the previous load balancing and risk balancing results, PDQN can not only achieve better resource balancing, but also maintain a low service blocking rate under high load scenarios. This shows that the PDQN method can better balance the relationship between network resource utilization and service assurance, and is more suitable for application in the business environment of power communication networks.

[0112] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0113] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A method for routing planning of a power communication network based on reinforcement learning, characterized in that: The method is implemented by the following steps: Step 1: construct a power communication network topology model G and a power communication network service model H; Step 2: Construct a load balancing and risk balancing optimization model for the power communication network; Step 3: Based on the power communication network topology model G and power communication network business model H constructed in step 1 and the power communication network load balancing and risk balancing optimization model constructed in step 2, a PDQN network model is constructed, and the state space S and action space A are defined. At each decision moment, the system selects a routing action from the action space A according to the current state, and selects a path in the Pareto optimal solution set, and finally realizes a routing strategy of load balancing and risk balancing. Step 4: Train the PDQN network model and finally output routing decisions based on business requirements.

2. The method for routing planning of a power communication network based on reinforcement learning according to claim 1, characterized in that: In step 1, the power communication network topology model G=(V, E, W), where: V={v i |i=1,2,...,N} is the node set of the communication station, v i is the communication site in the network, and N is the number of communication sites; E={e ij |v i ,v j ∈V,i≠j,e ij =e ji } is the set of optical cable links between communication stations, e ij Indicates that from v i to v j Is there a link between them? If v i to v j If there is a link between them, then e ij =1, otherwise e ij =0; Let W be the attribute set of the edges in E, where each edge includes two attributes: bandwidth capacity B and physical length D; where B = {b ij |e ij ∈E} represents edge e ij The bandwidth capacity of the corresponding link; D = {d ij |e ij ∈E represents edge e ij The physical length of the corresponding link.

3. The method for routing planning of a power communication network based on reinforcement learning according to claim 1, characterized in that: In step 1, the power communication network service model H=(T, O, F, I, C, Q); wherein: T={t k |k=1,2,...,M} is the set of business requirements, where t k represents the kth communication service demand, M is the total number of service demands, and for any service t k ∈T; O={v ko |v ko ∈V} represents the source node set, where v ko is the source node of the kth communication service demand; F={v kf |v kf ∈V} represents the destination node set, where v kf is the destination node of the kth communication service demand; I={i k |i k ∈T,0 k <1} represents the importance set of the business, i k is the business importance of the kth communication business requirement;​ C={c k |c k ∈T} represents the bandwidth requirement set of the service, c k The bandwidth capacity required for the kth communication service demand; Q = {q k |q k ∈T} represents the weight of the business, q k is the weight of the kth communication service demand, which is the inverse of the service proportion.

4. The method for routing planning of a power communication network based on reinforcement learning according to claim 1, characterized in that: In step 2, the load balance L and risk balance R are defined; The process of calculating the load balancing degree L is: First, it is known that Bij′ and Bij are the ij The used capacity and total capacity on link e ij The bandwidth utilization rate L ij Expressed as: Then, according to L ij Calculate the average bandwidth occupancy rate of the entire network link L using the number of communication sites N avg : Finally, according to the bandwidth occupancy mean L avg Calculate the load balancing degree L of the power communication network:

5. The method for routing planning of a power communication network based on reinforcement learning according to claim 4, characterized in that: In step 2, the specific process of calculating the risk balance R is: First, according to i k Link e ij The business importance of the kth communication business demand, q k Link e ij The weight of the kth communication service demand, K ij For link e ij The set of all communication services of link e ij The risk value R ij It is expressed as: Then, according to the risk value R ij Calculate the average bandwidth occupancy rate R of the entire network link based on the number of communication sites N avg : Finally, according to the bandwidth occupancy mean R avg Calculate the risk balance R of the power communication network:

6. The method for routing planning of a power communication network based on reinforcement learning according to claim 5, characterized in that: The specific process of step three is: Step 31: Construct the initial environment of the PDQN network model; the initial environment includes the state space S, the action space A and the reward function PR; Step 32: Design a PDQN network model with an independent output layer; Step 33: Action selection strategy; According to the current state s, the current business request (v ko ,v kf ,c k ) to select actions and finally achieve path selection in the Pareto optimal solution set.

7. The method for routing planning of a power communication network based on reinforcement learning according to claim 6, characterized in that: The specific process of step 31 is: First, define the state space S: use the bandwidth occupancy L of the links in the network ij and risk value R ij Define the state space S, which consists of the following three parts: Define the link load state vector WL = [wl ij ], where wl ij Indicates link e ij The bandwidth occupancy rate of the link; define the link risk state vector as WR = [wr ij ], where wr ij Indicates link e ij The risk value of Node position encoding P: defined as a one-hot vector NP, as follows: NP=[np n |np n ∈{0,1},n=1,2,...,N] In the routing decision process, for the business needs being processed, k The source node v ko and destination node v kf The element at the position is 1, and the elements corresponding to other nodes are 0; Then, the action space A is defined; after the system obtains the state space S, it will generate corresponding routing decision actions; the action space A corresponds to the communication service demand being processed in the routing decision process t k From the source node v ko To the destination node v kf The set of all feasible paths; Finally, set the reward function PR; After taking action a in the current state s, the environment enters the next state s' and returns the reward value. The reward function PR is as follows: PR=W1×L_R+W2×R_R Where L_R and R_R are defined as the reciprocals of the load balance degree L and the risk balance degree R, and W1 and W2 are the weight coefficients of the load balance degree and the risk balance degree, respectively.

8. The method for routing planning of a power communication network based on reinforcement learning according to claim 6, characterized in that: The specific process of step 32 is: According to the maximum load balancing Q value maxQL(s',a') of all possible actions a' in the next state s' and whether the terminal state done is reached, the load balancing Q value QL(s,a) of state s executing action a is defined, which can be expressed as follows: QL(s,a)=L_R+γmaxQL(s',a')(1-done) In the formula, γ is the discount factor; According to the maximum risk equilibrium Q value maxQR(s',a') of all possible actions a' in the next state s' and whether the terminal state done is reached, the risk equilibrium Q value QR(s,a) of state s performing action a is defined, which can be expressed as follows: QR(s,a)=R_R+γmaxQR(s',a')(1-done).

9. The method for routing planning of a power communication network based on reinforcement learning according to claim 6, characterized in that: The specific process of step 33 is: First, the path feasibility test is performed. According to the current state s, for each execution action a, the load balance degree L(s, a) and risk balance degree R(s, a) after bandwidth allocation are calculated. Secondly, construct the Pareto optimal solution set; Define the solution vector S_a of action a. For any action a1, a2∈A in the current state s, if L(s, a1) ≥ L(s, a2) and R(s, a1) ≥ R(s, a2) are satisfied, and at least one inequality holds, then the value of action a1 is said to dominate the value of action a2. Then, define the Pareto optimal solution set as the set of all non-dominated values ​​F(s) in the current state s: The ε-greedy strategy is expressed as: Where As is the action selected in state s; argmaxQ(s,a) is the action a corresponding to the maximum Q(s,a value in state s, and ε is the exploration probability; Finally, path selection is performed in the Pareto optimal solution set.

10. The method for routing planning of a power communication network based on reinforcement learning according to claim 9, characterized in that: The specific process of path selection in the Pareto optimal solution set is: Exploration phase: Randomly select a path p from the non-dominated value set F(s) under the current state s, as shown in the following formula: p = random(F)s)) Utilization stage: In the non-dominated value set F(s) under the current state s, the load balance Q value QL and the risk balance Q value QR are weighted into a dual-objective Q value Q(s,a), and the path with the largest weighted sum of Q(s,a) is selected, which can be expressed as follows: Q(s,a)=W1×QL+W2×QR Where W1 and W2 are the weight coefficients of QL and QR respectively.

Images