Communication equipment multi-link automatic switching control method based on reinforcement learning

Abstract

The invention relates to the field of communication, in particular to a reinforcement learning-based control method for multi-link automatic switching of communication equipment. The core of the method is that a Q-learning algorithm is adopted to train an intelligent agent. According to the method, through continuous interaction between an intelligent agent and an environment, firstly, a multi-dimensional decision strategy capable of comprehensively balancing bandwidth, time delay, switching times and cost is learned, and a switching parameter Q table is generated; in turn, this Q table is queried directly in the communication device to perform real-time handover decisions. The Q table supports enhanced training generation and also supports continuous optimization in use, so that the system can dynamically adapt to network environment changes, and optimal balance of communication functions is always guaranteed.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of communication, in particular to a control method for multi-link automatic switching of communication equipment based on reinforcement learning. BACKGROUND

[0002] In the field of communication, wired networks are widely used due to their advantages such as strong stability, high transmission rate and low cost, but they have poor mobility and are greatly limited by wiring. Mobile public networks, as the core of ground communication infrastructure, have the advantages of low deployment cost, high throughput, millisecond-level latency and good compatibility with standardized protocols. However, they are highly dependent on ground infrastructure and are vulnerable in the face of natural disasters, and are limited by the coverage density of cellular base stations, making it difficult to achieve continuous coverage in remote areas. Satellite networks rely on space segment platforms and have wide coverage, strong invulnerability and are not limited by geographical environment, which can effectively supplement the coverage blind area of ground networks. However, the long distance between satellites and the ground, limited spectrum resources, high transmission delay of up to hundreds of milliseconds, limited channel capacity and high operating costs make it difficult to achieve continuous coverage in remote areas.

[0003] In the face of all-round communication support needs, the industry consensus is to build communication equipment that supports wired, mobile public network and satellite multi-mode access, and to ensure business continuity through heterogeneous network fusion. However, multi-network collaboration still faces a key technical challenge: how to establish an automatic switching mechanism for network links to achieve the unity of communication continuity, stability and economy.

[0004] Existing research on wired / mobile public network / satellite multi-mode automatic switching still has some problems. First, the accuracy of the switching parameters is insufficient, mainly relying on engineers to configure based on experience, making it difficult to achieve precise adaptation. Second, different devices generally use the same set of switching parameters, which cannot adapt to different latitudes, climates and geographical environments, resulting in mismatch between parameters and actual scenarios. Therefore, how to generate adaptive switching parameters for devices through their own reinforcement learning based on specific environmental conditions has become a key problem that needs to be solved. SUMMARY

[0005] To overcome the problem of inaccurate parameter configuration and lack of environmental adaptability in multi-mode network (wired / mobile public network / satellite) switching of existing communication equipment, the present application proposes a control method for multi-link automatic switching of communication equipment based on reinforcement learning. This method aims to improve the adaptability of communication equipment in different environments through intelligent decision-making, thereby comprehensively ensuring the continuity, stability and economy of communication.

[0006] The technical solution adopted by the present application is as follows:

[0007] A control method for multi-link automatic switching of communication equipment based on reinforcement learning, comprising the following steps:

[0008] S1: simultaneously establishing and maintaining three independent communication links in the communication device, including wired network, mobile public network and satellite network;

[0009] S2: creating a state-action value table, i.e. Q table, the state s is determined according to the real-time signal quality of the three links, the action a is the decision of network switching, the Q table is used to store the long-term value evaluation value of the state-action pair, the Q-learning update function and the reward function are designed, and the Q table value and the Q-learning update function and the reward function parameters are initialized;

[0010] S3: real-time data acquisition, including link signal quality, link available bandwidth and link communication delay;

[0011] S4: observing the current state s, using the ε-greedy strategy to select and execute the action a; wherein the exploration rate ε is set to the initial value, and decreases with the increase of the training step number;

[0012] S5: after executing the action, switching to the corresponding network, collecting new network state data, updating the link performance indicators, and calculating the instant reward obtained by the action according to the reward function;

[0013] S6: using the Q-learning update formula, updating the corresponding Q value in the Q table according to the obtained reward r and the new state;

[0014] S7: updating the current state to the new state, and repeating S4 to S6, and iterating until the Q table converges;

[0015] S8: in the actual application stage, the current state is determined by real-time data, and the action with the highest Q value is selected as the switching decision by querying the Q table, so as to realize the optimal automatic switching of the link.

[0016] Further, in S2, the Q-learning update function is:

[0017]

[0018] In the formula, is the current Q table, which represents the state at time t, the action is executed, and the corresponding original Q table is is the learning rate, the value range is [0, 1], which represents the degree of new information covering the information; is the instant reward, which represents the instant reward feedback by the environment after executing the action . is the discount factor, with value range [0, 1], the smaller the parameter, the more concerned about the immediate reward, the larger the parameter, the more concerned about future rewards; indicates entering a new state , the maximum Q value corresponding to all possible actions after

[0019] Parameter initialization: set the learning rate and the discount factor ;

[0020] The reward function is:

[0021]

[0022] wherein,

[0023] is the basic reward:

[0024]

[0025] wherein, is the signal quality, i.e. the signal-to-noise ratio;

[0026] is the basic reward weight, with a range of [0, 1];

[0027] is the bandwidth reward:

[0028]

[0029] wherein, is the actual bandwidth, is the required bandwidth;

[0030] is the bandwidth reward weight, with a range of [0, 1];

[0031] is the delay penalty:

[0032]

[0033] wherein, is the typical delay of the current signal, is the actual delay;

[0034] is the cost penalty:

[0035]

[0036] is the switching penalty:

[0037]

[0038] Penalty for interruption:

[0039] ;

[0040] Parameter initialization: base reward weight and bandwidth reward weight .

[0041] The advantages of the present application compared with the prior art are:

[0042] (1) Global coverage: the present application effectively utilizes the wide area coverage characteristics of satellite communication, breaks through the geographical limitations of wired and mobile public networks, and realizes seamless communication in remote areas, oceans and air scenes.

[0043] (2) High reliability and continuity: the present application realizes automatic and non-sensing switching to the optimal link through intelligent rapid link switching mechanism when single network performance deteriorates or is interrupted, greatly improving the reliability and service continuity of communication.

[0044] (3) Dynamic optimal performance: the present application can adaptively learn and execute the optimal switching strategy under the current environment based on Q-learning data-driven decision, so as to guarantee the best user experience in bandwidth, delay, stability and other dimensions.

[0045] (4) Cost-effectiveness maximization: the cost penalty mechanism embedded in the reward function of the present application guides the system to preferentially select the ground network with lower cost under the premise of meeting performance requirements, significantly reduces the high cost of satellite link, and realizes the optimization of communication economy. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is the system topology diagram of the present application.

[0047] Figure 2 is the control method flow chart of the present application communication equipment multi-link automatic switching. DETAILED DESCRIPTION

[0048] The present application will be further explained and described below in conjunction with the drawings.

[0049] As shown in Figure 2 , a control method for communication equipment multi-link automatic switching based on reinforcement learning comprises the following steps:

[0050] S1: Multi-link parallel creation and maintenance:

[0051] Three independent communication links are simultaneously established and maintained in the communication device, respectively accessing wired network, mobile public network and satellite network. The architecture ensures that when network switching is needed, the link can quickly take over the communication task, providing seamless continuity guarantee for business. As shown in Figure 1

[0052] S2: Based on Q-learning intelligent decision-making framework:

[0053] The method adopts Q-learning algorithm as the core decision model, and the key elements are designed as follows:

[0054] State space:

[0055] The state s is determined according to the real-time signal quality (measured by signal-to-noise ratio) of the three links (wired, public network and satellite). The signal quality of each link is discretized into "good", "medium" and "poor" three levels. Therefore, the complete state of the system is composed of the permutation and combination of the states of the three links, a total of 27 possible states, which can be represented as: s=(s1, s2, …, s27). The state definition is shown in the table below.

[0056] Action space:

[0057] The action a is defined as the decision of network switching, which has 3 choices: switching to wired network (a1), switching to mobile public network (a2) and switching to satellite network (a3). It can be represented as: a=(a1, a2, a3).

[0058] Q table design:

[0059] The Q table is used to store the long-term value evaluation value (Q value) of the state-action pair. Its structure is a 27 row (state) x 3 column (action) matrix. An example of the structure of the Q table is as follows:

[0060] State s: wired, public network, satellite a1 : switch wired link a2: switch public network link a3: switch satellite link s1 : good, good, good Q1 Q2 Q3 s2: good, good, medium Q4 Q5 Q6 s3: good, good, bad Q7 Q8 Q9 s4: good, medium, good Q10 Q11 Q12 s5: good, medium, medium Q13 Q14 Q15 s6: good, medium, bad Q16 Q17 Q18 s7: good, bad, good Q19 Q20 Q21 s8: good, bad, medium Q22 Q23 Q24 s9: good, bad, bad Q25 Q26 Q27 s10: medium, good, good Q28 Q29 Q30 s11 : medium, good, medium Q31 Q32 Q33 s12: medium, good, bad Q34 Q35 Q36 s13: medium, medium, good Q37 Q38 Q39 s14: medium, medium, medium Q40 Q41 Q42 s15: medium, medium, bad Q43 Q44 Q45 s16: medium, bad, good Q46 Q47 Q48 s17: medium, bad, medium Q49 Q50 Q51 s18: medium, bad, bad Q52 Q53 Q54 s19: bad, good, good Q55 Q56 Q57 s20: bad, good, medium Q58 Q59 Q60 s21 : bad, good, bad Q61 Q62 Q63 s22: bad, medium, good Q64 Q65 Q66 s23: bad, medium, medium Q67 Q68 Q69 s24: bad, medium, bad Q70 Q71 Q72 s25: bad, bad, good Q73 Q74 Q75 s26: bad, bad, medium Q76 Q77 Q78 s27: bad, bad, bad Q79 Q80 Q81

[0061] Among them, the first table represents the current state (s), the second column, the third column and the fourth column represent the action (a), and the numerical value represents the Q value corresponding to the action pair in the corresponding state.

[0062] Initialize the Q table as a 27*3 matrix, and the Q value range is preset as [0,100]. All Q values are uniformly set to 10 at initialization.

[0063]

[0064] The Q-learning update function is:

[0065]

[0066] ​In the formula,

[0067] : Current Q table, representing the current state at time t , performing action ;

[0068] : Learning rate, value range [0, 1], representing the degree of new information covering old information;

[0069] : Immediate reward, the immediate reward given by the environment after performing action ;

[0070] : Discount factor, value range [0, 1], the smaller the parameter, the more concerned about immediate reward, the larger the parameter, the more concerned about future reward;

[0071] : New state entered after ;

[0072] : Maximum Q value corresponding to all possible actions after entering new state ; Parameter initialization: set learning rate (e.g. 0.1), discount factor

[0073] (e.g. 0.9).

[0074]

[0075] In the formula,

[0076] : Basic reward (encouraging high signal quality);

[0077]

[0078] , where is the signal quality (signal-to-noise ratio);

[0079] : Basic reward weight, range [0, 1];

[0080] : Bandwidth reward (encouraging high bandwidth)

[0081]

[0082] , where is the actual bandwidth, is the required bandwidth;

[0083] : Bandwidth reward weight, range [0, 1];

[0084] : Delay penalty (penalize high delay)

[0085]

[0086] where, is the typical delay of the current signal, is the actual delay;

[0087] : Cost penalty (penalize high-cost networks)

[0088]

[0089] : Switching penalty (strongly penalize unnecessary switches, avoid "ping-pong effect")

[0090]

[0091] : Interruption penalty (large penalty)

[0092]

[0093] Parameter initialization: base reward weight (e.g. 0.2), bandwidth reward weight (e.g. 0.3)

[0094] S3: Real-time data collection, including:

[0095] Signal quality (SNR) of each link: , , ;

[0096] Available bandwidth of each link: , , ;

[0097] Communication delay of each link: , , ;

[0098] S4: Q-learning decision and training:

[0099] Step 1: Determine the current state of the system according to the real-time collected data;

[0100] Step2: Observe the current network state s, and use the ε-greedy strategy to randomly select an action (exploration) with probability ε, and select the action with the maximum Q value in the Q table (utilization) with probability 1-ε; where the exploration rate ε can be set to an initial value of 1.0 and decay with the increase of training steps, such as decaying by 0.1 every 100 steps, and the minimum is 0.01;

[0101] Step3: Perform the selected action and switch to the corresponding network (if the target network is the current network, no actual switching is performed);

[0102] Step4: After entering the new state, collect new network data and calculate the immediate reward using the reward function.

[0103] Step5: Update the corresponding Q value in the Q table according to the Q-learning update formula;

[0104] Step6: Update the current state to the new state and repeat Steps 2 to 5 for continuous learning.

[0105] S5: Automatic switching and application based on Q table:

[0106] After sufficient offline or online training, the Q table will converge to a stable state. In the actual application stage, the system determines the current state through real-time data and queries the Q table to select the action with the highest Q value as the switching decision, thereby realizing the optimal automatic selection of links. In addition, the system supports fine-tuning the Q table during formal operation to adapt to long-term changes in the network environment.

Claims

1. A control method for automatic switching of a multi-link of a communication device based on reinforcement learning, characterized in that, The method comprises the following steps: S1: simultaneously establishing and maintaining three independent communication links in the communication device, including a wired network, a mobile public network and a satellite network; S2: creating a state-action value table, i.e., a Q table, wherein the state s is determined according to real-time signal quality of the three links, the action a is a network switching decision, the Q table is used for storing long-term value evaluation values of state-action pairs, a Q-learning update function and a reward function are designed, and Q table values and Q-learning update function and reward function parameters are initialized; S3: collecting data in real time, including signal quality, available bandwidth and communication delay of each link; S4: for the current state s, an action a is selected and executed by using an ε-greedy strategy; wherein an initial value of an exploration rate ε is set, and the exploration rate ε is attenuated with an increase in a training step number; S5: after the action is executed, the corresponding network is switched to, new network state data is collected, link performance indexes are updated, and an instant reward obtained by the action is calculated according to the reward function; S6: according to the obtained reward r and the new state, a corresponding Q value in the Q table is updated by using a Q-learning update formula; S7: the current state is updated to the new state, and S4 to S6 are repeated, and the iteration is circular until the Q table converges; S8: in the actual application stage, the current state is determined by real-time data, and the action with the highest Q value is selected as the switching decision by querying the Q table, so that the optimized automatic link switching is realized.

2. The control method of claim 1, wherein, In S2, the Q-learning update function is: where, Qcurrentis the current Q-table, representing the state Performing action The corresponding original Q-table; is the learning rate, taking value in the range [0, 1], representing the degree of new information covering old information; is the immediate reward, representing the immediate reward feedback from the environment after performing action ; is the discount factor, taking value in the range [0, 1], the smaller the parameter, the more concerned about the immediate reward, the larger the parameter, the more concerned about the future reward; represents the new state after performing action ; and is the maximum Q-value among all possible actions after performing action . Parameter initialization: Set learning rate and discount factor ; The reward function is: In the formula, Base reward: wherein is the signal quality, i.e. the signal-to-noise ratio; base reward weight, range [0, 1]; Bandwidth bonus: wherein, is the actual bandwidth, is the demand bandwidth; bandwidthrewardweight, range [0, 1]; For delay penalty: wherein, is a typical delay of the current signal, is an actual delay; Cost penalties: for switching penalty: Interrupt penalty: ； Parameter initialization: base reward weight and bandwidth reward weight .

Images