A DQN-assisted decision generation method for low-orbit satellite switching

Abstract

A decision generation method for DQN-assisted low-orbit satellite switching relates to the field of communications, and the method includes: modeling the communication system in which the communication link is located during the low-orbit satellite communication switching process; analyzing the feasibility of DQN-assisted low-orbit satellite switching; constructing the state space, action space and reward function in the deep reinforcement learning algorithm; training the constructed deep reinforcement learning algorithm model through data, and using the calculated deep reinforcement learning algorithm model to perform the low-orbit satellite switching target decision task. The present invention can achieve the following functions: During the switching process, if the user is covered by multiple low-orbit satellites at the same time, the network needs to improve the QoS of the communication link after the switching while minimizing the number of satellite switching according to the communication environment in which the user is located.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a DQN-assisted decision-making method for low-orbit satellite switching. Background Art

[0002] 3GPP (3rd Generation Partnership Project) initiated work in Rel-15 to provide network connectivity for user equipment using satellites and aerial platforms, and continued follow-up research in Rel-16 to adapt 5GNR to support non-terrestrial networks (NTN). Based on the research results of Rel-15 and Rel-16, 3GPP decided to initiate a work project on NTN in Rel-17, which is the first project to develop space technology as part of the 5GNR ecosystem. With the discussion of specifications in Rel-18, NTN is considered a long-term study.

[0003] Since satellites have the ability to provide global coverage, the combination of satellite communication networks and ground communication networks is considered to be an effective solution to achieve the vision of 6G global seamless network coverage and ultra-large-scale links. In the NTN scenario, the coverage capability of geosynchronous orbit satellites is very strong. Three geosynchronous orbit satellites can achieve global coverage. However, due to their high orbital altitude, geosynchronous orbit satellites face the problems of large communication delay and large link attenuation.

[0004] In recent years, low-orbit satellite communications have attracted widespread attention due to their low satellite orbit altitude (500-1500km), low latency required for communication, and low attenuation experienced by the link. Many companies have begun to deploy their own low-orbit satellite constellations, such as StarLink, OneWeb, etc. However, due to their high mobility, low-orbit satellites only cover a specific area for 5-10 minutes, which means that the use of low-orbit satellite communications faces the problem of frequent switching. In the 3GPP Rel-18 R2 working group, the mobility enhancement of NTN was widely discussed, among which the problem of low-orbit satellite communication switching trigger involved the problem of satellite switching. The 3GPP working group found that the change in the signal strength of the communication link in the coverage area provided by low-orbit satellites is not as obvious as that of ground cellular networks, which will cause the communication link to switch between satellites in a timely manner, resulting in the problem of dropped calls in the communication link. In order to solve this problem, the 3GPP working group proposed and adopted a switching decision based on the satellite service time and a switching decision based on positioning. The switching decision based on the satellite service time means that the low-orbit satellite will periodically broadcast the remaining time it can provide services to users in the coverage area. Users will judge whether to switch based on this remaining time and select the target satellite. The handover decision based on positioning means that the user obtains its own position through GPS positioning, and then obtains the current position of the satellite according to the satellite broadcast ephemeris. It determines whether a handover is needed by calculating the relative position between the user and the satellite, and selects the target satellite. The above two schemes proposed by the 3GPP working group clearly define the minimum requirements that need to be met when switching the communication link during low-orbit satellite communications. That is, regardless of the user's needs, the handover decision must be executed under the requirements of the above two schemes. In fact, if the user switches the communication link according to the above two minimum requirements, he may face the problem of poor communication quality.

[0005] At present, relevant research has proposed an optimization strategy for low-orbit satellite communication switching based on the minimum requirements for low-orbit satellite communication switching specified by 3GPP, which is used to improve the quality of service (QoS) of users during low-orbit satellite communication. This scheme comprehensively considers the bandwidth resources and satellite visibility time of the communication link that the target satellite can provide during switching. The user selects the target satellite for switching based on the above two factors affecting the quality of service of the communication link in combination with his own communication environment. In this scheme, the switching selection problem is modeled as a path selection problem of a weighted directed graph (Wang F, Jiang D, Wang Z, et al. Seamless Handover in LEO Based Non-Terrestrial Networks: Service Continuity and Optimization [J]. IEEE Transactions on Communications, 2022, 71 (2): 1008-1023.), the vertex of the graph is the target satellite, the directed edge of the graph represents the switching process from the source satellite to the target satellite, and the weight of the edge is composed of the weighted combination of the two parameters of bandwidth resources and satellite visibility time. Therefore, the choice of switching decision in satellite communication is converted into the problem of selecting the optimal path of the weighted directed graph, and finally the switching decision of the target satellite is obtained by solving the shortest path of the weighted directed graph. The above scheme converts the decision problem of target satellite selection into the problem of solving the shortest path in graph theory. However, using the above scheme, the user needs to first model the weighted directed graph during the switching process, and then execute the shortest path search strategy to obtain the decision of the target satellite during the switching. Under the influence of the high mobility characteristics of low-orbit satellites, each satellite can provide service to users for about 5-10 minutes, which means that users need to re-model the weighted directed graph according to the current communication environment every 5-10 minutes, and search for the shortest path, which generates a lot of signaling overhead for low-orbit satellite communications. And as the number of users in the network increases, the overhead of such signaling resources will continue to rise, which will impose a heavy burden on the satellite communication system. In addition, the above scheme does not take into account the changes of communication links over time during satellite communication. After modeling the weighted directed graph and searching for the shortest path, the user will execute the switching decision within a time period T according to the searched shortest path. However, within the T time period, the bandwidth resources that the satellite can provide will change with the increase of users in the network. Therefore, the switching decision corresponding to the shortest path searched may not be able to effectively guarantee the QoS of communication in real time. Summary of the invention

[0006] The purpose of the present invention is to provide a DQN-assisted low-orbit satellite switching decision-making method.

[0007] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0008] A DQN-assisted low-orbit satellite switching decision generation method of the present invention comprises the following steps:

[0009] Step S1: Modeling the communication system in which the communication link is located during the low-orbit satellite communication switching process;

[0010] Step S2: Analyze the feasibility of DQN-assisted low-orbit satellite switching;

[0011] Step S3: construct the state space, action space and reward function in the deep reinforcement learning algorithm;

[0012] Step S4: Use data to train the constructed deep reinforcement learning algorithm model, and use the calculated deep reinforcement learning algorithm model to perform the low-orbit satellite switching target decision-making task.

[0013] Furthermore, the specific operation process of step S1 is as follows:

[0014] Step S1.1: Determine the indicators that affect the user communication QoS;

[0015] The indicators include: the remaining service time that the satellite can provide to the user at the switching triggering moment, the channel capacity that the satellite can provide to the user at the switching triggering moment, and the load state of the satellite at the switching triggering moment;

[0016] Step S1.2: Determine the switching trigger condition;

[0017] The switching triggering condition is: the remaining service time that the current service satellite can provide is 0; or the data transmission rate requirement of the user exceeds the channel capacity that the current service satellite can provide;

[0018] Step S1.3: Problem modeling;

[0019] When the handover triggering conditions are not met, the optimal handover decision within the T time period during the user communication process is considered, and the following formula is used to quantify the pros and cons of each handover decision:

[0020]

[0021] in Indicates that at time t the user selects satellite LEO s As the reward obtained when switching targets; w1, w2 and w3 are all weight factors, w1+w2+w3=1 and 0≤w1,w2,w3≤1, and the weight factors are adjusted according to user needs;

[0022] Normalize the indicators that affect user communication QoS. Indicates satellite LEO s Can provide user UE n The ratio of the remaining service time provided to the maximum coverage time. The larger the ratio, the better the satellite LEO s The longer the communication service can be provided; Indicates that the satellite LEO is triggered when the switch is triggered s The load margin; Indicates the degree to which the user's data transmission rate requirements are met after the handover is completed; Indicates that at time t the user selects satellite LEO s As the switching target, Indicates that the user did not select satellite LEO at time t s As a switching target; It means that at time t, the user can only select one satellite as the access target;

[0023] Assume that the user needs to perform τ handovers within a time period of T, using Record the satellite information selected by the user for τ times of switching within the T time period. The problem to be solved is: the switching target decision problem of minimizing the number of switching times while meeting the user communication QoS index requirements. The problem is modeled as follows:

[0024]

[0025]

[0026]

[0027]

[0028]

[0029] in Limit the time for evaluating the user's switching decision; Indicates that at t i When the user selects a satellite The rewards obtained, Indicates that at t i At this moment, the user selects a satellite as a switching target.

[0030] Furthermore, the specific operation process of step S2 is as follows:

[0031] Step S2.1: reinforcement learning analysis;

[0032] Step S2.2: Q-leaning algorithm analysis;

[0033] Step S2.3: DQN analysis of deep reinforcement learning algorithm.

[0034] Furthermore, the specific operation process of step S2.1 is as follows:

[0035] The communication environment is mapped to the environment in reinforcement learning, the remaining service time of the satellite, the satellite load status and the user transmission rate requirement are constructed as the state space, the selection of the target satellite is constructed as the action space, and the maximum value to be optimized is constructed as the reward, so that the optimization problem solved by the present invention is converted into the optimal switching decision solving problem in the reinforcement learning algorithm; at the same time, the optimal switching decision solved by the reinforcement learning algorithm corresponds to an action sequence, which is consistent with the switching target decision problem of the user within the T time period considered in the present invention; and the optimal switching decision solved by the reinforcement learning algorithm is obtained based on the maximization of the cumulative reward. When the user switches using the strategy obtained by the reinforcement learning algorithm, the executed strategy is a decision that can maximize the reward obtained within the future T time period, which meets the goal of solving the optimization problem of the present invention; at the same time, when performing reinforcement learning, the value function is used to approximate and directly fit the state value function or the action value function.

[0036] Furthermore, the specific operation process of step S2.2 is as follows:

[0037] The Q-learning algorithm is essentially solving a Bellman optimality equation:

[0038]

[0039] Among them, S t =s means that at time t, it is in state s, A t =a means to execute action a at time t, S t+1 represents the state at the next moment, r t+1 represents the reward at the next moment, γ is the discount factor, and Q(s,a) represents the optimal action value function obtained by executing action a in state s under the optimal switching decision; the iterative algorithm is used to solve the Bellman optimal equation:

[0040]

[0041] where α k (s t ,a t ) represents the parameter corresponding to the kth iteration, which satisfies the following conditions:

[0042]

[0043]

[0044] When running this iterative algorithm, a set of data {(s t,a t ,r t ,s t+1 )} t , after using a large amount of data to execute the iterative algorithm, the optimal estimate of Q(s,a) is obtained, which is the optimal switching decision to be solved by the present invention.

[0045] Furthermore, in step S2.2, when using the Q-learning algorithm to solve the optimal switching decision, it is necessary to ensure that the state space is finite. Therefore, an approximate fitting function combined with the Q-learning algorithm is used to solve this problem. The idea of ​​combining the approximate fitting function with the Q-learning algorithm is expressed by the following formula:

[0046]

[0047] Where Q(S,A) represents the objective function, represents the approximate fitting function, where S is the state variable, A is the action variable, and w is the parameter to be optimized;

[0048] Constructing an approximate fitting function Solve the optimization equation using the stochastic gradient descent algorithm:

[0049]

[0050]

[0051] And because:

[0052]

[0053] therefore:

[0054]

[0055] Among them, w k and w k+1 represents the value of the optimization parameter w corresponding to the current state k and the next state k+1, α k represents the discount factor corresponding to state k, It means that the function J(w) performs gradient operation on w. Indicates that in state s t Execute action a t The optimization parameter w k The value function of .

[0056] Furthermore, the specific operation process of step S2.3 is as follows:

[0057] The DQN essentially solves the following optimization problem:

[0058]

[0059] Among them, (S t ,A t ,R t ,S t+1 ) is a random variable, and let:

[0060] Assume that w in y is a constant w T , and generate two neural networks at the same time: the main network and target network The optimization parameters of the main network will be updated as the data is input, and the w in the target network T It will be updated to w after w is updated a certain number of times T =w, after multiple training iterations, a target network corresponding to the optimized parameter w is obtained; the target network can approximately fit the optimal action value function Q(s,a) corresponding to the Q-learning algorithm, and when a state s is given, an action a is output, and the action a corresponds to the decision that maximizes the Q(s,a) value, which is the optimal switching decision to be solved by the present invention.

[0061] Furthermore, the specific operation process of step S3 is as follows:

[0062] During the T time period, the user will experience multiple handoffs. After each handoff, the state space will change. The T time period is divided into ζ time slots of equal length. The length of each time slot is 60 seconds, which is expressed as T = [(t1, t2), (t2, t3), ..., (t ζ-1 ,t ζ )]; the state space is modeled as in, Indicates t i The user is covered by satellite at all times. Indicates t i The remaining service time of multiple satellites that can cover the user at all times. Indicates that the user is in t i The transmission rate required at any time, Indicates t i The load status of multiple satellites covering users at all times, Indicates t i The channel capacity that each satellite can allocate to users at any time; Action space make Indicates that at t i When the user selects a satellite As the switching target, The reward function is defined as:

[0063]

[0064] Among them, w1, w2 and w3 are weight factors, w1+w2+w3=1 and 0≤w1,w2,w3≤1, and the weight factors are adjusted according to user needs;

[0065] Therefore, the optimal switching decision problem to be solved by the present invention is converted into the following optimization problem:

[0066]

[0067] Furthermore, the specific operation process of step S4 is as follows:

[0068] The Q(s,a) function is fitted through a deep neural network to enable the user to input the state space into the deep neural network model at time t. The deep neural network model then outputs the action value function corresponding to multiple actions, and then selects an action sequence that can maximize the cumulative return as the strategy π * To assist users to make the optimal switching decision starting from time t and within the time period [t, t+T].

[0069] The beneficial effects of the present invention are:

[0070] When using a large-scale low-orbit satellite constellation to provide communication services to the ground, the high mobility of low-orbit satellites will cause the communication link to undergo frequent switching between satellites. During the switching process, if the user is covered by multiple low-orbit satellites at the same time, the network needs to improve the QoS of the communication link after switching while minimizing the number of satellite switching according to the user's communication environment. In this regard, the present invention proposes a DQN (deep Q-network) assisted low-orbit satellite switching decision generation method to solve the problem of switching target selection faced by low-orbit satellite communications.

[0071] The present invention provides a DQN-assisted low-orbit satellite switching decision generation method. Based on the consideration of the switching decision problem within a period of time during the low-orbit satellite communication process, the switching decision problem is converted into a DQN optimal switching decision solution problem by combining the advantages of DQN, and the goal of reducing the number of switching times as much as possible while ensuring the user communication quality is achieved by solving the optimal switching decision. At the same time, a feasibility analysis is conducted on the problem of applying DQN to the switching decision of low-orbit satellite communication. The present invention can adjust the switching decision in response to changes in the environmental state, which is impossible for the low-orbit satellite switching decision based on graph theory. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 The present invention is a flow chart of a DQN-assisted low-orbit satellite switching decision-making generation method.

[0073] Figure 2 Schematic diagram of low-orbit satellite communication scenario.

[0074] Figure 3 Schematic diagram of reinforcement learning.

[0075] Figure 4 The schematic diagram is a method for generating a decision for switching low-orbit satellites assisted by DQN according to the present invention. DETAILED DESCRIPTION

[0076] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0077] like Figure 1 As shown, during the normal communication of users in the low-orbit satellite communication scenario, as the low-orbit satellite moves, the user's channel state will change. For example, when the user approaches the edge of the low-orbit satellite coverage, the channel state will deteriorate compared to the user at the center of the low-orbit satellite coverage. Even as the user leaves the coverage of the current service satellite, communication failures may occur. Therefore, it is necessary to switch between satellites in a timely manner to ensure the user's communication quality. In view of this problem, the present invention proposes a decision-making method for DQN-assisted low-orbit satellite switching, and further proposes a switching trigger condition that considers communication quality on the basis of the 3GPP organization specification. When the user's communication quality drops to meet the set switching trigger condition, the switching is triggered to select the switching target. In response to the problem of switching target selection, the present invention uses a deep reinforcement learning algorithm to assist in selecting the switching target. That is, after the user triggers the switch, the current channel state information needs to be sent to the ground station. The ground station inputs the received user channel state information and the low-orbit satellite constellation channel state information into the trained deep reinforcement learning algorithm model. The deep reinforcement learning algorithm model will output a set of switching decisions within a T time period. Finally, the ground station sends the switching decision to the user. The user selects the switching target and executes the switching operation, so that the user can communicate normally.

[0078] Specifically, consider a typical scenario where a low-orbit satellite constellation serves ground users for communication. The low-orbit satellite constellation includes S low-orbit satellites, using S = {LEO1, LEO2, ..., LEO s} indicates that the ground area contains N users who need low-orbit satellites to provide communication services, using N = {UE1, UE2, ..., UE n} indicates. Figure 2As shown, assuming that the user can ensure that there are at least two low-orbit satellites providing communication services for him at the same time, with the movement of the low-orbit satellites, the low-orbit satellite providing services for user UE2 in the current time period is LEO2, and the coverage of the low-orbit satellite LEO2 will gradually leave the ground area. At this time, it is necessary to switch between low-orbit satellites. Which one of the low-orbit satellites LEO1 and the low-orbit satellite LEO3 is selected as the satellite to provide communication services for user UE2 in the next time period so that the user can obtain better QoS in the future T time period is the problem to be solved by the present invention.

[0079] See also Figure 1 and Figure 4 To illustrate, a DQN-assisted low-orbit satellite switching decision generation method of the present invention mainly includes the following steps:

[0080] After the user accesses the low-orbit satellite communication network, he will receive the channel status information periodically broadcast by the current service satellite, which includes the satellite position and satellite load status in the current time period. The user calculates the remaining service time of the current service satellite through GPS positioning and the satellite's position information. The user calculates the channel capacity that the current service satellite can provide for itself through the satellite load status and its own environmental information. When the switching trigger condition is met, that is, when the user's data transmission rate requirement exceeds the channel capacity that the current service satellite can provide, or when the remaining service time of the current service satellite is zero, the current trigger switching is performed; after the trigger switching, the user sends its own data transmission rate requirement and location information to the ground station for a switching request; when the ground station receives the switching request sent by the user, it will The user's location information is combined with the ephemeris table to determine the set of satellites that can cover the user within the T time period, as well as the remaining service time and load status of each satellite in the satellite set. The set of satellites covering the user, the remaining service time of the satellites in the set, the load status of the satellites in the set, and the user's data transmission rate requirements are packaged to generate state space variables and sent to the data processing center. After the data processing center receives the state space variables sent by the ground station, the state space variables are input into the trained deep reinforcement learning algorithm model. The deep reinforcement learning algorithm model calculates and gives a switching decision that maximizes the cumulative reward within the T time period. After that, the data processing center feeds back the calculated switching decision to the ground station, and the ground station transmits it back to the user. The user executes the selection of the switching target according to the received switching decision.

[0081] A DQN-assisted low-orbit satellite switching decision generation method of the present invention has the following specific operation process:

[0082] Step S1: Modeling the communication system in which the communication link is located during the low-orbit satellite communication switching process;

[0083] Step S1.1: Determine the indicators that affect the user communication QoS;

[0084] In order to make a decision on the switching target in the low-orbit satellite switching scenario, it is necessary to determine the factors affecting the switching target considered during the switching. Since the satellite can send the ephemeris of the low-orbit satellite constellation and the load information of the current service satellite to the ground user in the form of broadcast, the present invention considers the following factors that may change the user communication QoS index from the perspective of improving the user communication QoS index:

[0085] 1. The remaining service time that the satellite can provide to users at the time of switching triggering

[0086] If the user chooses to switch to a satellite with a longer remaining service time, the number of handoffs between satellites experienced during the communication link establishment process can be reduced. Therefore, in order to determine the coverage of the user by satellite at time t, Indicates the state of the user being covered by satellite at time t, where:

[0087]

[0088] When the minimum elevation angle of low-orbit satellite communication is limited, λ is used to represent the maximum time that a single low-orbit satellite can serve ground users. Indicates the remaining service time of the user covered by the low-orbit satellite at time t, especially when = means the user is not in the satellite LEO at time t s Within the coverage area, at this time

[0089] 2. Channel capacity that the satellite can provide to users at the time of switching triggering

[0090] In order to avoid interference between signals in the same coverage area, the user communication link uses Orthogonal Frequency Division Multiplexing (OFDM). According to the Shannon formula R = B log2 (1 + CNR), CNR is the carrier-to-noise ratio (CNR). It can be seen that the channel capacity (unit: Mbps) that the satellite can provide to users is determined by the allocated bandwidth and the link carrier-to-noise ratio. Considering factors such as atmospheric attenuation, the Ku band (12-18GHz) is used as the low-orbit satellite communication working frequency band. In order to simplify the model, it is assumed that the frequency band resources allocated to each satellite are the same, all B (unit: Hz), and the multiple users served by the same low-orbit satellite share the frequency band resources of the low-orbit satellite equally, and the frequency band resources of the low-orbit satellite are used. Indicates the satellite LEO at time t s The frequency band resources allocated to n users within the coverage area are as follows:

[0091]

[0092] in, Represents the frequency band resources allocated by satellite s to user i at time t.

[0093] In order to calculate the channel capacity allocated by the satellite to the user, it is necessary to consider the calculation of the carrier-to-noise ratio CNR. When the user's transmission power is uniform and the satellite receiving antenna gain is the same, the main factor affecting the carrier-to-noise ratio CNR is the various attenuations experienced during the communication link transmission process. Here, PL is used to represent the attenuation experienced during the communication link transmission process (unit: dB), which mainly includes atmospheric absorption attenuation PL α , and the link loss PL caused by free space propagation β . Therefore PL can be expressed as:

[0094] PL=PL α +PL β

[0095] According to the 3GPP working group in the document (Study on New Radio (NR) to Support Non-Terrestrial Networks, document TR 38.811, 3GPP, (Release 15), 2020.) Section 6.6.4, the atmospheric attenuation PL α It can be expressed as:

[0096]

[0097] where f c Indicates the carrier frequency, Indicates user UE n}For satellite LEO s The elevation angle, L zenith (f c ) represents a function of the zenith attenuation as a function of the carrier frequency at different altitudes and environments on the earth. The main factors affecting this value are the absorption resonance lines of oxygen and water vapor. Its reference value under different weather conditions is given by ITU-R in the document (Attenuation by Atmospheric Gases, ITU-R, 2016. [Online]. Available: https: / / www.itu.int / dms_pubrec / itu-r / rec / p / R-REC-P.676-11-201609S!! PDF-E.pdf), which is usually less than 10dB.

[0098] Link loss PL caused by free space propagationβ It can be expressed as:

[0099]

[0100] in Refers to the user UE at time t n and satellite LEO s The distance between them can be expressed as:

[0101]

[0102] Here R e represents the radius of the Earth, h s It means satellite LEO s orbital altitude.

[0103] Based on the above modeling of channel link attenuation, the calculation of the carrier-to-noise ratio CNR (unit: dB) can be expressed as:

[0104] CNR=P1+G1+G2-PL-N

[0105] Where P1 represents the transmitting antenna power of the ground station (unit: dBW), G1 represents the gain of the transmitting antenna of the ground station (unit: dB), G2 represents the gain of the receiving antenna of the space station (unit: dB), and N represents the equivalent noise power, which is:

[0106]

[0107] Where k is the Boltzmann constant, which is approximately 1.380649×10 -23 J / K, T n Indicates the noise temperature (unit: K).

[0108] Users get satellite LEO s The number of users served at time t m s After that, the channel capacity that can be allocated after switching can be calculated. To simplify the model, assume that m s No change. Use Indicates that at time t, the satellite LEO s The channel capacity that can be allocated to users. It indicates the channel capacity that can be allocated to the user by the S satellites that can cover the user in the future T time period.

[0109] Using φ t represents the data transmission rate requirement of the user at time t. When , it indicates that the ground user's demand for data transmission rate exceeds the communication resources that the current service satellite can provide, so switching needs to be triggered.

[0110] 3. Satellite load status at the time of switching trigger

[0111] The definition of satellite load state is given below, and the specific calculation expression is:

[0112]

[0113] in is the satellite load status, Indicates user UE i The data transmission rate requirement at time t (i=1,2,...,m s ), Indicates the satellite LEO at time t s For user UE i The channel capacity provided is required here Indicates that the communication link is connected normally. The ratio of each user's data transmission rate requirement to the channel capacity at time t can represent the satellite LEO corresponding to the current user. s The channel status of the satellite LEO can be obtained by weighted averaging. s Load status at time t.

[0114] use It represents the load status of S satellites at time t. In order to simplify the model, it is assumed that the load status of the satellite does not change within the time period T. That is, the data received by the ground station is the result of calculation of each satellite in the low-orbit satellite constellation at time t according to the calculation expression of the satellite load status above. During the time period T of model training, the result value remains unchanged.

[0115] The above mentioned three indicators need to be considered when switching (the remaining service time that the satellite can provide to users at the time of switching triggering, the channel capacity that the satellite can provide to users at the time of switching triggering, and the load status of the satellite at the time of switching triggering). These three indicators can be used to solve the switching target decision problem of reducing the number of switching times as much as possible while meeting the user communication QoS indicator requirements.

[0116] Step S1.2: Determine the switching trigger condition;

[0117] In the present invention, the satellite is required to broadcast the number of users it serves and the load status as well as the current location of the satellite to ground users and ground stations with a period of ε (unit: s). The user can calculate the remaining service time of the satellite based on the received broadcast information combined with GPS positioning technology. At the same time, the user can calculate the channel capacity provided by the satellite for the user based on the environment in which the user is located and the number of users served by the satellite. Therefore, the user can determine whether it is necessary to trigger the switching between satellites based on whether the channel capacity meets the data transmission rate requirements of the user or the remaining service time of the satellite is close to 0. If the switching needs to be triggered, the switching triggering time is recorded as t.

[0118] Therefore, the following two switching trigger conditions can be summarized:

[0119] 1. The coverage area of ​​the current service satellite is about to leave the user's area, and the remaining service time that the current service satellite can provide is 0, that is,

[0120] 2. The data transmission rate demand of ground users exceeds the channel capacity that the current service satellite can provide, that is,

[0121] Step S1.3: Problem modeling;

[0122] In order to solve the switching target decision problem of minimizing the number of switching times while meeting the user's communication QoS index requirements, we must first ensure that the user's communication QoS index is met. Then try to reduce the number of user switching times, that is, choose the remaining service time The largest satellite is used as the switching target. This choice can well meet the switching target selection of a user between satellites at a time, but from the perspective of the T time period, the current switching target selection will affect the selection of the subsequent switching target. Considering the switching decision within a certain time period T during the user's communication process is more in line with the actual situation and has more research significance. Therefore, the present invention considers the optimal decision of multiple switching of users within the T time period, and the problem modeling is as follows:

[0123] If the switching trigger condition is not met, and In the case of , considering the optimal switching decision within a certain time period T during the user communication process, it is necessary to quantify the pros and cons of each switching decision, and use the following formula for quantification:

[0124]

[0125] in Indicates that at time t the user selects satellite LEO sAs the reward obtained when switching targets; w1, w2 and w3 are all weight factors, w1+w2+w3=1 and 0≤w1,w2,w3≤1, the weight factors can be adjusted according to user needs. In order to equally consider the three indicators that affect the user communication QoS indicators, these three indicators need to be normalized. Indicates satellite LEO s Can provide user UE n The ratio of the remaining service time provided to the maximum coverage time. The larger the ratio, the better the satellite LEO s The longer the communication service time can be provided, the fewer the number of handovers will be when following this strategy; Indicates that the satellite LEO is triggered when the switch is triggered s The load margin is normalized, so no additional normalization is required. Indicates the degree to which the user's data transmission rate requirements are met after the handover is completed; Indicates that at time t the user selects satellite LEO s As the switching target, Indicates that the user did not select satellite LEO at time t s As a switching target; It means that at time t, the user can only select one satellite as the access target.

[0126] Assume that the user needs to perform τ handovers within a time period of T, using The satellite information selected by the user for τ switching within the T time period is recorded. The problem to be solved by the present invention is: the switching target decision problem of reducing the number of switching times as much as possible while meeting the user communication QoS indicator requirements. The problem can be modeled as follows:

[0127]

[0128]

[0129]

[0130]

[0131]

[0132] in The switching decision evaluation time for the user is limited; Indicates that at t i When the user selects a satellite The rewards obtained, Indicates that at t i At this moment, the user selects a satellite as a switching target.

[0133] In the prior art, the use of graph theory to make decisions on the above-mentioned problem requires the ground station to perform a large amount of calculations. The present invention uses a deep reinforcement learning algorithm to solve this problem. The user and the ground station only need to send the communication link information to the data processing center, and the data processing center can generate corresponding switching decisions based on the trained deep reinforcement learning algorithm model. Model training and switching decisions are performed at different stages, thereby ensuring the feasibility of the present invention.

[0134] Step S2: Feasibility analysis of DQN-assisted low-orbit satellite switching;

[0135] Most of the existing studies that consider user switching decisions within a T time period are based on graph theory, that is, the target satellite of each switch is used as the vertex of the graph, the directed edge is used as the switching process, the overhead generated by executing the switching decision is used as the weight of the edge, and then the optimal switching decision within the T time period is obtained by searching the shortest path. However, the computational complexity of the switching decision increases with the increase in the number of users. Therefore, the following mainly introduces the principles and advantages of the deep reinforcement learning algorithm, which is used to provide a feasibility analysis of the application of the deep reinforcement learning algorithm to the low-orbit satellite switching decision problem within the T time period.

[0136] Step S2.1: reinforcement learning analysis;

[0137] Reinforcement learning (RL) discusses how an agent can maximize the rewards it can obtain in a complex and uncertain environment. Figure 3 As shown in the figure, reinforcement learning consists of two parts: the agent and the environment. In the process of reinforcement learning, the agent and the environment are constantly interacting. After the agent obtains a state s∈S in the environment, it will use this state to output an action a∈A, which is also called a decision π. This action is then executed in the environment, and the environment will output the next state and the reward r∈R brought by the current action based on the action taken by the agent. The purpose of the agent is to obtain as many rewards as possible from the environment.

[0138] If the communication environment in the present invention is mapped to the environment in reinforcement learning, the remaining service time of the satellite, the satellite load status and the user transmission rate requirements are constructed as the state space, the selection of the target satellite is constructed as the action space, and the maximum value to be optimized is constructed as the reward, the above optimization problem can be well converted into the optimal switching decision-solving problem in the reinforcement learning algorithm. At the same time, the optimal switching decision solved by the reinforcement learning algorithm corresponds to an action sequence, which is consistent with the user's switching target decision problem within the T time period considered in the present invention. And the optimal switching decision solved by the reinforcement learning algorithm is obtained based on the maximization of the cumulative reward, which means that when the user uses the strategy obtained by the reinforcement learning algorithm to switch, the executed strategy is a decision that can maximize the reward obtained in the future T time period, which is also in line with the goal of solving the optimization problem of the present invention.

[0139] However, there are still certain problems in directly using reinforcement learning algorithms to solve the above optimization problems. Traditional reinforcement learning algorithms use tables to store state value functions or action value functions, and this method has great limitations. For example, the low-orbit satellite communication scenario constructed in the present invention has an infinite number of states. In this case, the value function can no longer be stored in a table. The most effective solution at present is to use the value function to approximate the state value function or action value function directly. This reduces the requirements for storage space and also enhances the generalization ability of the model, effectively solving the problem of state space continuity.

[0140] Reinforcement learning algorithms are suitable for complex and uncertain environments where traditional rules and predefined algorithms may have difficulty adapting and solving problems. Reinforcement learning algorithms can autonomously learn and optimize strategies through interaction with the environment to make decisions in a constantly changing environment. At the same time, reinforcement learning algorithms have the ability to learn, accumulate experience from actual trial and error processes, and improve decision-making strategies through learning. This learning ability makes reinforcement learning algorithms perform well in tasks that require continuous optimization and adaptation.

[0141] Considering the above two advantages of the reinforcement learning algorithm, applying the reinforcement learning algorithm to the selection of switching targets for low-orbit satellite communications can well solve the problems of large real-time computing resource overhead and poor adaptability to environmental changes faced by existing related research. Therefore, the present invention proposes to use the reinforcement learning algorithm to improve the QoS of user communication as much as possible on the basis of reducing the number of handovers between satellites.

[0142] Step S2.2: Q-leaning algorithm analysis;

[0143] Before introducing the deep Q-network (DQN) with function fitting properties, it is necessary to introduce the Q-learning algorithm. The Q-learning algorithm is essentially solving a Bellman optimal equation:

[0144]

[0145] Among them S t =s means that at time t, it is in state s, A t =a means to execute action a at time t, S t+1 represents the state at the next moment, r t+1 represents the reward at the next moment, γ is the discount factor, and Q(s,a) represents the optimal action value function obtained by executing action a in state s under the optimal switching decision. In order to solve the above Bellman optimal equation, an iterative algorithm is required:

[0146]

[0147] where α k (s t ,a t ) represents the parameter corresponding to the kth iteration. This parameter will affect the convergence and divergence of the iterative algorithm. In order for the iterative algorithm to converge, the parameter needs to meet the following conditions:

[0148]

[0149]

[0150] Running this iterative algorithm requires a set of data {(s t ,a t ,r t ,s t+1 )} t After using a large amount of data to execute the iterative algorithm, the optimal estimate of Q(s,a) can be obtained. The purpose of using the Q-learing algorithm in the present invention is to use the optimized optimal estimate Q(s,a) to output an action a when a state s is given. The action a corresponds to the decision that maximizes the Q(s,a) value, which is the optimal switching decision to be solved by the present invention.

[0151] In summary, if we want to use the Q-learning algorithm to solve the optimal switching decision, we need to ensure that the state space is finite. In order to solve the problem that the state space is continuous, that is, the state space is infinite, we need to combine the idea of ​​approximate fitting function with the Q-learning algorithm.

[0152] Specifically, the idea of ​​combining the approximate fitting function with the Q-learing algorithm can be expressed as follows:

[0153]

[0154] Where Q(S,A) represents the objective function, Represents the approximate fitting function, where S is the state variable, A is the action variable, and w is the parameter to be optimized. Construct an approximate fitting function There are two ways. One is to construct a linear function. This method has high requirements for the selection of feature vectors, which means that a good understanding of the optimization problem is required. The other is to use a neural network to construct a nonlinear function. This method has better properties than the method of constructing a linear function. Therefore, the present invention uses a neural network to construct a nonlinear function to fit the target function.

[0155] Specifically, the stochastic gradient descent algorithm is used to solve the above optimization equation:

[0156]

[0157]

[0158] And because:

[0159]

[0160] therefore:

[0161]

[0162] Among them, w k and w k+1 represents the value of the optimization parameter w corresponding to the current state k and the next state k+1, α k represents the discount factor corresponding to state k, It means that the function J(w) performs gradient operation on w. Indicates that in state s t Execute action a t The optimization parameter w k The value function of .

[0163] In summary, a large amount of data can be used to perform the above iterative algorithm to fit an approximate fitting function.

[0164] Step S2.3: DQN analysis of deep reinforcement learning algorithm;

[0165] The DQN is a deep Q network with function fitting properties. DQN essentially solves the following optimization problem:

[0166]

[0167] Here (S t ,A t ,R t ,S t+1 ) is a random variable, and let:

[0168]

[0169] A problem with using the stochastic gradient descent algorithm is that the parameter w to be optimized does not only appear in It also appears in y, so we need to assume that w in y is a constant w T , so that the stochastic gradient descent algorithm can be used to solve the problem. In order to achieve the above operation, two neural networks need to be generated, one as the main network The other network is the target network The optimization parameters of the main network will be kept updated as the data is input, while the w in the target network T It will be updated to w after w is updated a certain number of times T =w, after multiple training iterations, a target network corresponding to the optimized parameter w can be obtained. The target network can approximately fit the optimal action value function Q(s,a) corresponding to the Q-learning algorithm, and output an action a when a state s is given. The action a corresponds to the decision that maximizes the Q(s,a) value, which is the optimal switching decision to be solved by the present invention.

[0170] Step S3: construct the state space, action space and reward function in the deep reinforcement learning algorithm;

[0171] First, the definition of the state space S, action space A and reward function in the present invention is introduced. Within the T time period, the user will experience multiple switching, and the state space will change after each switching. In order to evaluate the pros and cons of the switching decision within the T time period, the T time period needs to be divided into multiple time slots. The present invention divides the T time period into ζ equal-length time slots, each of which is 60 seconds long, specifically expressed as T = [(t1, t2), (t2, t3), ..., (t ζ-1 ,t ζ )], so the state space is modeled as in, Indicates t i The user is covered by satellite at all times. Indicates t i The remaining service time of multiple satellites that can cover the user at all times. Indicates that the user is in t i The transmission rate required at any time, Indicates t i The load status of multiple satellites covering users at all times, Indicates t i The channel capacity that each satellite can allocate to users at any given moment. Action Space make Indicates that at t i When the user selects a satellite As the switching target, The reward function is defined as follows:

[0172]

[0173] Among them, w1, w2 and w3 are all weight factors, w1+w2+w3=1 and 0≤w1,w2,w3≤1, and the weight factors are adjusted according to user needs.

[0174] Therefore, the optimal switching decision problem that the present invention seeks to solve is converted into the following optimization problem:

[0175]

[0176] Step S4: training the constructed deep reinforcement learning algorithm model through data, and using the calculated deep reinforcement learning algorithm model to perform the low-orbit satellite switching target decision task;

[0177] By fitting the Q(s,a) function through a deep neural network, the user can input the state space into the deep neural network model at time t, and then the deep neural network model outputs the action value function corresponding to multiple actions, and then selects an action sequence that can maximize the cumulative return as the strategy π * To assist users to make the optimal switching decision starting from time t in the time period [t, t+T]. This optimal switching decision can well guarantee the QoS of users during the communication process.

[0178] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A DQN-assisted low-orbit satellite switching decision generation method, characterized in that: The following steps are involved: Step S1: Modeling the communication system in which the communication link is located during the low-orbit satellite communication switching process; Step S1.1: Determine the indicators that affect the user communication QoS; The indicators include: the remaining service time that the satellite can provide to the user at the switching triggering moment, the channel capacity that the satellite can provide to the user at the switching triggering moment, and the load state of the satellite at the switching triggering moment; Step S1.2: Determine the switching trigger condition; The switching triggering condition is: the remaining service time that the current service satellite can provide is 0; or the data transmission rate requirement of the user exceeds the channel capacity that the current service satellite can provide; Step S1.3: Problem modeling; When the handover triggering conditions are not met, the optimal handover decision within the T time period during the user communication process is considered, and the following formula is used to quantify the pros and cons of each handover decision: in Indicates that at time t the user selects satellite LEO s As the reward obtained when switching targets; w1, w2 and w3 are all weight factors, w1+w2+w3=1 and 0≤w1,w2,w3≤1, and the weight factors are adjusted according to user needs; Normalize the indicators that affect user communication QoS. Indicates satellite LEO s Can provide user UE n The ratio of the remaining service time provided to the maximum coverage time, Indicates the satellite LEO at time t s Can provide user UE n The remaining service time provided, λ represents the satellite LEO s Can provide user UE n The maximum coverage time provided. The larger the ratio, the more LEO satellite s The longer the communication service can be provided; Indicates that the satellite LEO is triggered when the switch is triggered s The load margin, Indicates the satellite LEO at time t s Load status; Indicates the degree to which the user's data transmission rate requirements are met after the handover is completed. Indicates the satellite LEO at time t s The channel capacity that can be allocated to users, φ t represents the data transmission rate requirement of the user at time t; Indicates that at time t the user selects satellite LEO s As the switching target, Indicates that the user did not select satellite LEO at time t s As a switching target; It means that at time t, the user can only select one satellite as the access target; Assume that the user needs to perform τ handovers within a time period of T, using Record the satellite information selected by the user for τ times of switching within the T time period. The problem to be solved is: the switching target decision problem of minimizing the number of switching times while meeting the user communication QoS index requirements. The problem is modeled as follows: in Limit the time for evaluating the user's switching decision; Indicates that at t i When the user selects a satellite The rewards obtained, Indicates that at t i At this moment, the user selects a satellite As the switching target; the value range of S: The low-orbit satellite constellation contains S low-orbit satellites, using S = {LEO1,LEO2,...,LEO s } represents; the value range of T is: In step S3, the user will experience multiple switching in the T time period, and the state space will change after each switching. The T time period is divided into ζ equal-length time slots, and the length of each time slot is 60 seconds, which is expressed as T = [(t1, t2), (t2, t3), ..., (t ζ-1 ,t ζ )]; Step S2: Analyze the feasibility of DQN-assisted low-orbit satellite switching; Step S2.1: reinforcement learning analysis; The communication environment is mapped to the environment in reinforcement learning, the remaining service time of the satellite, the satellite load status and the user transmission rate requirements are constructed as the state space, the selection of the target satellite is constructed as the action space, and the maximum value to be optimized is constructed as the reward, so that the optimization problem to be solved is converted into the optimal switching decision-solving problem in the reinforcement learning algorithm; at the same time, the optimal switching decision solved by the reinforcement learning algorithm corresponds to an action sequence, which is consistent with the user's switching target decision problem within the T time period; and the optimal switching decision solved by the reinforcement learning algorithm is obtained based on the maximization of the cumulative reward. When the user uses the strategy obtained by the reinforcement learning algorithm to switch, the executed strategy is a decision that can maximize the reward obtained within the future T time period, which meets the goal of the optimization problem to be solved; at the same time, the value function is used to approximate the state value function or the action value function when performing reinforcement learning; Step S2.2: Q-leaning algorithm analysis; The Q-learning algorithm is essentially solving a Bellman optimality equation: Where S t =s means that the state is s at time t, A represents the action variable, a t represents the action performed at time t, A t =a means to execute action a at time t, S t+1 represents the state at the next moment, r t+1 represents the reward at the next moment, γ is the discount factor, and Q(s,a) represents the optimal action value function obtained by executing action a in state s under the optimal switching decision; the iterative algorithm is used to solve the Bellman optimal equation: where α k (s t ,a t ) represents the parameter corresponding to the kth iteration, which satisfies the following conditions: When running this iterative algorithm, a set of data {(s t ,a t ,r t ,s t+1 )} t , after using a large amount of data to execute the iterative algorithm, the optimal estimate of Q(s,a) is obtained, which is the optimal switching decision to be solved; In step S2.2, when using the Q-learning algorithm to solve the optimal switching decision, it is necessary to ensure that the state space is finite. Therefore, an approximate fitting function combined with the Q-learning algorithm is used to solve this problem. The idea of ​​combining the approximate fitting function with the Q-learning algorithm is expressed by the following formula: Where Q(S,A) represents the objective function, represents the approximate fitting function, where S is the state variable, A is the action variable, and w is the parameter to be optimized; Constructing an approximate fitting function Solve the optimization equation using the stochastic gradient descent algorithm: And because: therefore: Among them, w k and w k+1 represents the value of the optimization parameter w corresponding to the current state k and the next state k+1, α k represents the discount factor corresponding to state k, It means that the function J(w) performs gradient operation on w. Indicates that in state s t Execute action a t The optimization parameter w k The value function of Step S2.3: DQN analysis of deep reinforcement learning algorithm; The DQN essentially solves the following optimization problem: Among them, (S t ,A t ,R t ,S t+1 ) is a random variable, and let: Assume that w in y is a constant w T , and generate two neural networks at the same time: the main network and target network The optimization parameters of the main network will be updated as the data is input, and the w in the target network T It will be updated to w after w is updated a certain number of times T =w, after multiple training iterations, a target network corresponding to the optimized parameter w is obtained; the target network can approximately fit the optimal action value function Q(s,a) corresponding to the Q-learning algorithm, and output an action a when a state s is given. The action a corresponds to the decision that maximizes the value of Q(s,a), which is the optimal switching decision to be solved; Step S3: construct the state space, action space and reward function in the deep reinforcement learning algorithm; During the T time period, the user will experience multiple handoffs. After each handoff, the state space will change. The T time period is divided into ζ time slots of equal length. The length of each time slot is 60 seconds, which is expressed as T = [(t1, t2), (t2, t3), ..., (t ζ-1 ,t ζ )]; the state space is modeled as in, Indicates t i The user is covered by satellite at all times. Indicates t i The remaining service time of multiple satellites that can cover the user at all times. Indicates that the user is in t i The transmission rate required at any time, Indicates t i The load status of multiple satellites covering users at all times, Indicates t i The channel capacity that each satellite can allocate to users at any time; Action space make Indicates that at t i When the user selects a satellite As the switching target, The reward function is defined as: Among them, w1, w2 and w3 are weight factors, w1+w2+w3=1 and 0≤w1,w2,w3≤1, and the weight factors are adjusted according to user needs; Indicates t i Time Satellite The remaining service time that can be provided to users; Indicates that at t i Time Satellite The channel capacity that can be allocated to users; Indicates t i Time Satellite The ratio of the remaining service time that can be provided to users to the maximum coverage time; Indicates t i Time Satellite The load margin; Indicates t i The degree to which the user's data transmission rate requirements are met at all times; Therefore, the optimal switching decision problem to be solved is transformed into the following optimization problem: Step S4: training the constructed deep reinforcement learning algorithm model through data, and using the calculated deep reinforcement learning algorithm model to perform the low-orbit satellite switching target decision task; The deep neural network is used to fit the Q(s,a) function to enable the user to input the state space into the deep neural network model at time t. The deep neural network model then outputs the action value function corresponding to multiple actions, and then selects an action sequence that can maximize the cumulative return as the strategy π * To assist users to make the optimal switching decision starting from time t and within the time period [t, t+T].

Images