Low-orbit satellite communication hopping beam scheduling method based on deep reinforcement learning

Through the deep reinforcement learning model, the beam resource allocation of low-orbit satellite communication system is optimized, and the uneven signal coverage problem caused by limited beam resources is solved, and the communication effect of high throughput and low interference is achieved.

CN119010995APending Publication Date: 2024-11-22CHENGDU TUXUN TECH CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411292917.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-14
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The limited beam resources in low-orbit satellite communication systems lead to uneven signal coverage. How to improve system throughput while taking into account terminal service priorities and reducing inter-beam interference is a complex issue.

Method used

Using the low-orbit satellite communication beam hopping scheduling method based on deep reinforcement learning, the low-orbit satellite communication beam hopping scheduling method is adopted, and the deep reinforcement learning model is trained, and the scheduling wave bit decisions are generated based on the waiting time slot matrix and the cache packet matrix are optimized to optimize beam resource allocation, and the maximum To improve throughput and reduce inter-beam interference.

Benefits of technology

It realizes beam resource allocation according to business importance, improves system throughput, reduces inter-beam interference, and optimizes network capacity and performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119010995A_ABST
    Figure CN119010995A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of low-earth-orbit satellite communication, and discloses a low-earth-orbit satellite communication hopping beam scheduling method based on deep reinforcement learning, which comprises the following steps: acquiring a ground request service data packet; establishing a low-orbit satellite communication hopping beam optimization model according to a low-orbit satellite communication system architecture; the data packet waiting time slot number, the data packet number and the data packet service classification are used as states, the states are reconstructed into a waiting time slot number matrix and a cache data packet matrix, a satellite load scheduler is used as an intelligent agent, a selected wave position is used as an action, and a deep reinforcement learning model is constructed and trained; and utilizing the trained deep reinforcement learning model to generate a scheduling wave position decision according to the waiting time slot number matrix and the cache data packet matrix. According to the invention, the throughput rate of the system is improved while the inter-beam interference is reduced and the service priority of the terminal is considered.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of low-orbit satellite communications, and in particular to a low-orbit satellite communication beam hopping scheduling method based on deep reinforcement learning. Background Art

[0002] A low-orbit satellite communication system refers to a satellite payload communication system with an orbit altitude of less than 2,000 kilometers. Compared with ground base stations, the transmission distance of satellite communication is still relatively large, and it will face relatively large signal attenuation problems. Therefore, beamforming is one of the indispensable technologies for satellite communication. The formed beam can concentrate power to aim at a limited area, thereby achieving effective signal coverage for the area. The beam resources of a low-orbit satellite payload are limited, which means that the satellite signal cannot achieve signal coverage of the entire area at one time, but can only selectively cover part of the area. In order to ensure that every terminal in any area has the opportunity to communicate with the low-orbit satellite payload, the low-orbit satellite payload will selectively aim at different areas for signal transmission based on limited beam resources at different times. This is the underlying core logic of beam hopping scheduling.

[0003] Under the limitation of beam resources, how low-orbit satellite payloads choose which areas to cover with beams at a certain moment is the core issue of beam-hopping scheduling research. A poor beam-hopping scheduling method will lead to problems such as uneven distribution of communication resources. A good beam-hopping scheduling method can optimize the capacity and performance of the entire network. To design a good beam-hopping scheduling method, many factors need to be considered, such as the number of users in each area, the priority of communication needs in each area, and interference between beams. How to balance various factors to achieve relatively higher network capacity and performance is a relatively complex issue. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a low-orbit satellite communication beam-hopping scheduling method based on deep reinforcement learning, so as to improve the system throughput while reducing inter-beam interference and taking into account the terminal service priority.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0006] A low-orbit satellite communication beam-hopping scheduling method based on deep reinforcement learning comprises the following steps:

[0007] Obtain ground request service data packet;

[0008] According to the low-orbit satellite communication system architecture, a low-orbit satellite communication beam hopping optimization model is established;

[0009] The number of time slots that data packets are waiting for, the number of data packets, and the data packet service classification are taken as states, and the states are reconstructed into a matrix of the number of waiting time slots and a matrix of cached data packets. The satellite payload scheduler is taken as an intelligent agent, and the wave position selection is taken as an action. A deep reinforcement learning model is constructed and trained.

[0010] The trained deep reinforcement learning model is used to generate scheduling wave position decisions based on the waiting time slot number matrix and the cached data packet matrix.

[0011] As a preferred embodiment, the low-orbit satellite communication beam hopping optimization model is specifically as follows:

[0012]

[0013]

[0014]

[0015]

[0016] τ q ≤T th

[0017]

[0018] Among them, min represents the minimum function, T represents the set of all decision moments, C represents the set of all wave positions, λ1 and λ2 represent the initial allocation weights of real-time tasks and non-real-time tasks respectively. Indicates that at t j Time wave position c n Complete the transmission of the data packet of the mth real-time task, Indicates data packet The waiting delay from arrival to completion of the transmission, Indicates that at t j Time wave position c n Complete the transmission of the data packet of the mth non-real-time task, Indicates data packet The waiting delay from arrival to completion of the transmission, Indicates that at t j Time wave position c n The set of possible power values, p i represents an element in the power value set, Indicates that at t j Time wave position c n A 0-1 variable indicating whether there is beam illumination, K represents the maximum number of beams, Indicates that at t j Time wave position c n Distributed power, P totrepresents the total onboard power, τ q Indicates the waiting delay, T th Indicates the maximum tolerable delay.

[0019] As a preferred embodiment, when the state is reconstructed into a waiting slot number matrix and a cached data packet matrix,

[0020] The maximum delay of data waiting is evenly divided into multiple time slots.

[0021] Preferably, the action of selecting a wave position is as follows:

[0022] A(t j )={X(t j ), P(t j )}

[0023]

[0024]

[0025] Among them, A(t j ) indicates that at t j The action at the moment, X(t j ) indicates that at t j The wave position allocation vector at time, P(t j ) indicates that at t j The power allocation vector at time t, Indicates that at t j Time wave position c n A 0-1 variable indicating whether there is beam illumination, K represents the maximum number of beams, Indicates that at t j Time wave position c n Distributed power, P tot Indicates the total onboard power.

[0026] Preferably, the reward function of the deep reinforcement learning model is:

[0027] r(t j )=-∑∑(w m,n (t j )·d m,n (t j )) / L(t j )

[0028] Among them, r(t j ) indicates that at t j The reward of the moment, w m,n (t j ) indicates that all wave positions are in The packets cached in the time period, d m,n (t j) indicates that all wave positions are in The number of time slots to wait in the time period, m represents the wave position number, n represents the waiting time slot number, L(t j ) represents the sum of the distances between the selected wave positions.

[0029] As a preferred embodiment, the distance between the wave positions and the specific distance between the wave positions are:

[0030]

[0031] Among them, L(t j ) indicates that at t j The sum of the distances between the wave positions selected at the moment, O represents the wave position distance matrix, c i (t j ) indicates that at t j The i-th wave position selected at the moment, c k (t j ) indicates that at t j The kth wave position selected at time.

[0032] Preferably, when training the deep reinforcement learning model, a greedy algorithm is used to perform random beam allocation with probability ε, and the maximum Q value action output by the Q network is selected with probability 1-ε.

[0033] Preferably, when training a deep reinforcement learning model, the state update method of the deep reinforcement learning network is as follows:

[0034] Q * (s t , a t )←(1-α)·Q(s t , a t )+α(r+γ·max a Q(s t+1 , a))

[0035] Among them, Q * represents the updated state of the deep reinforcement learning network, α represents the impact factor of the expected reward of the future state on the updated network, Q represents the state of the deep reinforcement learning network, r represents the reward obtained by the current action, γ represents the attenuation factor, max represents the maximum value function, and s t represents the state at time t, a t Indicates the action at time t.

[0036] Preferably, when training a deep reinforcement learning model, the loss function of the deep reinforcement learning network is:

[0037]

[0038] Among them, L i represents the loss function, θi Represents the weight parameter of the network to be updated, E s,a,s′,r represents the expected value, r represents the reward obtained by the current action, γ represents the attenuation factor, max represents the maximum function, s′ represents the state after the current state s performs action a, a′ represents the action that s′ will perform, represents the weight parameter of the target network, Q represents the state of the deep reinforcement learning network, s represents the state, and a represents the action.

[0039] Preferably, when training a new deep reinforcement learning model, the updating method of the Q network to be trained is specifically as follows:

[0040]

[0041] in, represents the network state of the new deep reinforcement learning model, s t represents the state at time t, a t represents the action at time t, ω1, ω2, ω3 all represent migration factors, represents the network state of the original deep reinforcement learning model, represents the network state of the original target deep reinforcement learning model, r represents the reward obtained by the current action, γ represents the decay factor, and max represents the maximum value function. Represents the new target deep reinforcement learning model's network state.

[0042] The present invention has the following beneficial effects:

[0043] The present invention can allocate beam resources according to the importance of services by establishing a low-orbit satellite beam allocation optimization model with service weights; and establishes a reward model that simultaneously considers terminal service priority, throughput rate and beam spacing, and can reduce interference between beams while maximizing throughput rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flowchart of a beam-hopping scheduling method for low-orbit satellite communications based on deep reinforcement learning;

[0045] Figure 2 Schematic diagram of reconstructing state S into D matrix and W matrix;

[0046] Figure 3 It is the structure diagram of the deep Q network;

[0047] Figure 4 Schematic diagram of the distance between different wave positions. DETAILED DESCRIPTION

[0048] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0049] like Figure 1 As shown, a low-orbit satellite communication beam hopping scheduling method based on deep reinforcement learning provided by an embodiment of the present invention includes the following steps S1 to S4:

[0050] S1. Obtain ground request service data packet;

[0051] In an optional embodiment of the present invention, step S1 first obtains a ground request service data packet based on a low-orbit satellite communication system.

[0052] S2. According to the low-orbit satellite communication system architecture, a low-orbit satellite communication beam hopping optimization model is established;

[0053] In an optional embodiment of the present invention, step S2 divides the ground request service into real-time and non-real-time services according to different terminal service priorities in the low-orbit satellite communication system architecture, and establishes a low-orbit satellite communication beam hopping optimization model, which is expressed as:

[0054]

[0055]

[0056]

[0057]

[0058] τ q ≤T th

[0059]

[0060] Where min represents the minimum function, T = {t1, t2, ..., t n} represents the set of all decision moments, C represents the set of all wave positions, λ1 and λ2 represent the initial allocation weights of real-time tasks and non-real-time tasks respectively, and λ1+λ2=1, Indicates that at t j Time wave position c n Complete the transmission of the data packet of the mth real-time task, Indicates data packet The waiting delay from arrival to completion of the transmission, Indicates that at t j Time wave position c n Complete the transmission of the data packet of the mth non-real-time task, Indicates data packet The waiting delay from arrival to completion of the transmission, Indicates that at t j Time wave position c n The possible power value set, Indicates that at t j Time wave position c n A 0-1 variable indicating whether there is beam illumination, K represents the maximum number of beams, Indicates that at t j Time wave position c n Distributed power, P tot Indicates the total onboard power, T a Indicates the waiting delay, T th Indicates the maximum tolerable delay. The first constraint is the maximum number of beams, the second is the transmit power, the third is the waiting delay, and the fourth is the scheduling state.

[0061] S3, taking the number of time slots for data packets to wait, the number of data packets, and the data packet service classification as the state, and reconstructing the state into a waiting time slot number matrix and a cached data packet matrix, taking the satellite payload scheduler as the intelligent agent, and taking the wave position selection as the action, building a deep reinforcement learning model and training it;

[0062] In an optional embodiment of the present invention, step S3 first sets the maximum tolerable delay T th The data waiting delay is converted from time to time slots. The number of time slots, number of data packets, and data packet service classification of the data packet waiting are then used as the state, and the state is reconstructed into the waiting time slot number matrix and the cached data packet matrix. The specific state S is at t j The time can be divided into and Respectively represent all wave positions in The number of waiting slots and all wave positions in the time period The data packet status matrix cached in the time period. The state S is reconstructed into the D and W matrices as follows Figure 2 As shown in the figure, the dotted circle represents the real-time service data packet, and the solid circle represents the non-real-time service data packet. W(i, z) = λ1·n1+λ2·n2, where n1 and n2 represent the wave position j at t j-zThe number of real-time service data packets and non-real-time service data packets arriving at the time slot, λ1 and λ2 represent the weights of real-time service data packets and non-real-time service data packets respectively. For data packets that have exceeded the maximum waiting delay and have not been scheduled, they are accumulated to the maximum waiting time slot column. D(i, z) represents the number of real-time service data packets and non-real-time service data packets arriving at the time slot. j-z to j The waiting delay.

[0063] Step S3 selects the wave position as an action specifically as follows:

[0064] A(t j )={X(t j ), P(t j )}

[0065]

[0066]

[0067] Among them, A(t j ) indicates that at t j The action at the moment, X(t j ) indicates that at t j The wave position allocation vector at time, P(t j ) indicates that at t j The power allocation vector at time t, Indicates that at t j Time wave position c n A 0-1 variable indicating whether there is beam illumination, Indicates t j At this moment, there is a beam irradiating this wave position. Indicates t j At this moment, there is no beam irradiation at this wave position; K represents the maximum number of beams, Indicates the wave position c at time tj n Distributed power, P tot In order to simplify the model, it is assumed that

[0068] The deep reinforcement learning model constructed in step S3 can use a deep Q network, which consists of a residual block and a fully connected network (fc). The residual block consists of two convolutional layers and cross-layer connections to extract input features. The fully connected layer can increase the nonlinear fitting ability of the network and has a classifier function. Figure 3 shown.

[0069] The deep Q network of this embodiment has the reconstructed W and D matrices as input and the expected Q values ​​of different actions as output. The Q value is defined as the sum of the reward obtained by selecting the current action and the maximum expected reward in the future.

[0070] Step S3 defines the reward function that can be obtained by different actions, expressed as:

[0071] r(t j )=-∑∑(w m,n (t j )·d m,n (t j )) / L(t j )

[0072] Among them, r(t j ) indicates that at t j The reward of the moment, w m,n (t j ) indicates that all wave positions are in The packets cached in the time period, d m,n (t j ) indicates that all wave positions are in The number of time slots to wait in the time period, m represents the wave position number, n represents the waiting time slot number, L(t j ) represents the sum of the distances between the selected wave positions.

[0073] The distance between wave positions is defined as: According to the wave position distribution, construct the wave position distance matrix O∈R n×n , O is an upper triangular matrix, and O (i,i) = 0. The remaining elements of the matrix have a value of 0 if the two wave positions are adjacent, and a value of 1 if they are not adjacent. The wave position distance matrix is ​​as follows Figure 4 As shown. The distance between wave positions is as follows:

[0074]

[0075] Among them, L(t j ) indicates that at t j The sum of the distances between the wave positions selected at the moment, O represents the wave position distance matrix, ci(t j ) indicates that at t j The i-th wave position selected at the moment, c k (t j ) indicates that at t j The k-th wave position selected at time i≠k.

[0076] Step S3 introduces the greedy algorithm ε-greedy when training the deep Q network for the first time. The algorithm performs random beam allocation with probability ε and selects the maximum Q value action output by the Q network with probability 1-ε. In this way, the agent can explore more possibilities of actions to avoid falling into the local minimum. ε∈(0.05, 0.2). In this embodiment, the initial value of ε is 0.2. For every 100 iterations of the deep Q network, the value of ε is reduced to half of the original value.

[0077] To train the deep Q network described in S6, the network state update method is defined as follows:

[0078] Q * (s t , a t )←(1-α)·Q(s t , a t )+α(r+γ·max a Q(s t+1 , a))

[0079] Among them, Q * represents the updated state of the deep reinforcement learning network, α represents the impact factor of the expected reward of the future state on the updated network, Q represents the state of the deep reinforcement learning network, r represents the reward obtained by the current action, γ represents the attenuation factor, max represents the maximum value function, and s t represents the state at time t, a t Indicates the action at time t. In this embodiment, α=γ=0.5.

[0080] According to the above formula, the loss function of the network is defined as:

[0081]

[0082] Among them, L i represents the loss function, θ i Represents the weight parameter of the network to be updated, E s,a,s′,r represents the mathematical expectation, r represents the reward obtained by the current action, γ represents the attenuation factor, max represents the maximum function, s′ represents the current state s, the state after action a, a′ represents the action to be performed in state s′, represents the weight parameter of the target network, Q represents the state of the deep reinforcement learning network, s represents the state, and a represents the action. The target network is consistent with the network structure to be updated, and after every fixed number of steps, let

[0083] In order to train the deep Q network, a batch of quadruple (s, a, r, s′) is set. Among them, s represents the current state, a represents the executed action, r represents the reward obtained by executing a in state s, and s′ represents the updated state after executing a in state s. Each time an action is executed, the corresponding reward and the next state can be obtained. Therefore, a quadruple can be obtained after each action is executed and put into the experience pool. After the training starts, a small number (minibatch) of quadruple are randomly selected from the experience pool for training, and their order is disrupted to reduce the correlation. When the quadruple in the experience pool exceeds the set maximum capacity, the earliest stored data is cleared in the order of placement. In this embodiment, the minibatch is set to 32, and the maximum capacity of the experience pool is 60,000.

[0084] The process of training a deep Q network is as follows:

[0085] (1) Initialize the experience pool D;

[0086] (2) Use random weight parameters to initialize the parameters θ of the Q network; initialize the target network The parameter θ - , let θ - =θ;

[0087] (3) Starting from the first iteration, an action a is randomly selected with a small probability ε t , otherwise a t = argmax a Q(s t , a t θ t );

[0088] (4) Execute action a t , get the corresponding reward r and next state s t+1 , the quaternion (s t , a t , r, s t+1 ) into experience pool D;

[0089] (5) Set the label:

[0090]

[0091] in, represents the target network;

[0092] The loss function is

[0093] (y t -Q(s t , a t θ t )) 2

[0094] Then use the gradient descent algorithm to update θ;

[0095] Every C steps, let θ - =θ; until the maximum number of iterations N is reached.

[0096] Step S3: After the network training is completed, its parameters are fixed and it is deployed on the satellite side. For different satellites, when training a new Q network, the trained Q network is used as the initial target network to accelerate the training of the new Q network and save training resources.

[0097] Assume that the original Q network is The Q network to be trained for the new scene is The target Q network for the new scenario is The update strategy of the Q network to be trained is as follows:

[0098]

[0099] in, represents the network state of the new deep reinforcement learning model, st represents the state at time t, at represents the action at time t, ω1, ω2, ω3 all represent migration factors, and ω1+ω2+ω3=1, represents the network state of the original deep reinforcement learning model, represents the network state of the original target deep reinforcement learning model, r represents the reward obtained by the current action, γ represents the decay factor, and max represents the maximum value function. Indicates the network state of the new target deep reinforcement learning model. In the early stage of training, ω1 is large and ω3 is small. At this time, the original Q network is mainly used as the target. As the training progresses, ω1 decreases, ω3 increases, and the target Q network of the new scene gradually takes the leading role. In this embodiment, the initial values ​​are set: ω1=0.5, ω2=ω3=0.25. After that, ω1 gradually decreases to 0, and ω2 and ω3 gradually increase to 0.5.

[0100] After the Q network to be trained in the new scenario is trained, the trained new deep Q network is deployed to the satellite side.

[0101] S4. Use the trained deep reinforcement learning model to generate scheduling wave position decisions based on the waiting time slot number matrix and the cached data packet matrix.

[0102] In an optional embodiment of the present invention, when processing actual business, step S4 first converts the data packets in the buffer area into W and D matrices and sends them to the deep reinforcement learning model. After obtaining the output of the network, the wave positions corresponding to the K largest Q values ​​are selected as the wave positions for scheduling.

[0103] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0104] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0105] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0106] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0107] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A beam-hopping scheduling method for low-orbit satellite communications based on deep reinforcement learning, characterized in that: The following steps are involved: Obtain ground request service data packet; According to the low-orbit satellite communication system architecture, a low-orbit satellite communication beam hopping optimization model is established; The number of time slots that data packets are waiting for, the number of data packets, and the data packet service classification are taken as states, and the states are reconstructed into a matrix of the number of waiting time slots and a matrix of cached data packets. The satellite payload scheduler is taken as an intelligent agent, and the wave position selection is taken as an action. A deep reinforcement learning model is constructed and trained. The trained deep reinforcement learning model is used to generate scheduling wave position decisions based on the waiting time slot number matrix and the cached data packet matrix.

2. According to claim 1, a low-orbit satellite communication beam-hopping scheduling method based on deep reinforcement learning is characterized in that: The specific model of beam hopping optimization for low-orbit satellite communication is: t c ≤T th Among them, min represents the minimum function, T represents the set of all decision moments, C represents the set of all wave positions, λ1 and λ2 represent the initial allocation weights of real-time tasks and non-real-time tasks respectively. Indicates that at t j Time wave position c n Complete the transmission of the data packet of the mth real-time task, Indicates data packet The waiting delay from arrival to completion of the transmission, Indicates that at t j Time wave position c n Complete the transmission of the data packet of the mth non-real-time task, Indicates data packet The waiting delay from arrival to completion of the transmission, Indicates that at t j Time wave position c n The set of possible power values, p i represents an element in the power value set, Indicates that at t j Time wave position c n A 0-1 variable indicating whether there is beam illumination, K represents the maximum number of beams, Indicates that at t j Time wave position c n Distributed power, P tot represents the total onboard power, τ q Indicates the waiting delay, T th Indicates the maximum tolerable delay.

3. According to claim 1, a low-orbit satellite communication beam-hopping scheduling method based on deep reinforcement learning is characterized in that: When the state is reconstructed into a waiting time slot number matrix and a buffered data packet matrix, the maximum delay of data waiting is evenly divided into multiple time slots.

4. According to the method of claim 1, the beam hopping scheduling method for low-orbit satellite communication based on deep reinforcement learning is characterized in that: The action of selecting a wave position is as follows: A(t j )={X(t j ),P(t j )} Among them, A(t j ) indicates that at t j The action at the moment, X(t j ) indicates that at t j The wave position allocation vector at time, P(t j ) indicates that at t j The power allocation vector at time t, Indicates that at t j Time wave position c n A 0-1 variable indicating whether there is beam illumination, K represents the maximum number of beams, Indicates that at t j Time wave position c n Distributed power, P tot Indicates the total onboard power.

5. According to the method of claim 1, the beam hopping scheduling method for low-orbit satellite communication based on deep reinforcement learning is characterized in that: The reward function of the deep reinforcement learning model is specifically: r(t j )=-∑∑(w m,n (t j )·d m,n (t j )) / L(t j ) Among them, r(t j ) indicates that at t j The reward of the moment, w m,n (t j ) indicates that all wave positions are in The packets cached in the time period, d m,n (t j ) indicates that all wave positions are in The number of time slots to wait in the time period, m represents the wave position number, n represents the waiting time slot number, L(t j ) represents the sum of the distances between the selected wave positions.

6. According to claim 5, a low-orbit satellite communication beam-hopping scheduling method based on deep reinforcement learning is characterized in that: The distances between wave positions are as follows: Among them, L(t j ) indicates that at t j The sum of the distances between the wave positions selected at the moment, O represents the wave position distance matrix, c i (t j ) indicates that at t j The i-th wave position selected at the moment, c k (t j ) indicates that at t j The kth wave position selected at time.

7. According to claim 1, a low-orbit satellite communication beam-hopping scheduling method based on deep reinforcement learning is characterized in that: When training the deep reinforcement learning model, a greedy algorithm is used to perform random beam allocation with probability ε, and the action with the maximum Q value output by the Q network is selected with probability 1-ε.

8. According to claim 1, a low-orbit satellite communication beam-hopping scheduling method based on deep reinforcement learning is characterized in that: When training a deep reinforcement learning model, the state update method of the deep reinforcement learning network is as follows: Q * (s t ,a t )←(1-α)·Q(s t ,a t )+α(r+γ·max a Q(s t+1 ,a)) Among them, Q * represents the updated state of the deep reinforcement learning network, α represents the impact factor of the expected reward of the future state on the updated network, Q represents the state of the deep reinforcement learning network, r represents the reward obtained by the current action, γ represents the attenuation factor, max represents the maximum value function, and s t represents the state at time t, a t Indicates the action at time t.

9. According to a method for beam hopping scheduling of low-orbit satellite communications based on deep reinforcement learning in claim 1, it is characterized in that: When training a deep reinforcement learning model, the loss function of the deep reinforcement learning network is specifically: Among them, L i represents the loss function, θ i Represents the weight parameter of the network to be updated, E s,a,s′,r represents the expected value, r represents the reward obtained by the current action, γ represents the attenuation factor, max represents the maximum function, s′ represents the state after the current state s performs action a, a′ represents the action that s′ will perform, represents the weight parameter of the target network, Q represents the state of the deep reinforcement learning network, s represents the state, and a represents the action.

10. The method for beam hopping scheduling of low-orbit satellite communications based on deep reinforcement learning according to claim 1, characterized in that: When training a new deep reinforcement learning model, the update method of the Q network to be trained is as follows: in, represents the network state of the new deep reinforcement learning model, s t represents the state at time t, a t represents the action at time t, ω1, ω2, ω3 all represent migration factors, represents the network state of the original deep reinforcement learning model, represents the network state of the original target deep reinforcement learning model, r represents the reward obtained by the current action, γ represents the decay factor, and max represents the maximum value function. Represents the new target deep reinforcement learning model's network state.

Citation Information

Cited By

  • Beam hopping resource allocation method and device based on deep reinforcement learning

    CN120956320A

  • Low earth orbit satellite global beam position distribution scheduling system and method based on deep reinforcement learning

    CN121217198A