A communication satellite multi-beam resource management method based on deep reinforcement learning

Abstract

A method for multi-beam resource management and control of communication satellites based on deep reinforcement learning belongs to the field of satellite technology. This method utilizes a multi-agent reinforcement learning algorithm to dynamically adjust and optimize beam resource allocation strategies. By utilizing satellite antenna hopping technology and time-division multiplexing, it flexibly responds to changes in ground user demand and communication traffic, significantly improving the overall throughput of the onboard link. This dynamic optimization mechanism ensures efficient resource utilization and enhances the transmission rate and quality of the communication system.

Description

Technical Field

[0001] The present invention relates to a communication satellite multi-beam resource management and control method based on deep reinforcement learning, and belongs to the field of satellite technology. Background Art

[0002] Satellite-based communication systems offer long-range, all-weather coverage, wide coverage, long range, and are unaffected by geographical factors. They are a crucial component of modern communication systems, demonstrating unique advantages in situations where terrestrial communication networks are limited. However, with the advancement of satellite communication technology, spectrum scarcity and spatial congestion have become increasingly prominent. Multi-beam antenna technology has emerged as a new approach to addressing this issue. Multi-beam satellite payloads contain multiple sub-beams arranged in a specific topology to effectively cover target areas on the ground. Compared to single-beam antennas, multi-beam antennas offer smaller beamwidths and greater ground gain, significantly improving satellite communication signal quality. When a single satellite cannot fully cover a target area on the ground or when coverage needs to be extended, a constellation system consisting of multiple satellites becomes particularly valuable. Summary of the Invention

[0003] The technical problem solved by the present invention is: to overcome the shortcomings of the existing technology and provide a communication satellite multi-beam resource management method based on deep reinforcement learning, which improves the on-board link throughput, saves limited on-link resources, reduces latency and improves resource allocation efficiency.

[0004] The technical solution of the present invention is: a communication satellite multi-beam resource management method based on deep reinforcement learning, comprising:

[0005] Step 1: Construct a mathematical model of the beam resource allocation time window, user spacecraft, user mission, satellites in different orbits and beam resources, visible time window, and communication link establishment, and establish a communication and mission allocation environment model for multi-beam satellites;

[0006] Step 2: Based on the communication and task allocation environment model of the multi-beam satellite, simulate the generation and transmission of on-board tasks, and generate on-board tasks based on the simulated user needs; schedule resources to meet the task requirements, and evaluate the communication task throughput and latency during the scheduling process;

[0007] Step 3: Calculate the observation values ​​of the environment in steps 1 and 2 and input them into a high-dimensional observation vector to extract the characteristics of the overall environment and task, establish the relationship between satellite nodes and resources, and process the satellite node-resource characteristics to obtain the observation characteristics;

[0008] Step 4: Construct a Markov decision model for multi-agent cooperative reinforcement learning;

[0009] Step 5: Satellite node information, resource status, and mission requirements are used as observation inputs for reinforcement learning. The feature vectors are processed through an encoder-decoder and input into the Markov decision model to output the agent action.

[0010] Step 6: Train the policy network in the Markov decision model, calculate the advantage function of each action for back propagation, and optimize the policy parameters; in each decision cycle, the environment input features, the policy network outputs actions and advantage values, the environment is simulated according to the actions, and the network performs back propagation to update the weight parameters until the network converges; repeat steps 2 to 6 until the predetermined number of decision cycles is reached or the network converges; after each decision cycle, record and analyze the scheduling effect and optimize the resource allocation strategy.

[0011] Furthermore, the communication and task allocation environment model of the multi-beam satellite includes a multi-beam satellite, a gateway station and a multi-beam cell, as well as a forward link between the multi-beam satellite and the gateway station, and a reverse link between the multi-beam satellite and the multi-beam cell; the multi-beam satellite receives a resource request from the gateway station on the forward link, processes the request after resource scheduling tasks are allocated among multiple multi-beam satellites, and finally sends the information to the multi-beam cell via the reverse link.

[0012] Furthermore, in step 2, each satellite S i Initialize a task buffer q i , used to store the tasks that the satellite needs to process;

[0013] According to the needs of ground users k , determine the on-board missions that need to be generated;

[0014] Ground demand k Including users' communication requests and data transmission requirements;

[0015] At each moment t, according to the current ground demand r k , generate a new on-board mission T ki ;

[0016] Task T ki Contains specific parameters of the task, including data volume, priority, and expected completion time; then the task is assigned and the generated task T ki Assigned to the corresponding satellite s i The task buffer q i middle;

[0017] Task allocation is based on the satellite’s current load, task priority and other attributes;

[0018] Perform subsequent task scheduling.

[0019] Furthermore, in step 3, a linear perceptron and an encoder-decoder are used to process the satellite node-resource features to obtain the overall features, which are calculated as follows:

[0020]

[0021] Among them L encoder is the encoder network, L decoder is the decoder network, R is the reward function, V is the value function, Obs=FC{cov(Q),S q} is observed, by the task buffer content Q, channel state S q After convolution, the fully connected layer FC is used to process the initial features f i ;f i After the linear layer and attention mechanism, multiple attention groups are generated and added to the agent feature F i In , the attention mechanism is expressed as:

[0022]

[0023] Among them, Q, K, V correspond to the vectors of sequence, key and value, d K Indicates the dimensions of Q and K.

[0024] Furthermore, the Markov decision model is represented as a sextuple<O,A,R,P,N,γ> ; Where O is the observation set, A is the action set, R is the reward set, P is the state transition set, N is the number of agents, and γ is the discount factor;

[0025] Define the local observation space, action space, reward value space, agent set, state transition probability, and reward discount factor; multi-intelligence advantage decomposition adopts the trust distribution hybrid network and counterfactual baseline method:

[0026]

[0027] The multi-agent advantage decomposition theorem allows agents to make decisions one after another, so that each agent obtains the decision information of the previous agent when making a decision, transforming the multi-agent joint strategy optimization into a sequential strategy search process: Among them, A i is the advantage function value of agent i, o is the observation of agent i, a is the action of agent i, is the value function of agent i, b is the counterfactual baseline, n is the total number of agents, and m is the current agent.

[0028] Furthermore, the observation features obtained in step three are spliced ​​together with the channel throughput, occupied frequency band and coverage cell information as the input of the attention decoder network, and then output the agent action through the fully connected layer and softmax network.

[0029] Furthermore, step six uses a transformer network with an encoder-decoder structure and an independent proximal strategy algorithm to train the strategy network, calculates the advantage function of each action for back propagation, optimizes the strategy parameters based on the obtained data, and uses penalties to limit the strategy update amplitude; in each decision cycle, the environment inputs features, the strategy network outputs actions and advantage values, the environment is simulated according to the actions, the network performs back propagation to update weight parameters, and the network converges after multiple decision cycles.

[0030] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a communication satellite multi-beam resource management method based on deep reinforcement learning.

[0031] A communication satellite multi-beam resource management and control device based on deep reinforcement learning includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the communication satellite multi-beam resource management and control method based on deep reinforcement learning are implemented.

[0032] The advantages of the present invention compared with the prior art are:

[0033] (1) Improving on-board link throughput: This invention uses a multi-agent reinforcement learning algorithm to dynamically adjust and optimize beam resource allocation strategies. By utilizing satellite antenna wave-hopping technology and time-division multiplexing, it flexibly responds to changes in ground user demand and communication traffic, significantly improving the overall on-board link throughput. This dynamic optimization mechanism ensures efficient resource utilization and improves the transmission rate and quality of the communication system.

[0034] (2) Saving Limited On-Chain Resources: This invention constructs a Markov decision model for multi-agent cooperative reinforcement learning, which can effectively reduce resource waste during resource scheduling. The model processes the high-dimensional features of satellite nodes and resources, accurately calculates the advantage function of each action, and optimizes the policy parameters through backpropagation to achieve optimal resource allocation. This method not only improves resource utilization but also reduces the operating costs of the satellite system.

[0035] (3) Reduce latency and improve resource allocation efficiency: This invention uses the Independent Proximal Policy Optimization (IPPO) algorithm, combined with the Transformer network of the encoder-decoder structure, to quickly respond to and process onboard mission requirements. Through multi-level loss functions and backpropagation training, the policy network can continuously optimize and converge in each decision cycle, ensuring the real-time and efficient allocation of resources. This efficient resource scheduling mechanism effectively reduces communication latency and improves the response speed and resource allocation efficiency of the entire system. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0037] Figure 1 Multi-beam satellite resource scheduling scenario;

[0038] Figure 2 Sequential multi-agent reinforcement learning networks. DETAILED DESCRIPTION

[0039] In order to better understand the above technical solution, the technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0040] The following is a further detailed description of a communication satellite multi-beam resource management method based on deep reinforcement learning provided by an embodiment of the present invention in conjunction with the accompanying drawings. Specific implementation methods may include:

[0041] Step 1: Satellite environment simulation

[0042] Multi-beam satellites can generate a large number of beams and flexibly allocate these beam resources through beam hopping technology to adapt to unevenly distributed ground traffic needs and time-varying communication traffic characteristics, thereby improving communication quality and transmission rates. Satellite antenna beam hopping technology can simultaneously activate K beams and use time division multiplexing to cover N (N>K) cells. The set of active beams can change during each onboard time slot.

[0043] Its mathematical model mainly includes the beam resource allocation time window, user spacecraft, user mission, satellites in different orbits and beam resources, visible time window and the establishment of communication links.

[0044] Step 2: Onboard mission simulation

[0045] The generation and transmission of onboard missions are simulated based on real-world data. Onboard missions are generated based on user needs during the simulation. Once a mission is generated, the satellite schedules resources to meet the mission requirements. Metrics such as overall communication mission throughput and latency are used as evaluation criteria for scheduling effectiveness. The environmental simulation step also provides interfaces for calculating these metrics.

[0046] Step 3: Calculate high-dimensional features of the task and environment

[0047] Compute high-dimensional features of the task and environment. Specifically, the abstracted environment model from the first two steps is used as observation input, and further features are extracted. Satellites are used as nodes, and corresponding edges are established between satellites that can communicate via beams. Individual onboard tasks serve as the sequential input to the satellite task queue. Satellite node-resource observations are processed using a linear perceptron (MLP) and an encoder-decoder to obtain overall features.

[0048] Step 4: Construction of Markov decision model for multi-agent cooperative reinforcement learning

[0049] Furthermore, in order to realize resource allocation for multi-agent reinforcement learning, it is necessary to construct a Markov decision model for multi-agent cooperative reinforcement learning. The model is represented by a six-tuple<O,A,R,P,N,γ> , where O represents the set of local observation spaces of all agents, A represents the set of action spaces of all agents, R:O×A→{R_min,R_max} represents the reward value space of multi-agent joint actions, N represents the set of agents, P:O×A×O→R represents the state transition probability, and γ∈[0,1) represents the reward discount factor, which is used to distinguish long-term rewards from short-term rewards.

[0050] Step 5: Sequential reinforcement learning network construction

[0051] After constructing the Markov decision model, a sequential reinforcement learning network is constructed. The satellite environment from steps 1 and 2 serves as the training and execution environment for reinforcement learning. The reinforcement learning observations are the satellite node information, satellite resource status, and onboard mission requirements output by the environment. This information, combined with the previously extracted feature vectors, serves as the input for the reinforcement learning algorithm. The independent proximal policy optimization (IPPO) algorithm is used as the satellite resource scheduling algorithm. The encoder-decoder processed observation features are input, and the IPPO network outputs action probabilities and values.

[0052] Step 6: Multi-level loss function and back-propagation training and learning

[0053] Furthermore, the policy network is trained using a transformer network with an encoder-decoder structure and an independent proximal policy algorithm. The advantage function for each action is calculated and backpropagated. The policy parameters are optimized based on the obtained data, and penalties are used to limit the range of policy updates. In each decision cycle, the environment inputs features, and the policy network outputs an action and advantage value. The environment simulates the action, and the network performs backpropagation to update the weight parameters. After multiple decision cycles, the network converges.

[0054] Through these steps, the satellite environment can be simulated and the reinforcement learning algorithm can be used for resource scheduling, thereby improving the efficiency and performance of the satellite communication system.

[0055] In the solution provided in the embodiment of the present invention, Figure 1 As shown, the present invention implements a satellite beam resource allocation method based on multi-agent reinforcement learning, comprising the following steps:

[0056] Step 1: Satellite environment simulation

[0057] Modeling Figure 1 The communication and task allocation environment for a multi-beam satellite is shown. A mathematical model is developed to generate beam resource allocation time windows, user spacecraft, user tasks, satellites in different orbits and beam resources, visible time windows, and communication link establishment. Beam hopping (BH) technology and time-division multiplexing (TDD) methods are used to flexibly allocate beam resources to adapt to changes in ground transportation needs and communication traffic. A multi-beam satellite receives resource requests from a gateway via the forward link DVB-S2. After resource scheduling and task allocation among multiple multi-beam satellites, the satellite processes the request and finally transmits the information to the multi-beam cell via the reverse link DVB-S2.

[0058] The total bandwidth of the satellite is denoted as B_tot, which is divided into M blocks, with the bandwidth of each block being B_ch = B_tot / M. Assuming that each beam can only be assigned contiguous frequency blocks, this avoids excessive peak-to-average power ratio and improves the satellite's power efficiency. Bandwidth blocks assigned to different beams may overlap, leading to co-channel interference in the overlapping frequency bands. The overlapping factor α_t^(i,j) is defined to represent the co-channel interference from beam i to beam j at time slot t:

[0059]

[0060] The Signal-to-Interference-and-Noise-Ratio (SINR) of cell n is expressed as:

[0061]

[0062] Where N0 is the noise power spectral density, P b is the transmit power of each beam, denoted as P b =P tot / K. Calculate the channel capacity based on SINR:

[0063]

[0064] in Indicates whether cell n is covered. represents the bandwidth of beam k serving cell n. represents the total amount of tasks in the satellite's current beam buffer. The satellite's throughput in the current time slot is defined as:

[0065]

[0066] The system adjusts the cell coverage and frequency division multiplexing strategy of the phased array multi-beam antenna to achieve higher communication task throughput, lower task response delay, and ensure fairness in inter-cell delay as much as possible. The optimization goal is expressed as:

[0067]

[0068] Frequency division multiplexing is used between multiple beams to reduce communication interference, and coordinated decision-making between beams is required to maximize the efficiency of on-board resource utilization.

[0069] Step 2: Onboard mission simulation

[0070] Based on the modeled satellite infrastructure, the generation and transmission of onboard missions are simulated. Onboard missions are generated based on the simulated user needs. Resources are scheduled to meet mission requirements, and indicators such as communication mission throughput and latency are evaluated during the scheduling process. Each satellite S i Initialize a task buffer q i , used to store the tasks that the satellite needs to process. k , determine the onboard missions that need to be generated. Ground requirements k Including user communication requests and data transmission requirements. At each moment t, according to the current ground demand r k , generate a new on-board mission T ki . Task T ki Contains the specific parameters of the task, including data volume, priority, and expected completion time. Then the task is assigned and the generated task T ki Assigned to the corresponding satellite s i The task buffer q i The basis for task allocation is the current load of the satellite, the priority of the task, and other attributes. Further, subsequent task scheduling is carried out.

[0071] Step 3: Calculate high-dimensional features of the task and environment

[0072] The environmental observations calculated in steps 1 and 2 are input into a high-dimensional observation vector. The overall environment and task features are extracted, and the relationship between satellite nodes and resources is established. The satellite node-resource features are processed using a linear perceptron (MLP) and encoder-decoder to obtain the overall features. The calculation method is as follows:

[0073]

[0074] Among them L encoder is the encoder network, L decoder is the decoder network, R is the reward function, V is the value function, Obs=FC{cov(Q),S q} is observed, by the task buffer content Q, channel state S q After convolution, the fully connected layer FC is used to process the initial features f i .f i After the linear layer and attention mechanism, multiple attention groups are generated and added to the agent feature F i In , the attention mechanism is expressed as:

[0075]

[0076] Among them, Q, K, V correspond to the vectors of sequence, key and value, which can be obtained in representation learning, d K Indicates the dimensions of Q and K.

[0077] Step 4: Construction of Markov decision model for multi-agent cooperative reinforcement learning

[0078] Construct a Markov decision model for multi-agent cooperative reinforcement learning, represented as a sextuple<O,A,R,P,N,γ> .

[0079] Define the local observation space, action space, reward value space, agent set, state transition probability and reward discount factor.

[0080] Multi-intelligence advantage decomposition adopts the trust distribution hybrid network and counterfactual baseline method:

[0081]

[0082] The multi-agent advantage decomposition theorem allows agents to make decisions one after another, so that each agent can obtain the decision information of the previous agent when making a decision, transforming the multi-agent joint strategy optimization into a sequential strategy search process:

[0083]

[0084] Step 5: Sequential reinforcement learning network construction

[0085] Using the Independent Proximal Policy Optimization (IPPO) algorithm, a satellite resource scheduling algorithm based on reinforcement learning is constructed. Satellite node information, resource status, and mission requirements are used as observation inputs for reinforcement learning. The feature vectors are processed through an encoder-decoder and then input into the reinforcement learning algorithm.

[0086] The observed feature F obtained in step 3 i , the co-channel throughput, occupied frequency band and coverage cell information are spliced ​​as the input of the attention decoder network, and then output the agent action through the fully connected layer and softmax network.

[0087] Step 6: Multi-level loss function and back-propagation training and learning

[0088] The policy network is trained using an encoder-decoder network structure and an independent proximal policy algorithm. The advantage function for each action is calculated and backpropagated to optimize the policy parameters. During each decision cycle, the environment inputs features, and the policy network outputs an action and its advantage value. The environment simulates the action, and the network performs backpropagation to update the weight parameters until the network converges. Steps 2 to 6 are repeated until the predetermined number of decision cycles is reached or the network converges. After each decision cycle, the scheduling results are recorded and analyzed to optimize the resource allocation strategy.

[0089] The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, which, when executed on a computer, cause the computer to execute Figure 2 The method described.

[0090] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0091] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0092] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0094] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0095] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

[0096] The contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

Claims

1. A communication satellite multi-beam resource management method based on deep reinforcement learning, characterized in that: include: Step 1: Construct a mathematical model of the beam resource allocation time window, user spacecraft, user mission, satellites in different orbits and beam resources, visible time window, and communication link establishment, and establish a communication and mission allocation environment model for multi-beam satellites; Step 2: Based on the communication and task allocation environment model of the multi-beam satellite, simulate the generation and transmission of on-board tasks, and generate on-board tasks based on the simulated user needs; schedule resources to meet the task requirements, and evaluate the communication task throughput and latency during the scheduling process; Step 3: Calculate the observation values ​​of the environment in steps 1 and 2 and input them into a high-dimensional observation vector to extract the characteristics of the overall environment and task, establish the relationship between satellite nodes and resources, and process the satellite node-resource characteristics to obtain the observation characteristics; Step 4: Construct a Markov decision model for multi-agent cooperative reinforcement learning; Step 5: Satellite node information, resource status, and mission requirements are used as observation inputs for reinforcement learning. The feature vectors are processed through an encoder-decoder and input into the Markov decision model to output the agent action. Step 6: Train the policy network in the Markov decision model, calculate the advantage function of each action, perform backpropagation, and optimize the policy parameters. In each decision cycle, the environment inputs features, the policy network outputs actions and advantage values, the environment simulates the actions, and the network performs backpropagation to update weight parameters until the network converges. Repeat steps 2 to 6 until the predetermined number of decision cycles is reached or the network converges. After each decision cycle, record and analyze the scheduling effect and optimize the resource allocation strategy. The Markov decision model is represented as a sextuplet<O,A,R,P,N,γ> ; Where O is the observation set, A is the action set, R is the reward set, P is the state transition set, N is the number of agents, and γ is the discount factor; Define the local observation space, action space, reward value space, agent set, state transition probability, and reward discount factor; multi-intelligence advantage decomposition adopts the trust distribution hybrid network and counterfactual baseline method: The multi-agent advantage decomposition theorem allows agents to make decisions one after another, so that each agent obtains the decision information of the previous agent when making a decision, transforming the multi-agent joint strategy optimization into a sequential strategy search process: Among them, A i is the advantage function value of agent i, o is the observation of agent i, a is the action of agent i, is the value function of agent i, b is the counterfactual baseline, n is the total number of agents, and m is the current agent.

2. A communication satellite multi-beam resource management method based on deep reinforcement learning according to claim 1, characterized in that: The communication and task allocation environment model of the multi-beam satellite includes a multi-beam satellite, a gateway station and a multi-beam cell, as well as a forward link between the multi-beam satellite and the gateway station, and a reverse link between the multi-beam satellite and the multi-beam cell; The multi-beam satellite receives resource requests from the gateway on the forward link, processes the requests after allocating resource scheduling tasks among multiple multi-beam satellites, and finally sends the information to the multi-beam cell via the reverse link.

3. The communication satellite multi-beam resource management method based on deep reinforcement learning according to claim 1 is characterized in that: In the step 2, each satellite S i Initialize a task buffer q i , used to store the tasks that the satellite needs to process; According to the needs of ground users k , determine the on-board missions that need to be generated; Ground demand k Including users' communication requests and data transmission requirements; At each moment t, according to the current ground demand r k , generate a new on-board mission T ki ; Task T ki Contains specific parameters of the task, including data volume, priority, and expected completion time; then the task is assigned and the generated task T ki Assigned to the corresponding satellite s i The task buffer q i middle; Task allocation is based on satellite attributes including current load and task priority; Perform subsequent task scheduling.

4. The communication satellite multi-beam resource management method based on deep reinforcement learning according to claim 1 is characterized in that: In step 3, a linear perceptron and encoder-decoder are used to process the satellite node-resource features to obtain the overall features. The calculation method is as follows: Among them L encoder is the encoder network, L decoder is the decoder network, R is the reward function, V is the value function, Obs=FC{cov(Q),S q } is observed, by the task buffer content Q, channel state S q After convolution, the fully connected layer FC is used to process the initial features f i ;f i After the linear layer and attention mechanism, multiple attention groups are generated and added to the agent feature F i In , the attention mechanism is expressed as: Among them, Q, K, V correspond to the vectors of sequence, key and value, d K Indicates the dimensions of Q and K.

5. The method for multi-beam resource management of communication satellites based on deep reinforcement learning according to claim 1, characterized in that: The observation features obtained in step 3 are spliced ​​together with the channel throughput, occupied frequency band, and coverage cell information as the input of the attention decoder network, and then output the agent action through the fully connected layer and softmax network.

6. The method for multi-beam resource management of communication satellites based on deep reinforcement learning according to claim 1, characterized in that: The step six uses a transformer network with an encoder-decoder structure and an independent proximal strategy algorithm to train the strategy network, calculates the advantage function of each action for back propagation, optimizes the strategy parameters based on the obtained data, and uses penalties to limit the strategy update amplitude; in each decision cycle, the environment inputs features, the strategy network outputs actions and advantage values, the environment is simulated according to the actions, the network performs back propagation to update weight parameters, and the network converges after multiple decision cycles.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

8. A communication satellite multi-beam resource management and control device based on deep reinforcement learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

Images