Communication satellite multi-beam resource management and control method based on deep reinforcement learning

Abstract

The invention discloses a communication satellite multi-beam resource management and control method based on deep reinforcement learning, and belongs to the technical field of satellites. According to the invention, through the multi-agent reinforcement learning algorithm, the allocation strategy of the beam resources can be dynamically adjusted and optimized. A satellite antenna wave hopping technology and a time division multiplexing method are utilized to flexibly cope with changes of ground user requirements and communication flow, so that the overall throughput of a satellite link is remarkably improved. The dynamic optimization mechanism ensures efficient utilization of resources, and improves the transmission rate and quality of a communication system.

Description

Technical Field

[0001] The present invention relates to a communication satellite multi-beam resource management and control method based on deep reinforcement learning, and belongs to the field of satellite technology. Background Art

[0002] Satellite networking communication systems have the ability to provide long-distance, all-weather coverage, a wide coverage range, a long range, and are not affected by geographical factors. They are an important part of modern communication systems, especially in special circumstances where ground communication networks are limited. They show unique advantages. However, with the development of satellite communication technology, the problems of scarce spectrum resources and space congestion have become increasingly prominent. Multi-beam antenna technology has become a new direction to solve this problem. Multi-beam satellite payloads contain multiple sub-beams, which are arranged according to a specific topological structure to effectively cover the target area on the ground. Compared with single-beam antennas, multi-beam antennas have smaller beam widths and greater ground gain, which significantly improves the quality of satellite communication signals. When a single satellite cannot fully cover the target area on the ground or needs to expand coverage, a constellation system consisting of multiple satellites becomes particularly important. Summary of the invention

[0003] The technical problem solved by the present invention is: to overcome the shortcomings of the prior art and provide a communication satellite multi-beam resource management and control method based on deep reinforcement learning, which improves the on-board link throughput, saves limited on-board resources, reduces latency and improves resource allocation efficiency.

[0004] The technical solution of the present invention is: a communication satellite multi-beam resource management and control method based on deep reinforcement learning, comprising:

[0005] Step 1: Construct a mathematical model of the beam resource allocation time window, user spacecraft, user mission, satellites in different orbits and beam resources, visible time window, and communication link establishment, and establish a communication and task allocation environment model for multi-beam satellites;

[0006] Step 2: According to the communication and task allocation environment model of the multi-beam satellite, simulate the generation and transmission of on-board tasks, and generate on-board tasks according to the simulated user needs; schedule resources to meet the task requirements, and evaluate the communication task throughput and latency during the scheduling process;

[0007] Step 3: Calculate and input the observation values ​​of the environment in Step 1 and Step 2 into a high-dimensional observation vector, extract the characteristics of the overall environment and tasks, establish the relationship between satellite nodes and resources, process the satellite node-resource characteristics, and obtain the observation characteristics;

[0008] Step 4: Construct a Markov decision model for multi-agent cooperative reinforcement learning;

[0009] Step 5: Using satellite node information, resource status and task requirements as observation inputs for reinforcement learning, processing feature vectors through encoder-decoder, inputting them into the Markov decision model, and outputting agent actions;

[0010] Step six, train the policy network in the Markov decision model, calculate the advantage function of each action for back propagation, and optimize the policy parameters; in each decision cycle, the environment inputs features, the policy network outputs actions and advantage values, the environment simulates according to the actions, and the network performs back propagation to update weight parameters until the network converges; repeat steps two to six until the predetermined number of decision cycles is reached or the network converges; after each decision cycle, record and analyze the scheduling effect and optimize the resource allocation strategy.

[0011] Furthermore, the communication and task allocation environment model of the multi-beam satellite includes a multi-beam satellite, a gateway station and a multi-beam cell, as well as a forward link between the multi-beam satellite and the gateway station, and a reverse link between the multi-beam satellite and the multi-beam cell; the multi-beam satellite receives a resource request from the gateway station on the forward link, processes the request after resource scheduling tasks are allocated among multiple multi-beam satellites, and finally sends the information to the multi-beam cell through a reverse link.

[0012] Furthermore, in step 2, each satellite S i Initialize a task buffer q i , used to store the tasks that the satellite needs to process;

[0013] According to the needs of ground users k , determine the on-board tasks that need to be generated;

[0014] Ground Requirements k Including users' communication requests and data transmission requirements;

[0015] At each moment t, according to the current ground demand r k , generate a new on-board mission T ki ;

[0016] Task T ki Contains specific parameters of the task, including data volume, priority, and expected completion time; then the task is assigned and the generated task T ki Assigned to the corresponding satellite s i The task buffer q i middle;

[0017] Task allocation is based on the satellite’s current load, task priority and other attributes;

[0018] Perform subsequent task scheduling.

[0019] Furthermore, in step 3, a linear perceptron and an encoder-decoder are used to process satellite node-resource features to obtain overall features, and the calculation method is as follows:

[0020]

[0021] Where L encoder is the encoder network, L decoder is the decoder network, R is the reward function, V is the value function, Obs = FC{cov(Q),S q} is observed, based on the task buffer content Q, channel state S q After convolution, the fully connected layer FC is used to obtain the initial feature f i ;f i After the linear layer and the attention mechanism, multiple groups of attention are generated and added to the agent feature F i In , the attention mechanism is expressed as:

[0022]

[0023] Among them, Q, K, V correspond to the vectors of sequence, key and value, d K Denotes the dimensions of Q and K.

[0024] Furthermore, the Markov decision model is represented by a sextuple<O,A,R,P,N,γ> ; Where O is the observation set, A is the action set, R is the reward set, P is the state transition set, N is the number of agents, and γ is the discount factor;

[0025] Define the local observation space, action space, reward value space, agent set, state transition probability and reward discount factor; multi-intelligence advantage decomposition adopts the trust distribution hybrid network and counterfactual baseline method:

[0026]

[0027] The multi-agent advantage decomposition theorem allows agents to make decisions one after another, so that each agent obtains the decision information of the previous agent when making a decision, and transforms the multi-agent joint strategy optimization into a sequential strategy search process: Among them, A i is the advantage function value of agent i, o is the observation of agent i, a is the action of agent i, is the value function of agent i, b is the counterfactual baseline, n is the total number of agents, and m is the current agent.

[0028] Furthermore, the observation features obtained in the step three are concatenated with the channel throughput, occupied frequency band and coverage cell information as the input of the attention decoder network, and then output the agent action through the fully connected layer and the softmax network.

[0029] Furthermore, the step six uses a transformer network with an encoder-decoder structure and an independent proximal strategy algorithm to train the strategy network, calculates the advantage function of each action for back propagation, optimizes the strategy parameters based on the obtained data, and uses penalties to limit the strategy update amplitude; in each decision cycle, the environment inputs features, the strategy network outputs actions and advantage values, the environment is simulated according to the actions, the network performs back propagation to update weight parameters, and the network converges after multiple decision cycles.

[0030] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a communication satellite multi-beam resource management and control method based on deep reinforcement learning are implemented.

[0031] A communication satellite multi-beam resource management and control device based on deep reinforcement learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the communication satellite multi-beam resource management and control method based on deep reinforcement learning are implemented.

[0032] The advantages of the present invention compared with the prior art are:

[0033] (1) Improving on-board link throughput: The present invention uses a multi-agent reinforcement learning algorithm to dynamically adjust and optimize the allocation strategy of beam resources. By using satellite antenna wave hopping technology and time division multiplexing methods, it can flexibly respond to changes in ground user needs and communication traffic, thereby significantly improving the overall throughput of the on-board link. This dynamic optimization mechanism ensures efficient use of resources and improves the transmission rate and quality of the communication system.

[0034] (2) Saving limited on-chain resources: The present invention constructs a Markov decision model for multi-agent cooperative reinforcement learning, which can effectively reduce resource waste in the resource scheduling process. The model processes the high-dimensional features of satellite nodes and resources, accurately calculates the advantage function of each action, and optimizes the strategy parameters through back propagation, thereby achieving optimal resource allocation. This method not only improves resource utilization, but also reduces the operating cost of the satellite system.

[0035] (3) Reduce latency and improve resource allocation efficiency: The present invention adopts the independent proximal policy optimization (IPPO) algorithm, combined with the transformer network of the encoder-decoder structure, which can quickly respond to and process on-board mission requirements. Through multi-level loss functions and back-propagation training, the policy network can continuously optimize and converge in each decision cycle to ensure the real-time and efficient resource allocation. This efficient resource scheduling mechanism effectively reduces communication latency and improves the response speed and resource allocation efficiency of the overall system. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0037] Figure 1 Multi-beam satellite resource scheduling scenario;

[0038] Figure 2 Sequential multi-agent reinforcement learning networks. DETAILED DESCRIPTION

[0039] In order to better understand the above technical scheme, the technical scheme of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical scheme of the present invention, rather than limitations on the technical scheme of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0040] The following is a further detailed description of a communication satellite multi-beam resource management method based on deep reinforcement learning provided by an embodiment of the present invention in conjunction with the accompanying drawings of the specification. The specific implementation method may include:

[0041] Step 1: Satellite environment simulation

[0042] Multi-beam satellites can generate a large number of beams and flexibly allocate these beam resources through beam hopping technology to adapt to the uneven distribution of ground traffic needs and time-varying communication traffic characteristics, thereby improving communication quality and transmission rate. Satellite antenna beam hopping technology can activate K beams at the same time and use time division multiplexing to cover N (N>K) cells. The activated beam set can change in each satellite time slot.

[0043] Its mathematical model mainly includes the beam resource allocation time window, user spacecraft, user mission, satellites in different orbits and beam resources, visible time window and the establishment of communication links.

[0044] Step 2: Onboard mission simulation

[0045] The generation and transmission of onboard missions are simulated based on actual data. Onboard missions come from the needs of users in the simulation. After the onboard missions are generated, the satellite will schedule resources to meet the mission requirements. The overall communication mission throughput, latency and other indicators in the scheduling process will be used as evaluation criteria for the scheduling effect. The environmental simulation step will also provide relevant calculation interfaces for the indicators.

[0046] Step 3: Calculate high-dimensional features of the task and environment

[0047] Compute high-dimensional features of tasks and environments. Specifically, the environment modeling abstraction in the first two steps is used as observation input and further features are extracted. Satellites are used as nodes, and corresponding edges are established between satellites that can communicate by beam. Single on-board tasks are used as timing inputs of satellite task queues. Satellite node-resource observations are processed by linear perceptrons (MLPs) and encoder-decoders to obtain overall features.

[0048] Step 4: Construction of Markov decision model for multi-agent cooperative reinforcement learning

[0049] Furthermore, in order to realize resource allocation for multi-agent reinforcement learning, it is necessary to construct a Markov decision model for multi-agent cooperative reinforcement learning. The model is represented by a six-tuple<O,A,R,P,N,γ> , where O represents the set of local observation spaces of all agents, A represents the set of action spaces of all agents, R:O×A→{R_min,R_max} represents the reward value space of multi-agent joint actions, N represents the set of agents, P:O×A×O→R represents the state transition probability, and γ∈[0,1) represents the reward discount factor, which is used to distinguish long-term rewards from short-term rewards.

[0050] Step 5: Sequential reinforcement learning network construction

[0051] After constructing the Markov decision model, the sequential reinforcement learning network is further constructed. The satellite environment in steps one and two will serve as the training and execution environment for reinforcement learning. The observation of reinforcement learning is the satellite node information, satellite resource status, and on-board mission requirements output by the environment. This information is superimposed with the feature vector extracted previously and used as the input of the reinforcement learning algorithm. The independent proximal policy optimization (IPPO) algorithm is used as the satellite resource scheduling algorithm. The observation features processed by the encoder and decoder are input, and the independent proximal policy algorithm network outputs the action probability and value.

[0052] Step 6: Multi-level loss function and back propagation training and learning

[0053] Furthermore, the policy network is trained using a transformer network with an encoder-decoder structure and an independent proximal policy algorithm, the advantage function of each action is calculated for back propagation, the policy parameters are optimized based on the obtained data, and the policy update amplitude is limited by penalty. In each decision cycle, the environment inputs features, the policy network outputs actions and advantage values, the environment simulates the actions, the network performs back propagation to update the weight parameters, and the network converges after multiple decision cycles.

[0054] Through these steps, the satellite environment can be simulated and the reinforcement learning algorithm can be used for resource scheduling, thereby improving the efficiency and performance of the satellite communication system.

[0055] In the solution provided in the embodiment of the present invention, Figure 1 As shown, the present invention implements a satellite beam resource allocation method based on multi-agent reinforcement learning, comprising the following steps:

[0056] Step 1: Satellite environment simulation

[0057] Modeling Figure 1 The communication and task allocation environment of the multi-beam satellite shown in the figure. Generate a mathematical model for the beam resource allocation time window, user spacecraft, user tasks, satellites in different orbits and beam resources, visible time window, and communication link establishment. Utilize beam hopping (BH) technology and time division multiplexing method to flexibly allocate beam resources to adapt to changes in ground transportation needs and communication traffic. The multi-beam satellite receives resource requests from the gateway station on the forward link DVB-S2, processes the request after resource scheduling tasks are allocated among multiple multi-beam satellites, and finally sends the information to the multi-beam cell via the reverse link DVB-S2.

[0058] The total bandwidth of the satellite is denoted as B_tot, which is divided into M blocks, and the bandwidth of each block is B_ch = B_tot / M. It is assumed that each beam can only be assigned continuous frequency blocks to avoid excessive peak-to-average power ratio and improve the power efficiency of the satellite. The bandwidth blocks assigned to different beams may overlap, which leads to co-channel interference in the overlapping frequency bands. The overlapping factor α_t^(i,j) is defined to represent the co-channel interference from beam i to beam j in time slot t:

[0059]

[0060] The signal-to-interference-and-noise-ratio (SINR) of cell n is expressed as:

[0061]

[0062] Where, N0 is the noise power spectral density, P b is the transmit power of each beam, denoted as P b =P tot / K. Calculate the channel capacity based on SINR:

[0063]

[0064] in Indicates whether cell n is covered. represents the bandwidth of beam k serving cell n. represents the total amount of tasks in the satellite's current beam buffer. The satellite's throughput in the current time slot is defined as:

[0065]

[0066] The system adjusts the cell coverage and frequency division multiplexing strategy of the phased array multi-beam antenna to obtain higher communication task throughput, lower task response delay and ensure fairness of delay between cells as much as possible. The optimization goal is expressed as:

[0067]

[0068] Frequency division multiplexing is used between multiple beams to reduce communication interference, and the beams need to make coordinated decisions to maximize the efficiency of onboard resource utilization.

[0069] Step 2: Onboard mission simulation

[0070] Based on the modeled satellite infrastructure, simulate the generation and transmission of onboard missions. Generate onboard missions based on simulated user needs. Schedule resources to meet mission requirements and evaluate indicators such as communication mission throughput and latency during scheduling. Each satellite S i Initialize a task buffer q i , used to store the tasks that the satellite needs to process. k , determine the onboard tasks that need to be generated. Ground requirements k Including user communication requests and data transmission requirements. At each time t, according to the current ground demand r k , generate a new on-board mission T ki . Task T ki Contains specific parameters of the task, including data volume, priority, and expected completion time. Then the task is assigned and the generated task T ki Assigned to the corresponding satellite s i The task buffer q i The basis for task allocation is the current load of the satellite, the priority of the task, and other attributes. Further, subsequent task scheduling is performed.

[0071] Step 3: Calculate high-dimensional features of the task and environment

[0072] The environmental observation calculations in steps 1 and 2 are input into the high-dimensional observation vector. The features of the overall environment and tasks are extracted, and the relationship between satellite nodes and resources is established. The satellite node-resource features are processed using a linear perceptron (MLP) and encoder-decoder to obtain the overall features. The calculation method is as follows:

[0073]

[0074] Where L encoder is the encoder network, L decoder is the decoder network, R is the reward function, V is the value function, Obs = FC{cov(Q),S q} is observed, based on the task buffer content Q, channel state S q After convolution, the fully connected layer FC is used to obtain the initial feature f i .f i After the linear layer and the attention mechanism, multiple groups of attention are generated and added to the agent feature F i In , the attention mechanism is expressed as:

[0075]

[0076] Among them, Q, K, V correspond to the vectors of sequence, key and value, which can be obtained in representation learning, d K Denotes the dimensions of Q and K.

[0077] Step 4: Construction of Markov decision model for multi-agent cooperative reinforcement learning

[0078] Construct a Markov decision model for multi-agent cooperative reinforcement learning, represented as a sextuple<O,A,R,P,N,γ> .

[0079] Define the local observation space, action space, reward value space, agent set, state transition probability and reward discount factor.

[0080] The multi-intelligence advantage decomposition adopts the trust distribution hybrid network and counterfactual baseline method:

[0081]

[0082] The multi-agent advantage decomposition theorem allows agents to make decisions one after another, so that each agent can obtain the decision information of the previous agent when making a decision, and transforms the multi-agent joint strategy optimization into a sequential strategy search process:

[0083]

[0084] Step 5: Sequential reinforcement learning network construction

[0085] The independent proximal policy optimization (IPPO) algorithm is used to construct a satellite resource scheduling algorithm based on reinforcement learning. Satellite node information, resource status and mission requirements are used as observation inputs of reinforcement learning, and the feature vector is processed through the encoder-decoder and input into the reinforcement learning algorithm.

[0086] The observed feature F obtained in step 3 i , the co-channel throughput, occupied frequency band and coverage cell information are concatenated as the input of the attention decoder network, and then pass through the fully connected layer and softmax network to output the agent action.

[0087] Step 6: Multi-level loss function and back propagation training and learning

[0088] The policy network is trained using an encoder-decoder network structure and an independent proximal policy algorithm. The advantage function of each action is calculated for back propagation to optimize the policy parameters. In each decision cycle, the environment inputs features, the policy network outputs actions and advantage values, the environment simulates the actions, and the network back propagates to update the weight parameters until the network converges. Repeat steps 2 to 6 until the predetermined number of decision cycles is reached or the network converges. After each decision cycle, the scheduling effect is recorded and analyzed to optimize the resource allocation strategy.

[0089] The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer executes Figure 2 The method described.

[0090] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0091] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0092] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0094] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0095] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

[0096] The contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

Claims

1. A communication satellite multi-beam resource management method based on deep reinforcement learning, characterized in that: include: Step 1: Construct a mathematical model of the beam resource allocation time window, user spacecraft, user mission, satellites in different orbits and beam resources, visible time window, and communication link establishment, and establish a communication and task allocation environment model for multi-beam satellites; Step 2: According to the communication and task allocation environment model of the multi-beam satellite, simulate the generation and transmission of on-board tasks, and generate on-board tasks according to the simulated user needs; schedule resources to meet the task requirements, and evaluate the communication task throughput and latency during the scheduling process; Step 3: Calculate and input the observation values ​​of the environment in Step 1 and Step 2 into a high-dimensional observation vector, extract the characteristics of the overall environment and tasks, establish the relationship between satellite nodes and resources, process the satellite node-resource characteristics, and obtain the observation characteristics; Step 4: Construct a Markov decision model for multi-agent cooperative reinforcement learning; Step 5: Using satellite node information, resource status and task requirements as observation inputs for reinforcement learning, processing feature vectors through encoder-decoder, inputting them into the Markov decision model, and outputting agent actions; Step six, train the policy network in the Markov decision model, calculate the advantage function of each action for back propagation, and optimize the policy parameters; in each decision cycle, the environment inputs features, the policy network outputs actions and advantage values, the environment simulates according to the actions, and the network performs back propagation to update weight parameters until the network converges; repeat steps two to six until the predetermined number of decision cycles is reached or the network converges; after each decision cycle, record and analyze the scheduling effect and optimize the resource allocation strategy.

2. According to claim 1, a communication satellite multi-beam resource management method based on deep reinforcement learning is characterized in that: The communication and task allocation environment model of the multi-beam satellite includes a multi-beam satellite, a gateway station and a multi-beam cell, as well as a forward link between the multi-beam satellite and the gateway station, and a reverse link between the multi-beam satellite and the multi-beam cell; The multi-beam satellite receives resource requests from the gateway in the forward link, processes the requests after allocating resource scheduling tasks among multiple multi-beam satellites, and finally sends the information to the multi-beam cell via the reverse link.

3. According to claim 1, a communication satellite multi-beam resource management method based on deep reinforcement learning is characterized in that: In step 2, each satellite S i Initialize a task buffer q i , used to store the tasks that the satellite needs to process; According to the needs of ground users k , determine the on-board tasks that need to be generated; Ground Requirements k Including users' communication requests and data transmission requirements; At each moment t, according to the current ground demand r k , generate a new on-board mission T ki ; Task T ki Contains specific parameters of the task, including data volume, priority, and expected completion time; then the task is assigned and the generated task T ki Assigned to the corresponding satellite s i The task buffer q i middle; Task allocation is based on the satellite’s current load, task priority and other attributes; Perform subsequent task scheduling.

4. According to claim 1, a communication satellite multi-beam resource management method based on deep reinforcement learning is characterized in that: In step 3, a linear perceptron and an encoder-decoder are used to process satellite node-resource features to obtain overall features, which are calculated as follows: Where L encoder is the encoder network, L decoder is the decoder network, R is the reward function, V is the value function, Obs = FC{cov(Q),S q } is observed, based on the task buffer content Q, channel state S q After convolution, the fully connected layer FC is used to obtain the initial feature f i ;f i After the linear layer and the attention mechanism, multiple groups of attention are generated and added to the agent feature F i In , the attention mechanism is expressed as: Among them, Q, K, V correspond to the vectors of sequence, key and value, d K Denotes the dimensions of Q and K.

5. According to claim 1, a communication satellite multi-beam resource management method based on deep reinforcement learning is characterized in that: The Markov decision model is represented as a six-tuple<O,A,R,P,N,γ> ; Where O is the observation set, A is the action set, R is the reward set, P is the state transition set, N is the number of agents, and γ is the discount factor; Define the local observation space, action space, reward value space, agent set, state transition probability and reward discount factor; multi-intelligence advantage decomposition adopts the trust distribution hybrid network and counterfactual baseline method: The multi-agent advantage decomposition theorem allows agents to make decisions one after another, so that each agent obtains the decision information of the previous agent when making a decision, and transforms the multi-agent joint strategy optimization into a sequential strategy search process: Among them, A i is the advantage function value of agent i, o is the observation of agent i, a is the action of agent i, is the value function of agent i, b is the counterfactual baseline, n is the total number of agents, and m is the current agent.

6. According to claim 1, a communication satellite multi-beam resource management method based on deep reinforcement learning is characterized in that: The observation features obtained in the step 3 are concatenated with the channel throughput, occupied frequency band and coverage cell information as the input of the attention decoder network, and then output the agent action through the fully connected layer and softmax network.

7. According to claim 1, a communication satellite multi-beam resource management and control method based on deep reinforcement learning is characterized in that: The step six uses a transformer network with an encoder-decoder structure and an independent proximal strategy algorithm to train the strategy network, calculates the advantage function of each action for back propagation, optimizes the strategy parameters based on the obtained data, and uses penalties to limit the strategy update amplitude; in each decision cycle, the environment inputs features, the strategy network outputs actions and advantage values, the environment is simulated according to the actions, the network performs back propagation to update weight parameters, and the network converges after multiple decision cycles.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A communication satellite multi-beam resource management and control device based on deep reinforcement learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images