Satellite communication anti-jamming decision method and system based on hierarchical reinforcement learning

Abstract

This invention relates to the field of satellite communication anti-interference technology, and discloses a satellite communication anti-interference decision-making method and system based on hierarchical reinforcement learning. The method includes: sensing the current electromagnetic environment and acquiring interference noise intensity information for each channel across the entire frequency band; based on the interference noise intensity information and the current communication state, selecting a communication system and information rate combination suitable for the current electromagnetic environment through a master control agent; calling a sub-agent corresponding to the selected communication system and information rate combination, and having the sub-agent execute channel selection and power control decisions; calculating a reward value based on the communication results and storing the experience data from the decision-making process; and using the stored experience data, updating the policy network and value network of the master control agent through a near-end policy optimization algorithm to optimize subsequent decisions. This method reduces the dimensionality of the decision space, improves decision quality, and exhibits good adaptability and generalization performance.

Description

Technical Field

[0001] This invention relates to the field of satellite communication anti-jamming technology, and in particular to a satellite communication anti-jamming decision-making method and system based on hierarchical reinforcement learning. Background Technology

[0002] With the widespread application of satellite communication technology in military communications, emergency communications, ocean shipping, and aviation broadband access, the electromagnetic environment is becoming increasingly complex, and interference methods are constantly emerging, placing higher demands on the anti-interference capabilities of satellite communication systems. Traditional anti-interference technologies mainly include spread spectrum communication, frequency hopping communication, adaptive power control, and channel coding. These methods are usually based on preset fixed strategies or rules for parameter configuration and lack the ability to adapt to dynamically changing electromagnetic environments. When the type, intensity, or frequency band of interference changes, traditional methods struggle to adjust the communication system and parameters in a timely manner, leading to a decline in communication quality or even communication interruption.

[0003] In recent years, with the development of artificial intelligence technology, intelligent anti-interference decision-making methods based on reinforcement learning have gradually become a research hotspot. Reinforcement learning learns optimal decision strategies through the interaction between the agent and the environment, enabling adaptive anti-interference without relying on an accurate interference model. However, existing reinforcement learning-based anti-interference methods mainly adopt a single-agent architecture, handling multiple decision tasks such as communication system selection, channel selection, and power control in a unified manner, leading to an explosion in the dimensionality of the decision space, high training difficulty, and slow convergence speed. Furthermore, different communication systems have vastly different anti-interference mechanisms and parameter optimization spaces, making it difficult for a single agent to simultaneously master the optimization strategies of multiple systems, thus limiting the system's flexibility and scalability.

[0004] Therefore, there is an urgent need for an anti-interference decision-making scheme based on a hierarchical decision-making architecture, which can effectively compress the decision space, significantly improve training efficiency, and provide adaptive anti-interference capabilities to complex electromagnetic environments, thereby overcoming the problems of state space explosion and multi-system optimization difficulties faced by single-agent architectures in existing technologies. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a hierarchical reinforcement learning-based anti-interference decision-making method for satellite communication systems. This method employs a hierarchical decision-making architecture with a master agent and sub-agents working together. The anti-interference decision-making task is decomposed into a system selection layer and a parameter optimization layer, effectively reducing the dimensionality of the decision space and improving training efficiency and decision quality. Each sub-agent is independently pre-trained for different communication systems, fully leveraging the anti-interference advantages of each system and enhancing the system's adaptability and generalization performance in complex electromagnetic environments.

[0006] On the one hand, this invention provides a satellite communication anti-jamming decision-making method based on hierarchical reinforcement learning, comprising the following steps: S1: Sensing the current electromagnetic environment and acquiring information on the interference noise intensity of each channel across the entire frequency band; S2: Based on the interference noise intensity information and the current communication status, the pre-trained master control agent selects a combination of communication system and information rate that is suitable for the current electromagnetic environment; S3: Based on the selected combination of communication system and information rate, invoke the pre-trained sub-agent corresponding to the communication system, and have the sub-agent execute channel selection and power control decisions. S4: Calculate the reward value based on the communication results, and store the state, action, reward and next state in the decision-making process as experience data; S5: Using stored experience data, update the policy network and value network of the master agent through a near-end policy optimization algorithm to optimize subsequent decisions.

[0007] Furthermore, in step S2, the input state information of the master control agent includes the interference noise intensity vector of each channel in the full frequency band, the communication mode and rate combination used by the current communication system, the channel used by the current communication, and the power used by the current communication. The action information output by the master control agent is the selected combination of communication mode and rate.

[0008] Furthermore, in step S3, the sub-agent includes four pre-trained sub-agents, each corresponding to a different communication system: The first sub-agent corresponds to the non-spread spectrum communication system. Its input states include the interference noise intensity vector of each channel in the full frequency band, the channel used for current communication, and the power used for current communication. Its output actions include channel selection action and power selection action. The second sub-agent corresponds to the direct sequence spread spectrum communication system. Its input states include the interference noise intensity vectors of a preset number of channels, the channel used for the current communication, and the power used for the current communication. Its output actions include channel selection actions and power selection actions. The third sub-agent corresponds to a direct-sequence spread spectrum frequency hopping hybrid spread spectrum communication system and the frequency hopping range accounts for one-quarter of the entire frequency band. Its input states include the interference noise intensity vectors of a preset number of channels and the power used for current communication. The output action is a power selection action. The fourth sub-agent corresponds to a direct-sequence spread spectrum and frequency hopping hybrid spread spectrum communication system, and the frequency hopping range covers the entire frequency band. Its input states include the interference noise intensity vectors of a preset number of channels and the power used for current communication. Its output action is a power selection action.

[0009] Further, in step S5, the near-end policy optimization algorithm updates the policy network by minimizing the pruning objective function, which is expressed as: , in, Importance sampling ratio, For generalized dominance function estimation, For the maximum time step, For timing difference error, for Instant rewards for each moment For state Value function estimation, As a discount factor, To estimate the parameters for the generalized dominance function, For trimming parameters, These are the policy network parameters.

[0010] Preferably, in step S5, the near-end policy optimization algorithm updates the value network by minimizing the value loss function, which is expressed as: , in, Output for value network For target value; The total loss function of the policy optimization algorithm is composed of the policy loss, the value function loss, and the entropy regularization term. The total loss function is expressed as follows: , in, The value loss coefficient, Here is the entropy regularization coefficient. Let be the policy entropy.

[0011] Furthermore, in step S3, the sub-agent undergoes pre-training through the following steps: S31: Construct an independent training environment for each sub-agent and set the state space, action space and reward function under the corresponding communication system; S32: Based on the characteristics of each communication system, construct an independent Actor-Critic network structure for each sub-agent; S33: Employ a near-end strategy optimization algorithm to train each sub-agent independently until it reaches the preset performance index under the corresponding communication system; S34: Save the network parameters of each trained sub-agent for the master agent to use during decision-making.

[0012] Furthermore, in step S4, the reward value is a combined reward value calculated based on whether the communication is successful, the communication rate, the channel switching cost, and the power consumption.

[0013] Furthermore, in step S5, when updating the policy network and value network of the master agent, the near-end policy optimization algorithm uses an experience replay buffer to store sample data. When the number of samples in the buffer reaches a preset update threshold, it randomly samples from the buffer to perform multiple rounds of network parameter updates.

[0014] On the other hand, the present invention also provides a satellite communication anti-jamming decision system based on hierarchical reinforcement learning, comprising: The environmental sensing module is used to sense the current electromagnetic environment and obtain interference noise intensity information for each channel across the entire frequency band. The master control decision module is used to select a combination of communication system and information rate that is adapted to the current electromagnetic environment based on the interference noise intensity information and the current communication status through a pre-trained master control agent. The sub-agent invocation module is used to invoke a pre-trained sub-agent corresponding to the selected communication system and information rate combination, and the sub-agent performs channel selection and power control decisions. The experience storage module is used to calculate the reward value based on the communication results and store the state, action, reward and next state in the decision-making process as experience data. The network update module is used to update the policy network and value network of the master agent using stored experience data and a near-end policy optimization algorithm to optimize subsequent decisions.

[0015] In addition, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the satellite communication anti-interference decision-making method based on hierarchical reinforcement learning as described above.

[0016] Meanwhile, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the satellite communication anti-interference decision method based on hierarchical reinforcement learning as described above.

[0017] Compared with the prior art, the beneficial effects of the present invention are: This invention, through a hierarchical decision-making architecture that coordinates the master agent and sub-agents, decomposes anti-interference decision-making into a system selection layer and a parameter optimization layer, effectively solving the problem of decision space dimension explosion in single agent architecture, and significantly improving training efficiency and convergence speed. This invention dynamically selects the communication system by the master control agent and calls the corresponding pre-trained sub-agents to perform channel and power control, enabling each sub-agent to be specifically optimized for the characteristics of different systems, thereby improving the system's adaptability and generalization performance in complex electromagnetic environments. This invention enables the system to autonomously learn and adapt to the dynamic electromagnetic environment without relying on an accurate interference model through a closed-loop learning mechanism of environmental perception, hierarchical decision-making, experience storage and network updates, thereby enhancing the anti-interference robustness of the satellite communication system. This invention employs a near-end policy optimization algorithm, which balances the relationship between exploration and exploitation while ensuring the stability of policy updates through joint optimization of policy pruning, value loss, and entropy regularization, thereby further improving the convergence performance of the algorithm. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a satellite communication anti-interference decision-making method based on hierarchical reinforcement learning according to the present invention. Figure 2 This is a schematic diagram of the overall architecture of a satellite communication anti-interference decision system based on hierarchical reinforcement learning according to the present invention; Figure 3 This is a flowchart illustrating the training process of a master intelligent agent and sub-intelligent agents according to the present invention. Figure 4 This is a schematic diagram of the decision-making demonstration interface of the system of the present invention under different electromagnetic environments. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] The specific embodiments of the present invention will be described below with reference to the accompanying drawings and examples. Example 1

[0021] Please see Figure 1 The technical solution of the satellite communication anti-interference decision-making method based on hierarchical reinforcement learning provided in this embodiment includes the following steps: S1: Sensing the current electromagnetic environment and acquiring information on the interference noise intensity of each channel across the entire frequency band; S2: Based on the interference noise intensity information and the current communication status, the pre-trained master control agent selects a combination of communication system and information rate that is suitable for the current electromagnetic environment; S3: Based on the selected combination of communication system and information rate, invoke the pre-trained sub-agent corresponding to the communication system, and have the sub-agent execute channel selection and power control decisions. S4: Calculate the reward value based on the communication results, and store the state, action, reward and next state in the decision-making process as experience data; S5: Using stored experience data, update the policy network and value network of the master agent through a near-end policy optimization algorithm to optimize subsequent decisions.

[0022] To implement the above method, the hierarchical reinforcement learning system of the present invention consists of a master agent and four sub-agents, such as... Figure 2 As shown, the system adopts a hierarchical decision-making architecture, decomposing the communication anti-interference decision-making task into two levels. The master control agent is responsible for selecting the optimal combination of communication system and information rate based on the current electromagnetic environment. For the selected system, the corresponding sub-agents perform specific channel selection and power control.

[0023] The system supports four communication modes, each with different anti-interference capabilities and applicable scenarios: System 1 is a basic phase shift keying modulation with an operating frequency band of 400MHz, divided into 80 5MHz channels; this system has no spreading gain and is suitable for high-speed transmission in low-interference environments.

[0024] System 2 adopts direct sequence spread spectrum technology, operates in a frequency band of 400MHz, and is divided into 20 channels of 20MHz; it provides medium spread spectrum gain and is suitable for moderate interference environments.

[0025] System 3 combines direct sequence spread spectrum and frequency hopping technology, with a frequency hopping range of 100MHz (covering the first 5 20MHz channels); it provides strong spread spectrum gain and frequency hopping anti-interference capability, and is suitable for environments with strong interference.

[0026] System 4 employs full-band frequency hopping technology, covering the entire 400MHz band (20 20MHz channels). It provides the strongest anti-interference capability and is suitable for extreme interference environments.

[0027] like Figure 3 As shown, the complete process of sub-agent pre-training and master agent training in the hierarchical reinforcement learning system of this invention includes steps such as environment initialization, action sampling, reward calculation, experience storage, and network update. All agents adopt the Actor-Critic architecture of the proximal policy optimization (PPO) algorithm, which includes a policy network and a value network.

[0028] Specifically, in step S2, the input state information of the master agent includes the interference noise intensity vectors of each channel in the full frequency band, the communication system and rate combination used by the current communication system, the channel used for the current communication, and the power used for the current communication ([total_IN, current_system, current_ch, current_power]); The action information output by the master agent is the selected communication system and rate combination. Among them, the communication system and rate combination action are selected through the PPO policy network.

[0029] Secondly, in step S3, the sub-agent includes four pre-trained sub-agents corresponding to different communication systems: The first sub-agent corresponds to the non-spreading communication system. Its input state includes the interference noise intensity vectors of each channel in the full frequency band, the channel used for the current communication, and the power used for the current communication ([total_IN, current_ch, current_power]). The output actions include channel selection actions and power selection actions; The second sub-agent corresponds to the direct sequence spread spectrum communication system. Its input state includes the interference noise intensity vectors of a preset number of channels, the channel used for the current communication, and the power used for the current communication ([total_IN_DSSS, current_ch, current_power]), where total_IN_DSSS represents the interference noise intensity of M channels, and M < N). The output actions include channel selection actions and power selection actions; The third sub-agent corresponds to the direct spread frequency hopping hybrid spread spectrum communication system and the frequency hopping range occupies one-fourth of the full frequency band. Its input state includes the interference noise intensity vectors of a preset number of channels and the power used for the current communication ([total_IN_DSSS_1 / 4, current_power]). The output action is the power selection action; The fourth sub-agent corresponds to the direct spread frequency hopping hybrid spread spectrum communication system and the frequency hopping range covers the full frequency band. Its input state includes the interference noise intensity vectors of a preset number of channels and the power used for the current communication ([total_IN_DSSS, current_power]). The output action is the power selection action.

[0030] In step S5, the proximal policy optimization algorithm (PPO) updates the policy network by minimizing the clipped objective function, and the clipped objective function is expressed as: , Among them, Importance sampling ratio, This refers to the Generalized Advantage Estimation (GAE). For the maximum time step, For timing difference error, for Instant rewards for each moment For state Value function estimation, As a discount factor, To estimate the parameters for the generalized dominance function, For trimming parameters, These are the policy network parameters.

[0031] Furthermore, the Proximal Policy Optimization (PPO) algorithm updates the value network by minimizing a value loss function, which is expressed as: , in, Output for value network For target value; The total loss function of the policy optimization algorithm is composed of the policy loss, the value function loss, and the entropy regularization term. The total loss function is expressed as follows: , in, The value loss coefficient, Here is the entropy regularization coefficient. Let be the policy entropy.

[0032] In step S3, the sub-agent undergoes pre-training through the following steps: S31: Construct an independent training environment for each sub-agent and set the state space, action space and reward function under the corresponding communication system; S32: Based on the characteristics of each communication system, construct an independent Actor-Critic network structure for each sub-agent; S33: Employ a near-end strategy optimization algorithm to train each sub-agent independently until it reaches the preset performance index under the corresponding communication system; S34: Save the network parameters of each trained sub-agent for the master agent to use during decision-making.

[0033] In step S4, the reward value is a combined reward value calculated based on whether the communication was successful, the communication rate, the channel switching cost, and the power consumption. The specific combined reward... .

[0034] Finally, in step S5, when updating the policy network and value network of the master agent, the proximal policy optimization algorithm (PPO) uses an experience replay buffer to store sample data. When the number of samples in the buffer reaches a preset update threshold, it randomly samples from the buffer to perform multiple rounds of network parameter updates.

[0035] Specifically, in this embodiment, the detailed steps for training the sub-agent are as follows: (1) Initialization: Construct an independent training environment for each sub-agent, and set the state space, action space and reward function according to the corresponding communication system; (2) Network architecture construction: Design an independent Actor-Critic network structure according to the characteristics of each communication system, and adjust the number of network layers and neurons according to the state dimension and action dimension of different systems.

[0036] (3) Strategy sampling, for each time step Obtain the action probability distribution from the policy network Acquire sampling action Execute actions Get rewards and the next state Store and transfer tuples ; (4) Batch update: After collecting enough samples, perform multiple rounds of updates (K=8, 9, 10): (5) Performance evaluation: calculate the average return and training curve to determine whether the preset performance indicators have been met; (6) Model saving: Save the network parameters of the sub-agents that have been trained and have reached the preset performance indicators for use by the master agent.

[0037] The detailed steps for training the master control agent are as follows: This invention provides a visual testing system based on the Streamlit framework for the above-mentioned methods, such as... Figure 4 As shown, this diagram visually demonstrates the performance of the hierarchical reinforcement learning-based anti-interference decision-making method in practical applications, and it has the following characteristics: High degree of visualization: It comprehensively displays every aspect of the decision-making process through various forms such as spectrum charts, bar charts, and indicator cards; Highly interactive: It supports real-time parameter adjustment and scene switching, allowing users to quickly verify system performance under different conditions; Complete information: It displays both the final decision result and the intermediate calculation process, making it easier to understand the decision-making logic of the agent; Highly practical: The parameter configuration and environment generation logic are completely consistent with the training code, ensuring the authenticity and reliability of the test results; This demonstration system verifies that the hierarchical reinforcement learning method of this invention can make accurate anti-interference decisions in various complex electromagnetic environments, achieve effective collaboration between the master intelligent agent and the sub-intelligent agents, and fully demonstrate the intelligence, adaptability and practicality of the method.

[0038] Based on this, the present invention provides a satellite communication anti-jamming decision system based on hierarchical reinforcement learning, comprising: The environmental sensing module is used to sense the current electromagnetic environment and obtain interference noise intensity information for each channel across the entire frequency band. The master control decision module is used to select a combination of communication system and information rate that is adapted to the current electromagnetic environment based on the interference noise intensity information and the current communication status through a pre-trained master control agent. The sub-agent invocation module is used to invoke a pre-trained sub-agent corresponding to the selected communication system and information rate combination, and the sub-agent performs channel selection and power control decisions. The experience storage module is used to calculate the reward value based on the communication results and store the state, action, reward and next state in the decision-making process as experience data. The network update module is used to update the policy network and value network of the master agent using stored experience data and a near-end policy optimization algorithm to optimize subsequent decisions.

[0039] It should be noted that the steps in the satellite communication anti-interference decision-making method based on hierarchical reinforcement learning provided in this embodiment can be implemented based on the corresponding modules in the satellite communication anti-interference decision-making system based on hierarchical reinforcement learning. Those skilled in the art can refer to the technical solution of the system to implement the steps of the method. That is, the embodiments in the system can be understood as preferred examples of implementing the method, and will not be elaborated here.

[0040] Besides implementing the system and its various devices provided by this invention in purely computer-readable program code, the same functions can be achieved by logically programming the method steps, making the system and its various devices of this invention appear as logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices provided by this invention can be considered as a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0041] Finally, it should be noted that the above description is only a preferred embodiment of the present invention, and the scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be pointed out that for those skilled in the art, any improvements and modifications made without departing from the principle of the present invention should also be considered within the scope of protection of the present invention.

[0042] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A satellite communication anti-interference decision-making method based on hierarchical reinforcement learning, characterized in that, Includes the following steps: S1: Sensing the current electromagnetic environment and acquiring information on the interference noise intensity of each channel across the entire frequency band; S2: Based on the interference noise intensity information and the current communication status, the pre-trained master control agent selects a combination of communication system and information rate that is suitable for the current electromagnetic environment; S3: Based on the selected combination of communication system and information rate, invoke the pre-trained sub-agent corresponding to the communication system, and have the sub-agent execute channel selection and power control decisions. S4: Calculate the reward value based on the communication results, and store the state, action, reward and next state in the decision-making process as experience data; S5: Using stored experience data, update the policy network and value network of the master agent through a near-end policy optimization algorithm to optimize subsequent decisions.

2. The satellite communication anti-interference decision-making method based on hierarchical reinforcement learning according to claim 1, characterized in that, In step S2, the input state information of the master control agent includes the interference noise intensity vector of each channel in the full frequency band, the communication mode and rate combination used by the current communication system, the channel used by the current communication, and the power used by the current communication. The action information output by the master control agent is the selected combination of communication mode and rate.

3. The satellite communication anti-interference decision-making method based on hierarchical reinforcement learning according to claim 1, characterized in that, In step S3, the sub-agent includes four pre-trained sub-agents, each corresponding to a different communication system: The first sub-agent corresponds to the non-spread spectrum communication system. Its input states include the interference noise intensity vector of each channel in the full frequency band, the channel used for current communication, and the power used for current communication. Its output actions include channel selection action and power selection action. The second sub-agent corresponds to the direct sequence spread spectrum communication system. Its input states include the interference noise intensity vectors of a preset number of channels, the channel used for the current communication, and the power used for the current communication. Its output actions include channel selection actions and power selection actions. The third sub-agent corresponds to a direct-sequence spread spectrum frequency hopping hybrid spread spectrum communication system and the frequency hopping range accounts for one-quarter of the entire frequency band. Its input states include the interference noise intensity vectors of a preset number of channels and the power used for current communication. The output action is a power selection action. The fourth sub-agent corresponds to a direct-sequence spread spectrum and frequency hopping hybrid spread spectrum communication system, and the frequency hopping range covers the entire frequency band. Its input states include the interference noise intensity vectors of a preset number of channels and the power used for current communication. Its output action is a power selection action.

4. The satellite communication anti-interference decision-making method based on hierarchical reinforcement learning according to claim 1, characterized in that, In step S5, the near-end policy optimization algorithm updates the policy network by minimizing the pruning objective function, which is expressed as: ， in, Importance sampling ratio, For generalized dominance function estimation, For the maximum time step, For timing difference error, for Instant rewards for each moment For state Value function estimation, As a discount factor, To estimate the parameters for the generalized dominance function, For trimming parameters, These are the policy network parameters.

5. The satellite communication anti-interference decision-making method based on hierarchical reinforcement learning according to claim 4, characterized in that, In step S5, the near-end policy optimization algorithm updates the value network by minimizing the value loss function, which is expressed as: ， in, Output for value network For target value; The total loss function of the policy optimization algorithm is composed of the policy loss, the value function loss, and the entropy regularization term. The total loss function is expressed as follows: ， in, The value loss coefficient, Here is the entropy regularization coefficient. Let be the policy entropy.

6. The satellite communication anti-interference decision-making method based on hierarchical reinforcement learning according to claim 3, characterized in that, In step S3, the sub-agent undergoes pre-training through the following steps: S31: Construct an independent training environment for each sub-agent and set the state space, action space and reward function under the corresponding communication system; S32: Based on the characteristics of each communication system, construct an independent Actor-Critic network structure for each sub-agent; S33: Employ a near-end strategy optimization algorithm to train each sub-agent independently until it reaches the preset performance index under the corresponding communication system; S34: Save the network parameters of each trained sub-agent for the master agent to use during decision-making.

7. The satellite communication anti-interference decision-making method based on hierarchical reinforcement learning according to claim 1, characterized in that, In step S4, the reward value is a combined reward value calculated based on whether the communication is successful, the communication rate, the channel switching cost, and the power consumption.

8. The satellite communication anti-interference decision-making method based on hierarchical reinforcement learning according to claim 5, characterized in that, In step S5, when updating the policy network and value network of the master agent, the near-end policy optimization algorithm uses an experience replay buffer to store sample data. When the number of samples in the buffer reaches a preset update threshold, it randomly samples from the buffer to perform multiple rounds of network parameter updates.

9. A satellite communication anti-interference decision-making system based on hierarchical reinforcement learning, characterized in that, include: The environmental sensing module is used to sense the current electromagnetic environment and obtain interference noise intensity information for each channel across the entire frequency band. The master control decision module is used to select a combination of communication system and information rate that is adapted to the current electromagnetic environment based on the interference noise intensity information and the current communication status through a pre-trained master control agent. The sub-agent invocation module is used to invoke a pre-trained sub-agent corresponding to the selected communication system and information rate combination, and the sub-agent performs channel selection and power control decisions. The experience storage module is used to calculate the reward value based on the communication results and store the state, action, reward and next state in the decision-making process as experience data. The network update module is used to update the policy network and value network of the master agent using stored experience data and a near-end policy optimization algorithm to optimize subsequent decisions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the satellite communication anti-interference decision-making method based on hierarchical reinforcement learning as described in any one of claims 1-8.

11. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the satellite communication anti-jamming decision method based on hierarchical reinforcement learning as described in any one of claims 1-8.

Images