Aerial intelligent metasurface assisted air-space-ground-sea network reliable and safe communication method

Abstract

The invention discloses a reliable and safe communication method for an air intelligent metasurface-assisted air-space-ground-sea network, and belongs to the fields of intelligent metasurface-assisted communication, an air-space-ground-sea integrated network, air intelligent metasurface trajectory optimization, anti-interference and anti-eavesdropping safe communication and the like. According to the air-space-ground-sea integrated network security communication method based on deep reinforcement learning and assisted eavesdropping and malicious interference prevention of the air intelligent metasurface, important information data such as an air intelligent metasurface flight path, a user channel state and coordinates are combined; considering the constraints of multiple aspects such as safety communication requirements, air intelligent metasurface flight speed, communication reliability and the like in an air-space-ground-sea network, an air intelligent metasurface safety communication scheme based on a self-guidance post-empirical playback-Softmax depth bideterministic strategy gradient algorithm is provided. The invention aims to guarantee reliable and safe communication and improve communication quality to the greatest extent.

Description

Technical Field

[0001] The present invention belongs to the fields of intelligent metasurface assisted communication, integrated air-space-ground-sea network, aerial intelligent metasurface trajectory optimization, anti-interference and anti-eavesdropping secure communication, and specifically relates to an aerial intelligent metasurface assisted air-space-ground-sea network with anti-interference and anti-eavesdropping, and a reliable and secure communication method for aerial intelligent metasurface assisted air-space-ground-sea network based on deep reinforcement learning. Background Art

[0002] The smart metasurface is a uniform planar array composed of many low-cost passive reflective elements. Each element adaptively adjusts its reflection amplitude or phase, and the real-time programmable feature enables the metasurface to change its electromagnetic properties. By effectively designing the phase of the reflective unit, real-time controllable reflection beamforming can be achieved. The low cost, low energy consumption, programmability and easy deployment of smart metasurfaces make them applicable to various communication models, and they have great potential in improving communication performance, expanding coverage and reducing interference. Installing smart metasurfaces on drones as an aerial smart metasurface-assisted communication system can improve the communication reliability and spectrum efficiency of ground users, while also being highly dynamic. The characteristics of air-space-ground-sea links make users more susceptible to interference and eavesdropping. In an environment with eavesdropping and severe interference, the safe and reliable communication of users is a topic that needs to be discussed urgently, and the location deployment of aerial smart metasurfaces has a significant impact on the communication quality of users. Therefore, the reliable and secure communication of air-space-ground-sea networks assisted by aerial smart metasurfaces under interference attacks and eavesdropping is a particularly critical topic.

[0003] Intelligent metasurfaces can improve the anti-jamming performance of communications by dynamically adjusting the transmission characteristics of signals. [H.Yang, Z. Xiong, J. Zhao, "Intelligent Reflecting Surface Assisted Anti-Jamming Communications: A Fast Reinforcement Learning Approach," in IEEE Transactions on Wireless Communications, vol. 20, no. 3, pp. 1963-1974, March2021, doi: 10.1109 / TWC.2020.3037767.] A joint power allocation and reflection beamforming method based on fuzzy wolf climbing fast learning algorithm is proposed to learn the optimal anti-jamming strategy, which ensures the user's achievable rate and communication quality meet the standards under strong interference.

[0004] At the same time, it is a very effective anti-interference strategy to carry the smart metasurface on the drone to form an aerial smart metasurface and optimize the trajectory. [K. Lin, H. Yang, M. Zheng, L. Xiao, C. Huang and D. Niyato,"Penalized Reinforcement Learning-Based Energy-Efficient UAV-RIS Assisted Maritime Uplink Communications Against Jamming," in IEEE Transactions on Vehicular Technology, doi: 10.1109 / TVT.2024.3406896.] An optimization method based on penalty deep reinforcement learning is designed to optimize the trajectory of the aerial smart metasurface to maximize the energy collection efficiency while increasing the user's signal-to-noise ratio, improving the user's reachable rate under interference, and improving the system reliability performance.

[0005] In addition, eavesdroppers attempt to steal sensitive information, endangering the privacy of communications. Therefore, reflective beamforming is used on smart metasurfaces to improve the achievable confidentiality rate of users while suppressing the eavesdropper's eavesdropping data rate. [H. Yang, Z.Xiong, J. Zhao, D. Niyato, L. Xiao and Q. Wu, "Deep Reinforcement Learning-Based Intelligent Reflecting Surface for Secure Wireless Communications," inIEEE Transactions on Wireless Communications, vol. 20, no. 1, pp. 375-388,Jan. 2021, doi: 10.1109 / TWC.2020.3024860.] A secure beamforming algorithm based on deep post-decision state-priority experience replay learning is applied to assist secure communications, realizing the beamforming matrix at the base station and the phase shift matrix (reflective beamforming) at the smart metasurface, so that the system can operate safely and stably in the presence of eavesdropping. Summary of the invention

[0006] The purpose of the present invention is to provide a reliable and secure communication method for air-space-ground-sea networks assisted by an aerial intelligent metasurface. The method uses a deep reinforcement learning algorithm to optimize the flight trajectory of a drone, the parameters of an intelligent metasurface reflection unit, and the satellite transmission power in an environment with interference and eavesdropping, so as to adapt to the communication environment in a dynamic interference environment, suppress eavesdroppers, and ensure reliable and secure communication in an aerial intelligent metasurface-assisted communication system.

[0007] In order to achieve the above-mentioned invention object, the present invention provides the following technical solutions:

[0008] A method for reliable and secure communication assisted by an air-to-air intelligent metasurface in an air-to-ground integrated network under interference attack and eavesdropping comprises the following steps:

[0009] Step 1: Set up the system users are randomly distributed on the ground plane and the sea surface. The active eavesdropper is equipped with dual antennas, one of which is responsible for eavesdropping on all users' receiving signals, and the other interferes with the users by transmitting signals. The total flight time of the aerial intelligent metasurface is divided into equal intervals. In each time slot, the aerial intelligent metasurface analyzes the channel status and reflects the satellite transmission data to each user. At the same time, it adjusts the flight trajectory to deal with interference attacks and illegal eavesdropping in the wireless communication network. The reflection unit parameters of the aerial intelligent metasurface are , flight direction and satellite transmission power Together they form the action space;

[0010] Step 2: Build two main neural networks: Neural Networks , , Neural Network , The network parameters are , , and ; and the four target neural networks corresponding to these four neural networks , , , , the network parameters are , , , ; Initialize all network parameters to random parameters, and the target network parameters are the same as those of the main network; Neural network , Responsible for outputting the probability of each action based on the current state, and selecting appropriate actions to interact with the environment based on these probabilities; Neural Network , Responsible for evaluation , performance and guide its subsequent behavior; the target network is used to reduce training instability and improve learning efficiency;

[0011] Step 3: Initialize the experience replay buffer pool and The experience replay buffer pool significantly improves learning efficiency and stability by storing and sampling the interaction experience between the agent and the environment for model updating; it then receives the initial system state from the communication environment;

[0012] Step 4: time slot, all user channel information , the confidentiality rate of the signal received by the ground user , achievable rate , the coordinates of the ground user and the eavesdropper’s channel information Constructed as a state vector ,in is the collection of all users;

[0013] Step 5: Time slot, policy based , and self-guided exploration strategy selection with exploration noise Action ; is Gaussian noise, and the self-guided exploration strategy achieves the value of , system performance target value and the modification of the random value of the action to find a balance between exploration and exploitation; under the self-guided exploration strategy, the learned strategy is used to guide the network to quickly pass through familiar areas and then explore more complex stages; in this way, the network can learn tasks in stages, gradually improving its ability to adapt to dynamic interference environments and suppress eavesdroppers;

[0014] Step 6: Aerial Smart Metasurface Executes Actions , while controlling the satellite transmission power, choose to fly to the next position or hover, and reflect the satellite transmission signal to the user through the metasurface to obtain the next time slot state space ;

[0015] Step 7: Calculate the current communication rate of the system ,in for Compare with 0 and take the maximum value. is the user achievable rate, , are the satellite transmission power to the kth user and the jammer transmission power respectively; , , They are the channel from the satellite to the kth user, the channel from the interference signal to the metasurface reflected to the user, and the channel from the interference signal directly to the user; is the eavesdropping rate of the active eavesdropper, , , They are the channel from satellite to active eavesdropper, the channel from interference signal to metasurface reflection to eavesdropper, and the channel from interference signal directly to eavesdropper. is the additive complex Gaussian noise power;

[0016] Step 8: Calculate the current reward value , keep the confidentiality rate for each user The sum of the experience Store to the experience replay buffer pool Among them and is the current time slot state space and the next time slot state space, and is the total confidentiality rate achieved by the system and the expected total confidentiality rate, The action performed by the agent in the current time slot; when the confidentiality rate achieved is greater than the expected, the experience is stored in the experience replay buffer pool middle;

[0017] Step 9: If the number of data in the experience replay buffer pool is greater than a certain number, first , Randomly draw quantitative experience in In accordance with Probability replaces its expected goal with the achieved goal, improving the effectiveness of the learning process; In accordance with Probabilistically replacing the achieved target with the desired target further improves learning performance;

[0018] Step 10: Calculate the target value ,in is the discount factor, Represents the Q value of all possible actions in the next state, and its Q value is an estimate of the value of the state-action pair; is the expected Q function derived from the softmax operator, It's action The expected value operation of is the probability distribution of actions, Indicates that the relative weight of each action is calculated according to the size of the Q value. Control the "temperature" of softmax and adjust the degree of difference between different Q values;

[0019] Step 11: Using the calculated target value, update the neural network via Bellman loss , : , represents the parameter vector of the i-th Q network; Indicates updating the parameters of the Q network by minimizing the objective function; represents the number of mini-batch samples, Q network pair state and actions The estimated value of represents the corresponding random perturbation or strategy adjustment, represents the target value;

[0020] Step 12: Update the neural network according to the policy gradient , : , It is about the policy network parameters The objective function gradient of Is the output action of the policy network The gradient of Is Q Network about Action The gradient of

[0021] Step 13: Finally, after the subject network is updated several times, the target network is soft updated as follows:

[0022] ,

[0023] in and are the parameters of the target Q network and the target policy network; is the hysteresis coefficient, which controls the weight ratio of the target network and the main network update

[0024] Step 14: Repeat steps 5 to 13 until the algorithm converges.

[0025] Compared with the prior art, the present invention has the following outstanding advantages:

[0026] Different from the methods of countering interference and eavesdropping in the prior art, the present invention proposes a method for reliable and secure communication of aerial intelligent metasurfaces in air-space-ground-sea networks that simultaneously responds to interference attacks and illegal eavesdropping based on a deep reinforcement learning algorithm. This method suppresses eavesdropping and interference in the system by optimizing the flight trajectory and metasurface reflection unit parameters of the aerial intelligent metasurface, ensures safe and stable data transmission in the communication system, and improves the communication quality of users. Based on a deep reinforcement learning algorithm, the present invention jointly optimizes the reflection unit parameters, flight trajectory and satellite transmission power of aerial intelligent metasurfaces in air-space-ground communication networks to achieve safe and reliable communication in the presence of eavesdroppers and jammers. Simulation results show that it is comparable to the double-delay deep deterministic policy gradient algorithm [Fujimoto, Scott, Herke Hoof, and David Meger. "Addressingfunction approximation error in actor-critic methods." International conference on machine learning PMLR, 2018.] and Softmax deep double deterministic policy gradients [Pan, Ling, Qingpeng Cai, and Longbo Huang. "Softmax deep doubledeterministic policy gradients." Advances in neural information processing systems 33 (2020): 11767-11777.] Compared with the deep reinforcement learning algorithm provided by the present invention, namely the self-guided post-experience replay-Softmax deep dual deterministic policy gradient algorithm, it provides significant improvements in confidentiality rate and communication stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A flow chart of a reliable and secure communication method for an air-to-ground network assisted by an aerial intelligent metasurface under interference attacks provided by an embodiment of the present invention.

[0028] Figure 2 This is a system model diagram of the reliable and secure communication method described in an embodiment of the present invention.

[0029] Figure 3 This is a comparison chart of the cumulative rewards of the reliable and secure communication algorithm described in the embodiment of the present invention and the comparison algorithm at different numbers of iterations.

[0030] Figure 4 This is a comparison chart of the average user confidentiality rate of the reliable and secure communication algorithm described in the embodiment of the present invention and the comparison algorithm under different numbers of intelligent metasurface reflection units.

[0031] Figure 5This is a probability comparison diagram of user service quality satisfaction between the reliable and secure communication algorithm described in an embodiment of the present invention and the comparison algorithm under different numbers of intelligent metasurface reflection units. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the following embodiments will further illustrate the present invention in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0033] like Figure 1 , the embodiment of the present invention includes the following steps:

[0034] Step 1: Initialize the system model, Figure 2 This is a system model diagram of the reliable and secure communication method described in an embodiment of the present invention; the air-ground integrated communication system is provided with three single-antenna ground users and one single-antenna maritime user, and one active eavesdropper, which are randomly distributed on the plane and can move freely. As the transmitting end, the multi-antenna satellite dynamically adjusts the transmission power when the direct link to the ground is missing. At the same time, there is an active eavesdropper with an interference antenna and an eavesdropping antenna, which eavesdrops on the legitimate channels of the relay in the environment and interferes with the receiving users on the ground and at sea. As a relay, the intelligent metasurface in the air adjusts the flight trajectory according to the environmental information to reduce interference attacks and eavesdropping in the communication network. At the same time, the metasurface carried by the carrier improves the user signal-to-noise ratio by beamforming the satellite signal reflection. The safe and reliable operation of the communication system is guaranteed through joint optimization.

[0035] In each time slot, the intelligent metasurface in the air adjusts the flight trajectory according to the environment to reduce interference attacks and eavesdropping in the communication network; at the same time, the metasurface improves the user's signal-to-noise ratio by beamforming the satellite signal reflection; action space Parameters of reflection units of aerial smart metasurface , flight trajectory Republic Satellite Transmit Power Same composition;

[0036] Step 2: Build four main neural networks with two fully connected layers: Neural Network , , Neural Network , The network parameters are , , and ; The first fully connected layer consists of 400 neurons, and the second fully connected layer consists of 300 neurons; the input layer and the output layer are composed of state space and action space Size decision; build four target neural networks at the same time , , , , the network parameters are , , , ;The number and size of layers are consistent with the corresponding main network;

[0037] Step 3: Initialize the main function network parameters and the experience replay buffer pool, the size of which is 50,000; obtain the initial state from the environment; and set the target network parameters to be the same as the main network parameters;

[0038] Step 4: time slot, all user channel information , the confidentiality rate of the signal received by the ground user , achievable rate , the coordinates of the ground user and the eavesdropper’s channel information Constructed as the current state vector ;

[0039] Step 5: Under the self-guided exploration strategy, based on the strategy , Choose the exploration noise Action ; Intelligent metasurface performs actions in the air , choose to fly to the next location or hover, and reflect the satellite transmission signal to the user through the metasurface; the satellite performs the action , regulate the transmission power to the user;

[0040] Step 6: Calculate the current communication rate of the system ,in is the user achievable rate, , are the satellite transmission power to the kth user and the jammer transmission power respectively; , , They are the channel from the satellite to the kth user, the channel from the interference signal to the metasurface reflected to the user, and the channel from the interference signal directly to the user; is the eavesdropping rate of the active eavesdropper, , , They are the channel from satellite to active eavesdropper, the channel from jamming signal to metasurface reflection to eavesdropper, and the channel from jamming signal directly to eavesdropper; is the additive complex Gaussian noise power; calculate the current reward value ;

[0041] Step 7: Apply Experience Store to the experience replay buffer pool Among them and is the total confidentiality rate achieved by the system and the expected total confidentiality rate; when the achieved confidentiality rate is greater than the expected When the experience is stored in the experience playback buffer pool middle;

[0042] Step 8: Determine whether the number of data in the experience playback buffer pool is greater than the number of samples in each batch sampling ; If it is less than the number of samples, repeat steps 4 to 8; if it is greater than the number of samples, continue to step 9;

[0043] Step 9: First, in the experience replay buffer pool , 100 experiences are randomly drawn from the experience replay buffer pool. In accordance with Probability replaces its expected goal with the achieved goal, improving the effectiveness of the learning process; in the experience replay buffer pool In accordance with Probability replaces the achieved target with the expected target, further improving learning performance; calculate the target value ,in is the discount factor, Represents the Q value of all possible actions in the next state, and its Q value is an estimate of the value of the state-action pair; is the expected Q function derived from the softmax operator, It's action The expected value operation of is the probability distribution of actions, Indicates that the relative weight of each action is calculated according to the size of the Q value. Control the "temperature" of softmax and adjust the degree of difference between different Q values;

[0044] Step 10: Using the calculated target value, update the neural network via Bellman loss , : , represents the parameter vector of the i-th Q network; Indicates updating the parameters of the Q network by minimizing the objective function; represents the number of mini-batch samples, Q network pair state and actions The estimated value of represents the corresponding random perturbation or strategy adjustment, Represents the target value; updates the neural network according to the policy gradient , : , It is about the policy network parameters The objective function gradient of Is the output action of the policy network The gradient of It is the Q network about action Gradient of; Finally, after the main network is updated several times, the target network is soft updated with delay as follows: , ;in and are the parameters of the target Q network and the target policy network; is the hysteresis coefficient, which controls the weight ratio of the target network and the main network update;

[0045] Step 11: Repeat steps 4 to 10 until the algorithm converges.

[0046] In order to improve the communication security in the face of active eavesdroppers and interference in the air-space-ground-sea communication network, the present invention conducts a series of experiments, the core of which is to use the aerial intelligent surface to enhance the confidentiality and reliability of communication. In the experiment, a scene including satellites, ground users, aerial metasurfaces and active eavesdroppers is simulated, and different system parameters are set, such as different optimization algorithms and the number of metasurface elements. Through simulation, the impact of parameter changes on system performance is observed in different aspects.

[0047] Convergence comparison experiment design of different algorithms:

[0048] In this experiment, the maximum transmission power of the satellite is set to 40dBm, the interference power of the active eavesdropper is set to 30dBm, and the airborne intelligent metasurface is equipped with 64 surface reflection units. The double-delay deep deterministic policy gradient algorithm and the Softmax deep dual deterministic policy gradient algorithm are the comparative experimental groups of the present invention. The algorithm performance is compared with the self-guided post-experience playback-Softmax deep dual deterministic policy gradient algorithm proposed in the present invention. The experimental results of the comparison of the cumulative rewards of the reliable and secure communication algorithm and the comparative algorithm at different iteration times are shown in Figure 2. Figure 3 As shown in the figure, the cumulative reward of the proposed self-guided experience replay-Softmax deep dual deterministic policy gradient algorithm increases with the number of iterations. Compared with the deep deterministic policy gradient algorithm and the Softmax deep dual deterministic policy gradient algorithm, the proposed algorithm has faster convergence speed, higher cumulative reward, and smaller fluctuation. It shows that the algorithm has better reliability and efficiency.

[0049] Performance comparison experiment of different numbers of smart reflective surface units:

[0050] Purpose of the experiment: The experiment aims to evaluate the changes in system performance when the number of reflective units of the airborne intelligent meta-surface (IMS) increases from 16 to 81 under different algorithm optimizations. The focus is on the average confidentiality rate of users and the probability of user service quality satisfaction, and the Fixed Position-Self-Guided Post HocExperience Replay-Softmax Deep Deterministic Policy Gradient (FP-SG-PHER-Softmax DDPG) algorithm is introduced for comparison. The airborne intelligent meta-surface is fixed at the initial position, and the position of the airborne intelligent meta-surface is compared and observed to verify the necessity and importance of optimizing the position of the airborne intelligent meta-surface.

[0051] Experimental settings: maximum satellite transmission power: 40 dBm, active eavesdropper interference power: 30 dBm.

[0052] Variable: Number of aerial smart metasurface reflective units (16, 24, 36, 48, 64, 81).

[0053] Experimental steps: Set the satellite transmission power, eavesdropper interference power, and the initial position of the intelligent metasurface in the air. Configure and initialize each algorithm and different numbers of reflection units. Run the system under different conditions and record the average user confidentiality rate and user service quality satisfaction probability. Analyze the collected data and draw charts to show the system performance under different conditions.

[0054] Experimental results: Figure 4 and Figure 5 As shown, Figure 4 Represents the change of the average confidentiality rate of users under different numbers of metasurface units. Figure 5represents the probability of user service quality satisfaction under different numbers of metasurface elements, that is, the probability that all users in all time slots meet the confidentiality rate greater than 1 bit / s / Hz and the user's achievable rate is greater than 3 bit / s / Hz. As the number of intelligent metasurface reflection units increases, the confidentiality rate and the probability of user service quality satisfaction increase, which indicates that more intelligent metasurface reflection units can reflect more signal paths and signal power, thereby improving the signal reception strength at the user and reducing the eavesdropping signal strength at the eavesdropper. At the same time, the proposed algorithm is superior to the comparative algorithm in terms of user confidentiality rate and user service quality satisfaction probability, which once again demonstrates the efficiency and high reliability of the algorithm. However, the fixed position-self-guided post-experience replay-Softmax deep dual deterministic policy gradient algorithm cannot optimize the deployment position of the aerial intelligent metasurface. It is severely affected by interference and eavesdropping, and the performance of user confidentiality rate and user service quality satisfaction probability is seriously reduced, highlighting the reasonable optimization of the proposed algorithm for the deployment of aerial intelligent metasurface locations.

[0055] The above experiments show that the present invention can effectively improve the communication security and reliability of air-space-ground-sea networks in the face of active eavesdropping and interference by adopting aerial intelligent metasurface auxiliary technology and a deep reinforcement learning method based on self-guided post-experience replay-Softmax deep dual deterministic policy gradient algorithm.

[0056] The above embodiments are only preferred embodiments of the present invention and cannot be considered to limit the scope of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the present invention.

Claims

1. A reliable and secure communication method for air-space-ground-sea networks assisted by an aerial intelligent metasurface, characterized in that The following steps are involved: 1) Set the system to have users are randomly distributed on the ground plane and the sea surface, and an active eavesdropper is configured. The active eavesdropper is equipped with dual antennas, one of which is responsible for eavesdropping on all user receiving signals, and the other is used to transmit signals to interfere with the users; the total flight time of the aerial intelligent metasurface is divided into equal intervals In each time slot, the aerial intelligent metasurface analyzes the channel status and reflects the satellite transmission data to each user. At the same time, it adjusts the flight trajectory to deal with interference attacks and illegal eavesdropping in the wireless communication network. The reflection unit parameters of the aerial intelligent metasurface are , flight direction and satellite transmission power Together they form the action space; 2) Construct two main neural networks: including neural networks , , Neural Network , The network parameters are , , and ; And the four target neural networks corresponding to these four neural networks , , , , the network parameters are , , , ; Initialize all network parameters to random parameters to ensure that the target network parameters are the same as those of the main network; Neural network , Responsible for outputting the probability of each action based on the current state, and selecting appropriate actions to interact with the environment based on these probabilities; Neural Network , For evaluating neural networks , performance and guide its subsequent behavior; the target neural network is used to reduce training instability and improve learning efficiency; 3) Initialize the experience replay buffer pool and The experience replay buffer pool significantly improves learning efficiency and stability by storing and sampling the interaction experience between the agent and the environment for model updating; it then receives the initial system state from the communication environment; 4) No. time slot, all user channel information , the confidentiality rate of the signal received by the ground user , achievable rate , the coordinates of the ground user and the eavesdropper’s channel information Constructed as a state vector ,in is the collection of all users; 5) No. Time slot, based on , and self-guided exploration strategy selection with exploration noise Action ; is Gaussian noise, and the self-guided exploration strategy achieves the value of , system performance target value and the modification of action randomness to find a balance between exploration and exploitation; 6) Intelligent metasurfaces perform actions in the air , while controlling the satellite transmission power, choose to fly to the next position or hover, and reflect the satellite transmission signal to the user through the metasurface to obtain the next time slot state space ; 7) Calculate the current communication rate of the system and the eavesdropper's eavesdropping rate , where the communication rate includes the user's achievable rate, and the eavesdropping rate represents the rate at which the eavesdropper can eavesdrop on the information; 8) Calculate the current reward value , keep the confidentiality rate for each user The sum of the experience Store to the experience replay buffer pool Among them and is the current time slot state space and the next time slot state space, and is the total confidentiality rate achieved by the system and the expected total confidentiality rate, The action performed by the agent in the current time slot; when the confidentiality rate achieved is greater than the expected, the experience is stored in the experience replay buffer pool middle; 9) If the number of data in the experience replay buffer pool is greater than a certain number, first , Randomly extract a certain amount of experience in the experience replay buffer pool In accordance with Probability replaces its expected goal with the achieved goal, improving the effectiveness of the learning process; in the experience replay buffer pool In accordance with Probabilistically replacing the achieved target with the desired target further improves learning performance; 10) Calculate the target value ,in is the discount factor, Represents the Q value of all possible actions in the next state, and its Q value is an estimate of the value of the state-action pair; is the expected Q function derived from the softmax operator, It's the action The expected value operation of is the probability distribution of actions, Indicates that the relative weight of each action is calculated according to the size of the Q value. Control the "temperature" of softmax and adjust the degree of difference between different Q values; 11) Using the target value calculated in step 10), update the neural network using Bellman loss , : , represents the parameter vector of the i-th Q network; Indicates updating the parameters of the Q network by minimizing the objective function; represents the number of mini-batch samples, Q network pair state and actions The estimated value of represents the corresponding random perturbation or strategy adjustment, represents the target value; 12) Update the neural network based on the policy gradient , : , It is about the policy network parameters The objective function gradient of Is the output action of the policy network The gradient of It is the Q network about action The gradient of 13) After the subject network is updated several times, the target network is soft updated as follows: ， in, and are the parameters of the target Q network and the target policy network; is the hysteresis coefficient, which controls the weight ratio of the target network and the main network update 14) Repeat steps 5 to 13 until the algorithm converges.

2. A reliable and secure communication method for air-space-ground-sea networks assisted by an aerial intelligent metasurface as claimed in claim 1, characterized in that In step 5), the self-guided exploration strategy finds a balance between exploration and utilization by modifying the system performance achievement value, the system performance target value and the action random value, thereby guiding the network to learn the task in stages; under the self-guided exploration strategy, the learned strategy is used to guide the network to quickly pass through familiar areas and then explore more complex stages; in this way, the network learns the task in stages.

3. A reliable and secure communication method for air-space-ground-sea networks assisted by an aerial intelligent metasurface as claimed in claim 1, characterized in that In step 7), the specific steps of calculating the current communication rate of the system and the eavesdropper's eavesdropping rate are: calculating the current communication rate of the system ,in for Compare with 0 and take the maximum value. is the user achievable rate, , are the satellite transmission power to the kth user and the jammer transmission power respectively; , , They are the channel from the satellite to the kth user, the channel from the interference signal to the metasurface reflected to the user, and the channel from the interference signal directly to the user; is the eavesdropping rate of the active eavesdropper, , , They are the channel from satellite to active eavesdropper, the channel from interference signal to metasurface reflection to eavesdropper, and the channel from interference signal directly to eavesdropper. is the additive complex Gaussian noise power.

Images