Multi-agent cooperative anti-interference method based on spectrum data aggregation

Abstract

The invention discloses a multi-agent cooperative anti-interference method based on spectrum data aggregation. The method comprises the following steps: firstly, establishing a multi-agent cooperative anti-interference model based on spectrum data aggregation; secondly, joint actions are selected based on a greedy strategy; thirdly, executing a joint action, obtaining an award, sensing a current frequency spectrum, transferring a state to a next state, and meanwhile, storing experience into an experience pool of the intelligent agent; then, random batch sampling, target Q value calculation and loss function gradient calculation are carried out from the experience pool, a weight value is trained and updated, and when the maximum number of cycles is reached, the algorithm is ended; according to the method, the model is complete, the physical significance is clear, malicious interference and mutual interference between users can be avoided, the anti-interference performance of the ultra-dense network under incomplete sensing information can be remarkably improved, and the frequency utilization strategy is more fair.

Description

Technical Field

[0001] The present invention belongs to the technical field of wireless communication anti-interference, and in particular to a multi-agent collaborative anti-interference method based on spectrum data aggregation. Background Art

[0002] Thanks to the rapid development of wireless communication technology, the Internet of Things, Internet of Vehicles, Ad Hoc networks, etc. have also developed rapidly. The explosive growth in the number of wireless devices has led to the continuous expansion of network scale, the distribution of devices has become more crowded, and the already scarce and limited spectrum resources have become more difficult to allocate reasonably. The electromagnetic spectrum environment is extremely susceptible to interference and malicious attacks due to its natural openness. Therefore, it is very important to study the dynamic spectrum access method of large-scale, densely distributed wireless networks under malicious interference attacks.

[0003] The deep reinforcement learning anti-interference dynamic spectrum access method based on cognitive radio technology and machine learning methods has been widely studied in academia (Oroojlooyjadid A, Hajinezhad D. "A Review of Cooperative Multi-Agent Deep Reinforcement Learning", 2019.). However, the dynamic spectrum access decisions of these anti-interference methods are based on the premise that the sampling data obtained by spectrum sensing is complete and error-free. If the data obtained by spectrum sensing is incomplete or erroneous, the convergence time of the algorithm will be significantly extended and the performance will be seriously degraded. In the actual communication environment, due to the performance limitations of the hardware equipment and data transmission distortion, the data sensed by the communication equipment has varying degrees of missing and errors. The above-mentioned DRL-based anti-interference methods cannot guarantee the same reliable performance in the actual communication environment as in the simulation environment. Some papers focus on the perception problem. For example, the paper (Abbas W, Koutsoukos X. "Efficient Complete Coverage Through Heterogeneous Sensing Nodes," IEEE Wireless Communications Letters, 2015, 4 (1): 14-17.) studies the location deployment of communication nodes with heterogeneous spectrum sensing capabilities, reducing the cost required for full coverage. The paper (Hu M, Zhu Q. "Secondary user utility optimization algorithm on cooperative spectrum sensing," 2019 IEEE 5th International Conference on Computer and Communications (ICCC). IEEE, 2019: 463-467) comprehensively considers the perception energy consumption, detection probability, transmission distance and other factors of cognitive radio equipment, and obtains the optimal perception time of each CR device based on the game theory method to reduce the perception energy consumption.

[0004] In order to achieve collaborative anti-interference dynamic spectrum access under the condition of incomplete perception information, a feasible solution is to draw on the idea of ​​"win-win cooperation". All users upload the missing perception data to a central controller with powerful computing power. The central controller aggregates all the missing spectrum to obtain spectrum data with better integrity. The central controller trains a joint decision-making deep neural network based on the data uploaded by the users to guide the users' anti-interference dynamic spectrum access decisions. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-agent collaborative anti-interference method based on spectrum data aggregation, which well describes the anti-interference scenario of ultra-dense networks under incomplete perception information.

[0006] The technical solution to achieve the purpose of the present invention is: a multi-agent collaborative anti-interference method based on spectrum data aggregation, comprising the following steps:

[0007] Step 1: Establish a multi-agent collaborative anti-interference model based on spectrum data aggregation, which includes a group of interference devices, N users and a frequency control center;

[0008] Step 2: Take the spectrum waterfall diagram as the environment state, the spectrum sampling data aggregated by the spectrum center as the observation state, and the access channel selected by the user as the action. Define the reward function according to the environment state, observation state, and access state of other users, and model the multi-agent collaborative anti-interference process of spectrum data aggregation as a partially observable Markov decision process, where the environment state of time slot t is represented by S t The observed state is represented by o t The reward function for user n is expressed as

[0009] Step 3: Define the state value function of user n at time t Select actions for N users based on the greedy strategy loop

[0010] Step 4: For user n, sense the current spectrum and perform joint actions Get rewards and the next state O t+1 =[o t+1 ,o t ,...,o t-Φ+2 ] and the experience The data is stored in the experience pool E of user n;

[0011] Step 5: Random batch sampling from user n’s experience pool E With the state and reward as input and the user action as output, the neural network is trained using the gradient descent optimization method. During the neural network training process, the mellowmax operator is introduced to calculate the target state value TargetQ and define the loss function accordingly.

[0012] Step 6: loop step 4-step 5. When the number of loops reaches the total number of users, jump to step 3 until the simulation time reaches the maximum number of iterations.

[0013] Furthermore, in step 1, a multi-agent collaborative anti-interference model based on spectrum data aggregation is established. The specific method is as follows:

[0014] The multi-agent collaborative anti-interference model based on spectrum data aggregation is an ultra-dense wireless network scenario, which includes a group of interfering devices, N users and a frequency control center, where the user is represented by The entire ultra-dense network shares a frequency band, which is equally divided into M non-overlapping available channels with bandwidth b, expressed as The frequency range of channel k is [f k -b / 2,f k +b / 2], the SINR of user n when performing data transmission on channel k is:

[0015]

[0016] Among them, f k is the center frequency of channel k, b is the channel bandwidth; represents the data transmission power of user n on a channel, U n (f) represents the power spectrum density equation when user n transmits data; is the channel coefficient of user n in channel k;

[0017] I n,k is the power of the same-frequency interference received by other users on channel k to user n, J n,k is the power of malicious interference to user n in channel k, which are expressed as:

[0018]

[0019] Among them, U m (f) is the power spectral density equation of user m, is the channel coefficient of user m in channel k, N n is the set of transmitters other than user n, f m The center frequency of the transmission channel selected for user m; is the channel coefficient of interference signal j on channel k, J(f) is the power spectrum density equation of the jammer;

[0020] δ is the power of additive white Gaussian noise, expressed as:

[0021]

[0022] Where n(f) is the power spectral density equation of additive white Gaussian noise;

[0023] Considering that communication transmission has a minimum transmission quality requirement β th , the data transmission quality needs to be greater than β th is effective transmission, that is, the communication transmission rate C of user n when transmitting on channel k n,k It is expressed as:

[0024]

[0025] If the user can fully perceive the spectrum data of the entire available communication frequency band, user n performs a spectrum sensing result s n (f) is:

[0026]

[0027] Define the discrete spectrum sampling value as:

[0028]

[0029] Among them, Δf is the resolution of spectrum sampling, and the result of a spectrum sensing is o=[o1,o2,…,o X ] T , X is the number of sampling points;

[0030] The goal of each communication user is to access a channel with a higher cumulative transmission rate, which requires the communication user to avoid frequency conflicts with other users while avoiding malicious interference. That is, the optimization goal of communication user n is to find the maximum long-term cumulative discounted communication transmission rate, which can be expressed as:

[0031]

[0032] Among them, γ (0<γ<1) is the discount factor, π n is the policy for user n, The communication rate of user n in time slot t. If user n selects channel k to communicate in time slot t, then

[0033] Furthermore, in step 2, the spectrum waterfall diagram is used as the environment state, the spectrum sampling data aggregated by the spectrum center is used as the observation state, and the access channel selected by the communication user is used as the action. The reward function is defined according to the environment state, the observation state and the access state of the remaining users, and the multi-agent collaborative anti-interference process of spectrum data aggregation is modeled as a partially observable Markov decision process. The specific method is as follows:

[0034] (1) Environmental status

[0035] The environmental state is defined as the complete and accurate electromagnetic spectrum state. The environmental state of time slot t is represented by S t ;

[0036] (2) Observation status

[0037] The observation state O is a matrix sequence composed of spectrum sampling data aggregated by the spectrum center in the past historical time. The observation state of time slot t is expressed as O t :

[0038] O t =[o t ,o t-1 ,…,o t-Φ+1 ] T (8)

[0039] Among them, Φ is the historical lookback time, o t The observation data after the frequency control center aggregates the perceived results of all users in the t-time slot;

[0040] (3) Action

[0041] is the access channel for the user in time slot t,

[0042] (4) Reward value function

[0043] The joint reward set for time slot t is The reward value of user n and the state of the environment are S t , observation state O t , which is related to the access status of other users, expressed as:

[0044]

[0045] Among them, β n is the SINR of user n. During the transmission time slot, the SINR of user n is always greater than the minimum transmission quality requirement β th , then the transmission success reward value is 5; if the SINR of user n is lower than the minimum transmission quality requirement β at any time th , then the transmission failure reward value is -0.1;

[0046] (5) Partially Observable Markov Decision Process

[0047] A partially observable Markov decision process is described as a six-tuple in, Like the quadruple of MDP, it represents the state, joint action set, environment state transition probability and joint reward value function respectively, Ω is the observation space, and O is the observation state.

[0048] Further, step 3, define the state value function of user n at time t Select actions for N users based on the greedy strategy loop The details are as follows:

[0049] The state value function of user n at time t is approximated by reducing the dimension through the mean field method:

[0050]

[0051] Among them, a n Use length |A n | is represented by a one-hot code, that is, the element corresponding to the access channel position selected by user n is 1, and the rest are 0. represents the average action influence of the remaining users except user n, that is,

[0052]

[0053] Further, in step 4, random batch sampling is performed from the experience pool E of user n With state and reward as input and user action as output, the neural network is trained using gradient descent optimization, where:

[0054] Using the mellowmax operator mm w (·) to calculate the target state value TargetQ, expressed as:

[0055]

[0056] in, w (w>0) is the temperature parameter, is the reward value obtained by user n in time slot t;

[0057] The Q function update formula after introducing the mellowmax operator and mean field approximation is:

[0058]

[0059] in, is the action at time t, is the mean field action at time t;

[0060] The neural network nonlinearly fits the Q function and defines the loss function of the neural network as:

[0061]

[0062] The gradient descent algorithm is used to train the neural network, and the gradient of the loss function is:

[0063]

[0064] A multi-agent collaborative anti-interference system based on spectrum data aggregation, characterized in that the multi-agent collaborative anti-interference method based on spectrum data aggregation is implemented to achieve multi-agent collaborative anti-interference based on spectrum data aggregation.

[0065] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the multi-agent collaborative anti-interference method based on spectrum data aggregation is implemented to realize the multi-agent collaborative anti-interference based on spectrum data aggregation.

[0066] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the multi-agent collaborative anti-interference method based on spectrum data aggregation to achieve multi-agent collaborative anti-interference based on spectrum data aggregation.

[0067] Compared with the prior art, the present invention has the following significant advantages: (1) In view of the problem of large number of users and large dimension of Q function based on deep reinforcement learning, the mean field approximation method is used to reduce the dimension of Q function, reduce algorithm complexity and improve convergence speed. In view of the dependence of the classic DQN algorithm on the target network, the update method of mellwmax operator update algorithm is adopted to improve the stability of DRL algorithm update; (2) The model is complete and the physical meaning is clear. The proposed multi-agent collaborative anti-interference algorithm based on spectrum data aggregation can effectively solve the proposed model and find the strategy of long-term cumulative discounted communication rate; (3) It can effectively deal with dynamic interference and well characterize the multi-agent collaborative anti-interference scenario based on spectrum data aggregation. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a block diagram of the collaborative anti-interference algorithm under incomplete perception in the present invention.

[0069] Figure 2 This is the convergence analysis in Example 1 of the present invention.

[0070] Figure 3 It is the change of the normalized network throughput with the user's full-band perception ratio in Example 1 of the present invention.

[0071] Figure 4 This is the impact of the number of users with perception capability on the network throughput in Example 2 of the present invention. DETAILED DESCRIPTION

[0072] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0073] like Figure 1 As shown, the present invention is a multi-agent collaborative anti-interference method based on spectrum data aggregation, and the specific steps are as follows:

[0074] Step 1: Establish a multi-agent collaborative anti-interference model based on spectrum data aggregation;

[0075] The multi-agent collaborative anti-interference model based on spectrum data aggregation is an ultra-dense wireless network scenario. The network contains a group of interference devices, N communication users and a frequency control center. The communication user is represented by The entire ultra-dense network shares a frequency band. Considering that there are multiple communication channels in the network, the frequency band is divided into M non-overlapping available channels with bandwidth b. The channel set is expressed as

[0076] Furthermore, the channel The frequency range is [f k -b / 2,f k +b / 2], where the SINR of user n when performing data transmission on channel k is:

[0077]

[0078] Among them, f k is the center frequency of channel k, b is the channel bandwidth; represents the data transmission power of user n on a channel, U n (f) represents the power spectrum density equation when user n transmits data; is the channel coefficient of user n in channel k;

[0079] I n,k is the power of the same-frequency interference received by other users on channel k to user n, J n,k is the power of malicious interference to user n in channel k, which are expressed as:

[0080]

[0081] Among them, U m (f) is the power spectral density equation of user m, is the channel coefficient of user m in channel k, N n is the set of transmitters other than user n, f m The center frequency of the transmission channel selected for user m; is the channel coefficient of interference signal j on channel k, J(f) is the power spectrum density equation of the jammer;

[0082] δ is the power of additive white Gaussian noise, expressed as:

[0083]

[0084] Where n(f) is the power spectral density equation of additive white Gaussian noise;

[0085] Furthermore, considering that communication transmission has a minimum transmission quality requirement β th , data transmission quality is greater than βth This is the effective transmission, that is, the communication transmission rate C when user n transmits on channel k n,k It is expressed as:

[0086]

[0087] If the user can fully perceive the spectrum data of the entire available communication frequency band, user n performs a spectrum sensing result s n (f) is:

[0088]

[0089] Furthermore, the discrete spectrum sampling value is defined as:

[0090]

[0091] Among them, Δf is the resolution of spectrum sampling, and the result of a spectrum sensing is o=[o1,o2,…,o X ] T , X is the number of sampling points.

[0092] Step 2: The goal of each communication user is to access a channel with a higher cumulative transmission rate, which requires the user to avoid malicious interference while avoiding frequency conflicts with other users. That is, the optimization goal of user n is to find the maximum long-term cumulative discounted communication rate:

[0093]

[0094] Among them, γ (0<γ<1) is the discount factor, π n is the policy for user n, The communication rate of user n in time slot t. If user n selects channel k to communicate in time slot t, then

[0095] Step 3: Taking the spectrum waterfall diagram as the environment state, the spectrum sampling data aggregated by the spectrum center as the observation state, and the access channel selected by the user as the action, a reward function is defined according to the environment state, the observation state, and the access state of the remaining users, and the multi-agent collaborative anti-interference process of spectrum data aggregation is modeled as a partially observable Markov decision process.

[0096] (1) Environmental status

[0097] The environmental state is defined as the complete and accurate electromagnetic spectrum state. The environmental state of time slot t is represented by S t .

[0098] (2) Observation status

[0099] The observation state O is a matrix sequence composed of spectrum sampling data aggregated by the spectrum center in the past historical time. The observation state of time slot t is expressed as O t :

[0100] O t =[o t ,o t-1 ,…,o t-Φ+1 ] T (8)

[0101] Among them, Φ is the historical lookback time, o t It is the observation data after the frequency control center aggregates the perceived results of all users in the t time slot.

[0102] (3) Action

[0103] For each user’s access channel, a n ∈{1,2,…,M}.

[0104] (4) Reward value function

[0105] Joint Reward Set The reward value of user n and the state of the environment are S t , observation state O t , and the access status of other users. The reward value of user n is expressed as:

[0106]

[0107] Among them, β n is the SINR of user n. During the transmission time slot, the SINR of user n is always greater than the minimum transmission quality requirement β th , then the transmission success reward value is 5; if the SINR of user n is lower than the minimum transmission quality requirement β at any time th , then the transmission failure reward value is -0.1.

[0108] (5) Partially Observable Markov Decision Process

[0109] A partially observable Markov decision process can be described as a six-tuple in, Like the quadruple of MDP, it represents the state, joint action set, environment state transition probability and joint reward value function respectively, Ω is the observation space, and O is the observation state.

[0110] Step 4: Define the state value function of user n at time t Select actions for N users based on the greedy strategy loop The details are as follows:

[0111] The state value function of user n at time t can be approximated by reducing the dimension through the mean field method:

[0112]

[0113] Among them, a n Use length |A n | is represented by a one-hot code, that is, the element corresponding to the access channel position selected by user n is 1, and the rest are 0. represents the average action influence of the remaining users except user n, that is,

[0114]

[0115] Step 5: Sense the current spectrum and perform joint actions Get rewards The state transfers to the next state O t+1 =[o t+1 ,o t ,...,o t-Φ+2 ], the experience Stored into the experience pool E of agent n;

[0116] Step 6: Loop through the N users and randomly sample them in batches from the experience pool E The mellowmax operator is introduced to calculate the target state value TargetQ, and the gradient descent optimization method is used to train the neural network;

[0117] Using the mellowmax operator mm w (·) to calculate the target state value TargetQ, expressed as:

[0118]

[0119] in, w (w>0) is the temperature parameter, is the reward value obtained by user n in time slot t.

[0120] Furthermore, the Q function update formula after introducing the mellowmax operator and mean field approximation is:

[0121]

[0122] in, is the action at time t, is the mean field action at time t;

[0123] The neural network nonlinearly fits the Q function and defines the loss function of the neural network as:

[0124]

[0125] The gradient descent algorithm is used to train the neural network, and the gradient of the loss function is:

[0126]

[0127] Step 7, loop step 5-step 6. When the number of loops reaches the total number of users, jump to step 4 until the simulation time reaches the maximum number of iterations and the algorithm ends.

[0128] To summarize, each user samples the spectrum through a spectrum sensing device. Due to the limitations of hardware equipment, the user's perception ability is limited, and the perceived spectrum data may be missing or erroneous. The user then uploads the perceived spectrum sampling data and the communication throughput of the previous time slot to the frequency control center through a control link. The frequency control center will centrally aggregate the spectrum data uploaded by all users to obtain more complete spectrum data, and then use the aggregated spectrum data to input the trained joint decision neural network for joint channel access decision making. Finally, the joint access decision is sent to each communication user through the control channel. The present invention can effectively deal with dynamic interference and well characterize the multi-agent collaborative anti-interference scenario based on spectrum data aggregation.

[0129] Example 1

[0130] The first embodiment of the present invention is described in detail as follows. The system simulation adopts Python language and is based on TensorFlow neural network. The parameter setting does not affect the generality. This embodiment verifies the effectiveness of the proposed model and method. Figure 2 The effectiveness of the proposed algorithm is verified by three comparative algorithms. The parameters are set as follows: the entire communication network shares a 20MHz spectrum, which is divided into M non-overlapping channels. The number of users is N=20. The user samples the spectrum with a resolution of Δf=100kHz, generating the number of sampling points X=200. The frequency control center saves 100ms of spectrum data to construct a spectrum waterfall diagram. The user's communication signal uses a raised cosine wave with a lifting coefficient of 0.5. The user's power is 0dBm, and the minimum transmission quality requirement β for data transmission is th =10dB. Strong discount factor λ=0.9, learning rate α=0.01, temperature parameter w=0.01, algorithm iteration number is 21000 times. The interference mode transmitted by the jammer is frequency sweep interference, the interference signal is a raised cosine wave with a lifting coefficient of 0.5, a power of 40dBm, and a frequency sweep speed of 1GHz / s. The interference signal completes full-band interference every 20ms, and the power of the background noise is -90dBm / Hz.

[0131] First, the neural network Q is generated, the weights are randomly assigned, and the initial observation state O0 is set; secondly, the joint action is selected based on the greedy strategy; thirdly, the joint action is executed, the reward is obtained, the current spectrum is perceived, the state is transferred to the next state, and the experience is stored in the experience pool of the agent; then, random batch sampling is performed from the experience pool, the target Q value is calculated, the loss function gradient is calculated, and the weight value is updated through training. When the maximum number of cycles is reached, the algorithm ends. Figure 3 is the change of network throughput with the number of users. Compared with the centralized DQN algorithm that uses the traditional target random network mechanism, the proposed algorithm does not rely on the target neural network, the algorithm training and update are more stable, and the network throughput decreases less quickly than the centralized DQN algorithm.

[0132] Example 2

[0133] The second embodiment of the present invention is described in detail as follows. The system simulation adopts Python language and is based on TensorFlow neural network. The parameter setting does not affect the generality. This embodiment verifies the effectiveness of the proposed model and method. Figure 4 The effectiveness of the proposed algorithm is verified by three comparative algorithms. The parameters are set as follows: the entire communication network shares a 20MHz spectrum, which is divided into M non-overlapping channels. The number of users is N=20. The user samples the spectrum with a resolution of Δf=100kHz, generating the number of sampling points X=200. The frequency control center saves 100ms of spectrum data to construct a spectrum waterfall diagram. The user's communication signal uses a raised cosine wave with a lifting coefficient of 0.5. The user's power is 0dBm, and the minimum transmission quality requirement β for data transmission is th =10dB. Strong discount factor λ=0.9, learning rate α=0.01, temperature parameter w=0.01, algorithm iteration number is 21000 times. The interference mode transmitted by the jammer is frequency sweep interference, the interference signal is a raised cosine wave with a lifting coefficient of 0.5, a power of 40dBm, and a frequency sweep speed of 1GHz / s. The interference signal completes full-band interference every 20ms, and the power of the background noise is -90dBm / Hz.

[0134] When the user scale and user perception capability are fixed, the more devices with spectrum perception capability there are, the more complete the spectrum that the frequency management center can obtain when performing spectrum data aggregation processing.

[0135] Figure 4 It is the impact of the number of users with perception capabilities on the network throughput. As the number of iterations increases, the normalized network throughput continues to improve and converge. The more users with perception capabilities, the higher the normalized network throughput in the convergence stage.

[0136] In summary, the multi-agent collaborative anti-interference model of spectrum data aggregation proposed in the present invention mainly considers the attack of malicious interference equipment on the communication network and the actual situation of limited spectrum perception ability of communication equipment. It proposes an idea of ​​spectrum data aggregation for the limited perception ability of users who cannot perceive the complete spectrum, and centrally processes the spectrum data perceived by all users to obtain a more complete spectrum perception result, which is more practical than the traditional model; the proposed multi-agent collaborative anti-interference algorithm based on spectrum data aggregation can effectively solve the proposed model. For the problem of large number of users and large dimension of Q function based on deep reinforcement learning, the mean field approximation method is used to reduce the dimension of Q function, reduce the algorithm complexity and improve the convergence speed; for the dependence of the classic DQN algorithm on the target network, the update method of mellwmax operator update algorithm is adopted to improve the stability of DRL algorithm update. In the malicious interference environment, the proposed collaborative anti-interference algorithm can reduce the impact of incomplete spectrum perception on the performance and convergence of DRL algorithm, and significantly improve the network throughput of large-scale wireless communication networks.

Claims

1. A multi-agent collaborative anti-interference method based on spectrum data aggregation, characterized in that: The steps include: Step 1: Establish a multi-agent collaborative anti-interference model based on spectrum data aggregation, which includes a group of interference devices, N users and a frequency control center; Step 2: Take the spectrum waterfall diagram as the environment state, the spectrum sampling data aggregated by the spectrum center as the observation state, and the access channel selected by the user as the action. Define the reward function according to the environment state, observation state, and access state of other users, and model the multi-agent collaborative anti-interference process of spectrum data aggregation as a partially observable Markov decision process, where the environment state of time slot t is represented by S t , the observed state is represented by o t , the reward function of user n is expressed as Step 3: Define the state value function of user n at time t Select actions for N users based on the greedy strategy loop Step 4: For user n, sense the current spectrum and perform joint actions Get rewards and the next state O t+1 =[o t+1 ,o t ,...,o t-φ+2 ] and the experience The data is stored in the experience pool E of user n; Step 5: Random batch sampling from user n’s experience pool E With the state and reward as input and the user action as output, the neural network is trained using the gradient descent optimization method. During the neural network training process, the mellowmax operator is introduced to calculate the target state value TargetQ and define the loss function accordingly. Step 6: loop step 4-step 5. When the number of loops reaches the total number of users, jump to step 3 until the simulation time reaches the maximum number of iterations.

2. The multi-agent collaborative anti-interference method based on spectrum data aggregation according to claim 1 is characterized in that: Step 1: Establish a multi-agent collaborative anti-interference model based on spectrum data aggregation. The specific method is as follows: The multi-agent collaborative anti-interference model based on spectrum data aggregation is an ultra-dense wireless network scenario, which includes a group of interfering devices, N users and a frequency control center, where the user is represented by The entire ultra-dense network shares a frequency band, which is equally divided into M non-overlapping available channels with bandwidth b, expressed as The frequency range of channel k is [f k -b / 2,f k +b / 2], the SINR of user n when performing data transmission on channel k is: Among them, f k is the center frequency of channel k, b is the channel bandwidth; represents the data transmission power of user n on a channel, U n (f) represents the power spectrum density equation when user n transmits data; is the channel coefficient of user n in channel k; I n,k is the power of the same-frequency interference received by other users on channel k to user n, J n,k is the power of malicious interference to user n in channel k, which are expressed as: Among them, U m (f) is the power spectral density equation of user m, is the channel coefficient of user m in channel k, N n is the set of transmitters other than user n, f m The center frequency of the transmission channel selected for user m; is the channel coefficient of interference signal j on channel k, J(f) is the power spectrum density equation of the jammer; δ is the power of additive white Gaussian noise, expressed as: Where n(f) is the power spectral density equation of additive white Gaussian noise; Considering that communication transmission has a minimum transmission quality requirement β th , the data transmission quality needs to be greater than β th is effective transmission, that is, the communication transmission rate C of user n when transmitting on channel k n,k It is expressed as: If the user can fully perceive the spectrum data of the entire available communication frequency band, user n performs a spectrum sensing result s n (f) is: Define the discrete spectrum sampling value as: Among them, Δf is the resolution of spectrum sampling, and the result of a spectrum sensing is o=[o1,o2,…,o X ] T , X is the number of sampling points; The goal of each communication user is to access a channel with a higher cumulative transmission rate, which requires the communication user to avoid frequency conflicts with other users while avoiding malicious interference. That is, the optimization goal of communication user n is to find the maximum long-term cumulative discounted communication transmission rate, which can be expressed as: Among them, γ (0<γ<1) is the discount factor, π n is the policy for user n, The communication rate of user n in time slot t is: if user n selects channel k to communicate in time slot t, then 3. The multi-agent collaborative anti-interference method based on spectrum data aggregation according to claim 1 is characterized in that: Step 2: Take the spectrum waterfall diagram as the environment state, the spectrum sampling data aggregated by the spectrum center as the observation state, and the access channel selected by the communication user as the action. Define the reward function according to the environment state, observation state and access state of other users, and model the multi-agent collaborative anti-interference process of spectrum data aggregation as a partially observable Markov decision process. The specific method is as follows: (1) Environmental status The environmental state is defined as the complete and accurate electromagnetic spectrum state. The environmental state of time slot t is represented by S t ; (2) Observation status The observation state O is a matrix sequence composed of spectrum sampling data aggregated by the spectrum center in the past historical time. The observation state of time slot t is expressed as O t : The t =[o t ,o t-1 ,…,o t-Φ+1 ] T (8) Among them, Φ is the historical lookback time, o t The observation data after the frequency control center aggregates the perceived results of all users in the t-time slot; (3) Action is the access channel for the user in time slot t, (4) Reward value function The joint reward set for time slot t is The reward value of user n and the state of the environment are S t , observation state O t , which is related to the access status of other users, expressed as: Among them, β n is the SINR of user n. During the transmission time slot, the SINR of user n is always greater than the minimum transmission quality requirement β th , then the transmission success reward value is 5; if the SINR of user n is lower than the minimum transmission quality requirement β at any time th , then the transmission failure reward value is -0.1; (5) Partially Observable Markov Decision Process A partially observable Markov decision process is described as a six-tuple Among them, S, P, Like the quadruple of MDP, it represents the state, joint action set, environment state transition probability and joint reward value function respectively, Ω is the observation space, and O is the observation state.

4. The multi-agent collaborative anti-interference method based on spectrum data aggregation according to claim 1 is characterized in that: Step 3: Define the state value function of user n at time t Select actions for N users based on the greedy strategy loop The details are as follows: The state value function of user n at time t is approximated by reducing the dimension through the mean field method: Among them, a n Use length |A n | is represented by a one-hot code, that is, the element corresponding to the access channel position selected by user n is 1, and the rest are 0. represents the average action influence of the remaining users except user n, that is, 5. The multi-agent collaborative anti-interference method based on spectrum data aggregation according to claim 1 is characterized in that: Step 4: Random batch sampling from user n’s experience pool E With state and reward as input and user action as output, the neural network is trained using gradient descent optimization, where: Using the mellowmax operator mm w (·) to calculate the target state value TargetQ, expressed as: in, w (w>0) is the temperature parameter, is the reward value obtained by user n in time slot t; The Q function update formula after introducing the mellowmax operator and mean field approximation is: in, is the action at time t, is the mean field action at time t; The neural network nonlinearly fits the Q function and defines the loss function of the neural network as: The gradient descent algorithm is used to train the neural network, and the gradient of the loss function is:

6. A multi-agent collaborative anti-interference system based on spectrum data aggregation, characterized in that: Implement the multi-agent collaborative anti-interference method based on spectrum data aggregation as described in any one of claims 1-5 to achieve multi-agent collaborative anti-interference based on spectrum data aggregation.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the multi-agent collaborative anti-interference method based on spectrum data aggregation according to any one of claims 1 to 5 is implemented to realize multi-agent collaborative anti-interference based on spectrum data aggregation.

8. A computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the multi-agent collaborative anti-interference method based on spectrum data aggregation according to any one of claims 1 to 5 is implemented to achieve multi-agent collaborative anti-interference based on spectrum data aggregation.

Images