Intelligent spectrum online anti-interference frequency allocation method for inter-device communication

Abstract

The present invention discloses an intelligent spectrum online anti-interference frequency point allocation method for inter-device communication, including: a transmitting device sends a communication application to a receiving device; after receiving the application, the receiving device returns confirmation information and starts an intelligent online spectrum anti-interference algorithm, at the beginning of each time slot, the receiving device allocates a working frequency point for the user applying for network access before the beginning of each time slot, the receiving device learns the interference law of the external interference source until the algorithm converges, obtains the optimal strategy and performs frequency point selection and data transmission according to this strategy; if the interference pattern changes, the algorithm is re-learned until convergence, and the receiving device performs frequency point allocation and data transmission according to the re-learned strategy. The present invention can be used for dynamic detection of spectrum situation in inter-device communication, and in a communication system with unknown interference sources, the interference pattern in the online learning environment is used to make the optimal spectrum allocation strategy, so that the frequency user avoids the interference frequency point and successfully transmits data.

Description

Technical Field

[0001] The present invention belongs to the technical field of wireless communications, and in particular relates to an intelligent spectrum online anti-interference frequency point allocation method for inter-device communication. Background Art

[0002] With the further development of wireless communication technology, people have higher and higher requirements on the number of wireless devices and service quality. Due to the openness of wireless channels, the security issues of communication systems have attracted more and more attention. Faced with the increasingly complex electromagnetic environment, traditional anti-interference technologies, such as frequency hopping spread spectrum and direct sequence spread spectrum, require a large amount of available spectrum resources as the support of anti-interference technology. When a large amount of spectrum resources cannot be provided, the anti-interference effect will decrease, and it will not be able to cope with intelligent interference attacks. The research on anti-interference technology is becoming increasingly important in building a more robust and efficient wireless communication system.

[0003] In recent years, reinforcement learning technology has provided a general method to solve decision-making problems in unknown dynamic environments. The Q-learning algorithm is a model-free reinforcement learning algorithm that has been used by many scholars to solve the anti-interference problem in wireless communication networks.

[0004] The emerging reinforcement learning technology provides a general method to solve complex uncertainty decision-making problems. The Q-learning algorithm is a model-free reinforcement learning algorithm that is widely used in the anti-interference problem of wireless communication systems. However, due to the complex electromagnetic environment, traditional table-based algorithms face huge state and action spaces and thus fall into the curse of dimensionality. The prediction models based on neural networks or hidden Markov models proposed to solve such problems require a lot of computing resources for training and need to be retrained regularly. At the same time, a large number of parameters are required to obtain relevant communication environment information, which has certain limitations in actual wireless communication networks with service delay requirements. Although the latest proposed reinforcement learning online anti-interference algorithm based on kernel function approximation has a good performance in dynamically monitoring environmental information, analyzing relevant data in real time, and making optimal decisions on frequency resource allocation, there is still a lot of room for improvement in algorithm robustness and convergence speed.

[0005] The invention patent with publication number CN112672426B and titled “An Anti-interference Frequency Point Allocation Method Based on Online Learning” uses kernel functions to approximate the Q value table, avoiding the dimensionality curse problem of Q learning. However, the convergence and accuracy of this method in complex environments need to be improved. At the same time, when the interference environment changes, this method continues to add features to the original feature dictionary. This method cannot play a good anti-interference role when the feature dictionary size is limited. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide an intelligent spectrum online anti-interference frequency allocation method for inter-device communication in view of the deficiencies of the above-mentioned prior art. The transmitting device sends a communication request to the receiving device. After receiving the request, the receiving end returns confirmation information and starts the online spectrum anti-interference frequency allocation algorithm. At the beginning of each time slot, the receiving device allocates an available frequency to the transmitting device one time slot in advance. The transmitting device transmits data to the receiving device. After receiving the transmitted data, the receiving device obtains the usage information of the entire available frequency, generates the corresponding spectrum situation status and feedback, updates the intelligent online status related parameters and uses the historical status information to continue to allocate frequency points to the transmitting device until the algorithm converges. The present invention is aimed at wireless communication between devices in an interference environment, and can effectively solve the problem of efficient frequency allocation of the transmitting end user in the interference mode of the dynamically changing jammer.

[0007] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0008] An intelligent spectrum online anti-interference frequency point allocation method for inter-device communication includes:

[0009] Step 1: The sending device sends a communication request to the receiving device;

[0010] Step 2: After receiving the application, the receiving device returns confirmation information and starts the intelligent online spectrum anti-interference algorithm. At the beginning of each time slot, the receiving device allocates a frequency point as the working frequency point for the user applying for network access before the beginning of each time slot. Each working frequency point only allows one user to communicate. At the same time, the receiving device starts to learn the interference law of external interference sources until the algorithm converges. The receiving device obtains the optimal strategy and selects the frequency point and transmits data according to this strategy;

[0011] Step 3: After the anti-interference algorithm converges, if the receiving device determines that the interference pattern of the external interference source has changed, the transfer learning algorithm is started to perform online migration of the characteristic function, and the previous learning steps are repeated until the learned anti-interference algorithm converges again. The receiving device performs frequency allocation and data transmission in the next time slot according to the re-learned strategy.

[0012] To optimize the above technical solutions, the specific measures taken also include:

[0013] In the above step 2, firstly, the frequency allocation problem under interference environment is modeled by Markov modeling, and the environment state information is defined as T h The linear combination of the historical state characteristics of all available channels in time slots is s t , the channel a selected at each moment t , the transition probability p, the instantaneous reward is set as a piecewise function r t, the channel interference and successful transmission respectively obtain different reward values, the strategy of the Markov decision process is π, in order to find the optimal strategy, it is necessary to maximize the expectation of the cumulative discounted return at the frequency-using equipment in the communication network as the objective function, that is, the optimal Bellman equation: Where γ is the discount factor;

[0014] In step 2, an online spectrum anti-interference algorithm is used to solve the optimal Bellman equation. The online spectrum anti-interference algorithm uses an adaptive core aggregation anti-interference frequency point allocation method based on an asynchronous advantage actor-evaluator architecture.

[0015] In the above step 2, the process of using the adaptive core aggregation anti-interference frequency point allocation method based on the asynchronous advantage actor-judge architecture to solve the optimal Bellman equation to obtain the optimal strategy includes:

[0016] Step 1: The receiving end user obtains the signal-to-interference-to-noise ratio of each available frequency point of the frequency-using device, and obtains the spectrum characteristics within a preset time range in the past as the current state characteristics according to the channel state extraction algorithm based on moment estimation, and constructs a spectrum availability feature set based on this;

[0017] Step 2: The receiving device uses an asynchronous update architecture of an asynchronous advantage actor-judge based on kernel function approximation to set up k sub-threads, and each sub-thread has the same policy function network and value function network during the initialization process;

[0018] Set a preset fixed time period T 1 , at the beginning of each fixed time slot interval t = nT 1 , n=0,1,2,… randomly select a child thread to maximize the policy function, select the frequency for data transmission from all available frequency points, and during this time period, use the policy function of the thread selected at the beginning to make decisions;

[0019] Step 3: The sending device sends a data packet to the target according to the control instruction returned by the receiving device, obtains the received signal at the receiving device, and obtains the instantaneous return value of the action selection. The instantaneous return value refers to whether the frequency-using device in a certain time slot is connected to an undisturbed frequency point as the working frequency point. The state-action-return pair and the state are stored in the policy function experience pool and the state value function experience pool respectively; except for the selected thread, other threads select a part of samples from the policy function experience pool and the state value function experience pool to update the weights of the policy function and the value function, and update the feature dictionary of the policy function and the value function;

[0020] Step 4: Repeat steps 1-3, when t = nT 1 At the end of the time slot, the policy function feature dictionary and value function feature dictionary of the selected thread are updated;

[0021] Step 5: Repeat steps 1-4 until the preset fixed time period T is reached. 2 , T 2 =mT 1 , m is a positive integer, that is, t = uT 2 ,u is a positive integer, m>k, after the time slot ends, the policy function network of all threads in step 3 is updated, the policy function and state value function feature dictionary and weight aggregation are updated, and the updated dictionary and weight are copied to all threads;

[0022] Step 6: Repeat 1-5 until the algorithm converges, and use the action selection decision after convergence as the optimal decision for anti-interference frequency allocation, so that the receiving device obtains the optimal strategy and performs frequency selection and data transmission according to this strategy.

[0023] In the above step 1, the signal-to-interference-noise ratio extraction algorithm based on moment estimation obtains the current state characteristics, that is, the occupancy state of the channel, which is specifically:

[0024] Step 1.1: The receiving device receives the s Independent and identically distributed received signal samples are taken and the means of the first-order, second-order, and fourth-order terms of the samples are calculated as the first-order, second-order, and fourth-order estimates, respectively, which are expressed as and remember And calculate the expected value of the fourth power of the modulus of the received signal sample and record it as p u′ ;

[0025] Step 1.2: Calculate the signal-to-interference-noise ratio estimate

[0026] Signal-to-Interference-Noise Ratio Estimation There are two ways to calculate . When the algorithm is first executed, only one of the calculation methods is selected and remains unchanged in the following algorithms:

[0027] 1) When the interference signal model can be modeled as a Gaussian model,

[0028] 2) When the interference signal contains only one main component interference signal:

[0029]

[0030] Step 1.3: Set the signal-to-interference-noise ratio threshold and energy value threshold to determine the channel occupancy status.

[0031] The above step 2 is specifically as follows:

[0032] Step 2.1: Set up a network of k threads, each thread contains 2 independent networks: a policy function network and a state-value function network;

[0033] Among them, the strategy function satisfies the Gibbs sampling distribution or the Boltzmann strategy, and the characteristic parameters of each network are a linear combination of a set of Gaussian radial basis functions. The above expression also satisfies the formula:

[0034] Indicates the probability that the algorithm chooses action a in state s;

[0035] is the parameterized numerical preference for each state-action pair,

[0036] in, is the weight parameter of H. In each time slot, each action a has a different probability

[0037] Step 2.2: Set a preset fixed time period T 1 , at the beginning of each fixed time slot interval t = nT 1 , n=0,1,2,… randomly select a sub-thread’s policy function for decision making. The policy function is the probability value corresponding to each action. The sum of the probability values ​​corresponding to all actions is 1. For each time slot t, a uniformly distributed number with a value range of (0-1) is randomly generated. If the generated value falls within the probability value range corresponding to the action, communication is performed on the channel corresponding to the action. Otherwise, an available channel is selected from the set of available channels with random probability for communication.

[0038] In step 3 above, the update mode for the policy function and value function weights is:

[0039] Step 3.1: After each time slot t ends, the transmitting device sends a data packet to the target according to the control instruction returned by the receiving device, obtains the received signal at the receiving device, obtains the spectrum availability of the entire available frequency band according to the method in step 1, and takes the availability of the entire spectrum in a preset time period as the state.

[0040] Step 3.2: Set up a shared experience pool and store the state-action-reward pair after each decision as a sample into the shared experience pool; except for the selected thread, other threads select a part of samples from the policy function experience pool and the state value function experience pool, and incrementally update the policy function and value function weights of their respective threads.

[0041] In step 3 above, the update mode of the strategy function and value function feature dictionary is:

[0042] Step 3.3: Each thread sets up an independent strategy function feature dictionary and state value function dictionary;

[0043] Step 3.4: The strategy function uses the spectrum availability state-action pair as the feature vector, and the state-value function uses the spectrum availability state as the feature vector. The feature vectors in the strategy function feature dictionary, the feature vectors in the value function feature dictionary, and the feature vectors not currently added to the dictionary are respectively formed into Gaussian radial basis functions.

[0044] Step 3.5: Use the online kernel sparse method to sparse the dictionary.

[0045] In the above step 3.3, the approximate linear independence detection algorithm is used to sparse the feature dictionary of the strategy function and the value function. The algorithm determines whether to add the dictionary by judging the linear combination of the feature function of the feature vector and the linear correlation of the feature vector to be judged. Specifically,

[0046] Step 3.3.1: Find the expected difference between the linear combination of the feature vector to be judged and the feature vector in the feature dictionary.

[0047] Step 3.3.2: Define the expected difference threshold μ.

[0048] Step 3.3.3: Determine the difference between the difference obtained in step 3.3.1 and the threshold μ. If it is greater than μ, add the feature vector to the feature dictionary. If it is less than or equal to the threshold μ, do not add it to the feature dictionary.

[0049] In the above step 5, the aggregation update mode and weight aggregation update mode for the strategy function and state value function feature dictionary are:

[0050] Step 5.1: Select the dictionary with the largest number of entries among all threads as the initial dictionary, and use the approximate linear independence detection algorithm to judge the features in the dictionaries of the remaining threads that are different from the initial dictionary;

[0051] Step 5.2: Perform a weighted average operation on the weights corresponding to the feature vectors to obtain new weights;

[0052] Step 5.3: Copy the updated feature vector dictionary and weights to all threads.

[0053] The above step 3 is: when the receiving device determines that the interference pattern of the jammer has changed, repeat steps 1-5, and whenever t = uT 2 , start the transfer learning algorithm, dynamically adjust the strategy function feature dictionary and the state value function feature dictionary until the algorithm converges, and use the converged action selection decision as the optimal decision for anti-interference frequency allocation, so that the receiving device obtains the optimal strategy and performs frequency selection and data transmission according to this strategy. Specifically:

[0054] After the jammer interference pattern changes, at t = uT 2When , the feature dictionary is dynamically adjusted using a feature transfer learning algorithm based on similarity calculation:

[0055] Step 1): Take the first n eigenvectors r vectors and the latest added n c vectors form a feature set

[0056] Step 2): Define a feature set This collection is the collection removed The set of the c-th element in .

[0057] Step 3): By calculating and The cosine principal similarity of is calculated, and the z eigenvectors with the lowest similarity are removed to keep the number of eigenvectors unchanged.

[0058] The present invention has the following beneficial effects:

[0059] The present invention can be used for dynamic detection of spectrum status in inter-device communication. In a communication system with unknown interference sources, the interference pattern in the online learning environment can be used to make the optimal spectrum allocation strategy, so that frequency users can avoid interference frequencies and successfully transmit data.

[0060] The present invention proposes an intelligent online spectrum anti-interference frequency allocation method for inter-device communication, in which a transmitting device sends a communication application to a receiving device, and the receiving device returns confirmation information after receiving the application and starts an intelligent online spectrum anti-interference algorithm. Among them, the online spectrum anti-interference algorithm uses an adaptive core aggregation anti-interference frequency allocation method based on an asynchronous advantage actor-evaluator architecture to perform Markov modeling on the frequency allocation problem under interference environment, with the objective function of maximizing the expectation of the cumulative discounted return at the frequency-using device in the communication network. At the beginning of each time slot, the receiving device allocates a frequency as a working frequency to the user who applies for network access before the beginning of each time slot. Each working frequency only allows one user to communicate. At the same time, the receiving device begins to learn the interference law of the external interference source until the frequency allocation algorithm converges. The receiving device obtains the optimal strategy and performs frequency selection and data transmission according to this strategy. If, after the algorithm converges, the receiving device determines that the interference pattern of the external interference source has changed, it will start the transfer learning algorithm to perform online migration of the characteristic function and repeat the previous learning steps until the learned algorithm converges again. The receiving device will allocate frequencies and transmit data to the algorithm in the next time slot according to the re-learned strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a system model diagram of a specific implementation mode of the present invention;

[0062] Figure 2 It is an overall flow chart of a specific implementation mode of the present invention;

[0063] Figure 3 It is the time slot diagram for anti-interference communication;

[0064] Figure 4 It is a data interaction and execution module in the anti-interference communication of the present invention;

[0065] Figure 5 Schematic diagram of online update of anti-interference algorithm based on asynchronous advantage actor-judge;

[0066] Figure 6 It is a complete algorithm flow chart of the present invention;

[0067] Figure 7 It is a flow chart of the online feature dictionary sparse algorithm in the present invention;

[0068] Figure 8 Flow chart of online feature vector transfer learning in the present invention. DETAILED DESCRIPTION

[0069] The embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0070] Considering an inter-device communication environment with interference from jammers, an intelligent online spectrum anti-interference frequency allocation method for inter-device communication is proposed. At the beginning of each time slot, the frequency control center allocates a working frequency to a frequency-using device. At the same time, the center uses external spectrum detection equipment to analyze the spectrum situation of the available frequency band and learn the interference pattern of the external interference source. After a period of online learning, the anti-interference frequency allocation algorithm at the center converges and obtains the optimal spectrum allocation strategy. In the following time slots, the available frequency points are allocated to the frequency users according to this strategy for data transmission. After the algorithm converges, if the frequency control center senses that the interference pattern has changed, it will use a transfer learning-based method to dynamically adjust the elements in the feature vector dictionary until the algorithm converges again and allocates spectrum and transmits data according to the newly learned strategy. It can be used for dynamic detection of spectrum situation in inter-device communication. In a communication system with unknown interference sources, the interference pattern in the online learning environment can be used to make the optimal spectrum allocation strategy, so that the frequency users can avoid the interference frequency points and successfully transmit data. See Figure 1-3 , the method of the present invention specifically comprises:

[0071] Step 1: The sending device sends a communication request to the receiving device;

[0072] Step 2: After receiving the application, the receiving device returns a confirmation message and starts the intelligent online spectrum anti-interference algorithm.

[0073] In step 2, first, the frequency allocation problem in the interference environment is modeled by Markov modeling, and the environmental state information is defined as T h The linear combination of the historical state characteristics of all available channels in time slots is s t , the channel a selected at each moment t , the transition probability p, the instantaneous reward is set as a piecewise function r t , the channel interference and successful transmission respectively obtain different reward values, the strategy of the Markov decision process is π, in order to find the optimal strategy, it is necessary to maximize the expectation of the cumulative discounted return at the frequency-using equipment in the communication network as the objective function, that is, the optimal Bellman equation: Where γ is the discount factor;

[0074] Step 2 uses an online spectrum anti-interference algorithm to solve the optimal Bellman equation, wherein the online spectrum anti-interference algorithm uses an adaptive core aggregation anti-interference frequency allocation method based on an asynchronous advantage actor-evaluator architecture. At the beginning of each time slot, the receiving device allocates a frequency as a working frequency to the user applying for network access before the beginning of each time slot. Each working frequency only allows one user to communicate. At the same time, the receiving device begins to learn the interference rules of external interference sources until the frequency allocation algorithm converges. The receiving device obtains the optimal strategy and performs frequency selection and data transmission according to this strategy;

[0075] Step 3: If the receiving device determines that the interference pattern of the external interference source has changed after the algorithm converges, it will start the transfer learning algorithm to perform online migration of the characteristic function and repeat the previous learning steps until the learned algorithm converges again. The receiving device will allocate frequency points and transmit data in the next time slot according to the re-learned strategy;

[0076] In the specific implementation, when the adaptive core aggregation anti-interference frequency allocation method based on the asynchronous advantage actor-evaluator architecture is adopted in step 2, such as Figure 4 and Figure 6 As shown, the method includes:

[0077] Step 1: The receiving user obtains the signal-to-interference-noise ratio of each available frequency point of the frequency-using device, and obtains the spectrum characteristics within a preset time range in the past as the current state characteristics based on the channel state extraction algorithm based on moment estimation, and constructs a spectrum availability feature set based on this.

[0078] Step 2: The receiving device sets k sub-threads, each of which has the same policy function network and value function network during the initialization process. Set a preset fixed time period T 1 , at the beginning of each fixed time slot interval t = nT 1, n = 0, 1, 2, ... randomly select a sub-thread with the goal of maximizing the policy function, select the frequency for data transmission from all available frequencies, and within this time period, use the policy function of the thread selected at the beginning to make decisions.

[0079] Step 3: The sending device sends a data packet to the target according to the control instruction returned by the receiving device, obtains the received signal at the receiving device, and obtains the instantaneous return value of the action selection. The instantaneous return value refers to whether the frequency-using device in a certain time slot is connected to an undisturbed frequency point as the working frequency point, and stores the state-action-return pair and the state in the policy function experience pool and the state value function experience pool respectively; except for the selected thread, other threads select a part of samples from the policy function experience pool and the state value function experience pool to update the policy function and value function weights, as well as the policy function feature dictionary and the value function feature dictionary;

[0080] Step 4: Repeat steps 1-3, when t = nT 1 At the end of the time slot, the policy function feature dictionary and the value function feature dictionary of the selected thread are updated.

[0081] Step 5: Repeat steps 1-4 until a preset fixed time period T is reached. 2 , T 2 =mT 1 , m is a positive integer, that is, t = uT 2 ,u is a positive integer, m>k. After the time slot ends, the policy function network and value function network of all threads in step 3 are weighted and aggregated, and the policy function feature dictionary and value function feature dictionary are updated, and the updated dictionary and weights are copied to all threads.

[0082] Step 6: Repeat 1-5 until the algorithm converges, and use the action selection decision after convergence as the optimal decision for anti-interference frequency allocation.

[0083] In specific implementation, the signal to noise ratio extraction algorithm based on moment estimation is adopted for two different interference signal models in step 1 above, specifically:

[0084] Step 1.1: The receiving device receives the s Independent and identically distributed received signal samples are taken and the means of the first-order, second-order, and fourth-order terms of the samples are calculated as the first-order, second-order, and fourth-order estimates, respectively, which are expressed as and remember And calculate the expected value of the fourth power of the modulus of the received signal sample and record it as p u′ ;

[0085] Step 1.2: Calculate the signal-to-interference-noise ratio estimate

[0086] Signal-to-Interference-Noise Ratio Estimation There are two ways to calculate . When the algorithm is first executed, only one of the calculation methods is selected and remains unchanged in the following algorithms:

[0087] 1) When the interference signal model can be modeled as a Gaussian model,

[0088] 2) When the interference signal contains only one main component interference signal:

[0089]

[0090] Step 1.3: Set the signal-to-interference-noise ratio threshold and energy value threshold to determine the channel occupancy status.

[0091] In the specific implementation, in the above step 2, an asynchronous update architecture of asynchronous advantage actor-critic based on kernel function approximation is adopted, see Figure 5 , specifically including the following steps:

[0092] Step 2.1: Set up a network of k threads, each thread contains two independent networks, namely the policy function network and the state value function network, where the policy function satisfies the Gibbs sampling distribution or the Boltzmann strategy, and the characteristic parameters of each network are a linear combination of a set of Gaussian radial basis functions. The above expression also satisfies the formula: This function indicates that in the case of state s, the probability that the algorithm chooses action a is is the parameterized numerical preference for each state-action pair. In particular, the parameterized preference equation can be approximated by a linear characteristic function, i.e. in is the weight parameter of H. In each time slot, each action a has a different probability

[0093] Step 2.2: Set a preset fixed time period T 1 , at the beginning of each fixed time slot interval t = nT 1 , n=0,1,2,… randomly select a thread's policy function for decision making. The policy function is the probability value corresponding to each action. The sum of the probability values ​​corresponding to all actions is 1. For each time slot t, a uniformly distributed number with a value range of (0-1) is randomly generated. If the generated value falls within the probability value range corresponding to the action, communication is carried out on the channel corresponding to the action. Otherwise, an available channel is selected from the set of available channels with random probability for communication.

[0094] In step 3 above, the update mode for the policy function and the value function weights includes the following steps:

[0095] Step 3.1: After each time slot t ends, the transmitting device sends a data packet to the target according to the control instruction returned by the receiving device, obtains the received signal at the receiving device, obtains the spectrum availability of the entire available frequency band according to the method in step 1, and takes the availability of the entire spectrum in a preset time period as the state.

[0096] Step 3.2: Set up a shared experience pool and store the state-action-reward pair after each decision as a sample into the shared experience pool; except for the selected thread, other threads select a part of samples from the policy function experience pool and the state value function experience pool, and incrementally update the policy function and value function weights of their respective threads.

[0097] Step 3.3: Each thread sets up an independent strategy function feature dictionary and state value function dictionary.

[0098] Step 3.4: The strategy function uses the spectrum availability state-action pair as the feature vector, and the state-value function uses the spectrum availability state as the feature vector. The feature vectors in the strategy function feature dictionary, the feature vectors in the value function feature dictionary, and the feature vectors not currently added to the dictionary are respectively formed into Gaussian radial basis functions.

[0099] Step 3.5: Use the online kernel sparse method to sparse the dictionary.

[0100] In the above step 3.3, the approximate linear independent detection algorithm is used to sparse the feature dictionary of the strategy function and the value function, see Figure 7 , by judging the linear correlation between the linear combination of the characteristic function of the feature vector and the feature vector to be judged, the algorithm determines whether to add the dictionary. The specific steps are:

[0101] Step 3.3.1: Find the expected difference between the linear combination of the feature vector to be judged and the feature vector in the feature dictionary.

[0102] Step 3.3.2: Define the expected difference threshold μ.

[0103] Step 3.3.3: Determine the difference between the difference obtained in step 3.3.1 and the threshold μ. If it is greater than μ, add the feature vector to the feature dictionary. If it is less than or equal to the threshold μ, do not add it to the feature dictionary.

[0104] In the above step 5, the aggregation mode and weight aggregation mode of the feature dictionary of the strategy function and the state value function include:

[0105] Step 5.1: Select the dictionary with the largest number of entries among all threads as the initial dictionary, and judge the features in the dictionaries of the remaining threads that are different from the initial dictionary according to the approximate linear independence detection algorithm described in step 3.

[0106] Step 5.2: Perform a weighted average operation on the weights corresponding to the feature vectors to obtain new weights.

[0107] Step 5.3: Copy the updated feature vector dictionary and weights to all threads.

[0108] In the above step 3, when the receiving device determines that the interference pattern of the jammer has changed, steps 1-5 are repeated, and whenever t = uT 2 , start the transfer learning algorithm, dynamically adjust the strategy function feature dictionary and the state value function feature dictionary until the algorithm converges, and use the converged action selection decision as the optimal decision for anti-interference frequency allocation, that is, after the jammer style changes, at t = uT 2 When , the feature dictionary is transferred based on similarity calculation, refer to Figure 8 As shown, the specific steps include:

[0109] Step 1): Take the first n eigenvectors r vectors and the latest added n c vectors form a feature set

[0110] Step 2): Define a feature set This collection is the collection removed The set of the c-th element in .

[0111] Step 3): By calculating and The cosine principal similarity of is calculated, and the z eigenvectors with the lowest similarity are removed to keep the number of eigenvectors unchanged.

[0112] The specific analysis and examples are as follows:

[0113] like Figure 2 As shown in Figure 1, consider a device-to-device communication system in an interference environment consisting of multiple jammers, a transmitter user, and a receiver user. The entire system has C available frequency domain channels, and each channel is only allowed to be used by one user at a time. is the transmission signal sent to channel c at time t, is the interference signal sent by jammer l to channel c at time t, is the channel gain between the transmitting device and the receiving device about channel c at time t, is the channel gain between the jammer l and the receiving device about channel c at time t, so the expression of the received signal about channel c at time t is:

[0114]

[0115] Among them, λ t,c ∈{0,1} represents whether the user transmission signal is transmitted to channel c at time t, η t,c,l ∈{0,1} represents whether the jammer l sends an interference signal to channel c, n at time t 0 The mean is 0 and the variance is Gaussian white noise.

[0116] The transmission power is defined as The jammer interference power is Therefore, the signal-to-interference-to-noise ratio of the receiving user for channel c at time t can be expressed as

[0117] Assume that there is N s Independent and identically distributed samples Then we can get:

[0118] In order to be able to t,c The channel usage status is obtained by using a non-data-assisted estimation method to obtain the receiver's information about v t,c Specifically, two interference signal modes are considered:

[0119] 1)

[0120] make and By using the moment estimation method, ρ t,c Can be based on Get, where as well as

[0121] 2) There is a synchronous interference signal sent by the jammer in the channel make and By using the moment estimation method, ρ t,c Can be based on Get, where d is a priori constant that can be obtained through multiple simulation experiments.

[0122] In order to further obtain the availability of the frequency domain channel, that is, whether it is interfered by the jammer or occupied, it is necessary to define a power threshold p 0,th , which is significantly lower than the power of the received communication / interference signal but higher than the noise power.

[0123] ρ this the signal-to-interference-noise ratio demodulation threshold. Then, by using and The channel usage of the cth frequency domain channel at time t (ie, in the absence or presence of interference) can be extracted, as shown in Table 1.

[0124] Table 1 Channel usage estimation

[0125]

[0126] The dynamic spectrum anti-interference problem can be modeled as a Markov decision process, where the channel usage of channel c at time t is defined as s t,c , it indicates whether the channel is interfered by the jammer, a value of 1 means it is interfered, and a value of 0 means it is not interfered. t,c It can be obtained through the above method and Table 1.

[0127] Suppose the set of channel usage of all frequency domain channels is The set of all selectable channels is Complex interference patterns may be related to historical channel usage, so the environmental state information is defined as T h The historical status information of all available channels in time slots can be expressed as o t ={s t,1 ,...,s t,C} T , the transition probability p(s t+1 |s t ,a). The instantaneous return is set as a piecewise function r t , when the selected channel is interfered, r t =R f , when the selected channel is not interfered, r t =-R f , R s and R f All are positive values.

[0128] The strategy of the Markov decision process is defined as π. To find the optimal strategy, we need to solve the optimal Bellman equation: Where γ is the discount factor.

[0129] Since the jammer interference pattern is unknown, reinforcement learning method is used to solve the optimal strategy of the above optimal Bellman equation.

[0130] Specifically, the core-based distributed asynchronous advantage actor-judge proposed in the present invention is adopted. For the convenience of explanation, () v Represents the relevant parameters of the approximate function, () q Represents the relevant parameters of the approximate policy function.

[0131] Drawing on the update architecture of the asynchronous advantage actor-critic, K sub-threads are set up, each of which includes an actor network and a critic network.

[0132] The actor network uses the policy function to make decisions. In particular, the Boltzmann function is used as the policy function of child thread k: This function indicates that in the case of state s, the probability that the algorithm chooses action a is is the parameterized numerical preference for each state-action pair.

[0133] In particular, the parameterized preference equation can be approximated by a linear characteristic function, namely in is the weight parameter of H. In each time slot, each action a has a different probability The critic network can be set as a value function

[0134] In order to avoid the curse of dimensionality caused by a large state space and speed up the convergence, the Gaussian kernel function is used. to approximate H and V′, where f is the eigenvector of the dynamic environment.

[0135] The eigenvectors s and a can be represented as f s and f a .

[0136] Define the feature vector of the policy function The eigenvector of the value function f q =f s .

[0137] Specifically, the Gaussian kernel function is used To approximate H, where x t =(s t ,a t ),use To approximate V′, σ s and σ a The eigenvectors f are s and f a characteristic length range.

[0138] The present invention adopts an approximate linear independent detection method to perform online sparseness on the characteristic Gaussian kernel function. Specifically, two functions for storing f are defined for the strategy function network and the value function network, respectively. q and f v The feature vector dictionary and The corresponding characteristic function set can be expressed as {φ(fq )} and {φ(f v )}. For simplicity, we use the approximate linear independence detection algorithm to detect {φ(f q )} is taken as an example to illustrate the specific steps of this method. v )} can be obtained in a similar way.

[0139] definition As {φ(f q )}, a new feature vector added to the feature dictionary needs to satisfy the approximate linear independence detection condition, that is, Therefore, we need to find the optimal weight vector δ t It can be further written as Among them, <φ(x),φ(x′)>=g(x,x′), g(x t ,x t ) = g tt ,and So, when When t is the minimum value g tt -g t-1 (x t ) T α q In addition, it can be calculated by iterative Where, ot = [G t-1 ] -1 gt-1(xt) and G 1 =1. Let μ be the approximate linear independence analysis condition threshold, then if δ t >μ, then on the contrary, Remain unchanged.

[0140] The present invention divides the entire time slot into multiple time periods, each time period lasting T s time slots, T s ≥T h For the convenience of description, (*) is used to indicate the relevant parameters of each subthread. Before officially starting the algorithm, several spectrum decisions are made randomly and some parameters are obtained for initialization parameters. Initialize a t ,a t ,r t ,s t+1 ) experience pool. Initialize the global strategy function feature vector dictionary Eigenvector dictionary for global value functions and the global feature weight, i.e. and These parameters are then copied to all child threads. At the beginning of each time period, a child thread is randomly selected until the next time period begins and then reselected. The policy function of this thread is used to decide the channel that the sending user can use in the next time slot. In each time slot, the user uses the policy function of the child thread selected at the beginning of each time period to make a decision and store the obtained samples in the experience pool. The other child threads randomly select d from the experience pool. s Samples (s′ i,t ,a′ i,t ,r′ i,t ,s′ i,t+1 ),i=1,2,...,d s , the n-step temporal difference updating method is used to update the feature weights respectively, that is, and k is the line program number. From j = d s ,d s -1,...,1, execute the calculation of the cumulative gradient of the policy function:

[0141] in, Similarly, the cumulative gradient of the value function is: Afterwards you can get and Update: as well as in, and are the learning rates of the policy function and the value function respectively. s , When t=nT, the weight parameters and dictionary of the sub-thread used for decision-making are updated at the beginning of each time period. The only difference from the above sub-thread update step for each time slot is that all samples stored in the dictionary within the time slot decided by the sub-thread are used for update. a , T a =mT s , m ≥ K, perform global update on all sub-threads. Since the number of feature vector dictionaries of each sub-thread may be inconsistent, first remove the repeated feature vectors in all dictionaries, and remove their corresponding weight parameters, and then fill the weights of the corresponding positions with zeros to obtain the weight parameters of each sub-thread after zero filling. and The global weight parameters are obtained by weighted average update:

[0142] Taking the global update of the feature vector dictionary of the policy function as an example, the feature vector dictionary of the value function can be updated in the same way. The specific steps are to define a dictionary Used to store the feature vectors and matrices of all sub-threads It is the dictionary with the largest number of feature vectors among all corresponding sub-thread dictionaries. Obviously, right Execute the approximate linear independence detection algorithm and judge the set in turn Whether the feature vector in is added to the dictionary Finally, the global update parameters obtained are [G q ] -1 , [G v ] -1 , w q and w v Copy to all child threads and continue learning until the algorithm meets the convergence conditions.

[0143] Set a constant η 0 ∈(0,1), after the algorithm converges, when the decision error rate η>η 0 The receiving end will continue to execute algorithm learning at t = nT a , At this moment, the transfer learning algorithm is started, and the maximum number of feature vector dictionaries of the strategy function is defined as The maximum number of value function feature vector dictionaries is

[0144] Before executing the transfer learning algorithm in each time period, more than and The feature vector of will be temporarily stored in the corresponding dictionary. Similarly, taking the feature vector dictionary migration of the strategy function as an example, the migration of the feature vector dictionary of the value function can be obtained in the same way.

[0145] Specifically, when hour, The feature vector will be Removed.

[0146] The present invention adopts a kernel principal angle method to measure the similarity between a new feature vector and an original feature vector group.

[0147] The feature vectors in are arranged in time order, defining a set To store Middle front n r feature vectors and the latest n c The labels of the feature vectors.

[0148] Set the characteristic equation set {φ(f q )}

[0149] Assume that the set of labels of the feature vectors to be removed is Then and Written in column vector form U j and V j . j and V j Perform the QR decomposition operation, that is, and By solving To solve the cosine principal

[0150] Then, by calculating And remove The jth eigenvector in [G q ] -1 The row and column corresponding to the jth eigenvector need to be removed. The above steps are repeated until

[0151] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. An intelligent spectrum online anti-interference frequency point allocation method for inter-device communication, characterized in that: include: Step 1: The sending device sends a communication request to the receiving device; Step 2: After receiving the application, the receiving device returns confirmation information and starts the intelligent online spectrum anti-interference algorithm. At the beginning of each time slot, the receiving device allocates a frequency point as the working frequency point for the user applying for network access before the beginning of each time slot. Each working frequency point only allows one user to communicate. At the same time, the receiving device starts to learn the interference law of external interference sources until the algorithm converges. The receiving device obtains the optimal strategy and selects the frequency point and transmits data according to this strategy; Step 3: After the anti-interference algorithm converges, if the receiving device determines that the interference pattern of the external interference source has changed, the transfer learning algorithm is started to perform online migration of the characteristic function, and the anti-interference algorithm is re-learned until the algorithm converges. The receiving device performs frequency point allocation and data transmission in the next time slot according to the re-learned strategy; In step 2, first, the frequency allocation problem in the interference environment is modeled by Markov modeling, and the environmental state information is defined as T h The linear combination of the historical state characteristics of all available channels in time slots is s t , the channel a selected at each moment t , the transition probability p, the instantaneous reward is set as a piecewise function r t , the channel interference and successful transmission respectively obtain different reward values, the strategy of the Markov decision process is π, in order to find the optimal strategy, it is necessary to maximize the expectation of the cumulative discounted return at the frequency-using equipment in the communication network as the objective function, that is, the optimal Bellman equation: Where γ is the discount factor; Step 2: using an online spectrum anti-interference algorithm to solve the optimal Bellman equation, wherein the online spectrum anti-interference algorithm uses an adaptive core aggregation anti-interference frequency point allocation method based on an asynchronous advantage actor-evaluator architecture; The second step comprises: Step 1: The receiving end user obtains the signal-to-interference-to-noise ratio of each available frequency point of the frequency-using device, and obtains the spectrum characteristics within a preset time range in the past as the current state characteristics according to the channel state extraction algorithm based on moment estimation, and constructs a spectrum availability feature set based on this; Step 2: The receiving device uses an asynchronous update architecture of an asynchronous advantage actor-judge based on kernel function approximation to set up k sub-threads, and each sub-thread has the same policy function network and value function network during the initialization process; Set a fixed time period T1, at the beginning of each fixed time slot interval t = nT1, n = 0, 1, 2, ... randomly select a subthread to maximize the policy function as the goal, select the frequency for data transmission among all available frequency points, and in this time period, use the policy function of the thread selected at the beginning to make decisions; Step 3: The sending device sends a data packet to the target according to the control instruction returned by the receiving device, obtains the received signal at the receiving device, and obtains the instantaneous return value of the action selection. The instantaneous return value refers to whether the frequency-using device in a certain time slot is connected to an undisturbed frequency point as the working frequency point. The state-action-return pair and the state are stored in the policy function experience pool and the state value function experience pool respectively; except for the selected thread, other threads select a part of samples from the policy function experience pool and the state value function experience pool to update the weights of the policy function and the value function, and update the feature dictionary of the policy function and the value function; Step 4: Repeat steps 1-3. When the time slot t=nT1 ends, update the policy function feature dictionary and value function feature dictionary of the selected thread. Step 5: Repeat steps 1-4. When the preset fixed time period T2 is reached, T2=mT1, m is a positive integer, that is, t=uT2, u is a positive integer, m>k, after the time slot ends, the policy function network of all threads in step 3 is updated, the policy function and state value function feature dictionary and weight aggregation are updated, and the updated dictionary and weight are copied to all threads; Step 6: Repeat 1-5 until the algorithm converges, and use the action selection decision after convergence as the optimal decision for anti-interference frequency allocation, so that the receiving device obtains the optimal strategy and performs frequency selection and data transmission according to this strategy; The step 1 obtains the current state characteristics, that is, the occupancy state of the channel, based on the signal-to-interference-noise ratio extraction algorithm of moment estimation, which is specifically: Step 1.1: The receiving device receives the s Independent and identically distributed received signal samples are taken and the means of the first-order, second-order, and fourth-order terms of the samples are calculated as the first-order, second-order, and fourth-order estimates, respectively, which are expressed as and remember And calculate the expected value of the fourth power of the modulus of the received signal sample and record it as p u′ ; Step 1.2: Calculate the signal-to-interference-noise ratio estimate Signal-to-Interference-Noise Ratio Estimation There are two ways to calculate . When the algorithm is first executed, only one of the calculation methods is selected and remains unchanged in the following algorithms: 1) When the interference signal model is modeled as a Gaussian model, 2) When the interference signal contains only one main component interference signal: Step 1.3: Set the signal-to-interference-noise ratio threshold and energy value threshold to determine the channel occupancy status; Step 2 is as follows: Step 2.1: Set up a network of k threads, each thread contains 2 independent networks: a policy function network and a state-value function network; Among them, the strategy function satisfies the Gibbs sampling distribution or the Boltzmann strategy, and the characteristic parameters of each network are a linear combination of a set of Gaussian radial basis functions, satisfying the formula: Indicates the probability that the algorithm chooses action a in state s; is the parameterized numerical preference for each state-action pair, in, is the weight parameter of H. In each time slot, each action a has a different probability Step 2.2: Set a preset fixed time period T1. At the beginning of each fixed time slot interval t=nT1, n=0,1,2,..., randomly select a policy function of a sub-thread for decision-making. The policy function is the probability value corresponding to each action. The sum of the probability values ​​corresponding to all actions is 1. For each time slot t, randomly generate a uniformly distributed number with a value range of (0-1). If the generated value falls within the probability value range corresponding to the action, communication is performed on the channel corresponding to the action. Otherwise, an available channel is selected from the available channel set with random probability for communication. In step 3, the update mode of the policy function and value function weights is: Step 3.1: After each time slot t ends, the transmitting device sends a data packet to the target according to the control instruction returned by the receiving device, obtains the received signal at the receiving device, obtains the spectrum availability of the entire available frequency band according to the method in step 1, and takes the availability of the entire spectrum in a preset time period as the state; Step 3.2: Set up a shared experience pool, and store the state-action-reward pair after each decision as a sample in the shared experience pool; except for the selected thread, other threads select a part of samples from the policy function experience pool and the state value function experience pool, and incrementally update the policy function and value function weights of their respective threads; In step 3, the update mode of the strategy function and value function feature dictionary is: Step 3.3: Each thread sets up an independent strategy function feature dictionary and state value function dictionary; Step 3.4: The strategy function uses the spectrum availability state-action pair as a feature vector, and the state-value function uses the spectrum availability state as a feature vector. The feature vectors in the strategy function feature dictionary, the feature vectors in the value function feature dictionary, and the feature vectors not currently added to the dictionary are respectively formed into Gaussian radial basis functions; Step 3.5: Use the online kernel sparse method to sparse the dictionary; In step 3.3, the approximate linear independence detection algorithm is used to sparse the feature dictionary of the strategy function and the value function. The algorithm determines whether to add the dictionary by judging the linear combination of the feature function of the feature vector and the linear correlation of the feature vector to be judged. Specifically: Step 3.3.1: Find the expected difference between the linear combination of the feature vector to be judged and the feature vector in the feature dictionary; Step 3.3.2: Define the expected difference threshold μ; Step 3.3.3: Determine the difference between the difference obtained in step 3.3.1 and the threshold μ. If it is greater than μ, add the feature vector to the feature dictionary. If it is less than or equal to the threshold μ, do not add it to the feature dictionary. In step 5, the aggregation update mode and weight aggregation update mode for the strategy function and state value function feature dictionary are: Step 5.1: Select the dictionary with the largest number of entries among all threads as the initial dictionary, and use the approximate linear independence detection algorithm to judge the features in the dictionaries of the remaining threads that are different from the initial dictionary; Step 5.2: Perform a weighted average operation on the weights corresponding to the feature vectors to obtain new weights; Step 5.3: Copy the updated feature vector dictionary and weights to all threads; The step three is: when the receiving device determines that the interference pattern of the jammer has changed, repeat steps 1-5, and whenever t=uT2, start the transfer learning algorithm, dynamically adjust the strategy function feature dictionary and the state value function feature dictionary until the algorithm converges, and use the converged action selection decision as the optimal decision for anti-interference frequency allocation, so that the receiving device obtains the optimal strategy and performs frequency selection and data transmission according to this strategy. Specifically: After the jammer interference pattern changes, at t = uT2, the feature dictionary is dynamically adjusted using a feature transfer learning algorithm based on similarity calculation: Step 1): Take the first n eigenvectors r vectors and the latest n c vectors form a feature set Step 2): Define a feature set This collection is the collection removed The set of the cth element in ; Step 3): By calculating and The cosine principal similarity of is calculated, and the z eigenvectors with the lowest similarity are removed to keep the number of eigenvectors unchanged.

Images