A multi-user interference avoidance method and device

Abstract

The present invention discloses a multi-user interference avoidance method and device, the method comprising: using an interference background generation model to process modulation noise to obtain a frequency modulation interference signal; processing the frequency modulation interference signal to obtain an interference spectrum; constructing a multi-user interference avoidance network model, designing an action selection space, a state space, a reward function, and initializing hyperparameters; training the multi-user interference avoidance network model to obtain a training multi-user interference avoidance network model; using the training multi-user interference avoidance network model to process the interference spectrum to obtain a multi-user interference avoidance result; constructing a visualization platform, and using the visualization platform to display the multi-user interference avoidance result. The present invention can effectively improve the timeliness of multi-user interference avoidance, and at the same time, has a lower spectrum conflict rate in a complex interference environment.

Description

Technical Field

[0001] The present invention relates to the technical field of reinforcement learning, and in particular to a multi-user interference avoidance method and device. Background Art

[0002] The openness and sharing of wireless communications bring conveniences such as easy access and access to communication users, but also expose communication networks to wireless environments and are more vulnerable to interference attacks than wired networks. Interference attacks, as an important threat to wireless communication security, mainly affect and destroy the user's signal reception process by radiating electromagnetic signals, causing serious degradation of communication performance. Especially after integrating artificial intelligence and communication countermeasures technology, intelligent interference equipment has the ability to perceive, learn and make decisions, which brings unprecedented challenges to the development of wireless communication anti-interference technology. Therefore, in order to ensure the security, reliability and stability of wireless communications, it is necessary to study intelligent interference avoidance technology under complex electromagnetic environment conditions.

[0003] Interference avoidance mainly avoids interference by transforming the signal to be transmitted into a signal space with different dimensions from the interference signal, including frequency hopping spread spectrum, time hopping spread spectrum, direct sequence spread spectrum, etc. Currently, the commonly used multi-user interference avoidance technologies are reinforcement learning and game theory.

[0004] Game theory is usually used to analyze behavioral decisions and equilibrium problems among multiple individuals that influence each other. In the early days, zero-sum games and non-zero-sum games were used to model the adversarial relationship between users and interference. Each party aims to maximize its own utility and find a confrontational strategy under the equilibrium solution. Furthermore, considering the hierarchical behavioral characteristics between the communication party and the interference party, the Stackelberg game has been used in a large number of works to model the master-slave relationship between the two. Under the Stackelberg game framework, convex optimization, reinforcement learning and other methods are used to optimize the power or channel strategy in an interference environment to maximize the transmission utility of the communication party. Although game theory can better characterize the adversarial decision-making relationship between the two communication adversaries from a theoretical level and guide the design of algorithms for solving related equilibrium strategies, it often assumes that both users and interference know each other's decision-making utility model, relies too much on prior information or strict assumptions, and still has great limitations in practical applications.

[0005] Therefore, with the widespread application of machine learning technology in the field of wireless communications, methods represented by deep reinforcement learning have been widely used in the field of interference avoidance. Under conventional interference modes such as combing, frequency sweeping, regular blocking, and combination, deep reinforcement learning methods are used to learn the law of interference change, thereby optimizing the frequency, power, and beam strategies of the communication party to achieve the effect of anti-interference. Further considering the interference avoidance technology under multi-user conditions, multi-agent reinforcement learning methods are usually used to decide the interference avoidance methods for multiple users. The defect is that there is a weak or even no correlation between different agents. It performs well under simple interference, but it is difficult to cope with more complex interference and lacks adaptability and robustness.

[0006] The above two existing multi-user interference avoidance technologies have the following shortcomings:

[0007] (1) The multi-user interference avoidance technology based on game theory relies too much on prior knowledge. In the absence of prior information about the interferer, it is difficult to accurately identify the interference pattern.

[0008] (2) The multi-user interference avoidance technology based on multi-agent reinforcement learning performs well in simple environments. However, in actual applications, the jammer is also quite intelligent. It is difficult for this method to quickly design an efficient and targeted countermeasure strategy and quickly adapt and autonomously optimize as the jammer changes. Summary of the invention

[0009] The technical problem to be solved by the present invention is to provide a multi-user interference avoidance method and device. Different from the traditional intensive use of the maximum value of the network output as the training target, the learning target of the present invention is the first few maximum values, which is used as the basis to achieve multi-user interference avoidance. It is used to solve the defect that the existing multi-user interference avoidance method is difficult to adapt to complex interference environments. The present invention makes the enemy interference intelligent and simulates more complex regular sweep frequency interference, making the present invention more practical.

[0010] In order to solve the above technical problem, a first aspect of an embodiment of the present invention discloses a multi-user interference avoidance method, the method comprising:

[0011] S1, using the interference background generation model, the modulated noise is processed to obtain the FM interference signal;

[0012] S2, processing the frequency modulation interference signal to obtain an interference spectrum;

[0013] S3, build a multi-user interference avoidance network model, design the action selection space, state space, reward function, and initialize hyperparameters;

[0014] S4, training the multi-user interference avoidance network model to obtain an optimized multi-user interference avoidance network model;

[0015] S5, using the optimized multi-user interference avoidance network model to process the interference spectrum to obtain a multi-user interference avoidance result;

[0016] S6, constructing a visualization platform, and using the visualization platform to display the multi-user interference avoidance result.

[0017] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the interference background generation model is:

[0018]

[0019] Where J(t) is the frequency modulation interference signal, ω j is the central angular frequency, u(t) is the modulation noise, which is a zero-mean wide-sense stationary random process that conforms to the normal distribution. is the initial angle, which is a uniformly distributed and independent random variable on [0,2π], U j FM interference signal amplitude, K FM is the frequency modulation slope, t and t′ are time variables.

[0020] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the processing of the frequency modulation interference signal to obtain an interference spectrum includes:

[0021] S21, constructing first frequency sweep interference information and second frequency sweep interference information;

[0022] The first frequency sweep interference information is represented by a transfer matrix as [0.4, 0.3, 0.3], wherein the first column of the matrix is ​​the probability of upward transfer, the second column is the probability of unchanged, and the third column is the probability of downward transfer;

[0023] The second frequency sweep interference information is expressed by a transfer matrix as follows:

[0024]

[0025] The first column of the transfer matrix is ​​the probability of upward transfer, the second column is the probability of unchanged, and the third column is the probability of downward transfer;

[0026] S22: Process the first frequency sweeping interference information, the second frequency sweeping interference information, and the frequency modulation interference signal to obtain an interference spectrum.

[0027] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the multi-user interference avoidance network model includes an input layer, a first processing layer, a second processing layer, a third processing layer, a fourth processing layer, a fifth processing layer and an output layer;

[0028] The input dimension of the input layer is 1×15×10;

[0029] The output of the input layer is the input of the first processing layer, and the output dimension of the first processing layer is 32×8×5;

[0030] The output of the first processing layer is the input of the second processing layer, and the output dimension of the second processing layer is 64×5×3;

[0031] The output of the second processing layer is the input of the third processing layer, and the output dimension of the third processing layer is 64×4×3;

[0032] The output of the third processing layer is the input of the fourth processing layer, and the output dimension of the fourth processing layer is 1×448;

[0033] The output of the fourth processing layer is the input of the fifth processing layer, and the output dimension of the fifth processing layer is 1×512;

[0034] The output of the fifth processing layer is the input of the output layer, and the output dimension of the output layer is 1×10.

[0035] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the design of the action selection space, the state space, and the reward function includes:

[0036] S31, design the action selection space and obtain the action space matrix;

[0037] S32, design the state space and obtain the state space time-frequency diagram;

[0038] S33, designing a reward function and obtaining a reward function matrix;

[0039] The reward function matrix is:

[0040]

[0041] Where R1(s) is the reward function value.

[0042] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the training of the multi-user interference avoidance network model to obtain an optimized multi-user interference avoidance network model includes:

[0043] S41, random initial interference position, obtain the current interference state spectrum;

[0044] S42, using a multi-user interference avoidance network model, processing the current interference state spectrum to obtain a behavior matrix;

[0045] S43, stepping to the next spectrum allocation time period and obtaining the next interference state spectrum, executing the behavior matrix, and obtaining a reward value;

[0046] S44, the current interference state spectrum, behavior matrix, next interference state spectrum and reward value constitute a transformation process;

[0047] S45, storing the transformation process in a replay buffer;

[0048] S46, taking out the batch size conversion process from the replay buffer;

[0049] S47, using the batch size transformation process, training the multi-user interference avoidance network model to obtain an optimized multi-user interference avoidance network model.

[0050] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the multi-user interference avoidance network model is trained by using the batch size transformation process to obtain an optimized multi-user interference avoidance network model, including:

[0051] S471, processing the batch size transformation process to obtain an approximate target value;

[0052] The approximate target value is:

[0053]

[0054] Among them, y t is the approximate target value at time t, r t is the reward value at time t, γ is the discount factor, max_n is the operator, max_n means finding the largest n values, n is the number of users, is the behavior matrix a based on time t+1 t+1 Find the largest n values, s t+1 is the interference state spectrum at time t+1, Q(s t+1 ,a t+1 θ - ) is the value function, θ - is the multi-user interference avoidance network model parameter at time t+1;

[0055] S472, taking the sum of the largest n approximate target values ​​as the target value of the multi-user interference avoidance network model;

[0056] S473, constructing a predicted value of the multi-user interference avoidance network model;

[0057] The predicted value b t for:

[0058]

[0059] in, is the behavior matrix a based on time t t Find the largest n values, Q(s t ,a t ; θ) is the value function, s t is the interference state spectrum at time t, a t+1 is the behavior matrix at time t, θ is the multi-user interference avoidance network model parameter at time t;

[0060] S474, performing gradient descent using the loss function, processing the target value and the predicted value, and obtaining a training multi-user interference avoidance network model.

[0061] A second aspect of an embodiment of the present invention discloses a multi-user interference avoidance device, the device comprising:

[0062] A signal generation module is used to process the modulation noise using the interference background generation model to obtain a frequency modulation interference signal;

[0063] An interference spectrum generating module, used for processing the frequency modulation interference signal to obtain an interference spectrum;

[0064] The network model building module is used to build a multi-user interference avoidance network model, design the action selection space, state space, reward function, and initialize hyperparameters;

[0065] A training module, used for training the multi-user interference avoidance network model to obtain a trained multi-user interference avoidance network model;

[0066] An interference avoidance module, used to process the interference spectrum using the trained multi-user interference avoidance network model to obtain a multi-user interference avoidance result;

[0067] The visualization module is used to construct a visualization platform and use the visualization platform to display the multi-user interference avoidance result.

[0068] As an optional implementation manner, in the second aspect of the embodiment of the present invention, the interference background generation model is:

[0069]

[0070] Where J(t) is the frequency modulation interference signal, ω j is the central angular frequency, u(t) is the modulation noise, which is a zero-mean wide-sense stationary random process that conforms to the normal distribution. is the initial angle, which is a uniformly distributed and independent random variable on [0,2π], U j FM interference signal amplitude, K FMis the frequency modulation slope, t and t′ are time variables.

[0071] As an optional implementation manner, in the second aspect of the embodiment of the present invention, the processing of the frequency modulation interference signal to obtain an interference spectrum includes:

[0072] S21, constructing first frequency sweep interference information and second frequency sweep interference information;

[0073] The first frequency sweep interference information is represented by a transfer matrix as [0.4, 0.3, 0.3], wherein the first column of the matrix is ​​the probability of upward transfer, the second column is the probability of unchanged, and the third column is the probability of downward transfer;

[0074] The second frequency sweep interference information is expressed by a transfer matrix as follows:

[0075]

[0076] The first column of the transfer matrix is ​​the probability of upward transfer, the second column is the probability of unchanged, and the third column is the probability of downward transfer;

[0077] S22: Process the first frequency sweeping interference information, the second frequency sweeping interference information, and the frequency modulation interference signal to obtain an interference spectrum.

[0078] As an optional implementation, in the second aspect of the embodiment of the present invention, the multi-user interference avoidance network model includes an input layer, a first processing layer, a second processing layer, a third processing layer, a fourth processing layer, a fifth processing layer and an output layer;

[0079] The input dimension of the input layer is 1×15×10;

[0080] The output of the input layer is the input of the first processing layer, and the output dimension of the first processing layer is 32×8×5;

[0081] The output of the first processing layer is the input of the second processing layer, and the output dimension of the second processing layer is 64×5×3;

[0082] The output of the second processing layer is the input of the third processing layer, and the output dimension of the third processing layer is 64×4×3;

[0083] The output of the third processing layer is the input of the fourth processing layer, and the output dimension of the fourth processing layer is 1×448;

[0084] The output of the fourth processing layer is the input of the fifth processing layer, and the output dimension of the fifth processing layer is 1×512;

[0085] The output of the fifth processing layer is the input of the output layer, and the output dimension of the output layer is 1×10.

[0086] As an optional implementation, in the second aspect of the embodiment of the present invention, the design of the action selection space, the state space, and the reward function includes:

[0087] S31, design the action selection space and obtain the action space matrix;

[0088] S32, design the state space and obtain the state space time-frequency diagram;

[0089] S33, designing a reward function and obtaining a reward function matrix;

[0090] The reward function matrix is:

[0091]

[0092] Where R1(s) is the reward function value.

[0093] As an optional implementation manner, in the second aspect of the embodiment of the present invention, the training of the multi-user interference avoidance network model to obtain an optimized multi-user interference avoidance network model includes:

[0094] S41, random initial interference position, obtain the current interference state spectrum;

[0095] S42, using a multi-user interference avoidance network model, processing the current interference state spectrum to obtain a behavior matrix;

[0096] S43, stepping to the next spectrum allocation time period and obtaining the next interference state spectrum, executing the behavior matrix, and obtaining a reward value;

[0097] S44, the current interference state spectrum, behavior matrix, next interference state spectrum and reward value constitute a transformation process;

[0098] S45, storing the transformation process in a replay buffer;

[0099] S46, taking out the batch size conversion process from the replay buffer;

[0100] S47, using the batch size transformation process, training the multi-user interference avoidance network model to obtain an optimized multi-user interference avoidance network model.

[0101] As an optional implementation manner, in the second aspect of the embodiment of the present invention, the multi-user interference avoidance network model is trained by using the batch size transformation process to obtain an optimized multi-user interference avoidance network model, including:

[0102] S471, processing the batch size transformation process to obtain an approximate target value;

[0103] The approximate target value is:

[0104]

[0105] Among them, y t is the approximate target value at time t, r t is the reward value at time t, γ is the discount factor, max_n is the operator, max_n means finding the largest n values, n is the number of users, is the behavior matrix a based on time t+1 t+1 Find the largest n values, s t+1 is the interference state spectrum at time t+1, Q(s t+1 ,a t+1 θ - ) is the value function, θ - is the multi-user interference avoidance network model parameter at time t+1;

[0106] S472, taking the sum of the largest n approximate target values ​​as the target value of the multi-user interference avoidance network model;

[0107] S473, constructing a predicted value of the multi-user interference avoidance network model;

[0108] The predicted value b t for:

[0109]

[0110] in, is the behavior matrix a based on time t t Find the largest n values, Q(s t ,a t ; θ) is the value function, s t is the interference state spectrum at time t, a t+1 is the behavior matrix at time t, θ is the multi-user interference avoidance network model parameter at time t;

[0111] S474, performing gradient descent using the loss function, processing the target value and the predicted value, and obtaining a training multi-user interference avoidance network model.

[0112] A third aspect of an embodiment of the present invention discloses another multi-user interference avoidance device, the device comprising:

[0113] A memory storing executable program code;

[0114] a processor coupled to the memory;

[0115] The processor calls the executable program code stored in the memory to execute the multi-user interference avoidance method as described in the first aspect of the embodiment of the present invention.

[0116] A fourth aspect of an embodiment of the present invention discloses a computer storable medium, wherein the computer storable medium stores computer instructions, and when the computer instructions are called, they are used to execute the multi-user interference avoidance method as described in the first aspect of the embodiment of the present invention.

[0117] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0118] (1) The single-frequency, multi-frequency or swept-frequency interferences targeted by the current multi-user interference avoidance methods are relatively simple. Among them, the interference frequency bands in single-frequency and multi-frequency interferences are fixed, and the swept-frequency interference is also a simple linear swept-frequency interference. The interference avoidance effects obtained under these interferences are not representative and have no practical application value. In a real interference environment, the interferer is also intelligent, and its interference law is more complex and changeable. However, the swept-frequency interference simulated by the present invention has no obvious law, and one or more transfer probability matrices are used as the swept-frequency law. This law is more complex and can better reflect the current complex interference environment.

[0119] (2) In view of the current multi-user interference avoidance field, the present invention proposes a distributed prediction reinforcement learning method and applies it to the multi-user interference avoidance field. Compared with the multi-user interference avoidance method based on multi-agent reinforcement learning, this method consumes less computing resources and can effectively improve the timeliness of multi-user interference avoidance. At the same time, it has a lower spectrum conflict rate in complex interference environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0120] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0121] Figure 1 It is a flow chart of a multi-user interference avoidance method disclosed in an embodiment of the present invention;

[0122] Figure 2 It is a flow chart of another multi-user interference avoidance method disclosed in an embodiment of the present invention;

[0123] Figure 3 is a schematic diagram of an interference spectrum disclosed in an embodiment of the present invention;

[0124] Figure 4It is a structural diagram of a multi-user interference avoidance network model disclosed in an embodiment of the present invention;

[0125] Figure 5 is a schematic diagram of a visualization interface disclosed in an embodiment of the present invention;

[0126] Figure 6 is a conflict rate display diagram disclosed in an embodiment of the present invention;

[0127] Figure 7 It is a structural diagram of a multi-user interference avoidance device disclosed in an embodiment of the present invention;

[0128] Figure 8 It is a structural diagram of another multi-user interference avoidance device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0129] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0130] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or equipment.

[0131] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0132] The present invention discloses a multi-user interference avoidance method and device, the method comprising: using an interference background generation model to process the modulated noise to obtain a frequency modulation interference signal; processing the frequency modulation interference signal to obtain an interference spectrum; constructing a multi-user interference avoidance network model, designing an action selection space, a state space, a reward function, and initializing hyperparameters; training the multi-user interference avoidance network model to obtain a trained multi-user interference avoidance network model; using the trained multi-user interference avoidance network model to process the interference spectrum to obtain a multi-user interference avoidance result; constructing a visualization platform, and using the visualization platform to display the multi-user interference avoidance result. The present invention can effectively improve the timeliness of multi-user interference avoidance, and at the same time, has a lower spectrum conflict rate in a complex interference environment. The following are detailed descriptions.

[0133] Embodiment 1

[0134] See also Figure 1 , Figure 1 1 is a flow chart of a multi-user interference avoidance method disclosed in an embodiment of the present invention. Figure 1 The multi-user interference avoidance method described is applied to fields such as user interference avoidance applications, and the embodiments of the present invention do not limit this. Figure 1 As shown, the multi-user interference avoidance method may include the following operations:

[0135] S1, using the interference background generation model, the modulated noise is processed to obtain the FM interference signal;

[0136] S2, processing the frequency modulation interference signal to obtain an interference spectrum;

[0137] S3, build a multi-user interference avoidance network model, design the action selection space, state space, reward function, and initialize hyperparameters;

[0138] S4, training the multi-user interference avoidance network model to obtain an optimized multi-user interference avoidance network model;

[0139] S5, using the optimized multi-user interference avoidance network model to process the interference spectrum to obtain a multi-user interference avoidance result;

[0140] S6, constructing a visualization platform, and using the visualization platform to display the multi-user interference avoidance result.

[0141] Optionally, the interference background generation model is:

[0142]

[0143] Where J(t) is the frequency modulation interference signal, ω jis the central angular frequency, u(t) is the modulation noise, which is a zero-mean wide-sense stationary random process that conforms to the normal distribution. is the initial angle, which is a uniformly distributed and independent random variable on [0,2π], U j FM interference signal amplitude, K FM is the frequency modulation slope, t and t′ are time variables.

[0144] Optionally, the processing the frequency modulation interference signal to obtain an interference spectrum includes:

[0145] S21, constructing first frequency sweep interference information and second frequency sweep interference information;

[0146] The first frequency sweep interference information is represented by a transfer matrix as [0.4, 0.3, 0.3], wherein the first column of the matrix is ​​the probability of upward transfer, the second column is the probability of unchanged, and the third column is the probability of downward transfer;

[0147] The second frequency sweep interference information is expressed by a transfer matrix as follows:

[0148]

[0149] The first column of the transfer matrix is ​​the probability of upward transfer, the second column is the probability of unchanged, and the third column is the probability of downward transfer;

[0150] S22: Process the first frequency sweeping interference information, the second frequency sweeping interference information, and the frequency modulation interference signal to obtain an interference spectrum.

[0151] The method of processing the first frequency sweeping interference information, the second frequency sweeping interference information and the frequency modulation interference signal to obtain an interference spectrum is a prior art in the art.

[0152] Optionally, after obtaining the interference spectrum, the interference spectrum may be filtered to obtain a filtered interference spectrum.

[0153] The steps to filter the interference spectrum include:

[0154] The filter model expression is:

[0155]

[0156] Where W() represents the original interference spectrum, represents the interference spectrum after filtering, m represents the output pixel point, n represents the input pixel point, s represents the set rectangular frame area, which is used to traverse the sub-image, W is the two-dimensional template, and are two Gaussian functions, representing the spatial domain kernel and the pixel domain kernel respectively, M mThe specific expression is:

[0157]

[0158] and The specific expression is:

[0159]

[0160]

[0161] In the formula, a and b represent the horizontal and vertical coordinates of the input pixel, i and j represent the coordinates of the center of the box, and σ s , σ r Represents the standard deviation of the Gaussian function.

[0162] The VGG19 network is used to process the interference spectrum after filtering to obtain the sub-image feature information;

[0163] The sub-graph feature information is clustered using an agglomerative hierarchical clustering method to obtain a clustered interference spectrum; the clustered interference spectrum is then processed in sequence using steps S3, S4, and S5.

[0164] Optionally, the multi-user interference avoidance network model includes an input layer, a first processing layer, a second processing layer, a third processing layer, a fourth processing layer, a fifth processing layer and an output layer;

[0165] The input dimension of the input layer is 1×15×10;

[0166] The output of the input layer is the input of the first processing layer, and the output dimension of the first processing layer is 32×8×5;

[0167] The output of the first processing layer is the input of the second processing layer, and the output dimension of the second processing layer is 64×5×3;

[0168] The output of the second processing layer is the input of the third processing layer, and the output dimension of the third processing layer is 64×4×3;

[0169] The output of the third processing layer is the input of the fourth processing layer, and the output dimension of the fourth processing layer is 1×448;

[0170] The output of the fourth processing layer is the input of the fifth processing layer, and the output dimension of the fifth processing layer is 1×512;

[0171] The output of the fifth processing layer is the input of the output layer, and the output dimension of the output layer is 1×10.

[0172] Optionally, the multi-user interference avoidance network model is a deep learning knowledge distillation model, which is:

[0173] Set up the teacher model and the student model;

[0174] The teacher model is a YOLOv5l model; the student model is a YOLOv5s model;

[0175] Setting the temperature coefficient T, we get:

[0176]

[0177] In the formula, p i is the output probability of the i-th category, T is the temperature coefficient, z i is the output value of the i-th category, j is the number of classification categories;

[0178] Use Softmax function to calculate distillation loss Loss soft :

[0179] Loss soft =T 2 *KLdiv(P S ,P T )

[0180] Where P T is the probability distribution of the teacher model, P S is the probability distribution of the student model, KLdiv() is the KL divergence;

[0181] The student model loss Loss is calculated by the difference between the student model and the true label hard for:

[0182] Loss hard =H(P S ,y gt )

[0183] In the formula, y gt represents the true label, H(P S ,y gt ) indicates P S ,y gt The cross entropy of

[0184] The objective function of knowledge distillation is set as:

[0185] Loss = αLoss soft +(1-α)Loss hard

[0186] In the formula, Loss is the total loss, Loss soft is the distillation loss, Loss hardis the student model loss and α is the weight.

[0187] Optionally, the interference spectrum may be enhanced to obtain an enhanced interference spectrum, and the enhanced interference spectrum may be processed in sequence using steps S3, S4, and S5.

[0188] The steps of enhancing the interference spectrum include:

[0189] The interference spectrum is converted to the time-frequency domain using STFT (Short Time Fourier Transform), using a window length of 256 samples, an overlap rate of 75%, and a Hamming window. The length of the spectrum vector can be reduced to 129 by discarding frequency samples corresponding to negative frequencies. A fully convolutional network consisting of 16 convolutional layers is defined. The convolution kernel size of the first layer is 9×8, with 18 convolution kernels. The 2nd to 13th convolutional layers are groups of 3 layers repeated 4 times, with convolution kernel widths of 5, 9, and 9, respectively, and the number of convolution kernels is 64. The convolution kernel width of the last convolution layer is 129 and there is only 1 convolution kernel. In this network, convolution is performed only in the frequency dimension, and for all layers except the first layer, the convolution kernel width along the time dimension is set to 1. The convolution layer is followed by a batch normalization layer and a ReLU activation function layer, and finally output through a regression layer.

[0190] A convolutional neural network is established to complete the mapping F between the received interference spectrum STFT spectrum X(m,k) and the clean interference spectrum STFT spectrum S(m,k) to achieve the purpose of denoising. The input prediction variable signal and the network target signal are the amplitude spectrum of the received noisy interference spectrum and the clean interference spectrum respectively. The goal of the network is to learn the mapping F so that the network outputs the amplitude spectrum of the noisy interference spectrum. The regression network uses the prediction variable input to minimize the mean square error between its output and the input target. Therefore, the loss function of the denoising network can be expressed as

[0191]

[0192] Among them, B represents the batch size, M and K represent the total number of STFT time frames and frequency points, respectively. Finally, the denoised interference spectrum is converted back to the original domain using the amplitude spectrum output by the network and the phase of the received noisy interference spectrum.

[0193] Optionally, the design of the action selection space, state space, and reward function includes:

[0194] S31, design the action selection space and obtain the action space matrix;

[0195] S32, design the state space and obtain the state space time-frequency diagram;

[0196] S33, designing a reward function and obtaining a reward function matrix;

[0197] The reward function matrix is:

[0198]

[0199] Where R1(s) is the reward function value.

[0200] Optionally, the training of the multi-user interference avoidance network model to obtain an optimized multi-user interference avoidance network model includes:

[0201] S41, random initial interference position, obtain the current interference state spectrum;

[0202] S42, using a multi-user interference avoidance network model, processing the current interference state spectrum to obtain a behavior matrix;

[0203] S43, stepping to the next spectrum allocation time period and obtaining the next interference state spectrum, executing the behavior matrix, and obtaining a reward value;

[0204] S44, the current interference state spectrum, behavior matrix, next interference state spectrum and reward value constitute a transformation process;

[0205] S45, storing the transformation process in a replay buffer;

[0206] S46, taking out the batch size conversion process from the replay buffer;

[0207] S47, using the batch size transformation process, training the multi-user interference avoidance network model to obtain an optimized multi-user interference avoidance network model.

[0208] Optionally, the utilizing the batch size transformation process to train the multi-user interference avoidance network model to obtain an optimized multi-user interference avoidance network model includes:

[0209] S471, processing the batch size transformation process to obtain an approximate target value;

[0210] The approximate target value is:

[0211]

[0212] Among them, y t is the approximate target value at time t, r t is the reward value at time t, γ is the discount factor, max_n is an operator that is preset, and max_n means finding the largest n values, where n is the number of users. is the behavior matrix a based on time t+1 t+1Find the largest n values, s t+1 is the interference state spectrum at time t+1, Q(s t+1 ,a t+1 θ - ) is the value function, θ - is the multi-user interference avoidance network model parameter at time t+1;

[0213] S472, taking the sum of the largest n approximate target values ​​as the target value of the multi-user interference avoidance network model;

[0214] S473, constructing a predicted value of the multi-user interference avoidance network model;

[0215] The predicted value b t for:

[0216]

[0217] in, is the behavior matrix a based on time t t Find the largest n values, Q(s t ,a t ; θ) is the value function, s t is the interference state spectrum at time t, a t+1 is the behavior matrix at time t, θ is the multi-user interference avoidance network model parameter at time t;

[0218] S474, performing gradient descent using the loss function, processing the target value and the predicted value, and obtaining a training multi-user interference avoidance network model.

[0219] It can be seen that the frequency sweep interference simulated by the present invention has no obvious rules. One or more transition probability matrices are used as the frequency sweep rules. This rule is more complex and can better reflect the current complex interference environment. In view of the current multi-user interference avoidance field, the present invention proposes a distribution prediction reinforcement learning method and applies it to the multi-user interference avoidance field. Compared with the multi-user interference avoidance method based on multi-agent reinforcement learning, this method consumes less computing resources and can effectively improve the timeliness of multi-user interference avoidance. At the same time, in a complex interference environment, it has a lower spectrum conflict rate.

[0220] Embodiment 2

[0221] See also Figure 2 , Figure 2 FIG. 1 is a flow chart of another multi-user interference avoidance method disclosed in an embodiment of the present invention. Figure 2 The multi-user interference avoidance method described is applied to fields such as user interference avoidance applications, and the embodiments of the present invention do not limit this. Figure 2As shown, the multi-user interference avoidance method may include the following operations:

[0222] Step S1: construct an interference background generation model to generate an interference spectrum, such as Figure 3 As shown;

[0223] Step S1 includes the following sub-steps:

[0224] Step S11: According to the formula Generate FM interference signal, where ω j is the central angular frequency, u(t) is the modulation noise, which is a zero-mean wide-sense stationary random process that conforms to the normal distribution. is the initial angle, which is a uniformly distributed random variable on [0,2π] and independent of U j is the FM interference signal amplitude, K FM is the frequency modulation slope.

[0225] Step S12, simulate and construct two types of sweep frequency interference: RandomMove1_SP_Mut and RandomMove1_DiffProb interference, wherein RandomMove1-SP-Mut interference transfers the interference frequency band upward, unchanged and downward with a certain probability, the probability of upward transfer is 0.4, the probability of unchanged is 0.3, and the probability of downward transfer is 0.3, so the transfer matrix is ​​[0.4, 0.3, 0.3]; RandomMove1-DiffProb interference has 10 transfer probability matrices, and these 10 transfer probability matrices are used cyclically for interference state transfer. These 10 transfer matrices are:

[0226]

[0227] Step S13: Generate the time-frequency diagram of RandomMove1_SP_Mut and RandomMove1_DiffProb interference according to the frequency modulation interference signal generated in step S11 and the frequency sweep transfer rule in step S12. Figure 3 As shown in the figure, the size of the time-frequency diagram generated in this paper is 10*15, where dimension 10 is the frequency domain, which means that the available frequency band is divided into 10 sub-bands, and dimension 15 is the time domain, using the frequency domain of 15 historical time periods for regular prediction.

[0228] Step S2: construct the network structure, design the action selection space, state space, reward function and initialize the hyperparameters;

[0229] Step S2 includes the following sub-steps:

[0230] Step S21, design the action space as a 1*10 matrix with only 0 and 1, where the dimension 10 corresponds to the frequency domain dimension in S13, and if the value is 1, it means that the corresponding sub-band is used. The state space is a 10*15 interference time-frequency diagram generated by S13.

[0231] Step S22: Since the experiment of the present invention allocates frequency bands to three users, the reward function is defined as follows:

[0232]

[0233] Step S23, initialize the parameters, set the total number of training rounds to 100000, the number of observation rounds to 100, the number of exploration rounds to 20000, the maximum number of steps per round to 2000, the experience pool size to 20000, the batch size to 32, the online network learning rate to 0.0001, the discount factor to 0.99, the initial greed factor to 1, and the final greed factor to 0.01;

[0234] Step S24: construct a network structure, such as Figure 4 As shown, the input dimension is 1×15×10 and the output dimension is 1×10.

[0235] Step S3: Update the experience pool and network parameters through environmental interaction, and visualize the results;

[0236] Step S3 includes the following sub-steps:

[0237] Step S31: Randomly initialize the interference position to obtain the current interference state spectrum x t , after inputting the network, we get the behavior matrix a t , step to the next spectrum allocation time period and obtain the new interference state spectrum s t+1 , execute behavior a t And get the reward value r t This is a transformation process. t ,a t ,s t+1 ,r t , and store this transformation process in the replay buffer.

[0238] Step S32: Train the network using the idea of ​​distributed prediction reinforcement learning. Take out the batchsize from the replay buffer. If you want to allocate bandwidth for n users, define the operator max_n, which means to find the largest n values. Therefore, the approximate target value can be defined as θ -is the multi-user interference avoidance network model parameter at time t+1, where γ is the discount factor, which is defined in step S23. The sum of the largest n target values ​​is used as the target value of the distribution prediction reinforcement learning. The prediction value of the distribution prediction reinforcement learning is Since training is usually based on batch size, the calculation of the loss function is also based on batch size. The batch size is defined in S23 and is represented by m here, so the loss is The training of the multi-user interference avoidance network model can be completed by performing gradient descent with this loss, where θ is the multi-user interference avoidance network model parameter at time t.

[0239] Step S33: Use pygame to build a visualization platform and display the results. The display effect is as follows: Figure 5 The red part in the figure is the interference, and the green part is the frequency band used by the user.

[0240] Step S4: data statistics and analysis;

[0241] Figure 6 is the spectrum conflict rate under the two types of swept frequency interference defined in step S12 of the method.

[0242] Embodiment 3

[0243] See also Figure 7 , Figure 7 is a structural diagram of a multi-user interference avoidance device disclosed in an embodiment of the present invention. Figure 7 The multi-user interference avoidance device described is applied to fields such as user interference avoidance applications, and the embodiments of the present invention do not limit this. Figure 7 As shown, the multi-user interference avoidance device may include the following operations:

[0244] S301, a signal generation module, used to process the modulation noise using the interference background generation model to obtain a frequency modulation interference signal;

[0245] S302, an interference spectrum generating module, configured to process the frequency modulation interference signal to obtain an interference spectrum;

[0246] S303, a network model building module, used to build a multi-user interference avoidance network model, design an action selection space, a state space, a reward function, and initialize hyperparameters;

[0247] S304, a training module, used to train the multi-user interference avoidance network model to obtain a trained multi-user interference avoidance network model;

[0248] S305, an interference avoidance module, configured to process the interference spectrum using the trained multi-user interference avoidance network model to obtain a multi-user interference avoidance result;

[0249] S306, a visualization module is used to construct a visualization platform and use the visualization platform to display the multi-user interference avoidance result.

[0250] Embodiment 4

[0251] See also Figure 8 , Figure 8 is a structural diagram of another multi-user interference avoidance device disclosed in an embodiment of the present invention. Figure 8 The multi-user interference avoidance device described is applied to fields such as user interference avoidance applications, and the embodiments of the present invention do not limit this. Figure 8 As shown, the multi-user interference avoidance device may include the following operations:

[0252] A memory 401 storing executable program codes;

[0253] a processor 402 coupled to the memory 401;

[0254] The processor 402 calls the executable program code stored in the memory 401 to execute the steps in the multi-user interference avoidance method described in the first or second embodiment.

[0255] Embodiment 5

[0256] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program for electronic data exchange, wherein the computer program enables a computer to execute the steps of the multi-user interference avoidance method described in the first or second embodiment.

[0257] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0258] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0259] Finally, it should be noted that the multi-user interference avoidance method and device disclosed in the embodiments of the present invention disclose only the preferred embodiments of the present invention, which are only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-user interference avoidance method, characterized in that: The method comprises: S1, using the interference background generation model, the modulated noise is processed to obtain the FM interference signal; S2, processing the frequency modulation interference signal to obtain an interference spectrum, including: S21, constructing first frequency sweep interference information and second frequency sweep interference information; The first frequency sweep interference information is represented by a transfer matrix as [0.4, 0.3, 0.3], wherein the first column of the matrix is ​​the probability of upward transfer, the second column is the probability of unchanged, and the third column is the probability of downward transfer; The second frequency sweep interference information is expressed by a transfer matrix as follows: The first column of the transfer matrix is ​​the probability of upward transfer, the second column is the probability of unchanged, and the third column is the probability of downward transfer; S22, processing the first frequency sweeping interference information, the second frequency sweeping interference information and the frequency modulation interference signal to obtain an interference spectrum; S221, filtering the interference spectrum to obtain a filtered interference spectrum, including: The filter model expression is: Where W() represents the original interference spectrum, represents the interference spectrum after filtering, m represents the output pixel point, n represents the input pixel point, s represents the set rectangular frame area, which is used to traverse the sub-image, W is the two-dimensional template, and are two Gaussian functions, representing the spatial domain kernel and the pixel domain kernel respectively, M m The specific expression is: and The specific expression is: In the formula, a and b represent the horizontal and vertical coordinates of the input pixel, i and j represent the coordinates of the center of the box, and σ s , σ r represents the standard deviation of the Gaussian function; S222, using the VGG19 network to process the interference spectrum after filtering to obtain sub-image feature information; S223, clustering the sub-graph feature information using an agglomerative hierarchical clustering method to obtain a clustered interference spectrum; S3, build a multi-user interference avoidance network model, design the action selection space, state space, reward function, and initialize hyperparameters; S4, training the multi-user interference avoidance network model to obtain an optimized multi-user interference avoidance network model; S5, using the optimized multi-user interference avoidance network model to process the clustered interference spectrum to obtain a multi-user interference avoidance result; S6, constructing a visualization platform, and using the visualization platform to display the multi-user interference avoidance result.

2. The multi-user interference avoidance method according to claim 1, characterized in that: The interference background generation model is: Where J(t) is the frequency modulation interference signal, ω j is the central angular frequency, u(t) is the modulation noise, which is a zero-mean wide-sense stationary random process that conforms to the normal distribution. is the initial angle, which is a uniformly distributed and independent random variable on [0,2π], U j FM interference signal amplitude, K FM is the frequency modulation slope, t and t′ are time variables.

3. The multi-user interference avoidance method according to claim 1, characterized in that: The multi-user interference avoidance network model includes an input layer, a first processing layer, a second processing layer, a third processing layer, a fourth processing layer, a fifth processing layer and an output layer; The input dimension of the input layer is 1×15×10; The output of the input layer is the input of the first processing layer, and the output dimension of the first processing layer is 32×8×5; The output of the first processing layer is the input of the second processing layer, and the output dimension of the second processing layer is 64×5×3; The output of the second processing layer is the input of the third processing layer, and the output dimension of the third processing layer is 64×4×3; The output of the third processing layer is the input of the fourth processing layer, and the output dimension of the fourth processing layer is 1×448; The output of the fourth processing layer is the input of the fifth processing layer, and the output dimension of the fifth processing layer is 1×512; The output of the fifth processing layer is the input of the output layer, and the output dimension of the output layer is 1×10.

4. The multi-user interference avoidance method according to claim 1, characterized in that: The design of action selection space, state space, and reward function includes: S31, design the action selection space and obtain the action space matrix; S32, design the state space and obtain the state space time-frequency diagram; S33, designing a reward function and obtaining a reward function matrix; The reward function matrix is: Where R1(s) is the reward function value.

5. The multi-user interference avoidance method according to claim 1, characterized in that: The step of training the multi-user interference avoidance network model to obtain an optimized multi-user interference avoidance network model includes: S41, random initial interference position, obtain the current interference state spectrum; S42, using a multi-user interference avoidance network model, processing the current interference state spectrum to obtain a behavior matrix; S43, stepping to the next spectrum allocation time period and obtaining the next interference state spectrum, executing the behavior matrix, and obtaining a reward value; S44, the current interference state spectrum, behavior matrix, next interference state spectrum and reward value constitute a transformation process; S45, storing the transformation process in a replay buffer; S46, taking out the batch size conversion process from the replay buffer; S47, using the batch size transformation process, training the multi-user interference avoidance network model to obtain an optimized multi-user interference avoidance network model.

6. The multi-user interference avoidance method according to claim 5, characterized in that: The method of using the batch size conversion process to train the multi-user interference avoidance network model to obtain an optimized multi-user interference avoidance network model includes: S471, processing the batch size transformation process to obtain an approximate target value; The approximate target value is: Among them, y t is the approximate target value at time t, r t is the reward value at time t, γ is the discount factor, max_n is the operator, max_n means finding the largest n values, n is the number of users, is the behavior matrix a based on time t+1 t+1 Find the largest n values, s t+1 is the interference state spectrum at time t+1, Q(s t+1 ,a t+1 θ - ) is the value function, θ - is the multi-user interference avoidance network model parameter at time t+1; S472, taking the sum of the largest n approximate target values ​​as the target value of the multi-user interference avoidance network model; S473, constructing a predicted value of the multi-user interference avoidance network model; The predicted value b t for: in, is the behavior matrix a based on time t t Find the largest n values, Q(s t ,a t ; θ) is the value function, s t is the interference state spectrum at time t, a t+1 is the behavior matrix at time t, θ is the multi-user interference avoidance network model parameter at time t; S474, performing gradient descent using the loss function, processing the target value and the predicted value, and obtaining a training multi-user interference avoidance network model.

7. A multi-user interference avoidance device, characterized in that: The device comprises: A signal generation module is used to process the modulation noise using the interference background generation model to obtain a frequency modulation interference signal; The interference spectrum generation module is used to process the frequency modulation interference signal to obtain an interference spectrum, including: S21, constructing first frequency sweep interference information and second frequency sweep interference information; The first frequency sweep interference information is represented by a transfer matrix as [0.4, 0.3, 0.3], wherein the first column of the matrix is ​​the probability of upward transfer, the second column is the probability of unchanged, and the third column is the probability of downward transfer; The second frequency sweep interference information is expressed by a transfer matrix as follows: The first column of the transfer matrix is ​​the probability of upward transfer, the second column is the probability of unchanged, and the third column is the probability of downward transfer; S22, processing the first frequency sweeping interference information, the second frequency sweeping interference information and the frequency modulation interference signal to obtain an interference spectrum; S221, filtering the interference spectrum to obtain a filtered interference spectrum, including: The filter model expression is: Where W() represents the original interference spectrum, represents the interference spectrum after filtering, m represents the output pixel point, n represents the input pixel point, s represents the set rectangular frame area, which is used to traverse the sub-image, W is the two-dimensional template, and are two Gaussian functions, representing the spatial domain kernel and the pixel domain kernel respectively, M m The specific expression is: and The specific expression is: In the formula, a and b represent the horizontal and vertical coordinates of the input pixel, i and j represent the coordinates of the center of the box, and σ s , σ r represents the standard deviation of the Gaussian function; S222, using the VGG19 network to process the interference spectrum after filtering to obtain sub-image feature information; S223, clustering the sub-graph feature information using an agglomerative hierarchical clustering method to obtain a clustered interference spectrum; The network model building module is used to build a multi-user interference avoidance network model, design the action selection space, state space, reward function, and initialize hyperparameters; A training module, used for training the multi-user interference avoidance network model to obtain an optimized multi-user interference avoidance network model; An interference avoidance module, used to process the clustered interference spectrum using the optimized multi-user interference avoidance network model to obtain a multi-user interference avoidance result; The visualization module is used to construct a visualization platform and use the visualization platform to display the multi-user interference avoidance result.

8. A multi-user interference avoidance device, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the multi-user interference avoidance method according to any one of claims 1 to 6.

9. A computer storable medium, characterized in that: The computer storable medium stores computer instructions, and when the computer instructions are called, they are used to execute the multi-user interference avoidance method according to any one of claims 1 to 6.

Images