A path control method, device, system and storage medium of a jammer

Abstract

The application provides a path control method, device and system of a jammer and a storage medium, and belongs to the path control field.The method comprises the following steps: calculating a state space of jammer position, power information, received ACK quantity and spectrum information at all times;calculating an action space of decision information at all times;obtaining a state transition probability matrix according to a state transition probability matrix at all times;calculating a reward value set of the received ACK quantity at all times;and constructing an initial Q value table through the state space, the action space, the state transition probability matrix and the reward value set.The application can enable the jammer to make and execute different optimal decisions, thereby realizing efficient and dynamic interference, obtaining the most effective interference effect and the optimal utilization rate of interference resources, maximizing the interference benefit, and to a certain extent, solving the problem of "dimension disaster" and being suitable for the interference field.

Description

TECHNICAL FIELD

[0001] The present application mainly relates to the path control technical field, and in particular to a path control method, device and system of a jammer and a storage medium. BACKGROUND

[0002] Due to the rapid development of modern anti-jamming technology, the traditional single-domain jamming technology is difficult to play a good jamming effect in a complex environment, and the communication party and the jamming party are in a non-cooperative relationship, so that the jamming effect cannot be directly obtained by the jamming party. The jammer has the abilities of real-time situation awareness, learning and decision-making for the target communication system, and can adapt to different electromagnetic environments and cope with different jamming objects, but how to enable the jammer to make and execute different optimal decisions to achieve efficient and dynamic jamming has become an urgent problem to be solved. Reinforcement learning algorithm is the most suitable and skilled to solve the optimal decision problem, and the q algorithm has always been the most popular algorithm in reinforcement learning algorithm, but the q learning algorithm has a well-known "dimension disaster" problem, and the application is also limited to discrete problems. SUMMARY

[0003] The technical problem to be solved by the present application is to provide a path control method, device, system and storage medium of a jammer to solve the problems of the prior art.

[0004] The technical solution of the present application to solve the above technical problem is as follows: a path control method of a jammer, comprising the following steps:

[0005] obtaining the jammer position at multiple time points, the power information corresponding to each time point, the spectrum information, the decision information, the number of received ACKs and the state transition probability from the jammer;

[0006] calculating the state space of the jammer position, the power information, the number of received ACKs and the spectrum information at all time points;

[0007] calculating the action space of the decision information at all time points;

[0008] composing a matrix according to the state transition probability at all time points to obtain a state transition probability matrix;

[0009] calculating a reward value set of the number of received ACKs at all time points;

[0010] constructing an initial Q value table through the state space, the action space, the state transition probability matrix and the reward value set;

[0011] updating the initial Q value table according to the state information and the action information at each time point to obtain an updated Q value table;

[0012] Screen the maximum Q value from the updated Q value table, and take the decision information corresponding to the screened maximum Q value as the jammer flight path, and control the jammer to fly to the jammer flight path.

[0013] Another technical solution solving the above technical problems is as follows: a jammer path control device, comprising:

[0014] An information acquisition module is configured to obtain the jammer position at multiple time points, power information corresponding to each time point, spectrum information, decision information, the number of received ACKs, and state transition probability from the jammer;

[0015] A state space calculation module is configured to calculate the state space of the jammer position, power information, the number of received ACKs, and spectrum information at all time points;

[0016] An action space calculation module is configured to calculate the action space of the decision information at all time points;

[0017] A matrix composition module is configured to compose a matrix according to the state transition probability at all time points to obtain a state transition probability matrix;

[0018] A reward value set calculation module is configured to calculate the reward value set of the number of received ACKs at all time points;

[0019] A Q value table construction module is configured to construct an initial Q value table by using the state space, the action space, the state transition probability matrix, and the reward value set;

[0020] An update module is configured to update the initial Q value table according to the state information and action information at each time point to obtain an updated Q value table;

[0021] A control module is configured to screen the maximum Q value from the updated Q value table, take the decision information corresponding to the screened maximum Q value as the jammer flight path, and control the jammer to fly to the jammer flight path.

[0022] Based on the above-mentioned jammer path control method, the present application further provides a jammer path control system.

[0023] Another technical solution solving the above technical problems is as follows: a jammer path control system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, when the processor executes the computer program, the jammer path control method as described above is implemented.

[0024] Based on the above-mentioned jammer path control method, the present application further provides a computer readable storage medium.

[0025] Another technical solution of the present application to solve the above technical problems is as follows: a computer readable storage medium, the computer readable storage medium stores a computer program, when the computer program is executed by a processor, the path control method of the jammer is realized.

[0026] The present application has the advantages that: by calculating the state space of the jammer position, power information, received ACK quantity and spectrum information at all times, calculating the action space of the decision information at all times, obtaining the state transition probability matrix according to the state transition probability matrix composed of all times, calculating the reward value set of the received ACK quantity at all times, constructing the initial Q value table through the state space, the action space, the state transition probability matrix and the reward value set, updating the initial Q value table according to the state information and the action information at each time to obtain the updated Q value table, screening the maximum Q value from the updated Q value table, and taking the decision information corresponding to the screened maximum Q value as the jammer driving path, and controlling the jammer to fly to the jammer driving path, the jammer can make and execute different optimal decisions, so that efficient and dynamic interference is realized, the most effective interference effect and the optimal utilization rate of interference resources are obtained, the interference benefit is maximized, and the problem of curse of dimensionality is solved to some extent, and the present application is suitable for the interference field. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 A flowchart of a path control method of a jammer is provided for the embodiments of the present application.

[0028] Figure 2 A graph of the number of received ACKs after learning is provided for the embodiments of the present application.

[0029] Figure 3 A module block diagram of a path control device of a jammer is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0030] The principles and characteristics of the present application are described below in combination with the drawings, and the examples are only used to explain the present application, and are not used to limit the scope of the present application.

[0031] Figure 1 A flowchart of a path control method of a jammer is provided for the embodiments of the present application.

[0032] As shown in Figure 1 A path control method of a jammer includes the following steps:

[0033] Obtain the jammer position, the power information corresponding to each time, the spectrum information, the decision information, the received ACK quantity and the state transition probability from the jammer at multiple times;

[0034] calculating a state space of jammer position, power information, received ACK number and spectrum information at all times;

[0035] calculating an action space of decision information at all times;

[0036] obtaining a state transition probability matrix according to all-time state transition probability groups;

[0037] calculating a reward value set of received ACK number at all times;

[0038] constructing an initial Q value table by using the state space, the action space, the state transition probability matrix and the reward value set;

[0039] updating the initial Q value table according to state information and action information at each time to obtain an updated Q value table;

[0040] selecting a maximum Q value from the updated Q value table, and taking decision information corresponding to the selected maximum Q value as a jammer flight path, and controlling the jammer to fly to the jammer flight path.

[0041] It should be understood that the decision selection is made by using reinforcement learning, and a reinforcement learning system is generally defined by a four-tuple as {S, A, P, R}, S represents the state space, A represents the action space, P represents the state transition probability, and R represents the reward value (i.e., the reward value set).

[0042] Specifically, the state transition probability matrix P is defined as P = {P(s k+1 )|s k , a k +, s k+1 , s k ∈ S represents a probability of reaching state s k (i.e., the state information) by selecting action a k (i.e., the action information) from state s k+1 (i.e., the state information).

[0043] Specifically, the unmanned aerial vehicle learns to improve its own strategy by obtaining a reward value through interaction with the environment, i.e., selecting a flight position at the next step and whether to perform jamming, to obtain an optimal strategy update value (i.e., the selected maximum Q value).

[0044] In the above embodiment, the state space of the jammer position, power information, received ACK quantity and spectrum information at all times is calculated, the action space of the decision information at all times is calculated, the state transition probability matrix is obtained according to the state transition probability matrix composed of all times, the reward value set of the received ACK quantity at all times is calculated, the initial Q value table is constructed by the state space, the action space, the state transition probability matrix and the reward value set, the initial Q value table is updated according to the state information and the action information at each time to obtain the updated Q value table, the maximum Q value is screened from the updated Q value table, and the decision information corresponding to the screened maximum Q value is taken as the jammer flight path, and the jammer is controlled to fly to the jammer flight path, so that the jammer can make and execute different optimal decisions, thereby realizing efficient and dynamic jamming, obtaining the most effective jamming effect and the optimal utilization rate of jamming resources, and maximizing the jamming benefit, and to some extent, solving the problem of "dimension disaster" and being suitable for the field of jamming.

[0045] Optionally, as an embodiment of the present application, the power information includes the jammer flight power, the jammer jamming power and the jammer jamming power at the last time, and the spectrum information includes the spectrum of the environment and the user,

[0046] The process of calculating the state space of the jammer position, power information, received ACK quantity and spectrum information at all times includes:

[0047] The jammer residual energy value corresponding to the jammer flight power, jammer jamming power and received ACK quantity at each time is calculated to obtain the jammer residual energy value at each time.

[0048] The state information of the jammer position, spectrum of the environment and the user, jammer jamming power at the last time and jammer residual energy value at each time is calculated by the first formula to obtain the state information at each time, and the first formula is:

[0049] s k = (F(k), ω(k), P(k-1), E(k)),

[0050] wherein,

[0051] wherein, s k is the state information at the kth time, F(k) is the spectrum at the kth time, ω(k) is the jammer position at the kth time, P(k-1) is the jammer jamming power at the (k-1)th time, E(k) is the jammer residual energy value at the kth time, is the spectrum of the environment and the user at the kth time, is the preset frequency point at the kth time.

[0052] The state space of the state information at all time points is calculated by a second formula to obtain a state space, the second formula being:

[0053] s=[s1,s2…s k ],

[0054] wherein s is the state space, s k is the state information at the kth time point.

[0055] It should be understood that the state space S is defined. k ∈s:s=[s1,s2…s k ]。

[0056] Specifically, s k =(F(k),ω(k),P(k-1),E(k)),wherein, is the spectrum information of the perceived environment and the user, the unmanned aerial vehicle (i.e., the jammer) transmits a jamming signal at a specific frequency point , ω(k) represents the current position of the unmanned aerial vehicle (i.e., the position of the jammer at the kth time point), P(k-1) represents the jamming power of the unmanned aerial vehicle at the previous time point, and E(k) represents the energy value of the unmanned aerial vehicle at the current time point (i.e., the remaining energy value of the jammer at the kth time point).

[0057] In the above embodiment, the state space of the jammer position, power information, received ACK quantity, and spectrum information at all time points is calculated to obtain the state space, which lays a foundation for subsequent data processing and enables intelligent jamming of the communication link pairs on the ground, thereby solving the problem of optimal jamming waveform decision.

[0058] Alternatively, as an embodiment of the present application, the process of calculating the jammer remaining energy value corresponding to the jammer flight power, jammer jamming power, and received ACK quantity at each time point to obtain the jammer remaining energy value at each time point comprises:

[0059] The jammer remaining energy value corresponding to the jammer flight power, jammer jamming power, and received ACK quantity at each time point is calculated by a third formula to obtain the jammer remaining energy value at each time point, the third formula being:

[0060]

[0061] wherein p n =p f +p j ,

[0062] wherein E(k) is the jammer remaining energy value at the kth time point, ACK(k) is the received ACK quantity at the kth time point, T is a preset unit time, and pn p is the total power of the jammer f p is the flight power of the jammer corresponding to the current time j p is the jamming power of the jammer corresponding to the current time

[0063] It should be understood that the unmanned aerial vehicle (i.e. the jammer) must return to the starting point before the energy is exhausted, and the energy constraint can be expressed as:

[0064] Specifically, the goal of learning is to maximize the number of successful interference within a certain time, i.e. the least ACK received by the communication receiving end, which is expressed as:

[0065]

[0066] p is the flight power of the jammer corresponding to the current time n p is the flight power of the jammer corresponding to the current time f p is the jamming power of the jammer corresponding to the current time j p is the flight power of the jammer corresponding to the current time f p is the flight power of the unmanned aerial vehicle (i.e. the flight power of the jammer corresponding to the current time) j p is the jamming power of the unmanned aerial vehicle (i.e. the jamming power of the jammer corresponding to the current time)

[0067] In the above embodiment, the residual energy value of the jammer at each time is obtained by calculating the flight power of the jammer, the jamming power of the jammer and the number of received ACKs at each time through the third formula, which can maximize the number of successful interference of the unmanned aerial vehicle under a certain energy constraint, thereby realizing efficient and dynamic interference.

[0068] Optionally, as an embodiment of the present application, the decision information includes a flight decision of the jammer and a jamming decision of the jammer,

[0069] The action space of the decision information at all times is calculated, and the process of obtaining the action space includes:

[0070] The action information of the flight decision of the jammer and the jamming decision of the jammer at each time is calculated through the fourth formula, and the action information at each time is obtained, and the fourth formula is:

[0071] a k = (θ k , T k '),

[0072] wherein a k is the action information at the kth time, θ k is the flight decision of the jammer at the kth time, and T k ' is the jamming decision of the jammer at the kth time;

[0073] The action space of the action information at all time points is calculated by a fifth formula to obtain the action space, the fifth formula being:

[0074] A = [a1, a2, … a k ],

[0075] wherein A is the action space, a k is the action information at the kth time point.

[0076] It should be understood that the action space A is defined. k ∈ A: A = [a1, a2, … a k ].

[0077] Specifically, a k = (θ, T), θ = (θ0, θ1, θ2) represents the flight decision of the UAV (i.e., the jammer flight decision), θ0 represents that the UAV (i.e., the jammer) continues to fly forward at the current time point, θ1 and θ2 respectively represent that the UAV (i.e., the jammer) selects left turn and right turn at the current time point, and T = (T0, T1) represents the jamming decision of the UAV (i.e., the jammer decision), wherein T0 represents that the UAV (i.e., the jammer) selects not to emit jamming, and T1 represents that the UAV (i.e., the jammer) selects to emit jamming.

[0078] In the above embodiment, the action space of the decision information at all time points is calculated to obtain the action space, which lays a foundation for subsequent data processing and enables the jammer to formulate and execute different optimal decisions, thereby realizing efficient and dynamic jamming.

[0079] Optionally, as an embodiment of the present application, the process of calculating the reward value set of the number of received ACKs at all time points to obtain the reward value set comprises:

[0080] The reward value of the number of received ACKs at each time point is calculated by a sixth formula to obtain the initial reward value at each time point, and the initial reward values at all time points are collected to obtain the reward value set, the sixth formula being:

[0081]

[0082] wherein R k is the initial reward value at the kth time point, ACK(k) is the number of received ACKs at the kth time point, and k' is a constant.

[0083] It should be understood that the reward value R (i.e., the reward value set) is defined. R represents the number of received ACKs per unit time, and the size of the reward value is inversely proportional to the number of received ACKs.

[0084] In the above embodiment, the reward value of the received ACK quantity at each moment is calculated by the sixth formula to obtain a reward value set, which lays a foundation for subsequent data processing and enables the jammer to make and execute different optimal decisions, thereby realizing efficient and dynamic jamming.

[0085] Optionally, as one embodiment of the present application, the initial Q value table includes initial Q values at each moment, occurrence frequencies of state-action pairs corresponding to the initial Q values, and target reward values,

[0086] The process of updating the initial Q value table according to the state information and the action information at each moment to obtain an updated Q value table includes:

[0087] The maximum Q value is selected from all initial Q values at all moments, the action information corresponding to the selected Q value is taken as a target action, and the state information corresponding to the selected Q value is taken as a state to be processed;

[0088] The next state information of the state to be processed is taken as a target state, and the initial Q value with the action information being the target action and the state information being the state to be processed is selected from the initial Q value table to obtain a Q value to be processed after screening;

[0089] The updated Q value of the Q value to be processed and each initial Q value at each moment, the occurrence frequency of the state-action pair corresponding to each initial Q value, and the target reward value are calculated by the seventh formula to obtain the updated Q value at each moment, and the seventh formula is:

[0090] Q′ k (s,a)=[1-λ(s,a)]Q k (s,a)+λ[R′ k (s,a)+γS k (s′)],

[0091] wherein, S k (s′)=Q k (s′,a * ),

[0092] wherein, Q k ′(s,a) is the updated Q value of the kth moment of the s th state information and the a th action information, λ(s,a) is the learning rate of the s th state information and the a th action information, Q k (s,a) is the initial Q value of the kth moment of the s th state information and the a th action information, R′ k (s,a) is the target reward value of the kth moment of the s th state information and the a th action information, γ is a discount factor, and S k(s') is the state information of the s'th state information at the k'th moment, z(s, a) is the state-action pair occurrence number of the a'th action information of the s'th state information, Q k (s', a * ) is the to-be-processed Q value;

[0093] The updated Q value table is obtained by tabulating the updated Q values of all moments.

[0094] Specifically, the user updates the Q value (i.e., the updated Q value) according to the rule, and the update formula of the Q value table (i.e., the updated Q value table) is

[0095] Q k (s, a) = (1 - λ)Q k (s, a) + λ[R k + γS k (s')

[0096] λ represents a learning rate, which is used to adjust the influence of the new state and the instantaneous reward value on the Q value; γ represents a discount factor, i.e., the influence degree of the future reward on the currently selected action. z(s, a) represents the state-action pair occurrence number (i.e., the state-action pair occurrence number), R k represents a reward value, S k (s') = Q k (s', a * ) s' represents the next state after a is executed, a * represents an action corresponding to the maximum Q value, The user randomly selects an action with a probability of ε, and selects an action corresponding to the maximum Q value with a probability of 1-ε, and each user measures the reward value R k (s, a) (i.e., the target reward value), and then updates the Q value.

[0097] In the above embodiment, the initial Q value table is updated according to the state information and the action information of each moment to obtain the updated Q value table, which can enable the jammer to make and execute different optimal decisions, thereby realizing efficient and dynamic jamming, obtaining the most effective jamming effect and the optimal utilization rate of jamming resources, and maximizing the jamming benefit, and to a certain extent, solving the problem of "dimension disaster", and being applicable to the field of jamming.

[0098] Optionally, as another embodiment of the present application, the purpose of the present application is that the jammer can make the optimal jamming strategy and the optimal jamming trajectory by learning and training in a dynamically changing environment, thereby maximizing the jamming benefit, i.e., obtaining the most effective jamming effect and the optimal utilization rate of jamming resources. Under the energy constraint of a certain value, the number of successful jamming of the unmanned aerial vehicle is maximized.

[0099] Optionally, as another embodiment of the present application, the present application includes a communication link pair, jammer and the like device. The ground has multiple pairs of communication link pairs in communication with each other, and the air has a jamming unmanned aerial vehicle to interfere with the ground communication link. Assuming that the unmanned aerial vehicle knows the position of the ground communication node completely, the unmanned aerial vehicle makes a decision through reinforcement learning to determine the next flight position of the unmanned aerial vehicle and whether to interfere, and the total energy of the unmanned aerial vehicle is known and is a certain value. The goal is to optimize the trajectory of the unmanned aerial vehicle under the condition of a certain energy value, that is, to maximize the number of successful interference times per unit time, and finally the unmanned aerial vehicle can return to the starting position smoothly.

[0100] Optionally, as another embodiment of the present application, the present application considers a multi-user scenario in which the ground has multiple pairs of communication link pairs, and the communication users include transmitters and receivers. The positions and transmission powers of the transmitters are known, respectively, s i , p i where i∈{1,2…n}, the air has an intelligent jamming unmanned aerial vehicle, and the flight period of the unmanned aerial vehicle is T, The distance from the unmanned aerial vehicle to the ground node is:

[0101]

[0102] where ω0 represents the horizontal position of the unmanned aerial vehicle, H i represents the height of the unmanned aerial vehicle

[0103] The channel power gain between the unmanned aerial vehicle and the user is:

[0104]

[0105] where β0 represents the power gain when the reference distance is 1 m

[0106] The link loss gain from the transmitter to the receiver is P T→R The basis for judging whether the interference is successful is: When SnR < 10 dB, the communication is interrupted.

[0107] Optionally, as another embodiment of the present application, the present application uses interference benefit evaluation technology, perception state clustering technology, and reinforcement learning to intelligently interfere with the ground communication link pairs. Through interaction with the environment, the total energy of the unmanned aerial vehicle is certain and is a certain value. The jamming unmanned aerial vehicle selects the optimal path and the best interference decision to interfere, that is, the number of successful interference times per unit time is maximized, so that the interference benefit is maximized.

[0108] Optionally, as another embodiment of the present application, as Figure 2As shown, in the present application, it is assumed that there are three pairs of existing communication links on the ground, and the signal transmission power is 100 mw. There is an interference unmanned aerial vehicle at an altitude of 100 m, with coordinates (-1000, 200), an interference power of 20 mw, a speed of 40, delta_t = 1 s, a time slot N = 120, and a noise size of 10 -6 .

[0109] After the unmanned aerial vehicle flies into a specific area, the learning module starts learning and makes optimal interference decisions according to the selected optimal trajectory in the present application, Figure 2 The number of ACKs received by the user after learning is given. The simulation results show that in the early stage of the algorithm, the jammer has not yet learned, and the communication between the users is not interfered, so the number of ACKs received is not affected by the interference. After the learning module starts learning, the unmanned aerial vehicle starts to enter the specific area, constantly updates the learning, selects the unknown node with the strongest perceived signal to send an interference signal, and successfully interferes with the communication between the users after the unmanned aerial vehicle sends the interference signal. The number of ACKs received by the user also gradually tends to 0. The simulation results show that the proposed algorithm can explore the best interference strategy through continuous interaction with the environment.

[0110] Alternatively, as another embodiment of the present application, the present application relates to technical solutions including interference benefit evaluation technology, perception state clustering technology, and reinforcement learning-based optimization of unmanned aerial vehicle trajectory and interference decision learning. By using reinforcement learning, the unmanned aerial vehicle decides the optimal interference strategy and optimal interference trajectory after interacting with the environment, thereby maximizing the interference benefit, i.e., obtaining the most effective interference effect and the optimal utilization rate of interference resources.

[0111] Figure 3 A module block diagram of a path control device of a jammer is provided for the embodiments of the present application.

[0112] Alternatively, as another embodiment of the present application, as Figure 3 shown, a path control device of a jammer includes:

[0113] An information acquisition module is configured to obtain, from the jammer, the position of the jammer at multiple time points, power information corresponding to each time point, spectrum information, decision information, the number of received ACKs, and state transition probability.

[0114] A state space calculation module is configured to calculate the state space of the position of the jammer at all time points, power information, the number of received ACKs, and spectrum information.

[0115] An action space calculation module is configured to calculate the action space of the decision information at all time points.

[0116] The matrix composition module is configured to compose a matrix according to state transition probability at all time points to obtain a state transition probability matrix.

[0117] The reward value set calculation module is configured to calculate a reward value set of the number of ACKs received at all time points.

[0118] The Q value table construction module is configured to construct an initial Q value table by means of the state space, the action space, the state transition probability matrix and the reward value set.

[0119] The update module is configured to update the initial Q value table according to state information and action information at each time point to obtain an updated Q value table.

[0120] The control module is configured to screen a maximum Q value from the updated Q value table, and take decision information corresponding to the screened maximum Q value as a jammer flight path, and control the jammer to fly to the jammer flight path.

[0121] Optionally, as an embodiment of the present application, the power information includes jammer flight power, jammer interference power and jammer interference power at the previous time point, and the spectrum information includes spectrum of the environment and the user,

[0122] The state space calculation module is specifically configured to:

[0123] The state space calculation module is specifically configured to:

[0124] The state information at each time point is obtained by calculating the jammer position, the spectrum of the environment and the user, the jammer interference power at the previous time point and the jammer residual energy value at each time point by means of the first formula, and the first formula is:

[0125] s k =(F(k), ω(k), P(k-1), E(k)),

[0126] wherein,

[0127] wherein, s k is the state information at the kth time point, F(k) is the spectrum at the kth time point, ω(k) is the jammer position at the kth time point, P(k-1) is the jammer interference power at the (k-1)th time point, and E(k) is the jammer residual energy value at the kth time point, is the spectrum of the environment and the user at the kth time point, is a preset frequency point at the kth time point;

[0128] The state space of the state information at all time points is calculated by a second formula, and the state space is obtained, the second formula is:

[0129] s = [s1, s2…s k ],

[0130] Wherein, s is the state space, s k is the state information at the kth time point.

[0131] Optionally, another embodiment of the present application provides a path control system of a jammer, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the path control method of the jammer is realized. The system can be a computer or the like.

[0132] Optionally, another embodiment of the present application provides a computer readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the path control method of the jammer is realized.

[0133] It should be noted that, in this article, relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device.

[0134] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and unit can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0135] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0136] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may be distributed to multiple network units. Part or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0137] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0138] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that contributes to the present application, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the embodiments of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, and various program code storage media.

[0139] The above is only the preferred embodiment of the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A path control method of a jammer, characterized by, The method comprises the following steps: obtaining the jammer position at each time, power information corresponding to each time, spectrum information, decision information, the number of received ACKs and state transition probability from the jammer; calculating the state space of the jammer position, power information, the number of received ACKs and spectrum information at all times; calculating the action space of the decision information at all times; forming a matrix according to the state transition probability at all times to obtain a state transition probability matrix; calculating a reward value set of the number of received ACKs at all times; constructing an initial Q value table through the state space, the action space, the state transition probability matrix and the reward value set; updating the initial Q value table according to the state information and action information at each time to obtain an updated Q value table; selecting the maximum Q value from the updated Q value table and taking the decision information corresponding to the selected maximum Q value as the jammer flight path, and controlling the jammer to fly to the jammer flight path; the power information comprises the jammer flight power, the jammer jamming power and the jammer jamming power at the previous time, and the spectrum information comprises the spectrum of the environment and the user, the process of calculating the state space of the jammer position, power information, the number of received ACKs and spectrum information at all times comprises: calculating the jammer residual energy value of the jammer flight power, jammer jamming power and the number of received ACKs corresponding to each time to obtain the jammer residual energy value at each time; calculating the state information of the jammer position, the spectrum of the environment and the user, the jammer jamming power at the previous time and the jammer residual energy value at each time through a first formula to obtain the state information at each time, wherein the first formula is: ， wherein , wherein, is the first time point state information, is the first time point spectrum, is the first time point jammer position, is the first time point jammer interference power, is the first time point jammer residual energy value, is the first time point environment and user spectrum, is the first time point preset frequency point; calculating the state space of the state information at all times through a second formula to obtain the state space, wherein the second formula is: ， wherein, is the state space, is the state information at the time instant; the decision information comprises the jammer flight decision and the jammer jamming decision; the initial Q value table comprises the initial Q value at each time, the number of state-action pairs corresponding to each initial Q value and the target reward value, the process of updating the initial Q value table according to the state information and action information at each time to obtain the updated Q value table comprises: selecting the maximum Q value from all initial Q values at all times, taking the action information corresponding to the selected Q value as the target action, and taking the state information corresponding to the selected Q value as the state to be processed; taking the next state information of the state to be processed as the target state, selecting the initial Q value with the target action and the state information of the state to be processed from the initial Q value table to obtain the processed Q value; calculating the updated Q value of the processed Q value, the initial Q value at each time, the number of state-action pairs corresponding to each initial Q value and the target reward value through a seventh formula to obtain the updated Q value at each time, wherein the seventh formula is: ， wherein , , wherein, is the is the is the is the updated Q value of the is the is the is the learning rate of the is the is the is the initial Q value of the is the is the is the is the target reward value of the is the is the discount factor, is the is the is the state information of the is the is the is the number of state-action pairs of the is the Q value to be processed; drawing a table of all updated Q values at all times to obtain the updated Q value table.

2. The path control method of a jammer according to claim 1, characterized by, The process of calculating the jammer residual energy value corresponding to the jammer flight power, the jammer jamming power and the number of received ACKs at each moment includes: The jammer residual energy value at each moment is obtained by calculating the jammer flight power, the jammer jamming power and the number of received ACKs at each moment respectively through a third formula, and the third formula is: ， wherein , wherein, is the first is the jammer residual energy value at the moment, is the first is the number of received ACKs at the moment, is the preset unit time, is the total power of the jammer, is the flight power of the jammer corresponding to the current moment, is the jamming power of the jammer corresponding to the current moment.

3. The path control method of a jammer according to claim 1, characterized by, The process of calculating the action space of the decision information at all moments includes: The action information of the jammer flight decision and the jammer jamming decision at each moment is calculated through a fourth formula, and the action information at each moment is obtained, and the fourth formula is: ， wherein, is the action information at the is the jammer flight decision at the is the jammer jamming decision at the The action space of the action information at all moments is calculated through a fifth formula, and the action space is obtained, and the fifth formula is: ， wherein, is the action space, is the first action information at the time instant.

4. The path control method of a jammer according to claim 2, characterized by, The process of calculating the reward value set of the number of received ACKs at all moments includes: The reward value at each moment is calculated through a sixth formula, and the initial reward value at each moment is obtained, and the reward value set is obtained by collecting the initial reward values at all moments, and the sixth formula is: ， wherein, is the initial reward value at the time instant is the initial reward value at the time instant is the number of received ACKs at the time instant is the number of received ACKs at the time instant is a constant.

5. A path control device of a jammer, characterized by comprising: It includes: An information acquisition module is configured to obtain, from the jammer, the jammer positions at multiple moments, power information corresponding to each moment, spectrum information, decision information, the number of received ACKs and state transition probabilities; A state space calculation module is configured to calculate the state space of the jammer positions, the power information, the number of received ACKs and the spectrum information at all moments; An action space calculation module is configured to calculate the action space of the decision information at all moments; A matrix composition module is configured to compose a matrix according to the state transition probabilities at all moments, and obtain a state transition probability matrix; A reward value set calculation module is configured to calculate the reward value set of the number of received ACKs at all moments; A Q value table construction module is configured to construct an initial Q value table through the state space, the action space, the state transition probability matrix and the reward value set; An update module is configured to update the initial Q value table according to the state information and the action information at each moment, and obtain an updated Q value table; A control module is configured to select a maximum Q value from the updated Q value table, and take the decision information corresponding to the selected maximum Q value as a jammer driving path, and control the jammer to fly to the jammer driving path; The power information includes the jammer flight power, the jammer jamming power and the jammer jamming power at the previous moment, and the spectrum information includes the spectrum of the environment and the user, The state space calculation module is specifically configured to: Calculate the jammer residual energy value corresponding to the jammer flight power, the jammer jamming power and the number of received ACKs at each moment, and obtain the jammer residual energy value at each moment; The state information at each moment is obtained by calculating the jammer position, the spectrum of the environment and the user, the jammer jamming power at the previous moment and the jammer residual energy value at each moment respectively through a first formula, and the first formula is: ， wherein , in, For the first Status information at any given moment For the first Spectrum at time 10:00 For the first The location of the jammer at any given time. For the first The jamming power of the jammer at any given time. For the first The remaining energy value of the jammer at any given time. For the first The environment and user spectrum at any given time For the first The preset frequency point at any given time; The state space of the state information at all moments is calculated through a second formula, and the state space is obtained, and the second formula is: ， wherein, is the state space, is the state information at the time instant; The decision information comprises jammer flight decisions and jammer jamming decisions; The initial Q value table comprises initial Q values at respective time points, state-action pair occurrence frequencies corresponding to the initial Q values, and target reward values, The updating module is specifically configured to: select a maximum Q value from the initial Q values at all time points, and take action information corresponding to the selected Q value as a target action and state information corresponding to the selected Q value as a state to be processed; take next state information of the state to be processed as a target state, select initial Q values of action information and state information as the target action and the state to be processed from the initial Q value table, and obtain a processed Q value after screening; calculate updated Q values of the processed Q value, the initial Q values at respective time points, and the state-action pair occurrence frequencies corresponding to the initial Q values, and the target reward values by using a seventh formula, to obtain updated Q values at respective time points, the seventh formula being: ， wherein , , wherein, is the is the is the is the updated Q value of the is the is the is the learning rate of the is the is the is the initial Q value of the is the is the is the is the is the target reward value of the is the discount factor, is the is the is the state information of the is the is the is the number of state-action pairs of the is the Q value to be processed; draw a table for all the updated Q values at the respective time points to obtain an updated Q value table.

6. A path control system of a jammer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the path control method of the jammer according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium storing a computer program, wherein the computer program comprises the following steps of: receiving a request for a resource from a client; determining whether the client is authorized to access the resource; and if the client is authorized to access the resource, providing the resource to the client. When the computer program is executed by the processor, the path control method of the jammer according to any one of claims 1 to 4 is implemented.

Images