An intelligent anti-interference decision method for nonlinear swept frequency interference

Abstract

The present invention belongs to the technical field of communication anti-interference, and discloses an intelligent anti-interference decision method for nonlinear frequency sweeping interference, including: characterizing the communication anti-interference process under nonlinear frequency sweeping interference based on Markov decision process, defining the environment state, execution action, and immediate reward of the wireless communication system in any time slot; constructing a Q table based on the environment state and execution action, initializing the anti-interference strategy, initializing the Q table according to the anti-interference strategy, and forming an initial Q table; iteratively updating the initial Q table based on the Q learning algorithm, and obtaining the optimal Q table and corresponding actions when the algorithm converges; optimizing the execution actions of the wireless communication system based on the corresponding actions. In the unknown nonlinear frequency sweeping interference environment, the present invention embeds the existing anti-linear frequency sweeping interference strategy as prior knowledge into the Q learning anti-interference process, so as to realize the wireless communication system to quickly learn the unknown nonlinear interference behavior and improve the reliability and effectiveness of communication.

Description

Technical Field

[0001] The invention belongs to the technical field of communication anti-interference, and in particular relates to an intelligent anti-interference decision method for nonlinear frequency sweeping interference. Background Art

[0002] With the widespread use of various wireless devices, the electromagnetic environment is becoming increasingly complex. Malicious interference and communication anti-interference technology are contradictory, mutually reinforcing and developing alternately. Linear frequency sweep interference is a typical dynamic interference pattern, which has the advantages of concentrated interference energy, wide interference bandwidth, convenient implementation and high interference efficiency, and has been widely used. However, with the rapid development of software radio, interference cognition, pattern recognition, signal detection and other technologies, as well as the rapid improvement of the computing power of wireless communication systems, parameter perception algorithms, behavior prediction algorithms and efficient anti-interference strategies for conventional linear frequency sweep interference have been proposed and applied, which is expected to significantly improve the reliability and effectiveness of wireless communications under linear frequency sweep interference.

[0003] For this reason, nonlinear frequency sweep interference style came into being. It has the advantages of strong dynamics, weak interference regularity, and difficulty in perception and anti-interference processing, which significantly improves the interference efficiency. At present, for nonlinear frequency sweep interference, anti-interference decision algorithms based on machine learning are generally used, such as the anti-nonlinear frequency sweep interference algorithm based on Q learning. However, this type of algorithm has slow convergence speed, large amount of calculation, and poor real-time performance, which seriously restricts the improvement of wireless communication reliability and effectiveness. Summary of the invention

[0004] The purpose of the present invention is to provide an intelligent anti-interference decision method for nonlinear swept frequency interference to solve the problems existing in the above-mentioned prior art.

[0005] To achieve the above object, the present invention provides an intelligent anti-interference decision method for nonlinear frequency sweep interference, comprising the following steps:

[0006] Based on the Markov decision process, the communication anti-interference process under nonlinear frequency sweep interference is characterized, and the environmental state, execution action and immediate reward of the wireless communication system in any time slot are defined;

[0007] Based on the environmental state and the execution action, a Q table is constructed, an anti-interference strategy is initialized according to the start and end frequencies and the sweep period of the nonlinear sweep frequency interference according to the linear sweep frequency interference, and the Q table is initialized according to the initialized anti-interference strategy to form an initial Q table;

[0008] Iteratively updating the initial Q table based on a Q learning algorithm, obtaining an optimal Q table when the algorithm converges, and obtaining a corresponding action of the optimal Q table;

[0009] The execution actions of the wireless communication system are optimized based on the corresponding actions.

[0010] Optionally, the environmental state refers to an interference state faced by the wireless communication system, and the state of the tth time slot is defined as:

[0011] s(t)=[f J (t)]

[0012] Among them, s(t) represents the environmental state, f J (t)∈{1,2,...,M}, respectively representing the communication channel of the tth time slot and the channel where the interference is located; all possible states of the interference signal are s1,s2,...,s M , forming a state space S, where s1 = 1, s2 = 2, ..., s M =M;

[0013] The execution action refers to the action taken by the wireless communication system in the tth time slot:

[0014] a(t)=[f U (t)]

[0015] where f U (t)∈{1,2,...,M} represents the channel where the communication signal of the tth time slot is located, a(t) represents the execution action, and all possible actions of the wireless communication system are a1,a2,...a M , forming an action space A, where a1=1, a2=2, …, a M =M;

[0016] The instant reward refers to the instant reward value r(t) obtained when the wireless communication system performs action a(t) in the s(t) state:

[0017]

[0018] in,

[0019]

[0020] In the formula, is the channel switching cost. It is 0 when the transmission channel of the user's current time slot is the same as that of the previous time slot, otherwise it is 1. That is, when the previous time slot and the next time slot use different communication channels, a channel switching cost will be incurred. c f is the channel switching cost coefficient.

[0021] Optionally, after the Q table is constructed, the Q table is set to all zero values.

[0022] Optionally, the process of initializing the anti-interference strategy according to the start and end frequencies and the sweep period of the nonlinear sweep frequency interference according to the linear sweep frequency interference includes:

[0023] The wireless communication system senses and obtains the start and end frequencies and the sweeping period of the nonlinear sweeping frequency interference;

[0024] Based on the start and end frequencies and the frequency sweep period, and according to the anti-interference knowledge of linear frequency sweep interference, an anti-interference strategy is initialized to determine the frequency hopping point and the dwell time.

[0025] Optionally, the process of initializing the Q table according to the initialized anti-interference strategy to form an initial Q table includes:

[0026] Acquire prior knowledge based on frequency hopping and dwell time;

[0027] The prior knowledge is assigned to the Q table, the Q value is initialized, and an initial Q table is formed.

[0028] Optionally, the prior knowledge is obtained by:

[0029]

[0030] Where T is the interference period, T / 2 is the dwell time, t represents the tth time slot, A and B are both arbitrary constants, and A>B; the frequency hopping points f1 and f2 are calculated as follows:

[0031]

[0032]

[0033] Where f L 、f H are the start and end interference frequencies of the nonlinear frequency sweep interference; when the execution action a(t) is consistent with f1 or f2, the Q value is assigned to a higher reward value A; when the execution action a(t) is consistent with the environmental state s(t), the Q value is assigned to a lower reward value B. Under other conditions, the Q value is 0.

[0034] Optionally, the process of iteratively updating the initial Q table based on the Q learning algorithm includes:

[0035] Starting from the environmental state of the previous time slot, based on the anti-interference strategy of the previous time slot, the corresponding execution action is performed in the transmission sub-slot of the previous time slot to obtain the corresponding immediate reward. At the same time, the environmental state of the current time slot is perceived in the perception sub-slot of the current time slot, and the initial Q table is updated.

[0036] Optionally, the updating formula of the initial Q table is:

[0037] Q[s(t),a(t)]=(1-α)Q[s(t-1),a(t-1)]+αr(t)

[0038] Among them, Q[s(t), a(t)] is the Q value of the initial Q table in the current time slot after update, s(t-1) is the environmental state of the previous time slot, a(t-1) is the execution action of the transmission sub-slot of the previous time slot, r(t) is the corresponding immediate reward, α is the learning factor, and when all Q values ​​in the Q table no longer change significantly, it is considered that the Q table converges to the optimal.

[0039] The technical effects of the present invention are:

[0040] In an unknown nonlinear frequency sweeping interference environment, the present invention embeds the existing anti-linear frequency sweeping interference strategy as prior knowledge into the Q-learning anti-interference process, thereby enabling the wireless communication system to quickly learn the unknown nonlinear interference behavior, significantly improving the convergence speed, rapidly approaching the optimal transmission strategy, and improving the real-time performance of communication anti-interference processing under nonlinear frequency sweeping interference. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0042] Figure 1 It is a flow chart of an intelligent anti-interference decision method for nonlinear frequency sweeping interference of the present invention;

[0043] Figure 2 Schematic diagram of the communication signal frame structure in an embodiment of the present invention;

[0044] Figure 3 This is an example diagram of the time-frequency relationship of nonlinear swept frequency interference in an embodiment of the present invention;

[0045] Figure 4 Another example diagram of the time-frequency relationship of nonlinear swept frequency interference in an embodiment of the present invention;

[0046] Figure 5 Schematic diagram of the anti-linear frequency sweep interference strategy in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0048] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0049] Embodiment 1

[0050] like Figure 1-5 As shown, this embodiment provides an intelligent anti-interference decision method for nonlinear swept frequency interference.

[0051] like Figure 1 , the method comprises the following steps:

[0052] Step 1, modeling the communication anti-interference problem under nonlinear frequency sweep interference as a Markov decision process problem;

[0053] Step 2, according to the start and end frequencies and the sweep period of the perceived nonlinear sweep frequency interference, initialize its anti-interference strategy according to the linear sweep frequency interference;

[0054] Step 3, initialize the Q table in Q learning according to the initialized strategy;

[0055] Step 4: Iteratively update the Q table according to the conventional Q learning algorithm until the algorithm converges.

[0056] The detailed process of the method described in this embodiment is as follows:

[0057] First make the following assumptions:

[0058] 1. A wireless communication system (hereinafter referred to as the "system") has a transmitter and a receiver. The wireless communication frequency band is divided into M non-overlapping channels. Before each communication, the system establishes communication based on the communication channel given by the Q-learning anti-interference algorithm. The receiver has broadband spectrum sensing and Q-learning capabilities, and can sense whether there is interference in each channel in units of time slots, as well as the period and start and end frequencies of nonlinear frequency sweep interference, and can feedback the transmission action to be performed to the transmitter through a reliable public control channel. Neither the sender nor the receiver has prior information about the distribution of interference in the time and frequency domain.

[0059] 2. The sender and receiver communicate in a communication time slot (hereinafter referred to as "time slot") at a fixed information rate R. a Transmit data. The transmitted data is a fixed-length data packet, one frame per time slot, and each frame is divided into a sensing subframe, a learning subframe, and a data transmission subframe. Figure 2 The sensing subframe is used to complete spectrum sensing; the learning subframe is used to complete the machine learning process based on the proximal strategy optimization algorithm and provide the communication channel and transmission power; the data transmission subframe is used by the system to complete data transmission in a given communication channel using a given transmission power.

[0060] 3. The system communication reliability is affected by a nonlinear frequency sweep interference with an unknown pattern. This interference is periodic interference, and only one channel is interfered in each time slot; but within one period, the interference signal selects the channel to be interfered with in an arbitrary pattern. Figure 3 , Figure 4Two examples of what nonlinear swept-frequency interference might look like are shown.

[0061] The goal of the present invention is to embed the existing knowledge of anti-linear frequency sweeping interference into the Q learning algorithm, so that the system can obtain the optimal anti-interference strategy more quickly under unknown nonlinear frequency sweeping interference. Figure 1 Specific embodiments of the present invention are described below.

[0062] In step 1 (S1), the communication anti-interference problem under nonlinear frequency sweeping interference is modeled as a Markov decision process problem.

[0063] The environmental state refers to the interference state faced by the wireless communication system. The state of the tth time slot is defined as:

[0064] s(t)=[f J (t)] (1)

[0065] Among them, s(t) represents the environmental state, f J (t)∈{1,2,...,M}, respectively representing the communication channel of the tth time slot and the channel where the interference is located. All possible states of the interference signal are s1,s2,...,s M , forming a state space S, where s1 = 1, s2 = 2, ..., s M =M.

[0066] The action taken by the wireless communication system in the tth time slot is defined as:

[0067] a(t)=[f U (t)] (2)

[0068] Among them, f U (t)∈{1,2,...,M} represents the channel where the communication signal of the tth time slot is located. All possible actions of the wireless communication system are a1,a2,...a M , forming an action space A, where a1=1, a2=2, …, a M =M.

[0069] When the wireless communication system performs action a(t) in state s(t), the immediate reward value is r(t), and the global reward is defined as R(t). r(t) mainly depends on whether there is an interference signal in the selected transmission channel, and the channel switching cost is also taken into account.

[0070]

[0071] in,

[0072]

[0073] In the formula, is the channel switching cost, which is 0 when the user's current time slot transmission channel is the same as the previous time slot transmission channel, otherwise it is 1, that is, when the previous time slot and the next time slot use different communication channels, a channel switching cost will be incurred. f is the channel switching cost coefficient.

[0074] Define the Q table. The form of the Q table of the wireless communication system in the tth time slot is shown in Table 1.

[0075] Table 1

[0076] <![CDATA[a1]]> <![CDATA[a2]]> … <![CDATA[a M ]]> <![CDATA[s1]]> <![CDATA[Q t (s1,a1)]]> <h2 style=";text-align:left;direction:ltr"><![CDATA[Q <h2 style=";text-align:left;direction:ltr"> t <h2 style=";text-align:left;direction:ltr"> (s1,a2)]]><h2 style=";text-align:left;direction:ltr"> … <![CDATA[Q t (s1,a M )]]> <![CDATA[s2]]> <h2 style=";text-align:left;direction:ltr"><![CDATA[Q <h2 style=";text-align:left;direction:ltr"> t <h2 style=";text-align:left;direction:ltr"> (s2,a1)]]><h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr"><![CDATA[Q <h2 style=";text-align:left;direction:ltr"> t <h2 style=";text-align:left;direction:ltr"> (s2,a2)]]><h2 style=";text-align:left;direction:ltr"> <![CDATA[Q t (s2,a M )]]> … … … … … <![CDATA[s M ]]> <![CDATA[Q t (with M ,a1)]]> <![CDATA[Q t (s M ,a2)]]> … <![CDATA[Q t (s M ,a M )]]>

[0077] In Table 1, a1, a2, ... a M are the M possible actions of the wireless communication system, so the table has M columns (excluding the header); s1, s2, ..., s M There are M possible interference signal states, so the table has M rows (excluding the header). A row in Table 1 represents all Q values ​​corresponding to a state, and a column in the table represents all Q values ​​corresponding to a certain action.

[0078] After the Q table is established, it is first set to all 0 values. During the learning process, the user continuously interacts with the environment and explores the changing rules of interference to obtain the optimal transmission strategy. In the execution phase, communication is carried out based on the state information and the learned strategy.

[0079] In step 2 (S2), according to the start and end frequencies and the sweep period of the perceived nonlinear sweep frequency interference, an anti-interference strategy is initialized according to the linear sweep frequency interference.

[0080] The anti-linear frequency sweep interference strategy is known anti-interference knowledge, such as Figure 5 As shown. If the start and end interference frequencies of the nonlinear frequency sweep interference perceived by the system are f L and f H , the interference period is T, then according to the anti-interference knowledge of linear frequency sweep interference, the anti-interference strategy is adaptive frequency hopping, and its frequency hopping points are f1, f2:

[0081]

[0082]

[0083] Its residence time at the two frequency points is T / 2. Nonlinear frequency sweep interference start and end interference frequency f L 、f H , and the estimation method of the interference period T belong to the prior art known to professional and technical personnel in the field, and will not be described in detail in the present invention.

[0084] Step 3 (S3), initializing the Q table in Q learning according to the initialized strategy;

[0085] According to the calculated frequency hopping frequency and dwell time, the following value is assigned to the Q table as a priori knowledge to complete the initialization of the Q value and form an initial Q table:

[0086]

[0087] Where A and B are arbitrary constants and satisfy A>B. U (t)) is consistent with f1 or f2, the Q value is assigned a higher reward value A; when the action a(t) is consistent with the state s(t) (interference channel f J (t)) are consistent, the Q value is assigned to the lower reward value B. Under other conditions, the Q value is 0.

[0088] Step 4 (S4), iteratively update the Q table according to the conventional Q learning algorithm until the algorithm converges.

[0089] After obtaining the initial Q table, the Q table is continuously updated in the following transmission process. In each iteration, starting from the environment state s(t-1) of the previous time slot (t-1), firstly, based on the (t-1) time slot strategy, action a(t-1) is selected and executed in the transmission sub-slot of the time slot, and the corresponding reward r(t) is obtained. At the same time, the state s(t) of the t time slot is sensed in the perception sub-slot of the t time slot, and the Q value is updated as follows:

[0090] Q[s(t),a(t)]=(1-α)Q[s(t-1),a(t-1)]+αr(t) (8)

[0091] Among them, Q[s(t), a(t)] is the Q value of the initial Q table in the current time slot after update, s(t-1) is the environmental state of the previous time slot, a(t-1) is the execution action of the transmission sub-slot of the previous time slot, r(t) is the corresponding immediate reward, α is the learning factor, and when all Q values ​​in the Q table no longer change significantly, it is considered that the Q table converges to the optimal, and the action corresponding to the Q table at this time is output.

[0092] Regarding the exploration and utilization of the selected action a(t), the Softmax algorithm, UCB algorithm, etc. can be used.

[0093] The present invention provides an intelligent anti-interference decision method for nonlinear sweeping frequency interference, embeds the existing anti-linear sweeping frequency interference strategy into the Q learning algorithm as prior knowledge, realizes the wireless communication system to quickly learn the unknown nonlinear sweeping frequency interference behavior, quickly approaches the optimal transmission strategy, and improves the real-time anti-interference performance of communication under the condition of unknown nonlinear sweeping frequency interference.

[0094] The contents not described in detail in the present invention, such as broadband spectrum sensing algorithms, action exploration and utilization related algorithms, etc., belong to the prior art known to professional and technical personnel in the field.

[0095] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the intelligent anti-interference decision-making method for nonlinear swept frequency interference as described above is implemented.

[0096] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0097] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An intelligent anti-interference decision method for nonlinear frequency sweep interference, characterized in that: The following steps are involved: Based on the Markov decision process to characterize the communication anti-interference process under nonlinear frequency sweep interference, the environmental state, execution action and immediate reward of the wireless communication system in any time slot are defined; the environmental state refers to the interference state faced by the wireless communication system, and the environmental state of the tth time slot is defined as: s(t)=[f J (t)] Among them, s(t) represents the environmental state, f J (t)∈{1,2,...,M}, respectively representing the communication channel of the tth time slot and the channel where the interference is located; all possible states of the interference signal are s1,s2,...,s M , forming a state space S, where s1 = 1, s2 = 2, ..., s M =M; The execution action refers to the action taken by the wireless communication system in the tth time slot: a(t)=[f U (t)] Among them, a(t) represents the execution action, f U (t)∈{1,2,…,M} represents the channel where the communication signal of the tth time slot is located. All possible actions of the wireless communication system are a1,a2,…a M , forming an action space A, where a1=1, a2=2, …, a M =M; The instant reward refers to the instant reward value r(t) obtained when the wireless communication system performs action a(t) in the s(t) state: in, In the formula, is the channel switching cost. It is 0 when the transmission channel of the user's current time slot is the same as that of the previous time slot, otherwise it is 1. That is, when the previous time slot and the next time slot use different communication channels, a channel switching cost will be incurred. c f is the channel switching cost coefficient; Based on the environmental state and the execution action, a Q table is constructed, an anti-interference strategy is initialized according to the linear sweep frequency interference according to the start and end frequencies and the sweep frequency period of the nonlinear sweep frequency interference, and the Q table is initialized according to the initialized anti-interference strategy to form an initial Q table; the process of initializing the anti-interference strategy according to the linear sweep frequency interference according to the start and end frequencies and the sweep frequency period of the nonlinear sweep frequency interference includes: The wireless communication system senses and obtains the start and end frequencies and the sweeping period of the nonlinear sweeping frequency interference; Based on the start and end frequencies and the frequency sweep period, and according to the anti-interference knowledge of linear frequency sweep interference, an anti-interference strategy is initialized to determine the frequency hopping point and the dwell time; The process of initializing the Q table according to the initialized anti-interference strategy to form the initial Q table includes: Acquire prior knowledge based on frequency hopping and dwell time; Assigning the prior knowledge to the Q table, completing the initialization of the Q value, and forming an initial Q table; The formula for obtaining the prior knowledge is: Where T is the interference period, T / 2 is the dwell time, t represents the tth time slot, A and B are both arbitrary constants, and A>B; the frequency hopping points f1 and f2 are calculated as follows: Where f L 、f H is the start and end interference frequency of nonlinear frequency sweep interference; when the execution action a(t) is consistent with f1 or f2, the Q value is assigned to a higher reward value A; when the execution action a(t) is consistent with the environmental state s(t), the Q value is assigned to a lower reward value B, under other conditions, the Q value is 0; Iteratively updating the initial Q table based on a Q learning algorithm, obtaining an optimal Q table when the algorithm converges, and obtaining a corresponding action of the optimal Q table; The execution actions of the wireless communication system are optimized based on the corresponding actions.

2. The intelligent anti-interference decision method for nonlinear frequency sweep interference according to claim 1 is characterized in that: After the Q table is constructed, the Q table is set to all 0 values.

3. The intelligent anti-interference decision method for nonlinear frequency sweep interference according to claim 1 is characterized in that: The process of iteratively updating the initial Q table based on the Q learning algorithm includes: Starting from the environmental state of the previous time slot, based on the anti-interference strategy of the previous time slot, the corresponding execution action is performed in the transmission sub-slot of the previous time slot to obtain the corresponding immediate reward. At the same time, the environmental state of the current time slot is perceived in the perception sub-slot of the current time slot, and the initial Q table is updated.

4. The intelligent anti-interference decision method for nonlinear frequency sweep interference according to claim 3 is characterized in that: The update formula of the initial Q table is: Q[s(t),a(t)]=(1-α)Q[s(t-1),a(t-1)]+αr(t) Among them, Q[s(t), a(t)] is the Q value of the initial Q table in the current time slot after update, s(t-1) is the environmental state of the previous time slot, a(t-1) is the execution action of the transmission sub-slot of the previous time slot, r(t) is the corresponding immediate reward, α is the learning factor, and when all Q values ​​in the Q table no longer change significantly, it is considered that the Q table converges to the optimal.

Images