Intelligent decision-making method for radar anti-interference based on backtracking Q-learning

Abstract

The present invention proposes a radar anti-interference intelligent decision-making method based on backtracking Q learning, which mainly solves the problem of poor suppression effect and responsiveness of interference signals of existing radars when facing complex interference scenes. The scheme includes: 1) establishing a Q table; 2) the radar detects the echo signal and identifies all interference patterns therein; 3) according to the basic principle of first spatial domain and then time-frequency domain, a greedy algorithm is used to select an anti-interference method from the total anti-interference algorithm library; 4) the echo data is processed using the selected anti-interference algorithm, and the performance of the anti-interference method is evaluated using the interference suppression evaluation index; 5) the Q table is updated using the evaluation index; 6) after a corresponding anti-interference algorithm is selected for all interferences, the Q table is retrospectively updated according to the final performance evaluation index. The present invention can effectively help the radar quickly find the best anti-interference method, and improve the system robustness and interference suppression capability.

Description

Technical Field

[0001] The present invention belongs to the field of communication technology and relates to digital information transmission. Specifically, it is a radar anti-interference intelligent decision-making method based on backtracking Q learning, which can be used for radar to autonomously update and adjust anti-interference strategies and improve the radar's ability to resist complex interference. Background Art

[0002] Radar can detect distant targets both day and night, and is not blocked by fog, clouds and rain. It has the characteristics of all-weather and all-day, and has a certain penetration ability. Therefore, it has not only become an indispensable electronic equipment in the military, but also an important factor affecting the war situation, and is widely used in social and economic development; the party that takes the initiative in radar confrontation will gain a huge advantage. With the continuous advancement of interference and anti-interference technology, the electromagnetic environment is becoming more and more complex, the types of interference faced by radar have increased dramatically, and the interference scenes are complex. Even though many scholars have proposed various interference suppression methods for different interferences, traditional radars are extremely dependent on human control, so it is difficult to accurately suppress interference signals. Therefore, studying radar intelligent decision-making technology, enabling radar to adaptively formulate anti-interference strategy processes based on its own application environment, and reasonably using resources to achieve accurate suppression of interference signals has become one of the research hotspots in the radar field.

[0003] In recent years, with the continuous development of artificial intelligence technology, it has become possible for radar to have intelligent anti-interference. Wang Hao, Wang Feng and others from Hohai University proposed an intelligent anti-interference method based on a reinforcement learning model in the document "Application of Reinforcement Learning Algorithm in Radar Intelligent Anti-Interference" (doi:10.16592 / j.cnki.1004-7859.2020.03.009). The Q-learning algorithm is used to calculate and iterate the value function in the anti-interference model, so that the anti-interference strategy has the ability to update and optimize autonomously. However, there are problems with this algorithm: 1) The algorithm does not take into account the various errors in radar signal processing. Due to the existence of data errors, the algorithm will be difficult to converge due to its lack of robustness. 2) It does not take into account that when there are multiple options for methods in each domain, how to choose the method in each domain to achieve the best final anti-interference effect. Summary of the invention

[0004] The present invention combines reinforcement learning knowledge with radar anti-interference technology, takes into account the various errors in radar received data, improves the Q-learning algorithm, adds a backtracking update process, and applies it to radar intelligent anti-interference, so as to improve the radar's suppression effect on interference signals and the radar's rapid response capability.

[0005] The idea of ​​implementing the present invention is: first, the radar identifies all interference patterns in the echo signal, and then uses a greedy algorithm to select a suitable anti-interference method from the anti-interference algorithm library according to the Q table to process the echo signal. The performance of the anti-interference method is evaluated using the interference suppression evaluation index, and the Q table is updated using the evaluation results. After that, the three processing steps of selecting an anti-interference method, performance evaluation, and Q table update are continued for the remaining interference. After a corresponding anti-interference algorithm is selected for all interferences, the Q table is retroactively updated according to the final performance evaluation index. Based on this approach, it can effectively help the radar quickly find the best anti-interference method and improve the radar's ability to suppress interference. The specific steps are as follows:

[0006] (1) Create a P J Line, P A Q table of columns, where P J is the number of permutations and combinations of interference patterns, P A is the number of anti-interference methods;

[0007] (2) The radar detects the echo signal x(k) and identifies all interference patterns in the echo signal;

[0008] (3) The radar selects an anti-interference method from the total anti-interference algorithm library using a greedy algorithm according to the basic principle of first selecting an anti-interference method in the spatial domain and then in the time and frequency domain;

[0009] (4) The radar processes the echo signal using the selected anti-interference method to obtain processed data and construct an evaluation index for interference suppression; if the selected anti-interference method corresponds to a certain interference pattern in the echo signal, then regardless of whether the interference pattern has been suppressed, it is considered that the interference pattern has been eliminated and the interference state has changed;

[0010] (5) The radar determines whether the interference pattern currently being processed is the last one. If not, the Q table is updated according to the following formula:

[0011]

[0012] On the contrary, if the interference currently being processed is the last interference, the Q table is updated according to the following formula:

[0013] Q'(s t ,a i )=Q(s t ,a i )+α[ζ-Q(s t ,a i )],

[0014] Among them, s t Indicates interference status; a irepresents the i-th anti-interference method, i∈1…P A ;s t+1 Indicates that the system is in interference state s t When using anti-interference method a i The subsequent interference state; Q(s t ,a i ) indicates that the system is in the interference state s t When selecting anti-interference method a i Q value; Indicates that the system interference state is s t+1 When , the maximum Q value among all anti-interference methods; ζ represents the evaluation index, α is the learning rate, and γ is the discount coefficient;

[0015] (6) After processing each interference pattern in the echo signal, the radar performs a retrospective update on the key nodes in the current round of Q table. The key nodes are defined as: if the interference state of the echo signal at a certain moment is s t , the radar performs anti-jamming method a i After that, the interference state changes, then it is considered that Q(s t ,a i ) is the key node in this round of processing, and in each round of backtracking update, the backtracking update Q value of each key node is the same; the specific steps of backtracking update are as follows:

[0016] (6a) Calculate the backtracking update Q value:

[0017]

[0018]

[0019] Among them, w hs =[w1,w2,…,w P ] T represents the backtracking update weight, ζ hs =[ζ1,ζ2,…,ζ P ] T It represents the evaluation index of interference suppression, m is Q(s t ,a i ) is identified as a key node;

[0020] (6b) Backtrack and update each key node in the Q table according to the following formula:

[0021]

[0022] Among them, Q hs represents the Q value of the backtracking update, and α is the learning rate.

[0023] Compared with the prior art, the present invention has the following advantages:

[0024] First, the present invention organically combines reinforcement learning with radar anti-interference, so that the radar has intelligent anti-interference capabilities. Compared with traditional radars that rely on expert experience to manually arrange anti-interference processes, the present invention enables the radar to adaptively change anti-interference strategies when facing more complex interference scenarios, thereby improving system robustness.

[0025] Second, when the present invention applies the Q-learning algorithm to the field of radar anti-interference, a backtracking update step is proposed in the Q-learning algorithm based on a single round of training experience, which reduces the impact of data errors on the convergence of the algorithm and improves the robustness of the algorithm. The algorithm can also converge when there are certain errors in the received radar data, making the present invention more valuable for engineering applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flow chart of the implementation of the present invention;

[0027] Figure 2 It is an ideal performance simulation comparison chart of various anti-interference strategy processes in the present invention;

[0028] Figure 3 The algorithm proposed in this invention corresponds to Figure 2 The Q-value training simulation result diagram during the first step of processing;

[0029] Figure 4 The algorithm proposed in this invention corresponds to Figure 2 The Q value training simulation result diagram during the second step of processing; DETAILED DESCRIPTION

[0030] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0031] Reference Figure 1 The present invention proposes a radar anti-interference intelligent decision-making method based on backtracking Q learning, which specifically includes the following steps:

[0032] Step 1: Create a P J Line, P A Q table of columns, where P J is the number of permutations and combinations of interference patterns, P A is the number of anti-interference methods; in this embodiment, assuming that there are a interference patterns and b anti-interference methods, then P A = b;

[0033] Step 2: The radar detects the echo signal x(k) and identifies all interference patterns in the echo signal. In this embodiment, it is assumed that the initial interference state is J1+J2+…+J PIf the anti-interference measures corresponding to the interference J1 are used, the interference state becomes J2+…+J P ;

[0034] Step 3: The radar uses a greedy algorithm to select an anti-interference method from the total anti-interference algorithm library according to the basic principle of first spatial domain and then time-frequency domain. The specific steps are as follows:

[0035] (3a) Define a greediness coefficient ε, where 0<ε<1;

[0036] (3b) Let the system randomly select from the anti-interference algorithm library or select according to the Q table, where the probability of random selection is ε and the probability of selection according to the Q table is 1-ε;

[0037] The system makes a selection in the anti-interference algorithm library based on the Q table, and selects the anti-interference method corresponding to the row where the current interference state is located in the Q table; that is, with a probability of 1-ε, the column with the largest Q value is selected from the row where the state is the current interference state in the Q table, and the column number in the Q table is the number of the anti-interference method.

[0038] Step 4: The radar processes the echo signal using the selected anti-interference method to obtain processed data and construct an evaluation index for interference suppression; if the selected anti-interference method corresponds to a certain interference pattern in the echo signal, then regardless of whether the interference pattern has been suppressed, it is considered that the interference pattern has been eliminated and the interference state has changed;

[0039] The processed data is the signal after interference suppression; the radar uses the selected anti-interference method to process the echo signal, and the process follows the basic principle of first space domain and then time domain; the details are as follows:

[0040] If the first processing method for the echo signal is not the spatial domain anti-interference method, the processing is skipped, and an evaluation index of no higher than -50dB is given to the processing result as the evaluation result until a spatial domain anti-interference method is selected. When the spatial domain anti-interference method is selected, the adaptive weight vector w obtained by the spatial domain anti-interference method is used, and then the echo signal is processed according to the following formula to obtain the interference suppressed signal y(k):

[0041] y(k)=w H x(k),

[0042] Where x(k) represents the echo signal received by the radar antenna, k∈1…K, K is the number of received snapshots; x(k) and w are both N×1 column vectors, N represents the number of receiving array elements;

[0043] If the spatial domain anti-interference method has been used to process the echo signal before, and the spatial domain anti-interference method is selected again at this time, the processing will be skipped, and an evaluation index of no higher than -50dB will be given to the processing result as the evaluation result;

[0044] If the spatial domain anti-interference method has been used to process the echo signal before, and the anti-interference method selected at this time is the time-frequency domain method, then let the result of the previous processing be y'(k) and get the data after this processing

[0045]

[0046] Wherein, T[·] represents the use of a specific anti-interference method to process the echo data.

[0047] The evaluation index for constructing interference suppression is to determine the evaluation index for the spatial domain anti-interference method and the time-frequency domain anti-interference method respectively, as follows:

[0048] For the spatial domain anti-interference method, the output signal to interference and noise ratio SINR is used as the first evaluation indicator ζ1:

[0049]

[0050] Among them, R s is the target signal covariance matrix obtained by sample selection, R in is the interference and noise covariance matrix obtained through sample selection;

[0051] For time-frequency domain anti-interference, the processed echo signal Pulse compression is performed to obtain a pulse pressure signal Construct the second evaluation index ζ2:

[0052]

[0053] in, The signal after the pulse pressure The signal amplitude at the target location, express The mean value of the signal amplitude at locations other than the target signal location.

[0054] Step 5: The radar determines whether the interference pattern currently being processed is the last one. If not, that is, there is the next interference, the Q table is updated according to the following formula:

[0055]

[0056] On the contrary, if the interference currently being processed is the last interference, that is, there is no next interference, the Q table is updated according to the following formula:

[0057] Q'(s t ,a i )=Q(s t ,a i )+α[ζ-Q(s t ,a i )],

[0058] Among them, s t Indicates interference status; a i represents the i-th anti-interference method, i∈1…P A ;s t+1 Indicates that the system is in interference state s t When using anti-interference method a i The subsequent interference state; Q(s t ,a i ) indicates that the system is in the interference state s t When selecting anti-interference method a i Q value; Indicates that the system interference state is s t+1 When , the maximum Q value among all anti-interference methods; ζ represents the evaluation index, α is the learning rate, and γ is the discount coefficient;

[0059] Step 6: After processing each interference pattern in the echo signal, the radar backtracks and updates the key nodes in the current round of Q table. The key nodes are defined as: if the echo signal interference state at a certain moment is s t , the radar performs anti-jamming method a i After that, the interference state changes, then it is considered that Q(s t ,a i ) is the key node in this round of processing, and in each round of backtracking update, the backtracking update Q value of each key node is the same; the specific steps of backtracking update are as follows:

[0060] (6a) Calculate the backtracking update Q value:

[0061]

[0062]

[0063] Among them, w hs =[w1,w2,…,w P ] T represents the backtracking update weight, ζ hs =[ζ1,ζ2,…,ζ P ] T represents the evaluation index of interference suppression, that is, the evaluation index of each interference suppression constructed in step 4; m is Q(s t ,a i) is identified as a key node;

[0064] (6b) Backtrack and update each key node in the Q table according to the following formula:

[0065]

[0066] Among them, Q hs represents the Q value of the backtracking update, and α is the learning rate.

[0067] The present invention is further described below in conjunction with simulation experiments.

[0068] 1. Simulation parameter settings:

[0069] Table 1 gives the specific simulation parameters of the intelligent decision-making radar system. The parameters of the target signal and interference are given in Table 2. Table 3 gives the anti-interference methods in the spatial, time and frequency domains used in this simulation.

[0070] Table 1 Simulation parameters

[0071]

[0072] Table 2 Target parameters

[0073]

[0074] Table 3 Anti-interference methods

[0075]

[0076] 2. Simulation content and result analysis:

[0077] Simulation 1, under the simulation parameters of Table 1, Table 2 and Table 3 above, the performance simulation results of different anti-interference strategy processes (first in the spatial domain and then in the time domain) are as follows: Figure 2 As shown. In the figure, the names of the lines explain the order of the combination of anti-interference methods. For example, WCPO+Time_Threshold means that when processing data, the WCPO method in the spatial domain is used in the first step, and the time domain threshold zeroing method is used in the second step. WCPO+Time_Stft means that when processing data, the WCPO method in the spatial domain is used in the first step, and the time domain filtering method based on time-frequency analysis is used in the second step. In this simulation, the steering vector error is distributed in [-3°, 3°]. For different method combinations, 100 Monte Carlo runs are performed. It can be seen that from the first step to the second step of the processing flow, the interference suppression ratio of the combination of MatrixRec and the time domain filtering method is the highest. Taking this as a reference, the ideal convergence result of the algorithm proposed in the present invention should be: in the first step, the Q value of MatrixRec is the largest, and in the second step, the Q value of the time domain filtering method is the largest.

[0078] Simulation 2: Under the simulation parameters of Table 1, Table 2 and Table 3, simulation training was carried out using the technology of the present invention. The total number of training rounds was 500. The simulation results are as follows: Figure 3 , Figure 4 , as shown.

[0079] Referring to simulation 1, theoretically, the interference suppression ratio of the combination of MatrixRec and time domain filtering is the highest. Figure 3 As can be seen in the simulation, the convergence Q value of MatrixRec in the first step is the largest among several spatial domain methods. Figure 4 It can be seen that in the second step, the convergence Q value of the time domain filtering method is the largest in the time-frequency domain method, and this result is consistent with the result of simulation 1. Figure 3 , Figure 4 , the radar will first execute the MatrixRec method, and the second step will execute the Time_Stft method. Through these two steps, the radar will obtain a better interference suppression effect, that is, the selection of the optimal anti-interference strategy is completed. It is worth noting that the performance of the three anti-interference strategy processes [MatrixRec+Time_Stft], [WCPO+Time_Stft] and [Eigenspace+Time_Stft] are very similar, so there is no strict difference in the size of the Q values ​​of these three strategies. In addition, it can be seen that when the algorithm proposed by the present invention is adopted, the Q value of each method is basically stable at the time of the 50th round of training, and there is only a small fluctuation thereafter, which shows that the present invention has a certain robustness and engineering value.

[0080] The above simulation analysis and tests prove the correctness and effectiveness of the method proposed in the present invention.

[0081] Parts of the present invention that are not described in detail belong to common knowledge among those skilled in the art.

[0082] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Obviously, for professionals in this field, after understanding the content and principles of the present invention, they may make various modifications and changes in form and details without departing from the principles and structures of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

Claims

1. A radar anti-interference intelligent decision-making method based on backtracking Q learning, characterized in that: The steps include: (1) Create a P J Line, P A Q table of columns, where P J is the number of permutations and combinations of interference patterns, P A is the number of anti-interference methods; (2) The radar detects the echo signal x(k) and identifies all interference patterns in the echo signal; (3) The radar selects an anti-interference method from the total anti-interference algorithm library using a greedy algorithm according to the basic principle of first selecting an anti-interference method in the spatial domain and then in the time and frequency domain; (4) The radar processes the echo signal using the selected anti-interference method to obtain processed data and construct an evaluation index for interference suppression; if the selected anti-interference method corresponds to a certain interference pattern in the echo signal, then regardless of whether the interference pattern has been suppressed, it is considered that the interference pattern has been eliminated and the interference state has changed; (5) The radar determines whether the interference pattern currently being processed is the last one. If not, the Q table is updated according to the following formula: On the contrary, if the interference currently being processed is the last interference, the Q table is updated according to the following formula: Q'(s t ,a i )=Q(s t ,a i )+α[ζ-Q(s t ,a i )], Among them, s t Indicates interference status; a i represents the i-th anti-interference method, i∈1...P A ;s t+1 Indicates that the system is in interference state s t When using anti-interference method a i The subsequent interference state; Q(s t ,a i ) indicates that the system is in the interference state s t When selecting anti-interference method a i Q value; Indicates that the system interference state is s t+1 When , the maximum Q value among all anti-interference methods; ζ represents the evaluation index, α is the learning rate, and γ is the discount coefficient; (6) After processing each interference pattern in the echo signal, the radar performs a retrospective update on the key nodes in the current round of Q table. The key nodes are defined as: if the interference state of the echo signal at a certain moment is s t , the radar performs anti-jamming method a i After that, the interference state changes, then it is considered that Q(s t ,a i ) is the key node in this round of processing, and in each round of backtracking update, the backtracking update Q value of each key node is the same; the specific steps of backtracking update are as follows: (6a) Calculate the backtracking update Q value: Among them, w hs =[w1,w2,…,w P ] T represents the backtracking update weight, ζ hs =[ζ1,ζ2,…,ζ P ] T It represents the evaluation index of interference suppression, m is Q(s t ,a i ) is identified as a key node; (6b) Backtrack and update each key node in the Q table according to the following formula: Among them, Q hs represents the Q value of the backtracking update, and α is the learning rate.

2. The method according to claim 1, characterized in that: In step (3), an anti-interference method is selected from the anti-interference algorithm library, specifically: a greedy coefficient ε is defined, and 0<ε<1; the system is allowed to randomly select from the anti-interference algorithm library or select according to the Q table, where the probability of random selection is ε and the probability of selection according to the Q table is 1-ε; the system selects from the anti-interference algorithm library according to the Q table based on the row where the current interference state is located in the Q table, and selects the anti-interference method corresponding to the maximum Q value.

3. The method according to claim 1, characterized in that: The processed data in step (4) is the signal after interference suppression; the radar uses the selected anti-interference method to process the echo signal, and the process follows the basic principle of first spatial domain and then time domain; the details are as follows: If the first processing method for the echo signal is not the spatial domain anti-interference method, the processing is skipped, and an evaluation index of no higher than -50dB is given to the processing result as the evaluation result until a spatial domain anti-interference method is selected. When the spatial domain anti-interference method is selected, the adaptive weight vector w obtained by the spatial domain anti-interference method is used, and then the echo signal is processed according to the following formula to obtain the interference suppressed signal y(k): y(k)=w H x(k), Where x(k) represents the echo signal received by the radar antenna, k∈1…K, K is the number of received snapshots; x(k) and w are both N×1 column vectors, N represents the number of receiving array elements; If the spatial domain anti-interference method has been used to process the echo signal before, and the spatial domain anti-interference method is selected again at this time, the processing will be skipped, and an evaluation index of no higher than -50dB will be given to the processing result as the evaluation result; If the spatial domain anti-interference method has been used to process the echo signal before, and the anti-interference method selected at this time is the time-frequency domain method, then let the result of the previous processing be y'(k) and get the data after this processing Wherein, T[·] represents the use of a specific anti-interference method to process the echo data.

4. The method according to claim 3, characterized in that: The evaluation index for constructing interference suppression is to determine the evaluation index for the spatial domain anti-interference method and the time-frequency domain anti-interference method respectively, as follows: For the spatial domain anti-interference method, the output signal to interference and noise ratio SINR is used as the first evaluation indicator ζ1: Among them, R s is the target signal covariance matrix obtained by sample selection, R in is the interference and noise covariance matrix obtained through sample selection; For time-frequency domain anti-interference, the processed echo signal Pulse compression is performed to obtain a pulse pressure signal Construct the second evaluation index ζ2: in, The signal after the pulse pressure The signal amplitude at the target location, express The mean value of the signal amplitude at locations other than the target signal location.

Images