Automobile millimeter wave radar interference suppression method based on slow time phase coding

Abstract

The invention provides an automobile millimeter wave radar interference suppression method based on slow time phase coding, and the method achieves a better coherent interference suppression effect through optimizing the phase coding between adjacent pulses based on the idea of reinforcement learning. The method mainly comprises the following steps: establishing a Markov decision process of an automobile millimeter wave radar in a coherent interference suppression scene; deriving a Markov decision process reward function in the interference suppression scene; designing an observation matrix in a Markov decision process in a coherent interference suppression scene according to cumulative observation of a plurality of time steps; designing a reinforcement learning algorithm according to an inter-pulse phase coding scene to obtain action sampling output meeting reasonable probability distribution; designing a strategy network and a value network in combination with the observable problem of the environment part; training parameters are set, the network is trained, the trained network is used as decision output of coherent interference suppression, and result evaluation is carried out.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automatic driving, in particular to sensor perception in automatic driving, which combines deep reinforcement learning (DRL) and frequency modulated continuous wave (FMCW) radar signal interference cancellation technology to reduce the probability of mutual interference between vehicles and reduce the frequency of accidents of automatic driving vehicles. BACKGROUND

[0002] In the 21st century, with the breakthrough progress of sensor technology, artificial intelligence algorithm, high-precision map and high-performance computing platform, automatic driving technology has begun to upgrade by leaps and bounds. As the sensor of the car, the frequency modulated continuous wave radar has become the mainstream choice of car millimeter wave radar. However, since various millimeter wave radars installed on cars basically share the same frequency band, more and more vehicles are equipped with more radars to provide all-around situation awareness at different distances, which greatly increases the probability of mutual interference between radars, seriously affects the correctness of the detection results of car radars, and causes great safety hazards.

[0003] At present, the industry has proposed various signal processing methods to eliminate the mutual interference between car radars. However, such methods reduce the timeliness of target detection because they require additional processing of received signals. More importantly, as the main source of false targets, coherent interference has the same pattern as the echo signal of the detected target in these processing domains, so signal processing methods cannot distinguish and eliminate coherent interference and cannot solve the problem of false targets.

[0004] Fortunately, the above problems can be alleviated to some extent by inter-pulse coding. However, most methods of using phase coding to counter coherent interference use discrete coding sequences, and use random sequences, simple coding sequences or iterative solutions. In fact, discrete coding limits the solution space of optimal coding, while random coding can be further optimized. However, the existing iterative solution itself is an NP-hard problem, and its solution efficiency is difficult to guarantee, which will seriously affect the real-time performance in actual use. Reinforcement learning is a powerful method for solving sequential decision problems, and has been introduced into radar anti-interference methods in recent years. Therefore, the slow-time coding coherent interference suppression method based on reinforcement learning is a potential solution to the problems of low efficiency and high complexity in coding to suppress coherent interference. SUMMARY

[0005] This paper is inspired by the traditional inter-pulse phase iterative optimization encoding method to avoid coherent interference, according to the existing iterative optimization encoding method output is only discrete encoding value, and its existence solving space is limited, the problem of low iteration efficiency, this paper puts forward a kind of non-cooperative based on deep reinforcement learning of continuous value inter-pulse phase encoding optimization strategy, this paper according to the actual single vehicle and single interference source scene design and establish the signal level level Markov decision process, and in the value network and strategy network introduces the transformer network architecture to deal with time related waveform observation sequence. Simulation experiments show that the proposed method will be higher than the traditional random inter-pulse encoding method in the index of anti-interference performance based on the number of false targets and target missed detection number.

[0006] The automobile millimeter wave radar coherent interference suppression method of the application is as shown in the accompanying Figure 1 The automobile millimeter wave radar coherent interference suppression method of the application is as shown in the accompanying

[0007] Step 200, establish the Markov decision process of the automobile millimeter wave radar under the coherent interference suppression scene.

[0008] The phase encoded interference signal and the signal after desloping processing of the interfered radar's transmitting signal are as shown in formula (1):

[0009]

[0010] Wherein, each parameter is defined as follows:

[0011] x t (t;n) is the transmitting signal of a chirp radar, t represents fast time, and n represents slow time;

[0012] x i (t;n) is the interference signal of a chirp, t represents fast time, and n represents slow time;

[0013] is the phase offset introduced in a chirp signal;

[0014] h lp (t) is the impulse response function of a low-pass filter;

[0015] represents a phase encoding of the radar transmitting signal, and we set the encoding of a frame of radar signal as represents continuous phase encoding. Then the execution process of phase encoding optimization is modeled as a Markov decision process. For the setting of action, the action output at each time step is the continuous phase encoding value to be used for a frame of transmitting signal. Assuming that a frame of signal contains L chirps, the action is

[0016] In step 210, the reward function of the Markov decision process under the interference suppression scenario is derived. For each frame of signal, the constant false alarm detection and peak aggregation based on the range-Doppler map are performed, and the false target number and the missed detection number are calculated by comparing the results with the actual situation. The false target number and the missed detection number are respectively represented by n ft and n md .

[0017] In step 220, the observation matrix O k in the Markov decision process under the coherent interference suppression scenario is obtained according to the cumulative observation of multiple time steps.

[0018] O k is a matrix of (2m+1)*(L+3), and O k includes two parts: phase encoding and range-Doppler detection results, corresponding to the first vector and the last 2m vectors. The first bit of the (2m+1) vectors is the time step index of the observation. For the first phase encoding related vector, the second bit is fixedly filled with -1 as an identifier, and the third bit is the total number of targets n detected. The following L bits are the phase encoding. The last 2m vectors are divided into m groups, and each group includes two vectors containing the number of target detections on a range index and the range-Doppler map component or the peak aggregation result component.

[0019] In step 230, the action sampling output satisfying a reasonable probability distribution is derived, as shown in formula (2).

[0020]

[0021] wherein,

[0022] π(a|s) is the action probability density function of the final output;

[0023] μ(u|s) is the action probability density function before correction;

[0024] u is the action obtained by sampling the action probability density function;

[0025] a is the compressed phase action output, a=bsin(u);

[0026] dim(A) is the overall dimension of the action;

[0027] b is a scaling coefficient;

[0028] In step 240, the strategy network and the value network are designed.

[0029] Both policy network and value network are composed of information extraction module and decision module. Information extraction module is composed of encoder, the input of information module is single-step state observation matrix, and the output is state information extraction code, which is part of the input of decision module. Decision module is composed of decoder followed by deep neural network, the input of decoder is divided into two parts, one part is the output of information extraction module as its input key and value, the other part is all 1 vector or action to be evaluated as its input query.

[0030] Step 250, set training parameters, train value network and policy network.

[0031] In this paper, both target and jammer are set to 1, and the jammer and target are very close in the range dimension of the range-doppler map. Because of coherent jamming, the jammer and the radar under jamming use the same set of frequency sweeping parameters, and the center frequency and transmission time are fixed. During training, a total of 20,000 training rounds are performed, and each round includes 400,000 time steps.

[0032] Step 260, evaluate the jamming suppression effect of the trained value network and policy network.

[0033] In this paper, cumulative return and jamming suppression index are used to reflect the pros and cons of coherent jamming suppression, and the jamming suppression index is defined as formula (3):

[0034]

[0035] Wherein, The values of the missed detection number and the false target number obtained by performing one training simulation are smoothed, respectively, and w md , w ft Indicate the weights of the target missed detection number and the false target number, respectively.

[0036] Advantages

[0037] The coherent jamming suppression method of the automobile millimeter wave radar of the application establishes an inter-pulse phase coding anti-jamming model. A non-cooperative random continuous inter-pulse phase coding and its optimized strategy are proposed. The execution process of the proposed inter-pulse phase coding anti-jamming strategy is modeled as a Markov decision process, a deep reinforcement learning algorithm is designed to handle complex state space and output high-dimensional continuous-valued actions, and the phase coding sequence is optimized. Through simulation experiments, it is verified that the continuous inter-pulse phase coding strategy has good anti-jamming performance and real-time performance when facing coherent jamming, and can completely eliminate false alarms and greatly reduce the missed detection rate in a short time. The designed anti-jamming method does not require additional jamming sensing hardware and computing overhead, and can be completely implemented based on the existing automobile radar hardware system, and has good landing prospects. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to clearly and accurately explain the technical steps of the present application, all the drawings used in the description of the present application will be briefly described below. It should be noted that the drawings described below are only some examples of the implementation of the present application, and other ordinary skilled persons in the art can still obtain other drawings in other different scenarios based on these drawings.

[0039] Figure 1 is the implementation process of the present application; Figure 1 is the implementation process of the present application;

[0040] Figure 2 is an application scenario experimental parameter setting of the present application; Figure 2 is an application scenario experimental parameter setting of the present application;

[0041] Figure 3 is a structure diagram of the strategy network and the value network in the present application; Figure 3 is a structure diagram of the strategy network and the value network in the present application;

[0042] Figure 4 is a change of the cumulative reward return and the interference suppression index with the training round in the training process of the present application; Figure 4 is a change of the cumulative reward return and the interference suppression index with the training round in the training process of the present application;

[0043] Figure 5 is a change of the interference suppression index with the training round in the test process of the present application; Figure 5 is a change of the interference suppression index with the training round in the test process of the present application;

[0044] Figure 6 is a comparison effect diagram of the method proposed in the present application and random coding in the case of resisting coherent interference; Figure 6 is a comparison effect diagram of the method proposed in the present application and random coding in the case of resisting coherent interference; DETAILED DESCRIPTION

[0045] The steps and processes of the present application will be described in detail below in combination with the drawings in the present application. It is obvious that the examples described in the present application are only one example application scenario of the present application, and other results based on the content of the present application without substantial changes are also within the protection scope of the present application.

[0046] Figure 1 is the implementation process of the present application; Figure 2 is a parameter setting of an example scenario of the present application, which shows the basic elements contained in the present application and some parameters that will be used subsequently, including radar parameter setting, simulation scenario parameter setting, etc. In this example scenario, the interference source and the interfered radar use the same set of frequency sweeping parameters, and the center frequency and the transmission time are fixed. The number of targets and the number of interferences are both set to 1, and the interference and the target are very close in the distance dimension of the range-doppler diagram. The purpose of this setting is that when the interference and the target are in the close distance dimension, it is difficult for the traditional inter-pulse phase coding method to remove the false target through subsequent processing methods.

[0047] Figure 2 is an application scenario experimental parameter setting of the present application; Figure 2An example scenario is given, the main relevant parameters are: the radar center frequency is set to 77GHz, the radar bandwidth is set to 500MHz, the pulse repetition period is set to 60us, the sampling frequency is 20MHz, the maximum detection distance is set to 120m, the number of chirps per frame is set to 128, the maximum detection distance is set to 120m, the number of interference targets and interfered targets is one, in each round, the distance of the interfered target is randomly selected in the range of uniform distribution u(3, 120), the relative speed of the interfered target is also randomly selected in the range of u(-16, 16), and the relevant parameters of the target transmitting the interference signal are also in the range of Figure 2 The relevant parameters are set, which will not be described one by one here.

[0048] The specific steps of the coherent interference suppression method of the automobile millimeter wave radar in this paper are as follows:

[0049] Step 300, establish the Markov decision process of the automobile millimeter wave radar in the coherent interference suppression scene. First, define the radar transmitting signal:

[0050]

[0051] K represents the frequency modulation slope, f c represents the intermediate frequency of the signal, T c represents the period of a chirp signal, T sw represents the frequency sweeping duration of a single chirp, is the starting phase of the signal, and in the case of not considering the phase encoding first, it is considered that the starting phase of each chirp remains the same.

[0052] Interfere inter-pulse phase encoding, set the transmitting signal encoding sequence N is the number of chirps of a frame of transmitting signal. The transmitting signal of a frame after encoding is represented as:

[0053]

[0054] The interference signal received by the radar receiver for a single chirp is:

[0055]

[0056] The inter-pulse encoding sequence of the interference signal is The interference signal received by the radar receiver for a single chirp is:

[0057]

[0058] The interference signal after phase encoding and the transmitting signal of the interfered radar after desloping processing are as shown in formula (8):

[0059]

[0060] where x t (t;n) is the transmitted signal of a chirp, t denotes fast time, n denotes slow time, x i (t;n) is the jamming signal of a chirp, t denotes fast time, n denotes slow time, is the introduced phase offset in a chirp signal, h lp (t) is the impulse response function of a low-pass filter denotes a phase encoding of the radar transmitted signal, let the encoding of a frame of radar signal be denotes a continuous-valued phase encoding.

[0061] The execution process of phase encoding optimization is modeled as a Markov decision process, for the setting of actions, the output of each time step action is the continuous phase encoding value to be used for a frame of transmitted signal, let the number of chirps contained in a frame of signal be L, then the action

[0062] Step 310, derive the reward function of the Markov decision process under the jamming suppression scenario, for each frame of signal, constant false alarm detection and peak aggregation based on the range-Doppler map are performed, and the results are compared with the true situation to obtain the calculation results of the number of false targets and the number of missed detections, respectively using n ft and n md to represent the number of false targets and the number of missed detections, then the reward function is shown in equation (9):

[0063]

[0064] Step 320, according to the cumulative observation of multiple time steps, obtain the observation matrix O k of the Markov decision process under the coherent jamming suppression scenario.

[0065] The state s t in time step t is composed of all single-step observation matrices O k before the current training round in time step t, where k < t. The matrix O k contains both the phase encoding and target detection results of the time step, and also contains part of the corresponding range-Doppler map. Let the range-Doppler map of the kth time step be denoted as RDM k , which is expressed as in the component with distance dimension index i and Doppler dimension index j. Based on RDM k , two one-dimensional constant false alarm detection and peak aggregation are performed on the distance dimension and the Doppler dimension respectively, and the matrix obtained by peak aggregation is PC k, whose component with distance dimension index i and Doppler dimension index j is Its value is 1 or 0, indicating whether the unit has a target or not. Suppose that in the distance dimension, there are m distance units that detect peaks, and the indexes of these distance units are {r1, r2, ..., r m}, and the number of targets finally detected at these distances is {n1,n2,...,n m}, so O k is a (2m+1)*(L+3) matrix, O k It consists of two parts: phase encoding and range-Doppler detection results, corresponding to the first vector and the next 2m vectors respectively. The first bit of the (2m+1) vector is the time step index of the observation. For the first phase encoding-related vector, the second bit is fixedly filled with -1 as an identifier, the third bit is the total number of detected targets n, and the following L bits are phase encoding. The following 2m vectors are grouped into m groups of two each. The two vectors in each group contain the number of target detections at a range index and the range-Doppler map component or peak aggregation result component.

[0066] Step 330: Design a reinforcement learning algorithm based on the specific scenario to derive action sampling output that satisfies a reasonable probability distribution.

[0067] This paper uses the Soft Actor-Critic (SAC) algorithm, which has five networks: a policy network θ, two value networks ω1, ω2, and the corresponding target network The loss function of any value network is:

[0068]

[0069] The loss function of the policy network can be expressed as:

[0070]

[0071] where a = f θ (∈; s), ∈ is a noise random variable, and the action a is obtained by reparameterization technique, starting from a unit Gaussian distribution Sampling, then multiplying the sampled value by the standard deviation and adding the mean, and finally sampling. Finally, the mechanism of automatically adjusting the entropy regularization term is used to update α. The loss function of α is:

[0072]

[0073] Where -αlogπ(a|s) is the policy entropy, is the target entropy, which is generally set to the opposite of the action dimension, i.e., -dim(A). The phase of the strategy output designed in this paper is actually periodic, so a sine function is used instead of tanh during compression to reflect the periodicity. At the same time, in order to make the network better trained, the output is also scaled to an appropriate range, resulting in formula (2).

[0074] Step 340: Design the strategy network and value network. The construction of the strategy network and value network in this chapter is shown in the attached figure. Figure 3 As shown in Figure 2, both can be divided into an information extraction module and a decision module. The information extraction module consists of an encoder. The input to the information module is the single-step state observation matrix, and the output is the information extraction code of the state. This output serves as part of the input to the decision module. The decision module consists of a decoder followed by a deep neural network. The decoder input is divided into two parts: the output of the information extraction module, which serves as its key and value input, and the all-one vector or the action to be evaluated, which serves as its query input. Specifically, for the policy network, the decoder input is an all-one vector. Its output is then passed through a deep neural network and finally connected to two output heads, which output the mean μ and standard deviation σ of the Gaussian random policy, respectively. Finally, after normalization and resampling, a continuous-valued encoded action vector and its corresponding probability distribution are obtained. For the value network, the decoder input is the action to be evaluated, which is linearly transformed by a deep neural network module. Finally, the entire deep neural network module outputs the value estimate q(s,a) of the state-action pair input by the entire network.

[0075] Step 350: Set training parameters to train the value network and the policy network. Figure 2 As shown in (c), the number of chirps in a frame signal is set to 128, so the dimension of the agent action is also 128. k The dimension of the state is 131. In order to achieve multi-head attention, we will add another bit before the first bit of each state vector and fill it with 0, so that the dimension of the state is increased to 132. The parameter settings of the policy network are as follows: Figure 2 As shown in (d), "Linear Layer 1" is the first layer of the deep neural network. Its input is the output of the decoder. The activation function it uses is ReLU, followed by "Linear Layer_Mean" and "Linear Layer_Variance". "Linear Layer_Mean" directly outputs the mean of the Gaussian distribution. The output of "Linear Layer_Variance" also needs to go through the softplus activation function to adjust the output value range to positive numbers as the standard deviation of the Gaussian distribution. The parameter settings of the value network are shown in the attached figure. Figure 2(e) "Linear Layer 0" is a linear transformation layer preceding the decoder. It processes the action to be evaluated and adjusts its vector length to serve as the decoder's cross-attention query input. To fully preserve the action information, no activation function is used after "Linear Layer 0" for nonlinear transformation. The input of the deep neural network is the decoder output. The first fully connected layer, "Linear Layer 1," uses the ReLU activation function. The second fully connected layer, "Linear Layer 2," directly outputs a single value, which serves as the value estimate of the state-action pair input to the entire value network.

[0076] Step 360: Evaluate the interference suppression effect of the trained value network and policy network. This paper uses cumulative rewards and interference suppression index to reflect the effectiveness of coherent interference suppression. We define the interference suppression index as shown in formula (3).

[0077] in, They represent one round, that is, the numerical smoothing of the number of missed detections and false targets obtained by one training simulation, w md , w ft They represent the weights of missed target detection and false target detection, respectively.

[0078] The following explains some of the result graphs obtained in this example scenario.

[0079] Attachment Figure 4 The cumulative reward and interference suppression index change with the number of training rounds during the training process of the present invention. The agent in this paper was trained for a total of 20,000 rounds, a total of 400,000 time steps. During the training, this paper used the Doppler frequency shift coding method (comparison method 1) and the fixed discrete code table iterative optimization method (comparison method 2) to compare with the proposed inter-pulse optimization coding strategy. The change of cumulative reward with the number of training rounds is shown in the attached figure. Figure 4 As shown in (a), the baseline in the figure represents the average reward value that can be obtained by using the random inter-pulse coding method for anti-interference and based on the reward function designed in this paper. It can be seen from the figure that the average reward value of random inter-pulse coding is 0.39. The other three curves in the figure represent the changes in the cumulative rewards obtained by using different inter-pulse optimization coding strategies as the number of training rounds increases. It can be found that in the scenario designed in this experiment, the cumulative rewards of the proposed inter-pulse optimization coding strategy continue to increase with the training rounds, and the value of the cumulative rewards is much greater than that of random inter-pulse coding and the other two coding methods. In addition, the changes in the anti-interference index with the training rounds are shown in the attached figure. Figure 4 (b) As shown in the attached Figure 4(b)It can be seen that the anti-jamming index of the proposed method is also increasing with the increase of training rounds, and the maximum value of the anti-jamming index reaches 0.95. Therefore, in the simulated scenario in this paper, the anti-coherent jamming ability of the proposed inter-pulse optimization method is higher than that of the other encoding methods.

[0080] Figure 8 is a diagram showing the anti-jamming index of the proposed method in the test process of the present application with the change of training rounds. Figure 5 Figure 9 is a diagram showing the anti-jamming index of the proposed method in the test process of the present application with the change of training rounds. Figure 5 Figure 10 is a diagram showing the anti-jamming index of the proposed method in the test process of the present application with the change of training rounds.

[0081] Figure 11 is a diagram showing the anti-jamming index of the proposed method in the test process of the present application with the change of training rounds. Figure 6 Figure 12 is a diagram showing the anti-jamming index of the proposed method in the test process of the present application with the change of training rounds. Figure 6 (a)It can be seen that the anti-jamming index of the proposed method is also increasing with the increase of training rounds, and the maximum value of the anti-jamming index reaches 0.95. Therefore, in the simulated scenario in this paper, the anti-coherent jamming ability of the proposed inter-pulse optimization method is higher than that of the other encoding methods. Figure 6 (b)It can be seen that the anti-jamming index of the proposed method is also increasing with the increase of training rounds, and the maximum value of the anti-jamming index reaches 0.95. Therefore, in the simulated scenario in this paper, the anti-coherent jamming ability of the proposed inter-pulse optimization method is higher than that of the other encoding methods. Figure 6 (c)It can be seen that the anti-jamming index of the proposed method is also increasing with the increase of training rounds, and the maximum value of the anti-jamming index reaches 0.95. Therefore, in the simulated scenario in this paper, the anti-coherent jamming ability of the proposed inter-pulse optimization method is higher than that of the other encoding methods.

Claims

1. A method for suppressing interference of automotive millimeter-wave radar based on slow-time phase coding, characterized in that: include: Establish a Markov decision process for automotive millimeter-wave radar in the scenario of coherent interference suppression; Derive the reward function of the Markov decision process in the interference suppression scenario; design the observation matrix of the Markov decision process in the coherent interference suppression scenario based on the accumulated observations of multiple time steps; design a reinforcement learning algorithm based on the inter-pulse phase encoding scenario to obtain action sampling output that satisfies a reasonable probability distribution; design the policy network and value network based on the problem of partially observable environment; Set training parameters, train the network, use the trained network as the decision output for coherent interference suppression, and evaluate the results.

2. The Markov decision process for establishing an automotive millimeter-wave radar to suppress coherent interference through inter-pulse coding according to claim 1, characterized in that: In the scenario of coherent interference suppression, the traditional random inter-pulse phase coding method is remodeled into an optimized inter-pulse phase coding method based on reinforcement learning, and the phases corresponding to the 128 FM pulses contained in a frame signal are converted into the optimal inter-pulse phase coding method. The action space is defined as a continuous value. Through the optimization iteration of the reinforcement learning algorithm, each decision will output a 128-dimensional phase-optimized coded action vector per frame of radar signal.

3. The reward function in the Markov decision process according to claim 1, characterized in that Based on the constant false alarm detection and peak aggregation results of the range-Doppler image of the echo signal, the formula for deriving the reward function is: where n ft , n md The number of false targets and the number of missed targets are obtained by performing constant false alarm detection and peak aggregation on the echo signals received by millimeter waves.

4. The measurement matrix in the Markov decision process for designing a coherent interference suppression scenario according to claim 1, characterized in that: Based on the constraint that the state is partially observable in a real electromagnetic environment, multi-step observation information is added to the observation matrix to construct the observation matrix O k , let the modulus value of the distance-Doppler map at the kth time step be expressed as RDM k , whose component with distance dimension index i and Doppler dimension index j is expressed as In RDM k Based on the distance dimension and Doppler dimension, two one-dimensional constant false alarm detection and peak aggregation are performed respectively. Assume that n targets are finally detected, including real targets and false targets. The matrix obtained by peak aggregation is PC k , whose component with distance dimension index i and Doppler dimension index j is Assume that in the distance dimension, there are m distance units that detect peaks, and the indexes of these distance units are {r1, r2, ..., r m }, and the number of targets finally detected at these distances is {n1,n2,...,n m }.

5. According to claim 1, a reinforcement learning algorithm is designed to derive action sampling outputs that satisfy a reasonable probability distribution, characterized in that: The action probability density function is obtained using the following formula: Where π(a|s) represents the final output action probability density function, μ(u|s) represents the action probability density function before correction, u represents the action obtained by sampling the action probability density function, a=bsin(u), indicating that a is the compressed phase action output, dim(A) is the overall dimension of the action, and b represents the scaling factor.

6. According to claim 1, in combination with the problem of partially observable environment, the strategy network and value network are designed, characterized in that: In the context of partially observable inter-pulse optimized coding interference suppression, the transformer network architecture is introduced. In order to maintain the causality between adjacent codes, the word embedding and position encoding modules in the traditional network are discarded, and value evaluation and strategy optimization are directly achieved through encoder-decoder coupling.

Images