Intelligent passive monitoring method based on reinforcement learning

Abstract

The present invention discloses an intelligent passive monitoring method based on reinforcement learning, comprising the following steps: at the listener end, based on the equivalent channel and interference covariance matrix that the listener can pre-calculate, using a first DQN to generate an analog beamforming matrix, and inputting the analog beamforming matrix into a minimum mean square error module to calculate a digital beamforming matrix; using the analog and digital beamforming matrices to calculate the signal-to-interference-noise ratio of the received signal at the listener end, and using the signal-to-interference-noise ratio as the input of a second DQN; based on the listener's position, power level, and signal-to-interference-noise ratio information of the received signal, using a second DQN to generate the listener's moving direction and speed, when the power level reaches the critical level required for returning, the listener returns to the base to recharge; repeating the above learning process, the listener will eventually learn a trajectory with the highest average monitoring rate. The present invention ensures that the listener achieves a higher monitoring rate and successfully implements monitoring in a non-cooperative environment.

Description

Technical Field

[0001] The present invention belongs to the field of communication engineering, and specifically relates to an intelligent passive monitoring method based on reinforcement learning, and more specifically to an intelligent passive monitoring design method in a large-scale MIMO-OFDM (Multi Input Multi Output-Orthogonal Frequency Division Multiplexing) system based on reinforcement learning. Background Art

[0002] To prevent crime and terrorist attacks, government agencies need to lawfully monitor suspicious communication links to detect anomalous behavior.

[0003] With the rapid development of computer and wireless communication technologies, two main types of wireless monitoring techniques have emerged: active monitoring and passive monitoring. In passive monitoring, legitimate monitors maintain radio silence and monitor suspicious connections. Active monitoring, on the other hand, requires monitors to intentionally broadcast jamming signals to reduce the suspicious user's communication rate and increase their detection rate.

[0004] However, some existing monitoring schemes assume that the monitor has prior knowledge of the suspicious user's communication link, which is unrealistic. In 5G application scenarios, millimeter wave (mm-wave) is always combined with massive MIMO to provide high beamforming gain. The dense beam directivity of massive MIMO enhances the confidentiality of communications between the base station (BS) and the suspicious user. When the monitor and the suspicious user are not within the coverage of the same beam, both existing passive and active monitoring methods have difficulty achieving good performance. Therefore, in massive MIMO systems with directional beams, finding a suitable monitoring position is crucial. When the channel and the position of the suspicious user are unknown and change over time, finding the best monitoring position in a practical system is very time-consuming. In order to reduce the search time in practical applications, we adopt a strategy in which the monitor learns the mobile trajectory with the largest average monitoring rate offline and then applies it online.

[0005] Recently, several papers have investigated the application of reinforcement learning to optimize drone trajectories. The advantage of reinforcement learning algorithms is that the agent can achieve model-free learning, obtaining necessary feedback from the environment to optimize its execution strategy. In most papers on using reinforcement learning to optimize drone trajectories, the drone acts as a service provider, and its trajectory is optimized in a cooperative environment. Unlike most existing work, the drone's trajectory must be optimized in a non-cooperative environment as a listener, as shown in Figure 2. Summary of the Invention

[0006] Purpose of the invention: In response to the defects of the existing technology, the present invention proposes an intelligent monitoring method for large-scale MIMO-OFDM systems based on reinforcement learning to ensure that the monitor obtains a higher monitoring rate and successfully implements monitoring in a non-cooperative environment.

[0007] Technical Solution: A reinforcement learning-based intelligent monitoring method and performance analysis for massive MIMO-OFDM systems, including the following steps:

[0008] (1) At the listener side, based on the equivalent channel and interference covariance matrix that can be pre-calculated by the listener, the first DQN (Deep Q Network) is used to generate an analog beamforming matrix. The generated analog beamforming matrix is ​​input into the minimum mean square error (MMSE) module to calculate the digital beamforming matrix.

[0009] (2) calculating the signal-to-interference-plus-noise ratio (SINR) of the received signal at the listener using the analog and digital beamforming matrices, and using the SINR as input to a second DQN;

[0010] (3) Based on the location, power level, and signal-to-interference-noise ratio (SINR) of the listener, a second DQN is used to generate the direction and speed of movement of the listener. When the power level reaches the critical level required for returning, the listener returns to the base for charging.

[0011] (4) Repeat the above learning process, and eventually the monitor will learn a trajectory with the highest average monitoring rate.

[0012] Furthermore, the step (1) includes the following steps:

[0013] 1) On the listener side, use the first DQN based Generate the simulated beamforming matrix a1(t) of the listener at time t = {W E,rf (t)}, where s1(t) represents the state space, H E,k (t)f k (t) represents the equivalent composite channel of the kth subcarrier between the base station and the monitor, c k (t) represents the covariance matrix of interference from other base stations received by the monitor on the kth subcarrier, represents the set of all subcarriers assigned to suspicious users; a1(t) represents the action space, W E,rf (t) represents the listener's selection from the candidate basis vector The simulated beamforming matrix selected in ;

[0014] 2) When the listener outputs the simulated beamforming matrix W through the first DQN E,rf (t), WE,rf (t) is substituted into the minimum mean square error algorithm module to calculate the digital beamforming matrix of the monitor end for the kth subcarrier for where X k (t) = W E,rf (t)H E,k (t)f k (t), p k is the transmitted signal power, σ k is the noise power, and the superscript H indicates the conjugate transpose.

[0015] Furthermore, the step (2) includes the following steps:

[0016] Using the analog beamforming matrix W E,rf (t) and the digital beamforming matrix The signal-to-interference-noise ratio of the received signal at the kth subcarrier at the monitor end is calculated as Where I represents the identity matrix, and As the input of the second DQN;

[0017] Furthermore, the step (3) includes the following steps:

[0018] On the listener side, a second DQN based Generate the movement direction and speed of the listener a2(t) = {θ(t), v(t)}, where a2(t) represents the action space of the second DQN, s2(t) represents the state space of the second DQN, and γ kThe signal-to-interference-plus-noise ratio of the received signal on the \(k\)-th subcarrier at the listener side at time \(t\) is denoted as \((t)\), \(x(t)\) and \(y(t)\) jointly represent the position information of the listener at time \(t\), \(E(t)\) represents the remaining battery power of the listener at time \(t\), \(\theta(t)\) and \(v(t)\) respectively represent the moving direction and distance of the listener, and \(x(t)\), \(y(t)\), \(\theta(t)\) and \(v(t)\) satisfy the relationships \(x(t + 1)=x(t)+v(t)T\cos(\theta(t))\) and \(y(t + 1)=y(t)+v(t)T\sin(\theta(t))\), where \(T\) represents the duration of each time slot, and \(x(t + 1)\) and \(y(t + 1)\) jointly represent the position information of the listener at time \(t + 1\); there are 8 directions for the listener to move forward each time, namely upward, downward, left, right, upper left, lower left, lower right, and upper right; in addition, the listener can also choose to hover to listen at the original position, at this time \(v(t)=0\), and when the listener hovers, the power consumption is less than that when moving; the remaining battery power of the listener satisfies \(E(t + 1)=\max(0,E(t)-B(t)T)\), where \(B(t)\) represents the power consumption of the listener at time \(t\), and \(\max\) represents finding the maximum value; due to the limitation of the battery capacity, the listener needs to decide whether to move to a new position or hover at the original position to listen at each time \(t\), and at the same time ensure that there is enough power to return to the base, so the goal of the listener's learning is to find a trajectory with the maximum average listening rate.

[0019] Furthermore, the step (4) includes the following steps:

[0020] i After the second DQN outputs \(a2(t)\), calculate the reward \(r\) for this action t \(=\rho R0+(1-\rho)R(t)\), where \(R0\) represents the communication rate between the suspicious user and the base station, represents the channel capacity of the listening channel, \(\rho\) represents the listening success indication parameter, when \(R(t)\geq R0\), \(\rho = 1\), when \(R(t)<R0\), \(\rho = 0\);

[0021] ii In each learning episode, \(r\) t is simultaneously used as the reward for the two DQNs at each time \(t\). The listener starts from the base station, and the goal is to learn a path that maximizes the reward. When the remaining battery power of the listener can only meet the requirement of returning to the base, this learning episode ends and the next learning episode starts.

[0022] Beneficial Effects: The present invention is suitable for monitoring suspicious user communications in 5G massive MIMO-OFDM system scenarios. Through the design of a mobile monitor, it can effectively address the issue of enhanced confidentiality in communications between base stations and suspicious users caused by the strong beam directivity of massive MIMO. The intelligent monitoring solution proposed in the present invention has the advantages of high convergence speed and high average monitoring rate. It also solves the problem that traditional monitoring solutions require prior knowledge of the communication link between the base station and suspicious users. At the same time, a method for reducing the dimensionality of the action space using a serial DQN network is proposed, which improves the convergence speed of the monitor training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a diagram of an application scenario of intelligent passive monitoring in a massive MIMO-OFDM system of the present invention;

[0024] Figure 2 This is a structural diagram of a serial DQN-based listener receiving end of the present invention;

[0025] Figure 3 is a learning rate curve diagram of the monitor of the present invention under different learning algorithm schemes;

[0026] Figure 4 is a graph showing the average listening rate of the monitor of the present invention at different initial power levels;

[0027] Figure 5 It is a moving trajectory diagram of the monitor of the present invention in different training rounds. DETAILED DESCRIPTION

[0028] This invention proposes a reinforcement learning-based intelligent monitoring scheme for massive MIMO-OFDM systems, building upon traditional monitoring designs. This scheme maximizes the average monitoring rate of the monitor while meeting the monitor's power constraints. Computer simulations demonstrate the effectiveness of this monitoring scheme. At the monitor receiver, we jointly optimize the monitor's receive beamforming vector and movement direction. Assuming that the mobile monitor cannot accurately know incoming information between the base station and the suspicious user, we employ a deep reinforcement learning-based solution and propose a serial neural network-based monitor receiver design and estimation optimization scheme. In this scheme, the receiver first uses a DQN (Deep Query Network) to obtain the mapping between system state and analog beamforming vectors. Then, digital beamforming vectors are designed using MMSE (Multiple Multiple Sequencing) algorithms. After obtaining the analog and digital beamforming vectors, we perform dimensionality reduction on the system action space and use a DQN (Multiple Multiple Sequencing Network) to obtain the mapping between system state and movement direction. This approach ensures that the mobile monitor learns the optimal movement trajectory while meeting its power constraints.

[0029] The embodiments of the present invention are described in detail below with reference to the accompanying drawings:

[0030] Application scenarios of the present invention are as follows Figure 1 As shown in the figure, a suspicious user is communicating with a base station. Considering a multi-cell scenario, the mobile listener is also subject to interference from other nearby base stations. The mobile listener's goal is to monitor the communication information sent by the base station to the suspicious user. The key feature of this system is that the listener has no prior knowledge of the suspicious user's channel and location. Instead, the mobile listener uses a cascaded DQN network to generate its optimal receive beamforming vector, movement direction, and movement speed.

[0031] like Figure 2 As shown, in the intelligent monitoring solution of the present invention, the number of base station transmitting antennas is N t , the number of RF chains is N rf , where N rf <N t , the number of receiving antennas of the terminal is N U,r , the number of RF links is 1. Because the base station uses a large-scale antenna array, the communication beam between the base station and the suspicious user is very narrow. When the listener is not within the beam coverage, passive monitoring becomes very difficult. In addition, assuming that the listener does not know the location of the suspicious user and the channel state information between the base station and the suspicious user in advance, active monitoring is also relatively difficult. In this invention, a monitoring solution using an intelligent mobile listener is considered. The listener can find the optimal monitoring position by learning during the movement process.

[0032] The present invention adopts hybrid beamforming to solve the problem of high hardware cost of large-scale MIMO. In addition, in order to avoid interference, it is considered to assign different subcarriers to different users. Assigned to a suspicious user, the base station first uses a digital precoder For data symbol s k (t) precoding and processing the data using an analog precoder after the inverse fast Fourier transform (IFFT) Finally, a sending signal is generated.

[0033] Assume the number of monitor receiving antennas is N E,rf , the number of RF chains is N E,rf , and also uses hybrid beamforming method to monitor Since millimeter wave base stations are densely deployed, the monitor will also be interfered by other base stations around it. At time t, the received signal on the kth subcarrier is as follows

[0034] y E,k (t) = H E,k (t)f k(t)s k (t)+I E,k (t)+n E,k (1) in represents the channel matrix of the kth subcarrier between the monitor and the base station, I E,k (t) is the interference from other surrounding base stations on the kth subcarrier, I is the identity matrix.

[0035] Signals received by the listener First, through a common analog beamforming matrix Among them, W E,rf (t) can be obtained from a predefined discrete codebook This process can be implemented in a dynamic environment through deep reinforcement learning. Afterwards, the cyclic prefix of the signal is removed and N E,rf K-point fast Fourier transform is used to demodulate the signal. A low-dimensional digital beamforming matrix Combine the signals from all subcarriers, where Representing a collection The output of the kth beamformer is

[0036]

[0037] Assume that the listener can estimate the equivalent channel H based on the known reference signal E,k (t)f k (t), at time t, the listener can use this channel information to design the receive beamformer and calculate the signal-to-interference-noise ratio of the kth subcarrier

[0038]

[0039] Among them C k (t) represents I E,k The covariance matrix of (t), p k is the signal transmission power of the kth subcarrier of the base station, and the superscript H represents the conjugate transpose. Based on formula (4), the channel capacity of the monitoring link at time t is as follows

[0040]

[0041] In the intelligent monitoring solution of the present invention, let E(t) and B(t) represent the battery level of the monitor and the power consumption at time t respectively. B(t) includes the power consumption of the monitor for moving (or hovering) and monitoring, which are respectively represented by B v (t) and B r(t) is denoted as. Let the speed of the listener at time t be v(t). When v(t) = 0, B v (t) = b0, otherwise B v (t) = b1, where b0 and b1 satisfy b0 < b1, indicating that less power is consumed when the listener hovers and listens. The change in power satisfies E(t + 1) = max(0, E(t) - B(t)T), where T represents the duration of a time slot. Due to power limitations, at each time slot, the listener needs to decide whether to spend more power to move to a better listening position. At the same time, it is also necessary to ensure that the mobile listener (such as a drone) has enough power to return to the base. Therefore, the optimization objective is to maximize the average discounted "reward" sum under the constraints of power and moving speed, and this "reward" value is proportional to the listening rate.

[0042] To accelerate the learning speed of the listener in the early stage, the expression of the listening rate is defined as

[0043]

[0044] In the intelligent listening scheme of the present invention, let the position and moving direction of the mobile listener at time t be (x(t), y(t)) and θ(t) respectively. When the mobile listener moves at a constant speed, v(t) = v, and when the mobile listener is stationary, v(t) = 0, where v represents a fixed speed value. Therefore, the joint optimization problem of the receiving beamformer and the moving direction and speed of the mobile listener can be expressed as

[0045]

[0046] where α ∈ [0, 1) is the discount factor, represents the distance between the mobile listener and the charging base at time t + 1, θ represents the angle between the moving direction of the listener and its head direction, x(t + 1) = x(t) + v(t)Tcos(θ(t)), y(t + 1) = y(t) + v(t)Tsin(θ(t)), is the set of moving directions. The first part on the left side of constraint (7a) represents the power consumed by the listener during movement and listening, and the second part represents the power required for the listener to return to the charging base. Constraint (7a) ensures that the listener has enough power to return to the charging base.

[0047] Obviously, the optimal solution to optimization problem (7) is for the listener to find a suitable listening position, stop, and listen. The difficulty is that it is difficult to have sufficient time for training in an adversarial environment. When the suspicious user's position changes, the listener's trajectory may also change. Therefore, during the training phase, the suspicious user's position changes randomly to improve the stability of the strategy learned by the listener. Problem (7) can be modeled as a Markov process (MDP). When the system dynamics are unknown, reinforcement learning algorithms can be applied to solve problem (7).

[0048] For the reinforcement learning algorithm, let the system state be Action When the listener moves, its position and power will also change, which also leads to changes in the channel and interference. In addition, since the action space is large, it is difficult to achieve good performance by directly applying DQN to find the optimal mapping relationship between s(t) and a(t). In order to reduce the dimensions of the state space and action space, this paper proposes a listener receiver design based on cascaded DQN. Figure 2 As shown, the listener first decides the action a1(t) = {W E,rf (t)}, then the listener is based on Determine action a2(t) = {θ(t), v(t)}. The first DQN module is combined with the MMSE module to generate the simulated beamforming matrix W based on s1(t) E,rf (t) and the digital beamforming matrix To increase γ k (t), first use the minimum mean square error algorithm To calculate

[0049] For a given W E,rf (t) and The solution to the minimum mean square error is as follows

[0050]

[0051] where X k (t) = W E,rf (t)H E,k (t)f k (t). Next, substitute (8) into (4) and (5) to get the expression for R(t), and then use the first DQN to get from s1(t) Choose the best W E,rf (t), which maximizes the reward function in (6).

[0052] Get W E,rf (t) can then be calculated And use it as part of the input of the second DQN, and the second DQN is used to Determine the moving direction θ(t) and speed v(t). Then, problem (7) can be simplified to

[0053]

[0054] For problem (9), the system states and actions are simplified to With a2(t) = {θ(t), v(t)}, we can see that the dimensions of the action space and state space of the first and second DQNs are smaller when considering the serial scheme. In each training round, the listener starts from the base and aims to learn a path that maximizes the reward. The training round ends when the constraint (7a) is not satisfied.

[0055] like Figure 3 As shown, the average listening rate of the simulation The learning curve, where T m represents the time interval between a mobile listener leaving the base and returning to the base. The black and light gray curves represent the learning curves for the average listening rate for the tandem DQN scheme and the single DQN scheme, respectively. Clearly, due to the reduced dimensionality of the action space, the proposed scheme achieves faster convergence and a higher listening rate than the single DQN scheme. The dark gray curve is the learning curve for the average listening rate when the suspicious user's position is fixed. Clearly, this curve exhibits less fluctuation than the serial DQN scheme. If the suspicious user's position remains unchanged, allowing the listener sufficient learning time in a non-cooperative environment, the listener will learn an optimal listening strategy for this fixed user after approximately 300 rounds, consistent with theoretical expectations.

[0056] like Figure 4 As shown in the figure, the average listening rate curves under different movement and listening strategies are simulated at different initial power levels. When the position of the suspicious user is fixed, the optimal strategy learned by the listener based on the tandem DQN scheme is: the listener finds an optimal position and listens until it returns. The triangle and square solid lines represent the average listening rates based on the optimal strategy when the suspicious user is at (30m, -10m) and (70m, 10m), respectively. In actual application scenarios, a stable strategy trained by randomizing user positions is used to guide the movement and listening of the listener, and the corresponding listening rate is represented by a dotted line. It is obvious that when the listener uses this stable strategy, it can still find its ideal listening location and achieve performance close to that of the optimal strategy, which is consistent with theoretical expectations.

[0057] like Figure 5As shown in the figure, the simulated listener's movement trajectory after the 10th and 1000th rounds of learning, respectively. The suspicious user is fixed at the position (70m, 10m). It can be clearly seen that after a certain number of rounds of learning, the listener can learn a suitable trajectory and hover at a certain listening position until returning, which is consistent with the theoretical expectation.

[0058] Simulations have shown that the reinforcement learning-based intelligent monitoring method for large-scale MIMO-OFDM systems proposed in this invention can learn a suitable movement path to complete monitoring when the monitor cannot know the environmental information and the actual location of the suspicious user in advance. The proposed algorithm is used in the simulation scenario to solve the performance of the monitor, thereby verifying the correctness of the theory.

Claims

1. An intelligent passive monitoring method based on reinforcement learning, comprising the following steps: (1) At the listener, based on the equivalent channel and interference covariance matrix that can be pre-calculated by the listener, the first DQN is used to generate an analog beamforming matrix, and the analog beamforming matrix is ​​input into the minimum mean square error module to calculate the digital beamforming matrix; (2) Calculating the signal-to-interference-plus-noise ratio (SINR) of the received signal at the listener end using the analog beamforming matrix and the digital beamforming matrix, and using the SINR as the input of the second DQN; (3) Based on the location, power level, and signal-to-interference-noise ratio (SINR) of the listener, a second DQN is used to generate the direction and speed of movement of the listener. When the power level reaches the critical level required for returning, the listener returns to the base for charging. (4) Repeat the above learning process, and finally the monitor will learn a trajectory with the highest average monitoring rate; the step (1) includes the following steps: 1) On the listener side, use the first DQN based on Generate the simulated beamforming matrix a1(t) of the listener at time t = {W E，rf (t)}, where s1(t) represents the state space, H E，k (t)f k (t) represents the equivalent composite channel of the kth subcarrier between the base station and the monitor, C k (t) represents the covariance matrix of interference from other base stations received by the monitor on the kth subcarrier, represents the set of all subcarriers assigned to suspicious users; a1(t) represents the action space, W E，rf (t) represents the listener's selection from the candidate basis vector The simulated beamforming matrix selected in ; 2) When the listener outputs the simulated beamforming matrix W through the first DQN E，rf (t), W E，rf (t) is substituted into the minimum mean square error algorithm module to calculate the digital beamforming matrix of the monitor end for the kth subcarrier for where X k (t) = W E，rf (t)H E，k (t)f k (t), p k is the transmitted signal power, σ 2 is the noise power, and the superscript H represents the conjugate transpose; the step (2) includes the following steps: Using the analog beamforming matrix W E，rf (t) and the digital beamforming matrix Calculate the signal-to-interference-to-noise ratio γ of the received signal on the kth subcarrier at the monitor end at time t k (t) is Where I represents the identity matrix, and As the input of the second DQN; the step (3) includes the following steps: On the listener side, a second DQN based Generate the movement direction and speed of the listener a2(t) = {θ(t), v(t)}, where a2(t) represents the action space of the second DQN, s2(t) represents the state space of the second DQN, x(t) and y(t) together represent the location information of the listener at time t, E(t) represents the remaining power of the listener at time t, θ(t) and v(t) represent the movement direction and distance of the listener respectively, x(t), y(t), θ(t) and v(t) satisfy the relationship x(t+1) = x(t) + v(t)Tcos(θ(t)) and y(t+1) = y(t) + v(t)fsin(θ(t)), where T represents the duration of each time slot, x(t+1) and y(t+1) together represent the position of the listener at time t+1 Information; the listener has 8 possible directions of movement each time, namely up, down, left, right, upper left, lower left, lower right, and upper right; in addition, the listener can choose to hover to listen at the original position, at which time v(t) = 0. When the listener hovers, the power consumed is less than the power consumed when moving; the remaining power of the listener satisfies E(t+1) = max(0, E(t)-B(t)T), where B(t) represents the power consumed by the listener at time t, and max represents the maximum value; due to the limitation of battery capacity, the listener needs to decide at each time t whether to move to a new position or hover at the original position for listening, while also ensuring that there is enough power to return to the base, so the goal of the listener learning is to find a trajectory with the highest average listening rate.

2. The intelligent passive monitoring method based on reinforcement learning according to claim 1 is characterized in that: The step (4) includes the following steps: i After the second DQN outputs a2(t), calculate the reward r for this action t = ρR0 + (1 - ρ)R(t), where R0 represents the communication rate between the suspicious user and the base station represents the channel capacity of the listening channel, ρ represents the listening success indication parameter. When R(t) ≥ R0, ρ = 1; when R(t) < R0, ρ = 0; ii In each learning round, r t At each time t, as a reward for both DQNs, the listener starts from the base station with the goal of learning a path that maximizes the reward. When the remaining power of the listener is only enough to return to the base, this round of learning ends and the next round of learning begins.

Images