Reinforcement learning-based mobile antenna anti-active eavesdropping secure communication method and system

Abstract

The application discloses a mobile antenna anti-active eavesdropping security communication method and system based on reinforcement learning, relates to the technical field of communication, and comprises the following steps: acquiring base station configuration and a legal user channel; monitoring uplink signals to identify active eavesdroppers and measure their angles of arrival, and estimating the positions of the eavesdroppers in combination with the height of the base station; initializing the position of the mobile antenna by using circle filling; constructing a double-strategy network architecture, wherein a first network optimizes the antenna position, a second network optimizes beam forming, and the two networks are alternately iterated; constructing a reward function with the minimum secrecy rate of the legal user, updating the parameters of the two networks respectively, and outputting the joint strategy of the eavesdropping-resistant antenna position and beam forming until the secrecy rate converges. The application dynamically suppresses active interference without relying on accurate eavesdropping channel prior information by utilizing the degrees of freedom of the mobile antenna and the decision-making ability of reinforcement learning, improves the system secrecy rate, and is suitable for the scene of resisting malicious eavesdropping on the base station side.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and specifically to a mobile antenna anti-active eavesdropping secure communication method and system based on reinforcement learning. Background Technology

[0002] Movable antennas can make fuller use of the continuous spatial domain degrees of freedom, allowing antenna elements to move flexibly within a given area, thereby altering the channel response between the transmitter and receiver. By moving the antenna elements to locations with higher channel gain, the received signal power and signal-to-noise ratio are improved. In multipath propagation environments, the channel gain at different spatial locations may vary significantly. Movable antennas, through position optimization, can avoid deep fading locations and select locations with better channel conditions, validating their performance advantages in signal power enhancement. Furthermore, through movable antenna position optimization, the system can achieve interference suppression, flexible beamforming, and spatial multiplexing enhancement, thereby further improving the transmission performance of the wireless communication system.

[0003] However, in secure communication scenarios, active eavesdroppers can not only receive legitimate communication signals but also actively transmit jamming signals, interfering with legitimate communication links. This jamming signal reduces the quality of signals received by legitimate users and may force or induce the communication system to increase its transmission power and adjust its beam direction, thereby further enhancing the receivable signal strength of the eavesdropping link and reducing the system's security rate.

[0004] However, existing reinforcement learning-based secure communication methods using movable antennas are primarily geared towards passive eavesdropping scenarios. For example, eavesdroppers are modeled as passive nodes that only receive legitimate communication signals, and reinforcement learning methods are used to jointly optimize the position of the movable antenna and the transmission beamforming strategy to strengthen legitimate links and weaken eavesdropping links, thereby improving the system's security rate. However, these methods mainly focus on the passive eavesdropping model of "eavesdropping without interference," lacking systematic modeling and countermeasures against attacks by active eavesdroppers that simultaneously eavesdrop and interfere with transmissions. Furthermore, most secure communication methods assume that legitimate communication systems can accurately obtain prior information such as the eavesdropper's position and channel state. However, in real-world environments, this information is often difficult to obtain accurately. In scenarios where this prior information is missing or has significant measurement errors, the antenna position optimized by these methods may not be globally optimal, and may even lead to a severe deterioration in the signal quality at the legitimate receiver, failing to effectively guarantee the system's security rate.

[0005] Therefore, how to jointly optimize the position of movable antennas and beamforming strategies under the conditions of incomplete information from active eavesdroppers and the presence of active interference, so as to enhance legitimate communication links, suppress eavesdropping links, and reduce the impact of active interference on legitimate users, has become an urgent technical problem to be solved in the field of secure communication with movable antennas. Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes a mobile antenna anti-active eavesdropping secure communication method and system based on reinforcement learning. By constructing a dual-strategy deep reinforcement learning framework, the mobile antenna position and beamforming matrix are optimized separately, and collaborative iteration is performed with the minimum security rate as the reward function. This achieves a secure communication effect that maximizes the worst-case security rate of the system by dynamically adjusting the base station antenna layout and transmission strategy under active eavesdropping interference.

[0007] On the one hand, reinforcement learning-based mobile antenna anti-active eavesdropping secure communication methods include:

[0008] S1, obtain base station configuration parameters and legal user channel matrix, the configuration parameters including base station height, number of mobile antennas and mobile area range;

[0009] S2, monitor the received signals in the uplink communication frequency band, compare the characteristics of each received signal with the legitimate user channel matrix, determine that the mismatched signal is an illegitimate signal, extract the angle of arrival measurement data of the illegitimate signal, and identify the source of the illegitimate signal as an active eavesdropper;

[0010] S3, calculate the measurement location of the active eavesdropper based on the base station height and angle of arrival measurement data, and calculate the eavesdropping channel measurement value and the equivalent received interference power of the legitimate user based on the measurement location of the active eavesdropper;

[0011] Based on the number of mobile antennas and the range of movable area, the position of each mobile antenna is initialized using the circle filling method to obtain the initial mobile antenna position matrix;

[0012] S4, take the initial mobile antenna position matrix as the current mobile antenna position, and combine the legitimate user channel matrix, the active eavesdropper measurement position, the eavesdropping channel measurement value and the legitimate user equivalent received interference power into the first input state of the first strategy network.

[0013] S5, the first strategy network outputs the mobile antenna position offset based on the first input state and updates the mobile antenna position, and recalculates the legal user channel and eavesdropping channel measurement values ​​based on the updated mobile antenna position;

[0014] S6, combine the updated antenna position, the recalculated legitimate user channel, the eavesdropping channel measurement value, the active eavesdropper measurement position, and the legitimate user's equivalent received interference power into a second input state and input it into the second policy network;

[0015] S7, the second strategy network outputs the original beamforming matrix based on the second input state, performs power normalization on the original beamforming matrix to obtain the actual beamforming matrix, calculates the security rate of each legitimate user based on the actual beamforming matrix, constructs the first reward function corresponding to the mobile antenna position optimization and the second reward function corresponding to the beamforming optimization based on the minimum security rate, updates the parameters of the first strategy network based on the first reward function, updates the parameters of the second strategy network based on the second reward function, and iterates until the security rate converges, outputting the antenna position control strategy and beamforming strategy for countering active eavesdropping.

[0016] Furthermore, the positions of each mobile antenna are initialized using a circle-filling method, specifically as follows:

[0017] The number of small circles is determined based on the number of mobile antennas, and the radius of the small circles is determined based on the minimum spacing constraint between antennas. Within the movable area, the small circles are made to be non-overlapping and their radii are maximized. The center of each small circle is used as the initial position of each mobile antenna.

[0018] Furthermore, the location of the active eavesdropper is calculated based on the base station altitude and angle of arrival measurement data. The calculation formula is as follows:

[0019] ;

[0020] in, This is the azimuth measurement value; This is the measured pitch angle value; Indicates the height of the base station.

[0021] Furthermore, the original beamforming matrix is ​​normalized for power, and the calculation formula is as follows:

[0022] ;

[0023] in, To prevent extremely small positive numbers from being divided by zero; This is the normalized beamforming matrix; This is the original beamforming matrix; Indicates the maximum power of the base station; This represents the Frobenius norm of the matrix.

[0024] Furthermore, the confidentiality rate of each legitimate user is calculated based on the actual beamforming matrix, using the following formula:

[0025] ;

[0026] ;

[0027] ;

[0028] in, This indicates a value greater than 0; For the first The confidentiality rate of each legitimate user; For the first The received signal-to-interference-plus-noise ratio of each legitimate user; This indicates that the active eavesdropper is eavesdropping on the first... The received signal-to-interference-plus-noise ratio when there are valid user information; Indicates the first Channel vectors of each legitimate user; This represents the position matrix of the movable antenna at time t+1; Indicates that the base station is used to send to the first Beamforming vectors for information sent by legitimate users; Indicates that the base station is used to send to the first Beamforming vectors for information sent by legitimate users; This indicates that the jamming signal sent by the active eavesdropper is at the [number]th [time]. Received interference power measurements at each legitimate user location; Indicates the first Additive Gaussian noise power at the receiver of a legitimate user; This represents the eavesdropping channel vector of an active eavesdropper. This indicates the location measurement of the active eavesdropper; This represents the additive Gaussian noise power at the receiver of an active eavesdropper; This represents the conjugate transpose operation; Indicates a time index; J represents a legitimate user index; J represents a jamming signal; E represents an active eavesdropper.

[0029] Furthermore, with the minimum confidentiality rate Constructing the first reward function Second reward function The calculation formula is as follows:

[0030] ;

[0031] ;

[0032] ;

[0033] ;

[0034] in, The weighting coefficient representing the penalty for violating antenna spacing rules; The weighting coefficient represents the penalty for information leakage in the direction of the eavesdropping channel; This indicates the penalty for violating the minimum spacing requirement for mobile antennas; This indicates the penalty for information leakage in the direction of the eavesdropping channel; Indicates a time index; Indicates the number of mobile antennas; Indicates the number of legitimate users; Indicates a valid user index; This represents the eavesdropping channel vector of an active eavesdropper. This represents the conjugate transpose operation; Indicates that the base station is used to send to the first Beamforming vectors for information sent by legitimate users; This indicates the maximum transmit power of the base station; This represents the square of the second norm of a vector; It represents the smallest positive number that is not divisible by zero.

[0035] Furthermore, the parameters of the first policy network are updated based on the first reward function, and the parameters of the second policy network are updated based on the second reward function, specifically including:

[0036] Construct a first centralized Critic network and a second centralized Critic network, respectively corresponding to the first policy network and the second policy network. The first centralized Critic network obtains a first value measurement based on state information and calculates a first advantage function in combination with a first reward function, and updates the parameters of the first policy network based on the first advantage function. The second centralized Critic network obtains a second value measurement based on state information and calculates a second advantage function in combination with a second reward function, and updates the parameters of the second policy network based on the second advantage function. At the same time, update the parameters of the first centralized Critic network and the second centralized Critic network respectively.

[0037] On the other hand, a mobile antenna anti-active eavesdropping secure communication system based on reinforcement learning includes:

[0038] An initialization module is used to obtain base station configuration parameters and a legal user channel matrix. The configuration parameters include base station height, number of mobile antennas, and mobile area range.

[0039] The signal identification module is used to monitor the received signals in the uplink communication frequency band, compare the characteristics of each received signal with the legitimate user channel matrix, determine that the mismatched signal is an illegitimate signal, extract the angle of arrival measurement data of the illegitimate signal, and identify the source of the illegitimate signal as an active eavesdropper.

[0040] The initial matrix calculation module is used to calculate the measurement location of the active eavesdropper based on the base station height and angle of arrival measurement data, and to calculate the eavesdropping channel measurement value and the equivalent received interference power of the legitimate user based on the measurement location of the active eavesdropper.

[0041] Based on the number of mobile antennas and the range of movable area, the position of each mobile antenna is initialized using the circle filling method to obtain the initial mobile antenna position matrix;

[0042] The first state generation module is used to take the initial mobile antenna position matrix as the current mobile antenna position, and combine the legitimate user channel matrix, the active eavesdropper measurement position, the eavesdropping channel measurement value and the legitimate user equivalent received interference power into the first input state of the first policy network.

[0043] The antenna position update module is used by the first policy network to output the mobile antenna position offset based on the first input state and update the mobile antenna position, and to recalculate the measurement values ​​of the legitimate user channel and the eavesdropping channel based on the updated mobile antenna position.

[0044] The second state generation module is used to combine the updated antenna position, the recalculated legitimate user channel, the eavesdropping channel measurement value, the active eavesdropper measurement position, and the legitimate user's equivalent received interference power into a second input state and input it into the second policy network.

[0045] The strategy output module is used by the second strategy network to output the original beamforming matrix based on the second input state, perform power normalization on the original beamforming matrix to obtain the actual beamforming matrix, calculate the security rate of each legitimate user based on the actual beamforming matrix, construct a first reward function corresponding to mobile antenna position optimization and a second reward function corresponding to beamforming optimization based on the minimum security rate, update the parameters of the first strategy network based on the first reward function, update the parameters of the second strategy network based on the second reward function, and iterate until the security rate converges, and output the antenna position control strategy and beamforming strategy for combating active eavesdropping.

[0046] The present invention adopts the above technical solution and has the following beneficial effects:

[0047] (1) The present invention dynamically optimizes the position of the mobile antenna through the first strategy network and jointly optimizes the beamforming through the second strategy network. It also constructs a reward function with the minimum confidentiality rate of the legitimate user for iterative training. This can effectively suppress the interference signal sent by the active eavesdropper, maximize the confidentiality rate of the system in the worst case, and enhance communication security.

[0048] (2) The present invention extracts the angle of arrival measurement data of the active eavesdropper by uplink monitoring signal, estimates its position by combining the base station height, and then calculates the eavesdropping channel measurement value. It does not require channel statistics information or cooperative feedback from the eavesdropping end, is suitable for actual complex environments, and has good adaptability to changes in the position of the eavesdropper and the channel.

[0049] (3) This invention adopts a dual-strategy network architecture to control the antenna position and beamforming respectively, and updates the parameters through an independent reward function and a centralized Critic network to achieve alternating iteration and collaborative learning of the two sub-problems. It can make full use of the degrees of freedom and power resources of the mobile antenna, adjust the strategy in real time in a dynamic wireless environment, and ensure the continuous and secure communication of the system. Attached Figure Description

[0050] Figure 1 This is a flowchart of a reinforcement learning-based mobile antenna anti-active eavesdropping secure communication method according to an embodiment of the present invention.

[0051] Figure 2 This is a diagram of a mobile antenna anti-active eavesdropping secure communication system based on reinforcement learning, as described in an embodiment of the present invention.

[0052] Figure 3 This is a schematic diagram illustrating the reward changes during the training process in an embodiment of the present invention. Detailed Implementation

[0053] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0054] like Figure 1 As shown, the present invention provides a mobile antenna anti-active eavesdropping secure communication method based on reinforcement learning, comprising:

[0055] S1, obtain base station configuration parameters and legal user channel matrix, the configuration parameters include base station height, number of mobile antennas and mobile area range.

[0056] S2 monitors the received signals in the uplink communication frequency band, compares the characteristics of each received signal with the legitimate user channel matrix, determines that the mismatched signal is an illegitimate signal, extracts the angle of arrival measurement data of the illegitimate signal, and identifies the source of the illegitimate signal as an active eavesdropper.

[0057] Specifically, in this embodiment, the legitimate signal components in the received signal are eliminated based on known legitimate communication signals and legitimate user channel information to obtain a residual signal; the energy of the residual signal is calculated and compared with a preset interference detection threshold; when the residual signal energy is greater than the interference detection threshold, it is determined that an unexpected interference signal has been detected; further, the angle of arrival measurement data of the interference signal is extracted, and the source of the interference signal is identified as an active eavesdropper.

[0058] Specifically, to verify the feasibility of this method, in this embodiment, a movable antenna-assisted downlink secure transmission system is established. The system includes a base station, One legitimate user and one active eavesdropper. Base station configuration. One movable antenna is used; legitimate users are equipped with a single fixed-position antenna; active eavesdroppers are equipped with two fixed-position antennas, one for receiving secure signals from the base station and the other for transmitting jamming signals to legitimate users; a three-dimensional spatial coordinate system is established, with the base station located at an altitude of [missing information]. The reference location is: Legitimate users and active eavesdroppers are located in the ground area. Let the... The position of each movable antenna element within the two-dimensional moving area is represented as follows: The positions of all movable antennas form an antenna position matrix. , No. The three-dimensional global position of a movable antenna can be represented as: The estimated location of an active eavesdropper is represented as follows: The location of the active eavesdropper can be estimated from the angle of arrival, arrival delay, or characteristics of the received signal of the interference signal.

[0059] S3: Calculate the measurement position of the active eavesdropper based on the base station height and angle of arrival measurement data; calculate the eavesdropping channel measurement value based on the active eavesdropper measurement position; initialize the position of each mobile antenna using the circle filling method based on the number of mobile antennas and the range of the movable area to obtain the initial mobile antenna position matrix.

[0060] Specifically, the position of each mobile antenna is initialized using a circle-filling method, as follows:

[0061] The number of small circles is determined based on the number of mobile antennas, and the radius of the small circles is determined based on the minimum spacing constraint between antennas. Within the movable area, the small circles are made to be non-overlapping and their radii are maximized. The center of each small circle is used as the initial position of each mobile antenna.

[0062] Specifically, in this embodiment, the interference signal received by the base station from the active eavesdropper is processed, and the azimuth and elevation angles of each propagation path are measured using a two-dimensional MUSIC algorithm. Based on the angle measurement results, different propagation paths are separated, and the relative delay between different propagation paths is measured using the GCC-PHAT method. The earliest arrival path is determined as the Loss of Sight (LoS) path, and the azimuth and elevation angle measurements corresponding to the LoS path are obtained. Based on the azimuth and elevation angle measurements of the LoS path and the base station altitude, the measurement position of the active eavesdropper is calculated using the base station altitude and angle of arrival measurement data. The calculation formula is as follows:

[0063] ;

[0064] in, This is the azimuth measurement value; This is the measured pitch angle value; Indicates the height of the base station.

[0065] Specifically, initializing the position of movable antennas using the circle-filling method means treating each movable antenna as an equivalent small circle of equal radius, and the antenna position as the center of this small circle, arranging them within the movable area. A series of small circles with the largest possible radius, arranged so that they do not overlap. The center of each small circle is considered the initial position of the movable antenna. This circular filling method can provide uniformly distributed initial positions for all movable antennas within a limited moving area, and ensures that the initialized movable antenna positions meet the minimum spacing requirements between antennas. This initialization method avoids uneven antenna distribution and initial invalid exploration caused by random initialization.

[0066] Specifically, in estimating the location of an active eavesdropper, the base station uses the interference signal emitted by the active eavesdropper to estimate the angle of arrival, thereby obtaining an estimated azimuth angle value of the interference signal. and pitch angle estimates When the active eavesdropper is located on the ground plane, the base station further constructs an eavesdropping channel estimate based on the estimated location of the active eavesdropper and the channel model. (Note: The estimated values ​​in this application are the measured values.)

[0067] Specifically, in At any given time, the base station uses beamforming to simultaneously send... Send by a legitimate user These are independent information streams; the total signal transmitted by the base station is a linear superposition of all legitimate user signals. ,in For user information symbol vectors, To send to the Information symbols for a legitimate user. The beamforming matrix is ​​defined as follows: ,in For the first Beamforming vectors for legitimate users. The jamming signal emitted by the active eavesdropper is represented as: ,in This indicates the jamming transmission power of the active eavesdropper. This represents the normalized interference signal.

[0068] Specifically, the first A legitimate user receives signals transmitted from the base station:

[0069] ;

[0070] Wherein, the first term is the expected signal of the k-th legitimate user; the second term is the multi-user interference caused by the information flow of other legitimate users; the third term is the interference signal emitted by the active eavesdropper; and the fourth term is the... The received complex Gaussian white noise at each of the legal users satisfies zero mean and variance. .also, Indicates the base station to the The channel response between legitimate users is as follows:

[0071] ;

[0072] in, Indicates all movable antennas up to the 1st The launch site response matrix of a legitimate user, Indicates the base station to the The path response matrix of a legitimate user Indicates the first The received field response vector corresponding to a fixed-position antenna of a legitimate user.

[0073] S4, take the initial mobile antenna position matrix as the current mobile antenna position, and combine the legitimate user channel matrix, the active eavesdropper measurement position, the eavesdropping channel measurement value and the legitimate user equivalent received interference power into the first input state of the first strategy network.

[0074] S5, the first strategy network outputs the mobile antenna position offset based on the first input state and updates the mobile antenna position, and recalculates the legal user channel and eavesdropping channel measurement values ​​based on the updated mobile antenna position;

[0075] Specifically, in this embodiment, the first strategy network outputs the target position action of each mobile antenna based on the first input state, and maps the target position action to the absolute position coordinates of each mobile antenna within the preset mobile area according to the coordinate range of the preset mobile area, thereby obtaining the position of the mobile antenna in the next time time; based on the updated mobile antenna position, it recalculates the propagation distance difference and phase change corresponding to each propagation path between each mobile antenna and the legitimate user and the active eavesdropper, and reconstructs the field response vector accordingly, and further calculates the measurement values ​​of the legitimate user channel and the eavesdropping channel after the mobile antenna position is updated.

[0076] S6, the updated antenna position, the recalculated legitimate user channel, the eavesdropping channel measurement value, the active eavesdropper measurement position, and the legitimate user's equivalent received interference power are combined into a second input state and input into the second policy network.

[0077] S7, the second strategy network outputs the original beamforming matrix based on the second input state, performs power normalization on the original beamforming matrix to obtain the actual beamforming matrix, calculates the security rate of each legitimate user based on the actual beamforming matrix, constructs the first reward function corresponding to the mobile antenna position optimization and the second reward function corresponding to the beamforming optimization based on the minimum security rate, updates the parameters of the first strategy network based on the first reward function, updates the parameters of the second strategy network based on the second reward function, and iterates until the security rate converges, outputting the antenna position control strategy and beamforming strategy for countering active eavesdropping.

[0078] Specifically, in this embodiment, the original beamforming vectors corresponding to each legitimate user are used to form an original beamforming matrix. Power constraint processing is performed based on the total transmit power of the original beamforming matrix. When the total transmit power exceeds the preset maximum transmit power, normalization is performed; otherwise, the original beamforming matrix remains unchanged, thereby obtaining the actual beamforming matrix. The signal-to-interference-plus-noise ratio (SIR) of each legitimate user and the eavesdropper is calculated based on the actual beamforming matrix, and the security rate is further calculated.

[0079] Specifically, the power is normalized for the original beamforming matrix, and the calculation formula is as follows:

[0080] ;

[0081] in, To prevent extremely small positive numbers from being divided by zero; This is the normalized beamforming matrix; This is the original beamforming matrix; Indicates the maximum power of the base station; This represents the Frobenius norm of the matrix.

[0082] Specifically, the confidentiality rate of each legitimate user is calculated based on the actual beamforming matrix, using the following formula:

[0083] ;

[0084] ;

[0085] ;

[0086] in, This indicates a value greater than 0; For the first The confidentiality rate of each legitimate user; For the first The received signal-to-interference-plus-noise ratio of each legitimate user; This indicates that the active eavesdropper is eavesdropping on the first... The received signal-to-interference-plus-noise ratio when there are valid user information; Indicates the first Channel vectors of each legitimate user; This represents the position matrix of the movable antenna at time t+1; Indicates that the base station is used to send to the first Beamforming vectors for information sent by legitimate users; Indicates that the base station is used to send to the first Beamforming vectors for information sent by legitimate users; This indicates that the jamming signal sent by the active eavesdropper is at the [number]th [time]. Received interference power measurements at each legitimate user location; Indicates the first Additive Gaussian noise power at the receiver of a legitimate user; This represents the eavesdropping channel vector of an active eavesdropper. This indicates the location measurement of the active eavesdropper; This represents the additive Gaussian noise power at the receiver of an active eavesdropper; This represents the conjugate transpose operation; Indicates a time index; J represents a legitimate user index; J represents a jamming signal; E represents an active eavesdropper.

[0087] Specifically, with the minimum confidentiality rate Constructing the first reward function Second reward function The calculation formula is as follows:

[0088] ;

[0089] ;

[0090] ;

[0091] ;

[0092] in, The weighting coefficient representing the penalty for violating antenna spacing rules; The weighting coefficient represents the penalty for information leakage in the direction of the eavesdropping channel; This indicates the penalty for violating the minimum spacing requirement for mobile antennas; This indicates the penalty for information leakage in the direction of the eavesdropping channel; Indicates a time index; Indicates the number of mobile antennas; Indicates the number of legitimate users; Indicates a valid user index; This represents the eavesdropping channel vector of an active eavesdropper. This represents the conjugate transpose operation; This indicates that the base station is used to send to the first Beamforming vectors for information sent by legitimate users; This indicates the maximum transmit power of the base station; This represents the square of the second norm of a vector; It represents the smallest positive number that is not divisible by zero.

[0093] Specifically, the parameters of the first policy network are updated based on the first reward function, and the parameters of the second policy network are updated based on the second reward function, including:

[0094] Construct a first centralized Critic network and a second centralized Critic network, respectively corresponding to the first policy network and the second policy network. The first centralized Critic network obtains a first value measurement based on state information and calculates a first advantage function in combination with a first reward function, and updates the parameters of the first policy network based on the first advantage function. The second centralized Critic network obtains a second value measurement based on state information and calculates a second advantage function in combination with a second reward function, and updates the parameters of the second policy network based on the second advantage function. At the same time, update the parameters of the first centralized Critic network and the second centralized Critic network respectively.

[0095] Specifically, in this embodiment, when an active eavesdropper attempts to intercept confidential information sent by the base station to a legitimate user, the signal it receives is:

[0096] ;

[0097] When an active eavesdropper attempts to eavesdrop on the first Information symbols of a legitimate user At that time, the received signal can be equivalently represented as:

[0098] ;

[0099] in, The received complex Gaussian white noise at the active eavesdropper's location has zero mean and variance. .also, This represents the eavesdropping channel between the base station and the active eavesdropper, specifically:

[0100] ;

[0101] Specifically This represents the transmission field response matrix from the base station to the active eavesdropper. This represents the path response matrix from the base station to the active eavesdropper. Indicates the location The eavesdropper's received field response vector.

[0102] Specifically, in this embodiment, the construction parameters are: Antenna Position Strategy Network and parameters are Beamforming strategy network The construction parameters are: Centralized Critic Network The entire reinforcement learning process can be summarized into the following steps:

[0103] Step 1: Initialize reinforcement learning network parameters and buffers. Randomly initialize the policy network. parameters Policy Network parameter And centralized Critic network parameters Concurrently set an empty trajectory buffer. .

[0104] Step 2: Construct the global state of the centralized Critic network This is used as input to a centralized Critic network. The state includes the location of the movable antennas. Legitimate user channel matrix Estimate the eavesdropping channel Active eavesdroppers estimate location Equivalent received interference power for legitimate users Among them, the equivalent received interference power for legitimate users is , For legitimate users The equivalent received interference power at the location. Since neural networks typically process real-valued inputs, the complex channel is decomposed into real and imaginary parts, and the global state is represented as:

[0105] ; in, This indicates taking the real part of a complex number, complex vector, or complex matrix; This indicates taking the imaginary part of a complex number, complex vector, or complex matrix;

[0106] Step 3: Construct the policy network status :

[0107] ;

[0108] Step 4: Set the status Input to policy network ,according to Obtaining the position of the movable antenna: .in, for:

[0109]

[0110] Step 5: Perform antenna position movement. According to the new movable antenna position Recalculate legitimate user channels And estimating the eavesdropping channel .

[0111] Step 6: Construct the policy network status Policy Network The decision is made after the movable antenna position is updated, therefore the state Includes updated movable antenna locations Updated legitimate user channel matrix Updated estimated eavesdropping channel Active eavesdroppers estimate location Equivalent received interference power for legitimate users , is represented as:

[0112]

[0113] This state enables the neural network It can design beamforming matrices based on the moved antenna geometry and the updated channel state.

[0114] Step 7: Set the status Input to policy generation network ,according to The real and imaginary parts of the beamforming matrix are obtained: ,in For neural networks The output of the original beamforming matrix.

[0115] Step 8: To meet the total transmit power constraint of the base station, the neural network is... The original beamforming matrix output Power normalization is performed to obtain the actual beamforming matrix. :

[0116]

[0117] in, To prevent division by zero of extremely small positive numbers, after power normalization, the beamforming matrix satisfies: .

[0118] Step 9: The base station adjusts the position of the movable antenna according to the updated location. and beamforming matrix Send a confidential signal: And calculate the confidentiality rate for legitimate users:

[0119]

[0120] in, , For the first Received signal-to-interference-plus-noise ratio for each legitimate user:

[0121]

[0122] This indicates that the active eavesdropper is eavesdropping on the first... Received signal-to-interference-plus-noise ratio when there are 1 valid user information:

[0123]

[0124] Step 10: The system uses a policy network. and policy network The combined action of the two calculates the global reward. The global reward function is determined by the system's minimum security rate. Penalties for violations of antenna spacing regulations Composition, represented as:

[0125]

[0126] in, This is the minimum security rate of the system, used to ensure security and fairness among multiple legitimate users. This value serves as the main security performance indicator for reinforcement learning reward design and policy updates. This represents the weighting coefficient for penalties related to violations of antenna spacing.

[0127] Step 11: Construct the global state for the next time slot Constructing trajectory samples:

[0128] ;

[0129] Repeat execution Each time step yields a length of [number] times. The trajectory samples are stored in the trajectory buffer. .

[0130] Step 12: Based on the trajectory buffer Calculate the timing difference error:

[0131]

[0132] Based on the generalized dominance estimation method, the time-series difference errors of multiple time steps are weighted and accumulated to obtain the global dominance function of the current time step:

[0133]

[0134] Furthermore, by combining the global advantage function with the value estimate of the current global state by the centralized Critic network, a discounted reward for training the centralized Critic network is obtained:

[0135]

[0136] in, As a discount factor, For parameters to estimate the generalized dominance, The global advantage function for the current time step, used to simultaneously guide the policy network. and policy network Update; The discounted reward for the current time step is used to update the centralized Critic network.

[0137] Update policy network :

[0138] ;

[0139] in, It is the objective function of PPO. The probability ratio between the old and new strategies:

[0140] ;

[0141] Parameters are tailored for the PPO strategy. For policy networks The learning rate.

[0142] Next, update the policy network. :

[0143] ;

[0144] in, For policy networks The learning rate.

[0145] Update the centralized Critic network.

[0146] ;

[0147] in, The loss function for a centralized Critic network, The learning rate of the centralized Critic network is set and iterated continuously until the performance change in adjacent training stages meets the preset convergence condition, thus completing the policy network training; the reward change during the training process of the proposed algorithm is as follows: Figure 3 As shown, to verify the training effectiveness and convergence performance of the proposed reinforcement learning algorithm, the algorithm was trained for 100,000 environment interaction steps, and the reward value of each environment interaction step was recorded during the training process. At the same time, a sliding window with a length of 2048 environment steps was used to calculate the average reward value.

[0148] Depend on Figure 3 It can be seen that in the early stage of training, the single-step reward value fluctuates significantly because the policy network is in the exploration phase. As the training process continues, the moving average reward value generally shows a stable upward trend, gradually increasing from about 0.6 in the early stage to about 4.6 in the later stage. As the number of training steps increases, the growth rate of the average reward value gradually slows down, indicating that the first and second policy networks can gradually learn the mobile antenna position configuration and beamforming strategy through continuous interaction. The proposed algorithm has good learning ability and shows a gradually stabilizing convergence trend.

[0149] like Figure 2 As shown, this embodiment also discloses a mobile antenna anti-active eavesdropping secure communication system based on reinforcement learning, including:

[0150] Initialization module 21 is used to obtain base station configuration parameters and legal user channel matrix, wherein the configuration parameters include base station height, number of mobile antennas and mobile area range;

[0151] The signal identification module 22 is used to monitor the received signals in the uplink communication frequency band, compare the characteristics of each received signal with the legitimate user channel matrix, determine that the mismatched signal is an illegitimate signal, extract the angle of arrival measurement data of the illegitimate signal, and identify the source of the illegitimate signal as an active eavesdropper.

[0152] The initial matrix calculation module 23 is used to calculate the measurement position of the active eavesdropper based on the base station height and angle of arrival measurement data, and to calculate the eavesdropping channel measurement value and the equivalent received interference power of the legitimate user based on the measurement position of the active eavesdropper.

[0153] Based on the number of mobile antennas and the range of movable area, the position of each mobile antenna is initialized using the circle filling method to obtain the initial mobile antenna position matrix;

[0154] The first state generation module 24 is used to take the initial mobile antenna position matrix as the current mobile antenna position, and combine the legitimate user channel matrix, the active eavesdropper measurement position, the eavesdropping channel measurement value and the legitimate user equivalent received interference power into the first input state of the first policy network.

[0155] Antenna position update module 25 is used by the first policy network to output the mobile antenna position offset based on the first input state and update the mobile antenna position, and to recalculate the measurement values ​​of the legitimate user channel and the eavesdropping channel based on the updated mobile antenna position.

[0156] The second state generation module 26 is used to combine the updated antenna position, the recalculated legitimate user channel, the eavesdropping channel measurement value, the active eavesdropper measurement position, and the legitimate user's equivalent received interference power into a second input state and input it into the second policy network.

[0157] The strategy output module 27 is used for the second strategy network to output the original beamforming matrix based on the second input state, perform power normalization on the original beamforming matrix to obtain the actual beamforming matrix, calculate the security rate of each legitimate user based on the actual beamforming matrix, construct the first reward function corresponding to the mobile antenna position optimization and the second reward function corresponding to the beamforming optimization based on the minimum security rate, update the parameters of the first strategy network based on the first reward function, update the parameters of the second strategy network based on the second reward function, and iterate until the security rate converges, and output the antenna position control strategy and beamforming strategy for countering active eavesdropping.

[0158] The specific implementation of the reinforcement learning-based mobile antenna anti-active eavesdropping security communication system is the same as the reinforcement learning-based mobile antenna anti-active eavesdropping security communication method, and will not be described again in this embodiment.

[0159] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A mobile antenna anti-active eavesdropping secure communication method based on reinforcement learning, characterized in that, Includes the following steps: S1, obtain base station configuration parameters and legal user channel matrix, the configuration parameters including base station height, number of mobile antennas and mobile area range; S2, monitor the received signals in the uplink communication frequency band, compare the characteristics of each received signal with the legitimate user channel matrix, determine that the mismatched signal is an illegitimate signal, extract the angle of arrival measurement data of the illegitimate signal, and identify the source of the illegitimate signal as an active eavesdropper; S3, calculate the measurement location of the active eavesdropper based on the base station height and angle of arrival measurement data, and calculate the eavesdropping channel measurement value and the equivalent received interference power of the legitimate user based on the measurement location of the active eavesdropper; Based on the number of mobile antennas and the range of movable area, the position of each mobile antenna is initialized using the circle filling method to obtain the initial mobile antenna position matrix; S4, take the initial mobile antenna position matrix as the current mobile antenna position, and combine the legitimate user channel matrix, the active eavesdropper measurement position, the eavesdropping channel measurement value and the legitimate user equivalent received interference power into the first input state of the first strategy network. S5, the first strategy network outputs the mobile antenna position offset based on the first input state and updates the mobile antenna position, and recalculates the legal user channel and eavesdropping channel measurement values ​​based on the updated mobile antenna position; S6, combine the updated antenna position, the recalculated legitimate user channel, the eavesdropping channel measurement value, the active eavesdropper measurement position, and the legitimate user's equivalent received interference power into a second input state and input it into the second policy network; S7, the second strategy network outputs the original beamforming matrix based on the second input state, performs power normalization on the original beamforming matrix to obtain the actual beamforming matrix, calculates the security rate of each legitimate user based on the actual beamforming matrix, constructs the first reward function corresponding to the mobile antenna position optimization and the second reward function corresponding to the beamforming optimization based on the minimum security rate, updates the parameters of the first strategy network based on the first reward function, updates the parameters of the second strategy network based on the second reward function, and iterates until the security rate converges, outputting the antenna position control strategy and beamforming strategy for countering active eavesdropping.

2. The reinforcement learning-based mobile antenna anti-active eavesdropping secure communication method according to claim 1, characterized in that, In S3, the position of each mobile antenna is initialized using a circle-filling method, specifically as follows: The number of small circles is determined based on the number of mobile antennas, and the radius of the small circles is determined based on the minimum spacing constraint between antennas. Within the movable area, the small circles are made to be non-overlapping and their radii are maximized. The center of each small circle is used as the initial position of each mobile antenna.

3. The reinforcement learning-based mobile antenna anti-active eavesdropping secure communication method according to claim 1, characterized in that, In S3, the location of the active eavesdropper is calculated based on the base station altitude and angle of arrival measurement data. The calculation formula is as follows: ； in, This is the azimuth measurement value; This is the measured pitch angle value; Indicates the base station altitude.

4. The reinforcement learning-based mobile antenna anti-active eavesdropping secure communication method according to claim 1, characterized in that, In S7, the original beamforming matrix is ​​power normalized using the following formula: ； in, To prevent extremely small positive numbers from being divided by zero; This is the normalized beamforming matrix; This is the original beamforming matrix; Indicates the maximum power of the base station; This represents the Frobenius norm of the matrix.

5. The reinforcement learning-based mobile antenna anti-active eavesdropping secure communication method according to claim 1, characterized in that, In S7, the confidentiality rate of each legitimate user is calculated based on the actual beamforming matrix, using the following formula: ； ； ； in, This indicates a value greater than 0; For the first The confidentiality rate of each legitimate user; For the first The received signal-to-interference-plus-noise ratio of a legitimate user; This indicates that the active eavesdropper is eavesdropping on the first... The received signal-to-interference-to-noise ratio when there are 10 legitimate user information; Indicates the first Channel vectors of each legitimate user; This represents the position matrix of the movable antenna at time t+1; Indicates that the base station is used to send to the first Beamforming vectors for information sent by legitimate users; Indicates that the base station is used to send to the first Beamforming vectors for information sent by legitimate users; This indicates that the jamming signal sent by the active eavesdropper is at the [number]th [time]. Received interference power measurements at each legitimate user location; Indicates the first Additive Gaussian noise power at the receiver of a legitimate user; This represents the eavesdropping channel vector of an active eavesdropper. This indicates the location measurement of the active eavesdropper; This represents the additive Gaussian noise power at the receiver of an active eavesdropper; This represents the conjugate transpose operation; Indicates a time index; J represents a legitimate user index; J represents a jamming signal; E represents an active eavesdropper.

6. The mobile antenna anti-active eavesdropping secure communication method based on reinforcement learning according to claim 1, characterized in that, In S7, the minimum security rate is used. Constructing the first reward function Second reward function The calculation formula is as follows: ； ； ； ； in, The weighting coefficient representing the penalty for violating antenna spacing rules; The weighting coefficient represents the penalty for information leakage in the direction of the eavesdropping channel; This indicates the penalty for violating the minimum spacing requirement for mobile antennas; This indicates the penalty for information leakage in the direction of the eavesdropping channel; Indicates a time index; Indicates the number of mobile antennas; Indicates the number of legitimate users; Indicates a valid user index; This represents the eavesdropping channel vector of an active eavesdropper. This represents the conjugate transpose operation; Indicates that the base station is used to send to the first Beamforming vectors for information sent by legitimate users; This indicates the maximum transmit power of the base station; This represents the square of the second norm of a vector; It represents the smallest positive number that is not divisible by zero.

7. The mobile antenna anti-active eavesdropping secure communication method based on reinforcement learning according to claim 1, characterized in that, In S7, the parameters of the first policy network are updated based on the first reward function, and the parameters of the second policy network are updated based on the second reward function, specifically including: Construct a first centralized Critic network and a second centralized Critic network, respectively corresponding to the first policy network and the second policy network. The first centralized Critic network obtains a first value measurement based on state information and calculates a first advantage function in combination with a first reward function, and updates the parameters of the first policy network based on the first advantage function. The second centralized Critic network obtains a second value measurement based on state information and calculates a second advantage function in combination with a second reward function, and updates the parameters of the second policy network based on the second advantage function. At the same time, update the parameters of the first centralized Critic network and the second centralized Critic network respectively.

8. A mobile antenna anti-active eavesdropping secure communication system based on reinforcement learning, characterized in that, include: An initialization module is used to obtain base station configuration parameters and a legal user channel matrix. The configuration parameters include base station height, number of mobile antennas, and mobile area range. The signal identification module is used to monitor the received signals in the uplink communication frequency band, compare the characteristics of each received signal with the legitimate user channel matrix, determine that the mismatched signal is an illegitimate signal, extract the angle of arrival measurement data of the illegitimate signal, and identify the source of the illegitimate signal as an active eavesdropper. The initial matrix calculation module is used to calculate the measurement location of the active eavesdropper based on the base station height and angle of arrival measurement data, and to calculate the eavesdropping channel measurement value and the equivalent received interference power of the legitimate user based on the measurement location of the active eavesdropper. Based on the number of mobile antennas and the range of movable area, the position of each mobile antenna is initialized using the circle filling method to obtain the initial mobile antenna position matrix; The first state generation module is used to take the initial mobile antenna position matrix as the current mobile antenna position, and combine the legitimate user channel matrix, the active eavesdropper measurement position, the eavesdropping channel measurement value and the legitimate user equivalent received interference power into the first input state of the first policy network. The antenna position update module is used by the first policy network to output the mobile antenna position offset based on the first input state and update the mobile antenna position, and to recalculate the measurement values ​​of the legitimate user channel and the eavesdropping channel based on the updated mobile antenna position. The second state generation module is used to combine the updated antenna position, the recalculated legitimate user channel, the eavesdropping channel measurement value, the active eavesdropper measurement position, and the legitimate user's equivalent received interference power into a second input state and input it into the second policy network. The strategy output module is used by the second strategy network to output the original beamforming matrix based on the second input state, perform power normalization on the original beamforming matrix to obtain the actual beamforming matrix, calculate the security rate of each legitimate user based on the actual beamforming matrix, construct a first reward function corresponding to mobile antenna position optimization and a second reward function corresponding to beamforming optimization based on the minimum security rate, update the parameters of the first strategy network based on the first reward function, update the parameters of the second strategy network based on the second reward function, and iterate until the security rate converges, and output the antenna position control strategy and beamforming strategy for combating active eavesdropping.

Images