Improved ris assisted isac secure beamforming method of td3

Abstract

The present application relates to the field of wireless communication and radar perception technology, and particularly relates to an improved RIS assisted ISAC security beam forming method based on TD3, comprising: constructing an active RIS assisted ISAC security system, establishing BS-RIS, RIS-Bob / Eve channels based on a Rician fading channel; obtaining RIS unit amplitude and phase shift to construct a reflection matrix; calculating Bob / Eve received signal and SINR to obtain instantaneous secrecy rate; taking maximizing the secrecy rate as the target to construct a non-convex optimization problem and model it as a POMDP. The solving steps are: constructing a GAN to generate adversarial CSI sample adjustment state space; constructing an A-MPC-TD3 framework, iteratively inputting the current state to output actions, acting on the system and then transferring to the next state, and outputting the optimal solution when the maximum number of steps is reached. The present application improves the physical layer security performance, robustness and stealth capability of the communication and perception integrated system in a dynamic uncertain environment.

Description

Technical Field

[0001] This invention relates to the fields of wireless communication and radar sensing technology, and in particular to an improved RIS-assisted ISAC secure beamforming method for TD3. Background Technology

[0002] To address the severe challenges to physical layer security in 6G Integrated Sensing and Communications (ISAC) networks, existing solutions suffer from significant drawbacks: while artificial noise (AN) schemes can interfere with eavesdroppers, their high-power radiation is easily detected by spectrum monitoring equipment, failing to meet low observability requirements; passive reconfigurable intelligent surfaces (RIS), although capable of constructing virtual line-of-sight links, are limited by multiplicative fading effects, resulting in drastic deterioration of reflected signals under deep fading or long-distance conditions; and while emerging active RIS can overcome the fading bottleneck of passive RIS through signal amplification, it introduces the new problem of "amplification equals leakage," meaning that enhancing legitimate links may also improve the quality of eavesdropping links. Furthermore, existing research generally assumes ideal channel state information (CSI), failing to consider the high uncertainty of eavesdropper CSI in real-world environments, transmission distortion caused by hardware damage, and multiple non-convex constraints such as the need for RIS radiation to meet stealth requirements. Summary of the Invention

[0003] This invention provides an improved RIS-assisted ISAC security beamforming method for TD3, which addresses the problems in existing ISAC systems such as the easy exposure of artificial noise schemes, the limitation of passive RIS by multiplicative fading, the risk of "amplification equals leakage" of active RIS, and the lack of robust optimization for CSI uncertainty, hardware damage, and stealth constraints.

[0004] The objective of this invention can be achieved through the following technical solutions:

[0005] This invention provides an improved RIS-assisted ISAC security beamforming method for TD3, comprising: An active RIS-assisted ISAC security system is constructed; the ISAC security system consists of a base station, an active RIS, a legitimate user Bob, an eavesdropper Eve, and a sensing target; the active RIS contains multiple RIS units; a base station-active RIS channel matrix, an active RIS-Bob channel vector, and an active RIS-Eve channel vector are established based on the Ricean fading channel. Obtain the adjustable amplitude gain and adjustable phase shift of each RIS unit, and construct the reflection matrix of the active RIS of all RIS units based on the adjustable amplitude gain and the adjustable phase shift; The base station transmits the signal, and the theoretical received signals at Bob and Eve are calculated based on the base station-active RIS channel matrix, the reflection matrix of the active RIS, and the base station transmit signal. The signal-to-interference-plus-noise ratio (SIR) at Bob and Eve is calculated by decomposing the terms of the theoretical received signal. The instantaneous security rate at the current moment is obtained based on the SIR at Bob and Eve. Obtain the radar signal-to-interference-plus-noise ratio and the total radiated power of the RIS; With the goal of maximizing instantaneous security rate, and with the radar signal-to-interference-plus-noise ratio threshold, the legitimate user communication quality threshold, the base station transmit power upper limit, the stealth upper limit of the total radiated power of the RIS, the adjustable amplitude gain upper limit of the RIS unit, and the adjustable phase shift range as constraints, a joint constrained non-convex optimization problem is constructed. The optimization problem is modeled as POMDP; the state space, action space and reward function in POMDP are defined by the base station-active RIS channel matrix, active RIS-Bob channel vector, active RIS-Eve channel vector, instantaneous security rate, adjustable amplitude gain and adjustable phase shift of RIS unit. The specific steps for solving the optimization problem within the POMDP framework are as follows: An adversarial CSI generator for GAN is constructed. The synthesized CSI error output by the generator is superimposed onto the active RIS-Eve channel vector to obtain adversarial CSI samples. The state space in POMDP is adjusted using the adversarial CSI samples. Construct the A-MPC-TD3 framework, input the current state of POMDP into the A-MPC-TD3 framework, and output the current step action vector; apply the current step action vector to the ISAC security system, and then transfer to the next POMDP state to form a closed loop, and iterate in sequence; stop when the number of iterations reaches the preset maximum number of steps, and take the final action at the time of stopping as the optimal solution to the optimization problem.

[0006] Furthermore, the establishment of the base station-active RIS channel matrix, the active RIS-Bob channel vector, and the active RIS-Eve channel vector based on the Ricean fading channel includes: The BS-RIS channel matrix is ​​specifically expressed by the following formula:

[0007] In the formula, This indicates the preset Rice factor of the BS-RIS link. This represents the line-of-sight matrix of the BS-RIS channel. Represents the non-line-of-sight matrix of the BS-RIS channel; Represents the BS-RIS channel matrix; The RIS-Bob channel vector is specifically expressed by the following formula:

[0008] In the formula, This represents the preset Rice factor of the RIS-Bob link. This represents the line-of-sight vector of the RIS-Bob channel. Represents the non-line-of-sight vector of the RIS-Bob channel. Represents the RIS-Bob channel vector; The RIS-Eve channel vector is specifically expressed by the following formula:

[0009] In the formula, This represents the preset Rice factor of the RIS-Eve link. This represents the line-of-sight vector of the RIS-Eve channel. This represents the non-line-of-sight vector of the RIS-Eve channel. Represents the RIS-Eve channel vector; Among them, the line-of-sight matrix of the BS-RIS channel, the line-of-sight vector of the RIS-Bob channel, and the line-of-sight vector of the RIS-Eve channel are all calculated using array steering vectors; The elements in the non-line-of-sight matrix of the BS-RIS channel, the non-line-of-sight vector of the RIS-Bob channel, and the non-line-of-sight vector of the RIS-Eve channel are all randomly generated using a standard complex Gaussian distribution.

[0010] Further, the step of obtaining the adjustable amplitude gain and adjustable phase shift of each RIS unit, and constructing the reflection matrix of the active RIS of all RIS units based on the adjustable amplitude gain and the adjustable phase shift, includes:

[0011]

[0012]

[0013]

[0014] In the formula, This represents the reflection matrix of an active RIS. Indicates the first The reflection coefficient of each RIS unit. This indicates the total number of RIS units. This represents the diagonalization operator, which arranges a scalar sequence into a diagonal matrix. Indicates the first Adjustable amplitude gain of each RIS unit This represents the maximum value of the amplitude gain of the RIS unit; Indicates the first Adjustable phase shift of each RIS unit Represents a complex exponential function; represents phase rotation. Represents pi; The initial adjustable amplitude gain and initial adjustable phase shift of each RIS unit are... and Uniformly sampled and randomly generated within two constraint ranges; The adjustable amplitude gain and adjustable phase shift of each subsequent RIS unit are obtained by the A-MPC-TD3 framework through each iteration.

[0015] Further, the acquisition of the base station transmitted signal, and the calculation of the theoretical received signals at Bob and Eve based on the base station-active RIS channel matrix, the reflection matrix of the active RIS, and the base station transmitted signal, include: The expression for the base station's transmitted signal is:

[0016]

[0017] In the formula, Represents the base station's transmitted signal vector. Represents the base station beamforming vector. It is a complex scalar; This represents the additive distortion noise vector caused by hardware damage. Indicates the hardware damage coefficient. express The conjugate transpose of . This represents a cyclically symmetric complex Gaussian distribution. This represents the variance in the distribution; This means constructing a diagonal matrix from the diagonal elements of the outer product matrix of the base station beamforming vectors; Among them, the base station beamforming vector The initial value is randomly generated by sampling from a complex Gaussian distribution with a mean of 0 and a variance of 1; the subsequent w is obtained by the A-MPC-TD3 framework through each iteration; The theoretical expressions for the received signals at Bob and Eve are:

[0018]

[0019]

[0020] In the formula, This indicates the noise power of the active RIS amplifier. It is an M×M identity matrix; This represents the additive Gaussian noise vector introduced by the active RIS amplifier; Indicates user Thermal noise at the location, This indicates the thermal noise power at the user receiver. express The set of values ​​is ,gather In this context, 'b' represents the legitimate user Bob, and 'e' represents the eavesdropper Eve. Let H represent the channel vector from RIS to user u, where H is the conjugate transpose. Represents the BS-RIS channel matrix. This indicates the received signal at user u. This represents the reflection matrix of an active RIS; Among them, the noise power of the active RIS amplifier Thermal noise power at the user receiver This is the default value.

[0021] Further, the calculation obtains the signal-to-interference-plus-noise ratio (SIR) at Bob and at Eve; based on the SIR at Bob and at Eve, the instantaneous security rate at the current moment is obtained, including: The signal-to-interference-to-noise ratio at Bob is specifically expressed by the formula:

[0022] In the formula, Represents the RIS-Bob channel vector. express The conjugate transpose of . This represents the variance of the thermal noise at Bob. This indicates the signal-to-interference-plus-noise ratio at Bob's location; The signal-to-interference-to-noise ratio at Eve is specifically expressed by the formula:

[0023] In the formula, Represents the RIS-Eve channel vector. express The conjugate transpose of . This represents the variance of the thermal noise at Eve; Represents the square of the modulus. This represents the square of the second norm of a vector; This indicates the signal-to-interference-plus-noise ratio at Eve;

[0024] In the formula, Represents the logarithmic function with base 2. This is the positive part operator; it takes the value 0 when the value inside the parentheses is negative. This represents the instantaneous security rate at the current moment.

[0025] Further, obtaining the radar signal-to-interference-plus-noise ratio and the total radiated power of the RIS includes: The radar signal-to-interference-plus-noise ratio is specifically expressed by the formula:

[0026] In the formula, Represents the base station receive vector. This indicates the target's angle relative to RIS. Represents the complex reflection coefficient of the target. This represents the guidance vector pointing towards the target. Indicates clutter power. This indicates the thermal noise power of the base station radar receiver; For radar signal-to-interference-plus-noise ratio; Indicates the transpose symbol; in, , Let be the variance of the cyclic symmetric complex Gaussian distribution of the target complex reflection coefficient; where the target complex reflection coefficient is a preset value. The total radiated power of RIS can be expressed by the following formula:

[0027] In the formula, This is the trace operation of a matrix; Denotes the Frobenius norm; This represents the total radiated power of the RIS.

[0028] Furthermore, the construction process of the jointly constrained nonconvex optimization problem is as follows: The objective function in a jointly constrained nonconvex optimization problem is specifically expressed as:

[0029] In the formula, Describe the target optimization parameters in the output. This represents the value of the independent variable that maximizes the objective function; The constraints in the joint constraint nonconvex optimization problem are specifically expressed as follows:

[0030]

[0031]

[0032]

[0033]

[0034]

[0035] In the formula, Indicates the minimum SINR threshold of the radar. This indicates Bob's service quality threshold. This represents the base station's maximum transmit power budget. This indicates the upper limit of RIS radiation stealth. Indicates all All are true; This indicates the total number of RIS units. Indicates the first Adjustable amplitude gain of each RIS unit This represents the maximum value of the amplitude gain of the RIS unit; Indicates the first Adjustable phase shift of each RIS unit; Among them, the minimum radar SINR threshold, Bob's quality of service threshold, and the base station's maximum transmit power budget are all preset values; Among them, the upper limit of RIS radiation stealth. It is obtained by reverse calculation based on the preset false alarm rate. The specific process of reverse calculation to obtain the upper limit of RIS radiation stealth is as follows: The first step is to preset Eve's allowed false alarm rate. ; The second step is to analyze the statistical characteristics and false alarm rate of environmental noise. deduce the detection threshold Specifically, this can be expressed as a formula:

[0036] In the formula, This represents the mean signal energy at the background noise level. The standard deviation of the signal energy at the background noise level. For the standard Gaussian Q function, for inverse function, To detect threshold, False alarm rate; The third step is to establish the detection probability expression, specifically as follows:

[0037] In the formula, This represents the offset of RIS radiation relative to the noise floor threshold. This represents the probability that the eavesdropper will detect RIS radiation. By using the detection probability expression and taking the preset tolerance value as the probability value, the upper limit of RIS radiation stealth is derived. .

[0038] Furthermore, the construction process of the POMDP is as follows: The process of constructing the state space in POMDP is as follows:

[0039] In the formula, This represents the estimated value of the RIS-Eve channel vector at the current time t. This represents the estimated value of the RIS-Bob channel vector at the current time t. This represents the estimated value of the BS-RIS channel matrix at the current time t. This represents the true state space at the current time t. For matrix column vectorization operators, To take the real part of the complex number, To take the imaginary part of a complex number, It is the transpose symbol. This represents the instantaneous security rate at the previous moment. This represents the remaining available transmission power at the current time t. Indicates the hardware damage coefficient; Remaining available transmit power at current time t In the formula, This represents the base station's maximum transmit power budget; Specifically, the system includes preset BS-RIS channel matrix error, preset RIS-Bob channel vector error, and preset RIS-Eve channel vector error. The BS-RIS channel matrix is ​​estimated by adding the preset BS-RIS channel matrix error to obtain the estimated value of the BS-RIS channel matrix. The RIS-Bob channel vector is estimated by adding the preset RIS-Bob channel vector error to obtain the estimated value of the RIS-Bob channel vector. The RIS-Eve channel vector is estimated by adding the preset RIS-Eve channel vector error to obtain the estimated value of the RIS-Eve channel vector. The specific process of constructing the action space in POMDP is as follows:

[0040] In the formula, This represents the base station beamforming vector at the current time t. This represents the adjustable amplitude gain of the first RIS unit at the current time t. This represents the adjustable amplitude gain of the M-th RIS unit at the current time t. This represents the adjustable phase shift of the first RIS unit at the current time t. This represents the adjustable phase shift of the Mth RIS unit at the current time t; Represents the action space at the current time t; in, express The imaginary part is expanded; express The real part unfolds; The reward function in POMDP is specifically expressed by the following formula:

[0041] In the formula, For the reward function, The first weighting coefficient; This is the second weighting coefficient. The third weighting coefficient, and Used to punish violations of constraints; It is the fourth weighting coefficient. This indicates the violation amount when the radar SINR does not meet the threshold. This indicates the amount of violation when the RIS radiation power exceeds the stealth threshold; This represents the loss term that couples hardware damage with transmit power.

[0042] Furthermore, the adversarial CSI generator that constructs the GAN, by superimposing the synthesized CSI error output by the generator onto the active RIS-Eve channel vector, obtains adversarial CSI samples, including: An adversarial CSI generator based on a Generative Adversarial Network (GAN) is constructed, which generates adversarial errors only for active RIS-Eve channels. A generator is defined to map random noise vectors to synthetic CSI errors. A discriminator is defined to distinguish between real active RIS-Eve channel samples and the adversarial CSI samples output by the generator. An objective function for generative adversarial training is set. An auxiliary loss is introduced, which is the negative expectation of the Critic network output value in the TD3 algorithm, used to guide the generator to generate a worst-case perturbation that minimizes the Critic network's estimate. The synthetic CSI errors output by the generator are superimposed onto real active RIS-Eve channels to obtain adversarial CSI samples. The generator, discriminator, and TD3 algorithm are trained alternately. The objective function for generative adversarial training is specifically expressed as:

[0043] In the formula, Represents generator Minimize target, discriminator Maximize the goal; The objective function of GAN; Represents the true RIS-Eve channel vector The expected value is taken to follow the true data distribution. This represents the logarithm of the true probability output by the discriminator for a positive sample. This means that the adversarial CSI samples should follow a prior distribution and have the expected value. To counter CSI samples; The loss function in the generator is obtained by weighted fusion of the original generator adversarial loss and auxiliary loss; the auxiliary loss is specifically expressed as:

[0044] In the formula, These are weighting coefficients. This indicates the critical network for TD3. This represents the loss function of GAN during the training phase; Represents the state vector; This represents the actor network of TD3; The discriminator's loss is the binary cross-entropy loss.

[0045] Furthermore, the construction of the A-MPC-TD3 framework involves inputting the current state of the POMDP into the A-MPC-TD3 framework and outputting the current step action vector. After applying the current step action vector to the ISAC security system, the process transitions to the next POMDP state, forming a closed loop, and iterating sequentially. The iteration stops when the number of iterations reaches a preset maximum number of steps, and the final action at the point of cessation is taken as the optimal solution to the optimization problem, including: A framework A-MPC-TD3 is constructed, comprising a TD3 module and a Model Predictive Control (MPC) module. The following steps are iteratively executed until a preset maximum number of steps is reached: The current state of the POMDP is input into the Actor network of the TD3 module, outputting an initial action vector; the initial action vector is input into the MPC module, and a quadratic constraint programming problem is solved in the rolling time domain, with the objective of minimizing the distance between the MPC output action and the initial action vector, and with all constraints in the joint constraint non-convex optimization problem as the feasible region, resulting in a refined action vector; the initial action vector and the refined action vector are fused using convex combination weight coefficients to obtain a fused action vector; the amplitude gain of each RIS unit in the fused action vector is projected to its hardware upper limit interval, and the phase shift of each RIS unit is projected to its feasible region interval to obtain the current step action vector; the current step action vector is executed, the reward is calculated and experience samples are stored, and the parameters of the Actor network and Critic network of the TD3 module are updated; then, the next POMDP state is entered, and the next iteration continues; the iteration stops when the preset maximum number of steps is reached, and the final action at the time of stopping is taken as the optimal solution to the optimization problem. The quadratic constrained programming is specifically expressed by the following formula:

[0046] In the formula, H is the time domain length of MPC prediction; This is the relative time index within the MPC rolling time domain; The refined actions calculated by MPC; The original action given to the Actor in TD3; This represents the square of the second norm of a vector. It is a minimum value function; The specific formula for obtaining the fused action vector is as follows:

[0047] In the formula, These are the weighting coefficients. To refine the action vectors, As the initial action vector, To fuse action vectors.

[0048] Compared with existing technologies, the beneficial effects of this invention are: constructing an active RIS-assisted ISAC security system; the ISAC security system consists of a base station, an active RIS, a legitimate user Bob, an eavesdropper Eve, and a sensing target; the active RIS includes multiple RIS units; by introducing active RIS, the multiplicative fading limitation of passive RIS is overcome, enhancing link quality and system controllability; based on the Ricean fading channel, a base station-active RIS channel matrix, an active RIS-Bob channel vector, and an active RIS-Eve channel vector are established respectively; the adjustable amplitude gain and adjustable phase shift of each RIS unit are obtained, and the reflection matrix of the active RIS of all RIS units is constructed according to the adjustable amplitude gain and the adjustable phase shift; the signal transmission characteristics are accurately characterized, providing a precise mathematical basis for subsequent optimization; the base station transmitted signal is obtained, and the theoretical received signals at Bob and Eve are calculated according to the base station-active RIS channel matrix, the reflection matrix of the active RIS, and the base station transmitted signal; by decomposing the terms in the theoretical received signal, the signal-to-interference-plus-noise ratio at Bob and the signal-to-interference-plus-noise ratio at Eve are calculated; based on B The signal-to-interference-plus-noise ratio (SNR) at point ob and the SNR at point Eve are used to obtain the instantaneous security rate at the current moment; quantify the system security performance to provide a direct optimization target for maximizing the security rate; obtain the radar SNR and the total radiated power of the RIS; support the quantification of radar perception and stealth constraints to ensure that the system achieves a balance between perception and security; with the goal of maximizing the instantaneous security rate, the radar SNR threshold, the legitimate user communication quality threshold, the base station transmit power upper limit, the stealth upper limit of the total radiated power of the RIS, the adjustable amplitude gain upper limit of the RIS unit, and the adjustable phase shift range are used as constraints. This paper constructs a jointly constrained nonconvex optimization problem; it unifies the modeling of multiple requirements such as physical layer security, perception, power, hardware impairment, and stealth to achieve multi-objective collaborative optimization; the optimization problem is modeled as a POMDP; the state space, action space, and reward function in the POMDP are defined by the base station-active RIS channel matrix, the active RIS-Bob channel vector, the active RIS-Eve channel vector, the instantaneous security rate, and the adjustable amplitude gain and adjustable phase shift of the RIS unit; it adapts to partially observable environments and provides a standard framework for deep reinforcement learning solutions; The specific steps for solving the optimization problem within the POMDP framework are as follows: Construct an adversarial CSI generator using a GAN; superimpose the synthesized CSI error output by the generator onto the active RIS-Eve channel vector to obtain adversarial CSI samples; adjust the state space in POMDP using the adversarial CSI samples; enhance the robustness of the strategy against eavesdropper channel uncertainty through worst-case perturbation; construct an A-MPC-TD3 framework, inputting the current state of POMDP into the A-MPC-TD3 framework and outputting the current step action vector; and then... After being applied to the ISAC security system, it transitions to the next POMDP state, forming a closed loop, and iterates sequentially. When the number of iterations reaches the preset maximum number of steps, it stops, and the final action at the time of stopping is taken as the optimal solution to the optimization problem. Real-time secure beamforming under high-dimensional non-convex constraints is achieved, ensuring the feasibility of the action and improving the convergence speed and stability. It solves the problems in existing ISAC systems, such as the easy exposure of artificial noise schemes, the limitation of passive RIS by multiplicative fading, the risk of "amplification equals leakage" of active RIS, and the lack of robust optimization for CSI uncertainty, hardware damage, and stealth constraints. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This invention provides a schematic flowchart of the steps for an improved TD3 RIS-assisted ISAC security beamforming method; Figure 2 A diagram illustrating the relationship between cumulative rewards and the number of training rounds; Figure 3 This is a schematic diagram illustrating the relationship between the security rate and the number of active RIS units M. Figure 4 This is a schematic diagram illustrating the relationship between detection probability and RIS radiation power. Figure 5 A schematic diagram showing the relationship between radar output SINR and hardware impairment coefficient κ; Figure 6 For security rate and base station transmit power budget A diagram illustrating the relationship between the two. Detailed Implementation

[0051] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0052] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0053] To address the problems existing in the background technology, an improved RIS-assisted ISAC secure beamforming method for TD3 is designed. The aim is to jointly optimize the base station beamforming and the reflection coefficient of the active RIS in a dynamic and open wireless environment, maximize the system security rate and achieve robust online decision-making for CSI uncertainty while meeting multiple constraints such as radar perception quality, communication service quality, transmit power, hardware impairment tolerance and radiation stealth.

[0054] like Figure 1 As shown, this invention provides an improved RIS-assisted ISAC security beamforming method for TD3, comprising the following steps: Step S1: Construct an active RIS-assisted ISAC security system; the ISAC security system consists of a base station, an active RIS, a legitimate user Bob, an eavesdropper Eve, and a sensing target; the active RIS contains multiple RIS units; establish the base station-active RIS channel matrix, the active RIS-Bob channel vector, and the active RIS-Eve channel vector based on the Ricean fading channel; obtain the adjustable amplitude gain and adjustable phase shift of each RIS unit, and construct the reflection matrix of the active RIS for all RIS units based on the adjustable amplitude gain and the adjustable phase shift; obtain the base station transmitted signal, and based on the base station-active RIS channel matrix, the active RIS-Bob channel vector, and the active RIS-Eve channel vector, construct the reflection matrix of the active RIS for all RIS units. The reflection matrix of the IS and the base station transmitted signal are used to calculate the theoretical received signals at Bob and Eve. By decomposing the terms of the theoretical received signals, the signal-to-interference-plus-noise ratio (SIR) at Bob and Eve are calculated. Based on the SIR at Bob and Eve, the instantaneous security rate at the current moment is obtained. The radar SIR and the total radiated power of the RIS are obtained. With the goal of maximizing the instantaneous security rate, a joint constrained non-convex optimization problem is constructed with the radar SIR threshold, the legitimate user communication quality threshold, the base station transmit power upper limit, the stealth upper limit of the total radiated power of the RIS, the adjustable amplitude gain upper limit of the RIS unit, and the adjustable phase shift range as constraints.

[0055] It should be noted that due to the risks of "amplification equals leakage" of active RIS (Reconfigurable Smart Surface) in ISAC (Integrated Communication and Sensing) security systems, transmission distortion caused by hardware damage, performance coupling between radar and communication, and multiple conflicting non-convex constraints such as the need for RIS to satisfy radiation stealth, system modeling and problem construction are required in order to transform these real physical characteristics into a mathematically processable form and lay the foundation for subsequent design optimization algorithms. This is to clarify the optimization variables (base station beamforming vector and RIS reflection coefficient), objective function (maximizing security rate), and all constraints (radar SINR, communication QoS, transmission power, hardware damage, stealth, etc.).

[0056] Specifically, an ISAC security system is constructed. The ISAC (Intelligent Sensing and Communication) security system architecture includes a base station, an active RIS (Reconfigurable Intelligent Surface, acting as a relay), a legitimate user Bob, an eavesdropper Eve, and the sensing target. The direct transmission paths between the base station (BS) and the legitimate user Bob, and between the base station and the eavesdropper Eve, are blocked by obstacles such as buildings, preventing direct signal transmission. Therefore, an active RIS must be used as a relay to complete signal transmission via a reflection link. This configuration highlights the core role of the RIS in the system—it serves as both a signal relay and a shaper for secure beamforming.

[0057] In this embodiment, the base station consists of 64 ULA antennas; the active RIS consists of 32 RIS units (programmable units) and includes a power amplifier. The sensing target is a far-field static point scatterer, whose function is to enable the base station to perform radar detection simultaneously with communication—by transmitting signals and receiving the echoes reflected from the target, it estimates the target's position, velocity, and other information, achieving integrated communication and sensing (ISAC).

[0058] The direct links from the base station to Bob and from the base station to Eve are blocked, and an active RIS is used as a signal relay and security shaper. Based on the Rice fading channel, the base station-active RIS channel matrix, the active RIS-Bob channel vector, and the active RIS-Eve channel vector are established respectively.

[0059] The purpose of setting up the blockage is to force the base station to not communicate directly with Bob and Eve, and to force the signal to pass through the active RIS, thereby highlighting the core role of the RIS in the system (it is both a relay and a security controller).

[0060] The purpose of using an active RIS as a signal relay and security shaper: Relay: Amplifies and forwards base station signals to compensate for path loss caused by obstruction; Security Shaper: By adjusting the amplitude and phase of the reflector, the signal energy is focused on the legitimate user Bob, while the reception quality of the eavesdropper Eve is reduced, thus achieving physical layer security.

[0061] The three links BS-RIS, RIS-Bob, and RIS-Eve are all modeled as Ricean fading channels. Therefore, based on the Ricean fading channels, base station-active RIS channel matrix, active RIS-Bob channel vector, and active RIS-Eve channel vector are established respectively. Each channel is synthesized by weighting the line-of-sight (LoS) component and the non-line-of-sight (NLoS) component according to the Ricean factor.

[0062] The BS-RIS channel matrix is ​​specifically expressed by the following formula:

[0063] In the formula, This indicates the preset Rice factor of the BS-RIS link. This represents the line-of-sight matrix of the BS-RIS channel. Represents the non-line-of-sight matrix of the BS-RIS channel; This represents the BS-RIS channel matrix.

[0064] The RIS-Bob channel vector is specifically expressed by the following formula:

[0065] In the formula, This represents the preset Rice factor of the RIS-Bob link. This represents the line-of-sight vector of the RIS-Bob channel. Represents the non-line-of-sight vector of the RIS-Bob channel. This represents the RIS-Bob channel vector.

[0066] The RIS-Eve channel vector is specifically expressed by the following formula:

[0067] In the formula, This represents the preset Rice factor of the RIS-Eve link. This represents the line-of-sight vector of the RIS-Eve channel. This represents the non-line-of-sight vector of the RIS-Eve channel. This represents the RIS-Eve channel vector.

[0068] The line-of-sight matrix of the BS-RIS channel, the line-of-sight vector of the RIS-Bob channel, and the line-of-sight vector of the RIS-Eve channel are all calculated using array steering vectors (i.e., the portion of the signal energy that directly reaches the receiver from the transmitter). The rows and columns of the line-of-sight matrix of the BS-RIS channel are determined by the number of RIS elements and the number of base station antennas, respectively. The line-of-sight vector of the RIS-Bob channel consists of the number of RIS elements and one antenna of the legitimate user Bob. The line-of-sight vector of the RIS-Eve channel consists of the number of RIS elements and one antenna of the eavesdropper Eve.

[0069] The elements in the non-line-of-sight matrix of the BS-RIS channel, the non-line-of-sight vector of the RIS-Bob channel, and the non-line-of-sight vector of the RIS-Eve channel are all randomly generated using a standard complex Gaussian distribution.

[0070] In this embodiment It is 12. and The value is 7; the implementer can determine this based on the specific circumstances.

[0071] The reflection coefficient of each RIS unit is determined based on the adjustable amplitude gain and adjustable phase shift of each RIS unit; then the reflection coefficients of all RIS units are combined to form the reflection matrix of the active RIS; where the reflection matrix of the active RIS is a diagonal matrix.

[0072] The process of obtaining the reflection matrix of an active RIS is specifically expressed by the following formula:

[0073]

[0074]

[0075]

[0076] In the formula, This represents the reflection matrix of an active RIS. Indicates the first The reflection coefficient of each RIS unit. This indicates the total number of RIS units. This represents the diagonalization operator, which arranges a scalar sequence into a diagonal matrix. Indicates the first Adjustable amplitude gain of each RIS unit This represents the maximum value of the amplitude gain of the RIS unit (an inherent hardware parameter, which is 2dB in this embodiment). Indicates the first Adjustable phase shift of each RIS unit Represents a complex exponential function; represents phase rotation. It represents pi (π).

[0077] In this context, the adjustable amplitude gain and adjustable phase shift of each RIS unit are algorithm decision variables (obtained from the Actor network output of the A-MPC-TD3 framework algorithm, refined and projected by MPC). These are dynamically calculated by the proposed A-MPC-TD3 algorithm at each time step based on the current channel state, constraints, and optimization objective, rather than obtained through physical measurements. Furthermore, the initial adjustable amplitude gain and initial adjustable phase shift of each RIS unit are determined by... and Uniform sampling is randomly generated within the two constraint ranges.

[0078] Obtain the base station's transmitted signal; whereby the expression for the base station's transmitted signal is:

[0079]

[0080] In the formula, Represents the base station's transmitted signal vector. Represents the base station beamforming vector. It is a complex scalar that satisfies , The mean sign indicates the data sign. The average power is 1; This represents the hardware damage coefficient (arbitrarily set, 0.1 in this embodiment). express The conjugate transpose of . This represents a cyclically symmetric complex Gaussian distribution. This represents the additive distortion noise vector caused by hardware impairment. The corresponding variance in the distribution. This means constructing a diagonal matrix from the diagonal elements of the outer product matrix of the base station beamforming vectors.

[0081] The base station beamforming vector is dynamically calculated and output by the Actor network in the A-MPC-TD3 framework algorithm based on the current state, and is also a decision variable of the A-MPC-TD3 framework algorithm. The initial data of the base station beamforming vector is randomly initialized, that is, it is usually randomly generated by sampling from a complex Gaussian distribution with a mean of 0 and a variance of 1, and then the power is normalized.

[0082] in, It is randomly generated using a cyclic symmetric complex Gaussian distribution.

[0083] Among them, the data symbol for unit power A point is randomly selected uniformly from the QPSK constellation point set. The QPSK constellation point set is a well-known technique and will not be described in detail here.

[0084] Obtain the theoretical received signals at Bob and Eve, where the expression for the theoretical received signals is:

[0085]

[0086]

[0087] In the formula, This represents a cyclically symmetric complex Gaussian distribution. This represents the noise power of the active RIS amplifier (corresponding to the variance in the distribution; this value is artificially preset to a fixed value, which is -80dBm in this embodiment). The identity matrix is ​​M×M (diagonal elements are 1, off-diagonal elements are 0), representing that the noise of each RIS unit is independent and identically distributed. This represents the additive Gaussian noise vector introduced by the active RIS amplifier; Indicates user Thermal noise at the location, This represents the thermal noise power at the user receiver (which is preset to a fixed value in the simulation, and is -90dBm in this embodiment). express The set of values ​​is ,gather In this context, 'b' represents the legitimate user Bob, and 'e' represents the eavesdropper Eve. Let H represent the channel vector from RIS to user u, where H is the conjugate transpose. Represents the BS-RIS channel matrix. This indicates the received signal at user u. This represents the reflection matrix of an active RIS.

[0088] dBm (decibel-milliwatt) is the absolute unit of power, expressed logarithmically with 1 milliwatt as the reference.

[0089] It should be noted that, in order to quantify the channel quality difference between the legitimate link and the eavesdropping link, it can be analyzed by the difference between different power levels; and the analysis power can be calculated by the square of the signal amplitude. Therefore, the SINR at Bob and the SINR at Eve can be obtained by using the signal amplitude in the theoretically received signal.

[0090] The theoretical received signal at Bob and Eve's positions above can be divided into four terms: useful signal term. Hardware distortion item RIS Amplification Noise Term and receiver thermal noise. The power of the useful signal term forms the numerator of the subsequent SINR, and the sum of the other three terms forms the denominator of SINR. Based on this, let... =b, u=e, thus obtaining the SINR at the legitimate user Bob and the eavesdropper Eve, which can then be used to calculate the instantaneous security rate.

[0091] Since SINR is defined as the power ratio, and power is the square of the signal amplitude (or the square of the modulus), all terms in the numerator and denominator must be squared (or the square of the modulus) to represent power.

[0092] The signal-to-interference-plus-noise ratio (SINR) at Bob's location is obtained using the following formula:

[0093] In the formula, Represents the RIS-Bob channel vector. express The conjugate transpose of . This represents the variance of the thermal noise at Bob. This represents the signal-to-interference-plus-noise ratio (SINR) at Bob's location (a metric for the quality of the signal received by the legitimate user Bob).

[0094] The signal-to-interference-plus-noise ratio (SINR) at Eve is obtained using the following formula:

[0095] In the formula, Represents the RIS-Eve channel vector. express The conjugate transpose of . This represents the variance of the thermal noise at Eve; Represents the square of the modulus. Represents the square of the L2 norm; This represents the signal-to-interference-plus-noise ratio (SIR) at Eve (a measure of the signal quality received by Eve, the eavesdropper).

[0096] In this embodiment, the thermal noise variance at Bob and the thermal noise variance at Eve are fixed parameters that are set manually. 90dBm.

[0097] Since the vectors in the formulas for SINR at Bob and SINR at Eve are both complex Gaussian vectors, the square of a complex Gaussian vector is the quadratic expectation of the complex Gaussian vector, which is equal to the square of the vector magnitude multiplied by the noise variance.

[0098] The first item after splitting The square of is Since s is unit power, its squared value is 1, so s is not displayed here; and because middle The size is 1×M. The size is M×M, The size is M×N, The size is N×1; therefore, the overall It is a complex scalar, so it can be used directly. Just calculate its square.

[0099] The second item after splitting middle It is obedience The cyclically symmetric complex Gaussian distribution has the second term. The square of is the quadratic expectation of the complex Gaussian vector, which is equal to the square of the vector's magnitude multiplied by the noise variance. Since the variance includes a diagonal matrix, it can be directly expanded.

[0100] Among them, the third item after splitting In It is obedience The cyclically symmetric complex Gaussian distribution has the third term. The square of is the quadratic expectation of the complex Gaussian vector, which is equal to the square of the vector modulus multiplied by the noise variance. Therefore, it is the square of the L2 norm of the complex Gaussian vector multiplied by the variance in the distribution.

[0101] Among them, the fourth item after splitting It is obedience Since it follows a cyclic symmetric complex Gaussian distribution, the variance of the distribution is directly obtained after squaring it.

[0102] The instantaneous security rate is obtained based on the signal-to-interference-plus-noise ratio (SIR) at Bob's location and the SIR at Eve's location; the specific process of obtaining the instantaneous security rate is expressed by the following formula:

[0103] In the formula, Represents the logarithmic function with base 2. This is the positive part operator; it takes the value 0 when the value inside the parentheses is negative. This represents the instantaneous security rate at the current moment.

[0104] Among them, the instantaneous security rate is the secure communication capability of the system at the current moment, and the object is the information transmission security of the legitimate user Bob relative to the eavesdropper Eve.

[0105] Radar perception modeling: The base station operates in monostatic radar mode, transmitting dual-function waveforms. The target echo propagates along a bidirectional path: BS→RIS→target→RIS→BS. The radar receive SINR (Signal-to-Interference-Ratio) expression is derived, including RCS, uplink / downlink gain, clutter, and noise terms.

[0106] The radar signal-to-interference-plus-noise ratio (SINR) is specifically expressed by the formula:

[0107] In the formula, This represents the base station receive vector (obtained by sampling the echo signal from the base station radar receiver). This indicates the target's angle relative to RIS. This represents the target complex reflection coefficient (0.5 in this embodiment, but the coefficient is not specifically limited). This represents the guidance vector pointing towards the target. Indicates clutter power (in) 100dBm to (Choose any value from 80dBm) This represents the thermal noise power of the base station radar receiver (in this embodiment, it is taken as...). 90dBm), This represents the noise power of the active RIS amplifier (corresponding to the variance in the distribution; this value is artificially set to a fixed value, which is -80dBm in this embodiment). SINR (Signal Intensity Reduction) is a metric used in radar reception to measure the quality of the received signal. It represents the ratio of useful signal power to interference and noise power, usually expressed in decibels (dB). A high SINR value indicates better signal quality, which helps improve the radar's target detection and tracking capabilities. This represents the transpose symbol.

[0108] in, , Let V be the variance of the cyclic symmetric complex Gaussian distribution of the target complex reflection coefficient.

[0109] Stealth constraint modeling: Calculate the total radiated power at the RIS output (including useful signal, hardware distortion, and RIS self-noise). Based on the Neyman-Pearson criterion for energy detectors, establish a probabilistic expression for the detection of RIS radiation by an eavesdropper, and set an upper limit constraint for stealth.

[0110] The total radiated power of the RIS is specifically expressed by the formula:

[0111] In the formula, This is the trace operation of a matrix, which is the sum of the diagonal elements; The square of the Frobenius norm is equal to the sum of the squares of the amplitudes of the elements in a diagonal matrix. This represents the total radiated power of the RIS.

[0112] With the goal of maximizing instantaneous security rate, and constrained by the radar signal-to-interference-plus-noise ratio threshold, the legitimate user communication quality threshold, the base station transmit power upper limit, the stealth upper limit of the total radiated power of the RIS, the upper limit of the adjustable amplitude gain of the RIS unit, and the adjustable phase shift range, a jointly constrained non-convex optimization problem is constructed.

[0113] The construction process of the jointly constrained nonconvex optimization problem is as follows: The objective function in a jointly constrained nonconvex optimization problem is specifically expressed as:

[0114] In the formula, Describe the target optimization parameters in the output. This represents the value of the independent variable that maximizes the objective function; The constraints in the joint constraint nonconvex optimization problem are specifically expressed as follows:

[0115]

[0116]

[0117]

[0118]

[0119]

[0120] In the formula, This indicates the minimum SINR threshold for the radar (10dB in this embodiment). This indicates Bob's quality of service threshold (8dB in this example). This represents the base station's maximum transmit power budget (30 dBm in this example). This indicates the upper limit of RIS radiation stealth. Indicates all All are true; This indicates the total number of RIS units. Indicates the first Adjustable amplitude gain of each RIS unit This represents the maximum value of the amplitude gain of the RIS unit (an inherent hardware parameter, which is 2dB in this embodiment). Indicates the first Adjustable phase shift of each RIS unit; The square of the Frobenius norm of the base station beamforming vector; Among them, the minimum radar SINR threshold, Bob's quality of service threshold, and the base station's maximum transmit power budget are all preset values; Among them, the first formula in the constraints is the radar SINR constraint, the second formula is the legitimate user QoS (Quality of Service) constraint, the third formula is the BS power constraint, the fourth formula is the RIS stealth constraint, the fifth formula is the RIS element amplitude constraint, and the sixth formula is the RIS element phase constraint.

[0121] Among them, the RIS stealth constraint is a stealth constraint introduced in this scheme: it explicitly sets an upper bound for the RIS output radiation, making it difficult to distinguish from the environmental noise floor. Two points need to be clarified here: First, some early drafts showed... The constraint has been removed because These are fixed hardware properties, not optimization variables. Second, the rank-one constraints used in the semidefinite relaxation (SDR) method... This is automatically satisfied in the single-stream scenario described in this paper, and is therefore redundant.

[0122] Eve, the eavesdropper, deploys an energy detector to determine if the RIS is radiating signals. This detector works by measuring the total signal energy in the environment; if it exceeds a certain threshold, the RIS is considered to be radiating (i.e., "detected"). Anomaly detection is performed using an energy detector based on the Neyman-Pearson criterion. The anomaly detection process involves inputting the RIS radiated power, environmental noise statistics (mean and standard deviation), and outputting a false alarm probability (a measure of the probability that the RIS has been detected by the eavesdropper).

[0123] Among them, the upper limit of RIS radiation stealth. It is obtained by reverse calculation based on the preset false alarm rate. The specific process of reverse calculation to obtain the upper limit of RIS radiation stealth is as follows: The first step is to preset Eve's allowed false alarm rate. In this embodiment, Eve is preset to allow a false alarm rate. The value is set to 0.01. In this embodiment, the false alarm rate allowed by the preset Eve is not specifically limited, and the implementer can determine it according to the specific situation. The second step is to analyze the statistical characteristics and false alarm rate of environmental noise. deduce the detection threshold Specifically, this can be expressed as a formula:

[0124] In the formula, This represents the mean signal energy at the background noise level. The standard deviation of the signal energy at the background noise level. For the standard Gaussian Q function, for inverse function, To detect threshold, False alarm rate; The third step is to establish the detection probability expression, specifically as follows:

[0125] In the formula, This represents the offset of RIS radiation relative to the noise floor threshold. This represents the probability that the eavesdropper will detect RIS radiation. By using the detection probability expression and taking the preset tolerance value as the probability value, the upper limit of RIS radiation stealth is derived. .

[0126] It should be noted that by controlling the actual radiated power of RIS to not exceed a certain upper limit, it becomes difficult to distinguish from environmental noise, thereby reducing the risk of being detected by Eve.

[0127] Solving the joint constrained nonconvex optimization problem P1 presents four difficulties: the objective function... and Neither of them are convex; Constrained by non-convex amplitude-phase coupling constraints (the fifth and sixth constraints); Eve's CSI (mainly referring to...) The channel at the Eve end (i.e., the channel itself) is typically known only statistically, making the problem partially observable; the time-varying nature of the channel requires millisecond-level online decision-making. These characteristics lead us to propose an adversarial deep reinforcement learning method in the following steps.

[0128] This step establishes an active RIS-assisted ISAC security system model, including Ricean fading channel, active RIS reflection model, signal transmission model under hardware impairment, radar sensing model, and radiation stealth constraint model. It also forms a joint optimization problem with the goal of maximizing the security rate while taking into account radar SINR, communication QoS, transmit power, hardware impairment, and stealth constraints.

[0129] Step S2: Model the optimization problem as POMDP; define the state space, action space and reward function in POMDP through the base station-active RIS channel matrix, active RIS-Bob channel vector, active RIS-Eve channel vector, instantaneous security rate, adjustable amplitude gain and adjustable phase shift of RIS unit.

[0130] It should be noted that, due to the characteristics of the joint constrained nonconvex optimization problem (P1) being high-dimensional, nonconvex, and constrained coupled, and the fact that the channel state information (CSI) of the eavesdropper cannot be directly observed (only partially observable) in the actual dynamic environment, in order to transform this complex dynamic decision problem into a form suitable for solving by a deep reinforcement learning framework, it is modeled as a partially observable Markov Decision Process (POMDP). The state space (containing observable channel estimates, historical information, remaining resources, and hardware states) and action space (containing the base station beamforming vector and RIS reflection coefficient to be optimized) are explicitly defined, thus laying the foundation for the subsequent online learning and decision-making of the agent.

[0131] Specifically, define a POMDP tuple: construct the tuple. ;in, For the real state space, For the action space, Let be the state transition probability. For the reward function, This is the discount factor. Wherein, the discount factor... Round length (e.g., 3000 steps).

[0132] The optimization problem is modeled as a POMDP. The state space, action space, and reward function in the POMDP are defined by the base station-active RIS channel matrix, the active RIS-Bob channel vector, the active RIS-Eve channel vector, the instantaneous security rate, the adjustable amplitude gain of the RIS unit, and the adjustable phase shift.

[0133] The system presets the BS-RIS channel matrix error, the RIS-Bob channel vector error, and the RIS-Eve channel vector error. It then adds the BS-RIS channel matrix to the preset BS-RIS channel matrix error to obtain an estimate of the BS-RIS channel matrix. Finally, it adds the RIS-Bob channel vector to the preset RIS-Bob channel vector error to obtain an estimate of the RIS-Bob channel vector. Finally, it adds the RIS-Eve channel vector to the preset RIS-Eve channel vector error to obtain an estimate of the RIS-Eve channel vector.

[0134] In this embodiment, the Frobenius norms of the preset BS-RIS channel matrix error, the preset RIS-Bob channel vector error, and the preset RIS-Eve channel vector error are all less than or equal to the preset error thresholds. In this embodiment, the preset error thresholds are all 1 dB. In this embodiment, the preset error thresholds are not specifically limited, and the implementer can determine them according to the specific circumstances.

[0135] The specific process of constructing the state space in POMDP is as follows:

[0136] In the formula, This represents the estimated value of the RIS-Eve channel vector at the current time t. This represents the estimated value of the RIS-Bob channel vector at the current time t. This represents the estimated value of the BS-RIS channel matrix at the current time t. This represents the true state space at the current time t. For matrix column vectorization operators, To take the real part of the complex number, To take the imaginary part of a complex number, It is the transpose symbol. This represents the instantaneous security rate at the previous moment. This represents the remaining available transmission power at the current time t. This represents the hardware impairment coefficient (which can be used to characterize the drift of hardware state over time).

[0137] Wherein, the remaining available transmit power at the current time t In the formula, This represents the base station's maximum transmit power budget, i.e., the maximum total power that the base station is allowed to output; in this embodiment... It is 30dBm, where in this embodiment... No specific restrictions are imposed; implementers can decide based on the specific circumstances.

[0138] The specific process of constructing the action space in POMDP is as follows:

[0139] In the formula, This represents the base station beamforming vector at the current time t. This represents the adjustable amplitude gain of the first RIS unit at the current time t. This represents the adjustable amplitude gain of the M-th RIS unit at the current time t. This represents the adjustable phase shift of the first RIS unit at the current time t. This represents the adjustable phase shift of the Mth RIS unit at the current time t; This represents the action space at the current time t.

[0140] in, express The expansion of the imaginary part is... 3D real vector; express The expansion of the real part is, i.e., 3D real vector.

[0141] Among them, the action space is A vector space of real dimensions. The output of the Actor network. Mapped to via the Sigmoid function , Scaling the mapping using the tanh function .

[0142] The reward function in POMDP is specifically expressed by the following formula:

[0143] In the formula, For the reward function, This is the first weighting coefficient, used to prioritize the security gain; This is the second weighting coefficient. The third weighting coefficient, and Used to punish violations of constraints; The fourth weighting coefficient is used to constrain the transmit power to improve energy efficiency; these four weighting coefficients are used to balance the objective term and each constraint penalty term. This indicates the violation amount when the radar SINR does not meet the threshold. This indicates the amount of violation when the RIS radiation power exceeds the stealth threshold; This represents the loss term resulting from the coupling between hardware impairment and transmit power. By appropriately selecting weights, the objective function prioritizes security gain while imposing heavy penalties for constraint violations. The four weight coefficients were determined through simulation debugging (trial and error method) to achieve a balance between the penalty for each constraint and the security rate gain.

[0144] In this embodiment, the first weighting coefficient Set to 1, second weighting coefficient Take 6, the third weighting coefficient Take 10, the fourth weighting coefficient The value is set to 0.1, where in this embodiment the first weighting coefficient is... Second weighting coefficient Third weighting coefficient and the fourth weighting coefficient No specific limitations are set; the implementer can decide based on the specific circumstances.

[0145] This step models the original optimization problem as a partially observable Markov decision process, defining a state space that includes channel estimation, historical security rate, remaining power, and hardware impairment coefficient, as well as an action space that includes the base station beamforming vector and the amplitude / phase of the RIS unit.

[0146] The specific steps for solving the optimization problem within the POMDP framework are as follows: An adversarial CSI generator for GAN is constructed. The synthesized CSI error output by the generator is superimposed onto the active RIS-Eve channel vector to obtain adversarial CSI samples. The state space in POMDP is adjusted using the adversarial CSI samples. The A-MPC-TD3 framework is constructed. The current state of POMDP is input into the A-MPC-TD3 framework (Adaptive ModelPredictive Control-based Twin Delayed Deep Deterministic Policy Gradient), and the current step action vector is output. After the current step action vector is applied to the ISAC security system, it is transferred to the next POMDP state, forming a closed loop, and iterating sequentially. When the number of iterations reaches the preset maximum number of steps, the iteration stops, and the final action at the time of stopping is taken as the optimal solution to the optimization problem.

[0147] Step S3: Construct an adversarial CSI generator for GAN, and superimpose the synthesized CSI error output by the generator onto the active RIS-Eve channel vector to obtain adversarial CSI samples; adjust the state space in POMDP using the adversarial CSI samples.

[0148] It should be noted that, since the channel state information (CSI) of the eavesdropper (Eve) in a real system cannot be directly observed and is subject to uncertainty, and traditional methods usually assume ideal CSI or random error, the trained policy is not robust enough in the face of the worst case. Therefore, in order to simulate the extreme channel error that may occur in the real environment and force the decision network to learn the ability to cope with the worst conditions, a generative adversarial network (GAN) is introduced to synthesize the eavesdropper channel error in the worst case. Through the game between the generator and the discriminator and the guidance of auxiliary loss, the most challenging perturbation to the decision network is generated, thereby enhancing the policy's robustness to CSI uncertainty.

[0149] Specifically, an adversarial CSI generator based on a Generative Adversarial Network (GAN) is constructed, which generates adversarial errors only for active RIS-Eve channels. A generator is defined to map random noise vectors to synthetic CSI errors. A discriminator is defined to distinguish between real active RIS-Eve channel samples and adversarial CSI samples output by the generator. An objective function for generative adversarial training is set. An auxiliary loss is introduced, which is the negative expectation of the Critic network output value in the TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm, used to guide the generator to generate worst-case perturbations that minimize the Critic network's estimate. The synthetic CSI errors output by the generator are superimposed on real active RIS-Eve channels to obtain adversarial CSI samples. The generator, discriminator, and TD3 algorithm are trained alternately. GAN stands for Generative Adversarial Network.

[0150] The process of obtaining the random noise vector includes random sampling from a standard normal distribution with a mean of 0 and a variance of the identity matrix.

[0151] The adversarial CSI samples are obtained by superimposing the CSI error onto the RIS-Eve channel vector. The RIS-Eve channel vector is then used as a positive sample. The adversarial CSI samples and the positive samples are then input into the discriminator to output the true probability.

[0152] The generator and discriminator are alternately optimized to make the CSI error generated by the generator as close as possible to the true error distribution, and the discriminator to distinguish between real and generated samples as much as possible.

[0153] The objective function for generative adversarial training is specifically expressed as:

[0154] In the formula, Represents generator Minimize target, discriminator Maximize the goal; The objective function (value function) of GAN; Represents the true RIS-Eve channel vector The expected value is taken to follow the true data distribution. This represents the logarithm of the true probability output by the discriminator for a positive sample. This means taking the expected value of the adversarial CSI samples following a prior distribution (usually a standard normal distribution); To counter CSI samples.

[0155] To guide To generate the most adversarial error for TD3, this paper adds an auxiliary loss, that is, the loss function in the generator is obtained by weighted fusion of the original generator adversarial loss and the auxiliary loss; the original generator adversarial loss is a well-known technique and will not be described in detail here.

[0156] The auxiliary loss is specifically expressed as follows:

[0157] In the formula, This is the weighting coefficient (a preset value, set to 0.1 in this example). This indicates the critical network for TD3. This represents the loss function of GAN during the training phase; This represents the state vector. This represents the actor network of TD3.

[0158] The discriminator's loss is the binary cross-entropy loss; the binary cross-entropy loss is a well-known technique and will not be elaborated on here.

[0159] The generator, discriminator, and TD3 algorithm are trained alternately.

[0160] The specific process of alternating training is as follows: First, initialize the generator, discriminator, TD3 Actor network, Critic network and its target network, and experience replay buffer. Then, repeat the following steps until convergence: after each round of GAN training (fix the generator to update the discriminator, then fix the discriminator to update the generator, and use the auxiliary loss to guide the generator to generate the worst-case CSI in the direction most unfavorable to the current TD3 policy), i.e., use the current generator to generate adversarial CSI samples and inject them into the Eve channel; then, perform a round of TD3 training (the agent outputs the initial action in the current state through the Actor, refines and projects it through MPC to obtain the final action, interacts with the environment to obtain the reward and the next state and stores them in the experience replay buffer, samples small batches to update the Critic, delays the update of the Actor, and softly updates the target network); this process is repeated alternately.

[0161] In this step, a generative adversarial network is introduced. The generator synthesizes the worst-case eavesdropper channel error and uses a discriminator to distinguish it. At the same time, the auxiliary loss guides the generator to generate the most challenging perturbation to the decision network, so as to enhance the robustness of the policy to CSI uncertainty.

[0162] Step S4: Construct the A-MPC-TD3 framework, input the current state of POMDP into the A-MPC-TD3 framework, and output the current step action vector; apply the current step action vector to the ISAC security system, and then transfer to the next POMDP state to form a closed loop, and iterate in sequence; stop when the number of iterations reaches the preset maximum number of steps, and take the final action at the time of stopping as the optimal solution to the optimization problem.

[0163] It should be noted that since the initial actions output by the TD3 algorithm may violate multiple non-convex constraints such as radar SINR, communication QoS, transmit power, and RIS radiation stealth, direct execution will lead to a decrease in system performance or constraint failure. Therefore, in order to achieve high-dimensional joint optimization while satisfying all physical constraints, and to take into account the efficient exploration of deep reinforcement learning and the feasibility of model predictive control, an MPC module is introduced to solve the quadratic constraint programming in the short time domain to refine the actions. The TD3 output and the MPC output are then combined convexly and projected onto the feasible amplitude and phase set, thereby achieving safe, reliable and real-time online beamforming decision-making.

[0164] Specifically, an A-MPC-TD3 framework is constructed. The current state of the POMDP is input into the A-MPC-TD3 framework, and the current step action vector is output. After the current step action vector is applied to the ISAC security system, it is transferred to the next POMDP state to form a closed loop, and the iteration is performed sequentially. When the number of iterations reaches the preset maximum number of steps, the iteration stops, and the final action at the time of stopping is taken as the optimal solution to the optimization problem.

[0165] The specific process of iterative optimization is as follows: A framework A-MPC-TD3 is constructed, comprising a TD3 module and a Model Predictive Control (MPC) module. The following steps are iteratively executed until a preset maximum number of steps is reached: The current state of the POMDP is input into the Actor network of the TD3 module, outputting an initial action vector; the initial action vector is input into the MPC module, and a quadratic constraint programming problem is solved in the rolling time domain, with the objective of minimizing the distance between the MPC output action and the initial action vector, and with all constraints in the joint constraint non-convex optimization problem as the feasible region, resulting in a refined action vector; the initial action vector and the refined action vector are fused using convex combination weight coefficients to obtain a fused action vector; the amplitude gain of each RIS unit in the fused action vector is projected to its hardware upper limit interval, and the phase shift of each RIS unit is projected to its feasible region interval to obtain the current step action vector; the current step action vector is executed, the reward is calculated, and experience samples are stored, updating the Actor network and Critic network parameters of the TD3 module; then, the next POMDP state is entered, and the next iteration continues; the iteration stops when the preset maximum number of steps is reached, and the final action at the time of stopping is taken as the optimal solution to the optimization problem.

[0166] The Critic network parameters update themselves by minimizing the error between the predicted Q value and the TD target; the specific process is as follows: The objective of temporal differential (TD) is:

[0167] The noise smoothing of the target action is processed as follows:

[0168] The Critic network parameters are updated in the following manner:

[0169] In the formula, The target value; For instant rewards; Discount factor; The state at the next moment; (i=1,2) is a pair of target Critic networks in the TD3 algorithm. The target Critic network is a delayed copy of the current Critic network and is slowly updated through Polyak soft update to provide a stable target Q-value estimate and alleviate the overestimation problem. Let be the trainable parameter vector of the Critic network for the i-th target; The target Actor network is a delayed copy of the current Actor network. Its function is to generate target actions with smoothed noise, further suppressing overestimation of target values ​​and improving training stability. For the trainable parameter vector of the target Actor network; The action after applying smoothed noise to the target action; To truncate Gaussian smoothed noise, Its standard deviation, To truncate the boundary, clip(·) is the truncation function; This refers to the number of samples in a small batch. This is the current Critic's valuation of the state-action pair. This represents the value of the independent variable that makes the function reach its minimum value.

[0170] Among them, the TD objective is the core formula used to update the Critic network in the TD3 algorithm, and it belongs to the training process of the TD3 module.

[0171] Specifically, the TD3 Actor outputs the initial action: the current state is input into the TD3 Actor network, and the initial action vector is output. Includes the real / imaginary part of the base station beamforming vector w and the amplitude gain of the RIS unit. and phase shift .

[0172] The quadratic constrained programming is specifically expressed by the following formula:

[0173] In the formula, H is the time domain length of MPC prediction; This is a relative time index within the MPC rolling time domain. ; The refined actions calculated by MPC; The original (potentially constraint-violation) action given to the TD3 Actor. This represents the square of the vector's 2-norm, measuring the distance between MPC and TD3 actions. It is a minimum value function.

[0174] The refined action is obtained by using quadratic constraint programming. .

[0175] The specific formula for obtaining the fused action vector in convex combination fusion is as follows:

[0176] In the formula, These are the weighting coefficients. To refine the action vectors, As the initial action vector, To fuse action vectors.

[0177] In this embodiment, the weighting coefficient The value is set to 0.7, where in this embodiment the weighting coefficient is... No specific restrictions are imposed; implementers can decide based on the specific circumstances.

[0178] First project to the feasible set: The amplitude gain in the projection to Interval, phase shift projection to The interval is used to obtain the final action. Next, environmental interaction is performed: Execution The reward is calculated and the experience replay buffer is updated. Then, the TD3 network is updated: batch data is sampled from the buffer, and the Actor and Critic network parameters are updated (using standard TD3 procedures such as delayed update strategy and target network smoothing). Finally, MPC and TD3 alternate: after every K steps of TD3 update, MPC refinement is performed again.

[0179] In this step, the initial action is output using a TD3 dual-critic network. The model predictive control module is introduced to solve the quadratic constraint programming in the short time domain to refine the action to satisfy various constraints. The TD3 output and the MPC output are then combined convexly and projected onto the feasible amplitude phase set to realize online decision-making for safe beamforming under high-dimensional non-convex constraints.

[0180] The specific simulation parameters for this scheme are given in Table 1 below.

[0181] Table 1 Simulation Parameter Table

[0182] The value of 256-256-256 for the width of the TD3 hidden layer indicates that there are three hidden layers, each with 256 neurons.

[0183] Figure 2 The changes in cumulative reward during training are illustrated, and the proposed A-MPC-TD3 is compared with three baseline learning algorithms. The proposed algorithm exhibits the fastest learning speed in the early stages of training and reaches stable convergence in far fewer rounds than other methods, while all baseline methods converge slowly and achieve significantly lower stable reward levels. The advantage of the proposed scheme in the early stages of training stems from the MPC refiner: it ensures the feasibility of actions during the random exploration phase of the TD3 algorithm, greatly improving the efficiency of action exploration. In the later stages of training, the adversarial GAN ​​module continuously exposes the algorithm to extreme channel error scenarios, effectively avoiding performance degradation caused by overfitting and ensuring more stable convergence performance and less fluctuation. Although the BCD warm-start method initializes the learning process with a feasible solution, it is trapped in a local optimum and cannot achieve further performance improvement due to the suboptimal nature of the BCD solution under hardware impairment conditions. Overall, the proposed algorithm not only achieves significantly higher cumulative rewards than all baseline methods, but also converges at a faster speed, fully demonstrating that the integrated design of the three modules, GAN, TD3 and MPC, plays a complementary role, rather than just an incremental performance improvement.

[0184] Figure 3The curves depicting the confidentiality rate as a function of the number of active RIS units are presented. All considered schemes exhibit a gradual and marginally decreasing performance improvement with increasing unit number, but significant performance gaps exist between different methods. The proposed A-MPC-TD3 maintains a significant performance advantage across all unit sizes. This advantage is particularly pronounced when reflection hardware resources are limited, thanks to the optimization amplitude adjustment strategy supported by the MPC module. In contrast, traditional comparative schemes employ conservative power allocation, leading to insufficient resource utilization and limited secure transmission performance. As the number of RIS units continues to increase, the performance improvement of all schemes gradually saturates, indicating that further increases in reflection units can no longer effectively suppress eavesdropping channels, and the quality of the legitimate channel gradually becomes the dominant factor. Furthermore, the active RIS scheme using artificial noise assistance achieves a moderate confidentiality gain by interfering with eavesdropping signals, but this is accompanied by a decrease in stealth performance. Meanwhile, the passive RIS scheme consistently exhibits the worst confidentiality capacity, further validating the inherent fading limitations of passive reflection transmission.

[0185] Figure 4 The stealth performance, measured by the probability of eavesdropper detection, was evaluated under different RIS radiated powers. The detection probability of all schemes showed a gradually increasing trend, but the critical power level required to trigger effective eavesdropping detection varied greatly. The active RIS scheme using AN suffered severe stealth performance degradation at low radiated powers due to the active injection of high-power noise, making it highly detectable. Traditional RIS baseline schemes reached the detectability threshold at medium power levels, exhibiting limited stealth capabilities. In stark contrast, the proposed A-MPC-TD3 maintained good stealth over a wide power range and delayed the appearance of detectable signals to higher power levels, achieving a significant safety margin against eavesdropping detection. Under typical operating conditions, the proposed method had extremely low detection risk, far surpassing other comparative methods in stealth performance, validating its inherent "stealth defense" capability. By coordinating the amplitude and phase of the reflector units, the proposed framework focused signal energy on legitimate users and sensing targets while suppressing unnecessary external radiation, reducing overall signal exposure.

[0186] Figure 5The radar SINR performance under different hardware impairment levels was characterized. As the hardware impairment coefficient gradually increases, the proposed A-MPC-TD3 only exhibits a slight performance degradation, maintaining stable radar detection capability. In contrast, all baseline RIS-based methods suffer more severe SINR attenuation, with the passive RIS scheme showing the weakest damage resistance. The active RIS-assisted scheme using AN maintains competitive performance under all impairment conditions, approaching the proposed method, but still showing a stable performance gap. The strong hardware robustness of the proposed framework stems from two core designs. First, the carefully designed reward constraint suppresses excessive power allocation to directions sensitive to hardware distortion, guiding the strategy to select reliable transmission directions with good RIS reflection gain and distortion tolerance. Second, the inherent amplification capability of the active RIS effectively offsets the increased noise floor caused by hardware defects, a unique advantage that passive RIS cannot match. This superior performance under severe hardware impairment demonstrates that active RIS brings a fundamental enhancement to structural robustness, not just marginal performance improvement.

[0187] Figure 6 The performance variation with increasing BS transmit power was investigated. The security rate of the proposed scheme steadily increases with increasing power budget. All baseline methods exhibit a clear performance saturation trend at high transmit power: once the power exceeds a certain threshold, further power increases offer limited improvement to legitimate communication while also benefiting the eavesdropping link to a similar degree. The proposed method maintains a stable performance advantage over other schemes across the entire power range, maintaining continuous gain relative to both active and passive RIS baselines. Two key phenomena were observed. First, the performance advantage of the proposed algorithm is more pronounced at low transmit power. In legitimate channels with low SNR scenarios, the amplification capability of active RIS can be fully utilized, effectively enhancing secure transmission. As transmit power increases, the performance gap gradually narrows. Second, the AN-based scheme demonstrates increasingly strong competitiveness at high power levels, surpassing the active RIS baseline. This indicates that under high power conditions, interference suppression against eavesdroppers plays a more dominant role than the limitations of fixed beam design. Overall, the proposed A-MPC method consistently provides optimal security performance across all power settings, fully validating the effectiveness of its dedicated security amplification defense mechanism.

[0188] This concludes the embodiment.

[0189] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0190] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, systems, and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0191] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0192] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0193] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the present invention.

Claims

1. An improved RIS-assisted ISAC secure beamforming method for TD3, characterized in that, include: Construct an active RIS-assisted ISAC security system; The ISAC security system consists of a base station, an active RIS, a legitimate user Bob, an eavesdropper Eve, and a sensing target; the active RIS contains multiple RIS units; a base station-active RIS channel matrix, an active RIS-Bob channel vector, and an active RIS-Eve channel vector are established based on the Ricean fading channel. Obtain the adjustable amplitude gain and adjustable phase shift of each RIS unit, and construct the reflection matrix of the active RIS of all RIS units based on the adjustable amplitude gain and the adjustable phase shift; The base station transmits the signal, and the theoretical received signals at Bob and Eve are calculated based on the base station-active RIS channel matrix, the reflection matrix of the active RIS, and the base station transmit signal. The signal-to-interference-plus-noise ratio (SIR) at Bob and Eve is calculated by decomposing the terms of the theoretical received signal. The instantaneous security rate at the current moment is obtained based on the SIR at Bob and Eve. Obtain the radar signal-to-interference-plus-noise ratio and the total radiated power of the RIS; With the goal of maximizing instantaneous security rate, and with the radar signal-to-interference-plus-noise ratio threshold, the legitimate user communication quality threshold, the base station transmit power upper limit, the stealth upper limit of the total radiated power of the RIS, the adjustable amplitude gain upper limit of the RIS unit, and the adjustable phase shift range as constraints, a joint constrained non-convex optimization problem is constructed. The optimization problem is modeled as POMDP; the state space, action space and reward function in POMDP are defined by the base station-active RIS channel matrix, active RIS-Bob channel vector, active RIS-Eve channel vector, instantaneous security rate, adjustable amplitude gain and adjustable phase shift of RIS unit. The specific steps for solving the optimization problem within the POMDP framework are as follows: An adversarial CSI generator for GAN is constructed. The synthesized CSI error output by the generator is superimposed onto the active RIS-Eve channel vector to obtain adversarial CSI samples. The state space in POMDP is adjusted using the adversarial CSI samples. Construct the A-MPC-TD3 framework, input the current state of POMDP into the A-MPC-TD3 framework, and output the current step action vector; apply the current step action vector to the ISAC security system, and then transfer to the next POMDP state to form a closed loop, and iterate in sequence; stop when the number of iterations reaches the preset maximum number of steps, and take the final action at the time of stopping as the optimal solution to the optimization problem.

2. The improved TD3 RIS-assisted ISAC secure beamforming method according to claim 1, characterized in that, The base station-active RIS channel matrix, active RIS-Bob channel vector, and active RIS-Eve channel vector are established based on the Ricean fading channel. include: The BS-RIS channel matrix is ​​specifically expressed by the following formula: In the formula, This indicates the preset Rice factor of the BS-RIS link. This represents the line-of-sight matrix of the BS-RIS channel. Represents the non-line-of-sight matrix of the BS-RIS channel; Represents the BS-RIS channel matrix; The RIS-Bob channel vector is specifically expressed by the following formula: In the formula, This represents the preset Rice factor of the RIS-Bob link. This represents the line-of-sight vector of the RIS-Bob channel. Represents the non-line-of-sight vector of the RIS-Bob channel. Represents the RIS-Bob channel vector; The RIS-Eve channel vector is specifically expressed by the following formula: In the formula, This represents the preset Rice factor of the RIS-Eve link. This represents the line-of-sight vector of the RIS-Eve channel. This represents the non-line-of-sight vector of the RIS-Eve channel. Represents the RIS-Eve channel vector; Among them, the line-of-sight matrix of the BS-RIS channel, the line-of-sight vector of the RIS-Bob channel, and the line-of-sight vector of the RIS-Eve channel are all calculated using array steering vectors; The elements in the non-line-of-sight matrix of the BS-RIS channel, the non-line-of-sight vector of the RIS-Bob channel, and the non-line-of-sight vector of the RIS-Eve channel are all randomly generated using a standard complex Gaussian distribution.

3. The improved TD3 RIS-assisted ISAC secure beamforming method according to claim 1, characterized in that, The process of obtaining the adjustable amplitude gain and adjustable phase shift of each RIS unit, and constructing the reflection matrix of the active RIS of all RIS units based on the adjustable amplitude gain and the adjustable phase shift, includes: In the formula, This represents the reflection matrix of an active RIS. Indicates the first The reflection coefficient of each RIS unit. This indicates the total number of RIS units. This represents the diagonalization operator, which arranges a scalar sequence into a diagonal matrix. Indicates the first Adjustable amplitude gain of each RIS unit This represents the maximum value of the amplitude gain of the RIS unit; Indicates the first Adjustable phase shift of each RIS unit Represents a complex exponential function; represents phase rotation. Represents pi; The initial adjustable amplitude gain and initial adjustable phase shift of each RIS unit are... and Uniformly sampled and randomly generated within two constraint ranges; The adjustable amplitude gain and adjustable phase shift of each subsequent RIS unit are obtained by the A-MPC-TD3 framework through each iteration.

4. The improved TD3 RIS-assisted ISAC secure beamforming method according to claim 1, characterized in that, The acquisition of the base station transmitted signal, and the calculation of the theoretical received signals at Bob and Eve based on the base station-active RIS channel matrix, the active RIS reflection matrix, and the base station transmitted signal, include: The expression for the base station's transmitted signal is: In the formula, Represents the base station's transmitted signal vector. Represents the base station beamforming vector. It is a complex scalar; This represents the additive distortion noise vector caused by hardware damage. Indicates the hardware damage coefficient. express The conjugate transpose of . This represents a cyclically symmetric complex Gaussian distribution. This represents the variance in the distribution; This means constructing a diagonal matrix from the diagonal elements of the outer product matrix of the base station beamforming vectors; Among them, the base station beamforming vector The initial value is randomly generated by sampling from a complex Gaussian distribution with a mean of 0 and a variance of 1; the subsequent w is obtained by the A-MPC-TD3 framework through each iteration; The theoretical expressions for the received signals at Bob and Eve are: In the formula, This indicates the noise power of the active RIS amplifier. It is an M×M identity matrix; This represents the additive Gaussian noise vector introduced by the active RIS amplifier; Indicates user Thermal noise at the location, This indicates the thermal noise power at the user receiver. express The set of values ​​is ,gather In this context, 'b' represents the legitimate user Bob, and 'e' represents the eavesdropper Eve. Let H represent the channel vector from RIS to user u, where H is the conjugate transpose. Represents the BS-RIS channel matrix. This indicates the received signal at user u. This represents the reflection matrix of an active RIS; Among them, the noise power of the active RIS amplifier Thermal noise power at the user receiver This is the default value.

5. The improved TD3 RIS-assisted ISAC secure beamforming method according to claim 4, characterized in that, The calculation yields the signal-to-interference-plus-noise ratio (SIR) at Bob and the SIR at Eve. Based on the signal-to-interference-plus-noise ratio (SIR) at Bob's location and the SIR at Eve's location, the instantaneous security rate at the current moment is obtained, including: The signal-to-interference-to-noise ratio at Bob is specifically expressed by the formula: In the formula, Represents the RIS-Bob channel vector. express The conjugate transpose of . This represents the variance of the thermal noise at Bob. This indicates the signal-to-interference-plus-noise ratio at Bob's location; The signal-to-interference-to-noise ratio at Eve is specifically expressed by the formula: In the formula, Represents the RIS-Eve channel vector. express The conjugate transpose of . This represents the variance of the thermal noise at Eve; Represents the square of the modulus. This represents the square of the second norm of a vector; This indicates the signal-to-interference-plus-noise ratio at Eve; In the formula, Represents the logarithmic function with base 2. This is the positive part operator; it takes the value 0 when the value inside the parentheses is negative. This represents the instantaneous security rate at the current moment.

6. The improved TD3 RIS-assisted ISAC secure beamforming method according to claim 5, characterized in that, The acquisition of radar signal-to-interference-plus-noise ratio and total radiated power of RIS includes: The radar signal-to-interference-plus-noise ratio is specifically expressed by the formula: In the formula, Represents the base station receive vector. This indicates the target's angle relative to RIS. Represents the complex reflection coefficient of the target. This represents the guidance vector pointing towards the target. Indicates clutter power. This indicates the thermal noise power of the base station radar receiver; For radar signal-to-interference-plus-noise ratio; Indicates the transpose symbol; in, , Let be the variance of the cyclic symmetric complex Gaussian distribution of the target complex reflection coefficient; where the target complex reflection coefficient is a preset value. The total radiated power of RIS can be expressed by the following formula: In the formula, This is the trace operation of a matrix; Denotes the Frobenius norm; This represents the total radiated power of the RIS.

7. The improved TD3 RIS-assisted ISAC secure beamforming method according to claim 6, characterized in that, The process of constructing the joint constraint nonconvex optimization problem is as follows: The objective function in a jointly constrained nonconvex optimization problem is specifically expressed as: In the formula, Describe the target optimization parameters in the output. This represents the value of the independent variable that maximizes the objective function; The constraints in the joint constraint nonconvex optimization problem are specifically expressed as follows: In the formula, Indicates the minimum SINR threshold of the radar. This indicates Bob's service quality threshold. This represents the base station's maximum transmit power budget. This indicates the upper limit of RIS radiation stealth. Indicates all All are true; This indicates the total number of RIS units. Indicates the first Adjustable amplitude gain of each RIS unit This represents the maximum value of the amplitude gain of the RIS unit; Indicates the first Adjustable phase shift of each RIS unit; Among them, the minimum radar SINR threshold, Bob's quality of service threshold, and the base station's maximum transmit power budget are all preset values; Among them, the upper limit of RIS radiation stealth. It is obtained by reverse calculation based on the preset false alarm rate. The specific process of reverse calculation to obtain the upper limit of RIS radiation stealth is as follows: The first step is to preset Eve's allowed false alarm rate. ; The second step is to analyze the statistical characteristics and false alarm rate of environmental noise. deduce the detection threshold Specifically, this can be expressed as a formula: In the formula, This represents the mean signal energy at the background noise level. The standard deviation of the signal energy at the background noise level. For the standard Gaussian Q function, for inverse function, To detect threshold, False alarm rate; The third step is to establish the detection probability expression, specifically as follows: In the formula, This represents the offset of RIS radiation relative to the noise floor threshold. This represents the probability that the eavesdropper will detect RIS radiation. By using the detection probability expression and taking the preset tolerance value as the probability value, the upper limit of RIS radiation stealth is derived. .

8. The improved TD3 RIS-assisted ISAC secure beamforming method according to claim 7, characterized in that, The construction process of the POMDP is as follows: The process of constructing the state space in POMDP is as follows: In the formula, This represents the estimated value of the RIS-Eve channel vector at the current time t. This represents the estimated value of the RIS-Bob channel vector at the current time t. This represents the estimated value of the BS-RIS channel matrix at the current time t. This represents the true state space at the current time t. For matrix column vectorization operators, To take the real part of the complex number, To take the imaginary part of a complex number, It is the transpose symbol. This represents the instantaneous security rate at the previous moment. This represents the remaining available transmission power at the current time t. Indicates the hardware damage coefficient; Remaining available transmit power at current time t In the formula, This represents the base station's maximum transmit power budget; Specifically, the system includes preset BS-RIS channel matrix error, preset RIS-Bob channel vector error, and preset RIS-Eve channel vector error. The BS-RIS channel matrix is ​​estimated by adding the preset BS-RIS channel matrix error to obtain the estimated value of the BS-RIS channel matrix. The RIS-Bob channel vector is estimated by adding the preset RIS-Bob channel vector error to obtain the estimated value of the RIS-Bob channel vector. The RIS-Eve channel vector is estimated by adding the preset RIS-Eve channel vector error to obtain the estimated value of the RIS-Eve channel vector. The specific process of constructing the action space in POMDP is as follows: In the formula, This represents the base station beamforming vector at the current time t. This represents the adjustable amplitude gain of the first RIS unit at the current time t. This represents the adjustable amplitude gain of the M-th RIS unit at the current time t. This represents the adjustable phase shift of the first RIS unit at the current time t. This represents the adjustable phase shift of the Mth RIS unit at the current time t; Represents the action space at the current time t; in, express The imaginary part is expanded; express The real part unfolds; The reward function in POMDP is specifically expressed by the following formula: In the formula, For the reward function, The first weighting coefficient; This is the second weighting coefficient. The third weighting coefficient, and Used to punish violations of constraints; It is the fourth weighting coefficient. This indicates the violation amount when the radar SINR does not meet the threshold. This indicates the amount of violation when the RIS radiation power exceeds the stealth threshold; This represents the loss term that couples hardware damage with transmit power.

9. The improved TD3 RIS-assisted ISAC secure beamforming method according to claim 1, characterized in that, The adversarial CSI generator that constructs the GAN, by superimposing the synthesized CSI error output by the generator onto the active RIS-Eve channel vector, obtains adversarial CSI samples, including: An adversarial CSI generator based on a Generative Adversarial Network (GAN) is constructed, which generates adversarial errors only for active RIS-Eve channels. A generator is defined to map random noise vectors to synthetic CSI errors. A discriminator is defined to distinguish between real active RIS-Eve channel samples and the adversarial CSI samples output by the generator. An objective function for generative adversarial training is set. An auxiliary loss is introduced, which is the negative expectation of the Critic network output value in the TD3 algorithm, used to guide the generator to generate a worst-case perturbation that minimizes the Critic network's estimate. The synthetic CSI errors output by the generator are superimposed onto real active RIS-Eve channels to obtain adversarial CSI samples. The generator, discriminator, and TD3 algorithm are trained alternately. The objective function for generative adversarial training is specifically expressed as: In the formula, Represents generator Minimize target, discriminator Maximize the goal; The objective function of GAN; Represents the true RIS-Eve channel vector The expected value is taken to follow the true data distribution. This represents the logarithm of the true probability output by the discriminator for a positive sample. This means that the adversarial CSI samples should follow a prior distribution and have the expected value. To counter CSI samples; The loss function in the generator is obtained by weighted fusion of the original generator adversarial loss and auxiliary loss; the auxiliary loss is specifically expressed as: In the formula, These are weighting coefficients. This indicates the critical network for TD3. This represents the loss function of GAN during the training phase; Represents the state vector; This represents the actor network of TD3; The discriminator's loss is the binary cross-entropy loss.

10. The improved TD3 RIS-assisted ISAC secure beamforming method according to claim 1, characterized in that, The A-MPC-TD3 framework is constructed by inputting the current state of the POMDP into the A-MPC-TD3 framework and outputting the current step action vector. After applying the current step action vector to the ISAC security system, the process transitions to the next POMDP state, forming a closed loop, and iterating sequentially. The iteration stops when the number of iterations reaches a preset maximum number of steps, and the final action at the point of cessation is taken as the optimal solution to the optimization problem, including: A framework A-MPC-TD3 is constructed, comprising a TD3 module and a Model Predictive Control (MPC) module. The following steps are iteratively executed until a preset maximum number of steps is reached: The current state of the POMDP is input into the Actor network of the TD3 module, outputting an initial action vector; the initial action vector is input into the MPC module, and a quadratic constraint programming problem is solved in the rolling time domain, with the objective of minimizing the distance between the MPC output action and the initial action vector, and with all constraints in the joint constraint non-convex optimization problem as the feasible region, resulting in a refined action vector; the initial action vector and the refined action vector are fused using convex combination weight coefficients to obtain a fused action vector; the amplitude gain of each RIS unit in the fused action vector is projected to its hardware upper limit interval, and the phase shift of each RIS unit is projected to its feasible region interval to obtain the current step action vector; the current step action vector is executed, the reward is calculated and experience samples are stored, and the parameters of the Actor network and Critic network of the TD3 module are updated; then, the next POMDP state is entered, and the next iteration continues; the iteration stops when the preset maximum number of steps is reached, and the final action at the time of stopping is taken as the optimal solution to the optimization problem. The quadratic constrained programming is specifically expressed by the following formula: In the formula, H is the time domain length of MPC prediction; This is the relative time index within the MPC rolling time domain; The refined actions calculated by MPC; The original action given to the Actor in TD3; This represents the square of the second norm of a vector. It is a minimum value function; The specific formula for obtaining the fused action vector is as follows: In the formula, These are the weighting coefficients. To refine the action vectors, As the initial action vector, To fuse action vectors.

Images