Waveform generation method and system of intelligent reflecting surface auxiliary safety communication integrated system based on reinforcement learning

Abstract

The invention relates to a reinforcement learning-based waveform generation method and system for an intelligent reflector-assisted safety sensing integrated system. The method comprises the following steps of: 1, constructing the intelligent reflector-assisted safety sensing integrated system ISAC; 2, constructing an optimization problem; according to known channel state information, obtaining an expression of a signal received by the LU and an expression of a signal received by the BS reflected by the target object, then obtaining a signal to interference plus noise ratio or a signal to noise ratio of the two, and writing an eavesdropping signal to interference plus noise ratio of an eavesdropper target relative to the LU; finally, an objective function and constraint conditions of the optimization problem are established, and optimization parameters are determined at the same time; and step 3, introducing an RL to solve an objective function of an optimization problem, and generating a waveform. According to the waveform design method, the node calculation complexity can be greatly reduced, the system performance is improved, and the performance compromise between communication and sensing is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of wireless communication perception integration, and in particular to a waveform generation method and system for an intelligent reflective surface assisted safety interawareness integrated system based on reinforcement learning. Background Art

[0002] The new generation of wireless networks has been seen as a key enabler for many emerging applications that require high-quality wireless connectivity as well as highly accurate and powerful sensing capabilities. A common theme in the assumptions about B5G / 6G networks is that sensing will play a more important role than ever before. The concept of communication-sensing integration (abbreviated as "synesthesia integration") is gaining increasing attention from academia, industry, and standardization organizations, and has been formally included in the six major usage scenarios of 6G by ITU-R. In future communication systems, the use of higher frequency bands (from centimeter waves, millimeter waves to terahertz), wider bandwidths, and denser large-scale antenna arrays will enable communication and perception to be integrated and mutually enhanced. The entire communication system can act as a sensor to explore the transmission, reflection, and scattering characteristics of wireless signals to "understand" the physical world and provide a completely new user experience. High-precision positioning, imaging, and environmental reconstruction achieved through passive sensing can improve communication performance, achieve more accurate beamforming, faster beam failure recovery, and track channel state information with less signal overhead. This approach is often referred to as "perception-assisted communication."

[0003] The power communication network is an important infrastructure supporting the development of smart grids, ensuring the security, real-time, accuracy and reliability of various power services. With the advancement of smart grid construction, new services such as robot inspection, visual operation and online monitoring of substation status have emerged. These services generally show the characteristics of large bandwidth, mobility and large connection, which cannot be met by traditional communication methods. ISAC has the characteristics of wide bandwidth, low latency and high reliability, and at the same time has high-quality wireless connection and highly accurate sensing capabilities, which provides good technical support for smart grid planning, design and operation.

[0004] RIS (Reconfigurable intelligent surface) is usually composed of a flat metasurface equipped with passive, cost-effective and efficient reflective elements. The introduction of RIS helps to establish additional non-line-of-sight (NLoS) links, thereby providing more degrees of freedom (DoF) to improve the overall system performance. In view of this, the integration of RIS in ISAC (Integrated sensing and communication) systems has attracted a lot of research attention. The advantages of RIS in communication and sensing come from two main factors: new DoF in the channel and reduced hardware costs. RIS acts as a supplementary anchor point to provide additional channel DoF by purposefully modifying the transmitted signal, allowing the receiver to partially understand the electromagnetic environment. However, the existing ISAC system model only considers the environmental congestion of communication or sensing and does not consider both in a congested environment at the same time, with high path loss, and the waveform design of the ISAC system with RIS and the beamforming problem of RIS have high dimensionality and uncertainty. The traditional optimization method has high complexity, and its optimized performance needs to be improved. Summary of the invention

[0005] In view of the shortcomings of existing system design and optimization methods, the present invention proposes a waveform generation method for an intelligent reflective surface assisted safety synaesthesia integrated system based on reinforcement learning.

[0006] The system design considers scenarios closer to the real environment. The communicating LUs (Legitimate users) and the sensing targets are in a congested environment. The signal propagation of the direct link is based on high loss and fading, so they can only establish indirect links through RIS. For the waveform design of the system model and the beamforming of RIS, the decoupling method is used to separate different variables into sub-problems for separate optimization, and the final optimization result is obtained through iteration; RL is introduced to optimize the optimization problem, avoiding the high dimensionality and uncertainty of the environment, and has high generalization.

[0007] The present invention also provides a waveform generation system for an intelligent reflective surface assisted safety synaesthesia system based on reinforcement learning.

[0008] In the present invention, a waveform generation method of an intelligent reflective surface auxiliary safety synaesthesia integrated system based on reinforcement learning includes:

[0009] Step 1: Build an intelligent reflective surface assisted security inter-sensing system ISAC, and introduce RIS to assist communication and sensor to establish a link; in which multiple LUs and BSs perform downlink communication, and at the same time, the BS's transmission waveform is transmitted to the target and reflected back to the BS to complete the function of detecting the target. Assume that the target may also be a potential eavesdropper.

[0010] Step 2: The normal operation of the system requires the BS (Base station) to broadcast a suitable signal waveform through multiple antennas. The reflected signal reaches the RIS through the transmission channel, and reaches the LU and the target object respectively after being reflected by the RIS. The channels of the links in the process are in the same environment. Assuming that the channel state information of all links is known a priori and the channel fading is block fading, it obeys the statistical channel distribution characteristics, so it can be assumed that the channel obeys the Rayleigh distribution.

[0011] Step 3: Based on the known channel state information, we can get the expression of the signal received by LU and the expression of the received signal that is reflected by the target object and reaches the BS. Then we can get the signal-to-interference-plus-noise ratio or signal-to-noise ratio of the two. We can also write the secure signal-to-interference-plus-noise ratio of the eavesdropper's target relative to the LU. Finally, we can establish the objective function and constraints of the optimization problem and determine the optimization parameters.

[0012] Step 4: The optimization problem is non-convex, and can be processed into a convex optimization problem through mathematical tools, and solved using a convex optimization solver. Considering the complexity and high dimensionality of convex optimization, RL is introduced to solve the optimization problem. An intelligent agent is set to interact with the environment, and a better strategy is learned through continuous trial and error, and a reward and penalty mechanism feedback is set. In order to handle continuous actions, the deep deterministic policy gradient algorithm in RL is used to solve the optimization problem in step 3.

[0013] Terminology explanation:

[0014] ISAC, Integrated sensing and communication, integrated communication and perception;

[0015] RIS, Reconfigurable intelligent surface, intelligent reflective surface;

[0016] BS, Base station;

[0017] LU, Legitimate users, legal users.

[0018] The purpose of the present invention is achieved through the following technical solutions:

[0019] A waveform generation method for an intelligent reflective surface assisted safety synaesthesia system based on reinforcement learning, comprising:

[0020] Step 1: Build an intelligent reflective surface auxiliary safety interawareness system ISAC;

[0021] Step 2: Construct the optimization problem; based on the known channel state information, obtain the expression of the signal received by the LU and the expression of the received signal of the BS after being reflected by the target object, and then obtain the signal-to-interference-to-noise ratio or signal-to-noise ratio of the two, and write the eavesdropper's eavesdropping signal-to-interference-to-noise ratio relative to the LU; finally, establish the objective function and constraints of the optimization problem, and determine the optimization parameters;

[0022] Step 3: Introduce RL to solve the objective function of the optimization problem and generate the waveform.

[0023] Preferably, according to the present invention, a smart reflective surface-assisted secure communication perception integrated system based on reinforcement learning includes a BS, multiple single-antenna LUs, a potential eavesdropper target, and a RIS;

[0024] The BS is equipped with M receive / transmit antennas arranged as a uniform linear array (ULA) to serve K single-antenna LUs, i.e., the BS transmits information-carrying transmit signals through a uniform linear array, establishes indirect paths through the RIS to reach multiple LUs, completes the downlink communication process, and detects point targets at the same time; an indirect link is established between the BS and the LU through a RIS equipped with N reflective units.

[0025] Preferably, according to the present invention, the waveform transmitted by the BS is a joint communication sensing waveform;

[0026] The joint radar communication transmission signal X is given by the following formula:

[0027] X=W c s c +W r s r (1);

[0028] Among them, s c ∈C K Represents the information symbol matrix and satisfies C K represents a set of K×1 complex matrices; K represents the number of LUs, E(·) represents the expected operation, and I K represents the K-order identity matrix; Wc∈C M × K represents the corresponding communication beamforming matrix, C M×K represents the set of M×K complex matrices; s r ∈C M It includes M independent radar waveforms and satisfies I M represents the M-order identity matrix; CM represents the set of M×1 complex matrices; W r ∈C M×M represents the radar beamforming matrix; C M×M represents a set of M×M complex matrices;

[0029] Define the overall symbolic vector as The transmit beamforming matrix is

[0030] Finally, the joint radar communication transmission signal X is in the form of:

[0031] X = Ws (2);

[0032] The received signal y of the kth LU k That is, the signal expression received by LU is:

[0033]

[0034] Among them, h r,k ∈C N represents the channel coefficient from RIS to the kth user, G represents the equivalent channel coefficient from BS to RIS, Φ=diag{φ1,φ2,...,φ N} T is the reflection coefficient matrix of RIS and satisfies nk represents the zero-mean additive white Gaussian noise (AWGN) at the kth LU, with a variance of The composite channel coefficient between the BS and the kth user is defined as h k =G H Φh r,k ;definition where w k represents the kth column, w j represents the jth column;

[0035] Therefore, the received signal-to-interference-plus-noise ratio SINR at the kth LU is k That is, the signal-to-interference-noise ratio of the signal received by LU is:

[0036]

[0037] The echo signal y received through the BS-RIS-target-RIS-BS path r That is, the received signal expression of BS is expressed as:

[0038]

[0039] Among them, h r,t is the channel coefficient between RIS and target, n rrepresents zero-mean additive white Gaussian noise with variance α0 depends on the target radar cross section (RCS) and the path loss coefficient of the BS-target link, satisfying The composite channel coefficient between the BS-RIS-target-RIS-BS path is defined as: Therefore, the BS's received signal-to-noise ratio SNR r That is, the signal-to-noise ratio of the BS received signal is:

[0040]

[0041] If the target is regarded as a potential eavesdropper, the signal y received by the eavesdropper is e yes:

[0042]

[0043] Among them, n e represents zero-mean Gaussian white noise with variance The composite channel coefficient between BS and target is defined as: The eavesdropping signal-to-interference-and-noise ratio (SINR) of the kth LU e,k That is, the eavesdropper's signal-to-interference-noise ratio relative to the LU is given by the following formula:

[0044]

[0045] The optimization problem is formulated with the goal of maximizing the radar output signal-to-noise ratio by jointly optimizing the transmit beamforming matrix and the RIS reflection coefficient under the communication performance requirements, secure transmission constraints, transmit power budget, and reflection coefficient constraints of each LU. The optimization problem is as follows:

[0046]

[0047] Among them, Γ k is the signal-to-interference-noise ratio requirement of the kth LU, Γ e,k is the eavesdropping signal-to-interference-to-noise ratio threshold used to achieve the required security transmission performance, P BS represents the transmit power budget; Substituting expressions (1)-(8) into the optimization problem (9) yields equation (10):

[0048]

[0049] According to the preferred embodiment of the present invention, RL is introduced to solve the optimization problem, i.e., formula (10), and an MDP process is established. An MDP process includes four key factors, including:

[0050] State: The state s∈S includes all channel state information and phase shift matrices at each RIS:

[0051]

[0052] Among them, s t represents the state at time t, represents the channel coefficient from RIS to the kth LU at time t-1, G t-1 represents the channel coefficient from BS to the RIS at time t-1, represents the channel coefficient from RIS to the target at time t-1, Φ t-1 represents the RIS reflection matrix at time t-1;

[0053] Action: The action space A includes action a t , so that after the agent has taken an action, the agent goes from s t To the next state s t+1 Therefore, action a t for:

[0054] a t =[W,Φ] (12)

[0055] Among them, a t represents the action at time t;

[0056] Reward: The reward is to maximize the radar receiving signal-to-noise ratio of the BS. The reward function is defined as: r t =SNR r t ;

[0057] State Transition Probability: If the current state is s t , and the agent is in s t Take action t , then the state transition probability P(s t+1 |s t ,a t ) gives the arrival at the new state s t+1 The probability of

[0058] Build a deep deterministic policy gradient algorithm to solve the optimization problem;

[0059] Initialize all neural networks and experience replay, and use offline strategies to generate some experience data;

[0060] For each interaction sequence between the agent and the environment, the environment is initialized to the initial state, actions are performed according to the current strategy, and the actions include exploratory action noise. At the same time, rewards and new states are obtained, and the rewards and states are stored in the experience replay;

[0061] Randomly sample a portion of data from the experience replay and input it into the agent. The current state s is input into the executor network to obtain action A. Both the current state s and action A are input into the critic network to obtain the estimated Q value.

[0062] The new state s' is input into the executor target network to obtain action A'. Both the new state s' and action A' are input into the critic target network to obtain TD target.

[0063] The difference between the obtained TD target and the Q value is the TD error. The TD error is minimized by the gradient descent algorithm to obtain new parameters, which are used to update the critic neural network parameters.

[0064] The policy of the actor network is updated based on the sampled gradients, and new continuous actions are output for the next moment of environmental changes.

[0065] Preferably, according to the present invention, the training process of the deep deterministic policy gradient algorithm includes:

[0066] 1) Initialization: Randomly initialize the critic network Q(s, aω) and the executor network μ(sθ), where ω and θ are the weight parameters of the critic network and the executor network, respectively, for approximate estimation of the value function; initialize the target network Q of the critic network and the executor network T and μ T , the parameter is updated to ω T ←ω,θ T ←θ,ω T represents the weight parameter of the critic target network, θ T Represents the weight parameters of the executor target network; initializes the experience playback R;

[0067] 2) Iterative training:

[0068] Outer loop: For each round, first initialize the environment state to s0, and then train the parameters of each time period to update;

[0069] Inner loop: For each time period t, first select a according to the current strategy and exploration noise t =μ(s t |θ)+Ν t , N t Indicates exploration noise, execute a t And get reward t , get the state s for the next time period t+1 , storage tuple (s t ,a t ,r t ,s t+1 ) to the experience playback R; sample a portion of data (s) from the experience playback Ri ,a i ,r i ,s i+1 );Calculate the timing difference error: y i =r i +γQ T (s i+1 ,μ T (s i+1 |θ T )ω T );

[0070] 3) Train the critic network: y i is the time difference target, Q(s i ,a i |ω) is the value estimated by the critic network, that is, the estimated value, to establish the time series difference error between the target value and the estimated value, and use the algorithm to minimize the time series difference error. Thus, the critic network parameters obtained make the output value closer to the target value, and the required ω is obtained by minimizing the time series difference error. L represents the loss function, which is the average value of the square of the time series difference error:

[0071]

[0072] Where ρ is the number of times sampled from the experience replay;

[0073] Directly use the gradient descent algorithm for minimization:

[0074]

[0075] Among them, α ω is the step size of gradient descent; represents the gradient of the action-value function at time t with respect to ω;

[0076] 4) Train the executor network: Use the gradient descent algorithm to update the executor strategy based on the strategy gradient:

[0077]

[0078] Among them, α θ is the step size of gradient descent; represents the gradient of the action value function at time i with respect to a, Represents the gradient of the action function at time i with respect to θ;

[0079] 5) Update target network parameters:

[0080]

[0081] Where τ represents the network learning rate; then step 2) is performed again. If the error tolerance is met or the maximum number of iterations is reached, the iteration ends; Q(s,a|ω) represents the action value function of the critic network; μ(sθ) represents the action function of the executor network; ω represents the weight parameter of the critic network; θ represents the weight parameter of the executor network; Q T represents the critic target network; μ T represents the actor-target network; ω T represents the weight parameter of the critic target network; θ T represents the weight parameter of the actor target network; t represents the exploration noise; ρ represents the number of sampling times; y i represents the temporal difference error; L represents the loss function; represents the gradient of the action-value function at time t with respect to ω; Represents the gradient of the action-value function at time i with respect to a; represents the gradient of the action function at time i with respect to θ; τ represents the network learning rate.

[0082] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a waveform generation method of an intelligent reflective surface auxiliary safety synaesthesia integrated system based on reinforcement learning are implemented.

[0083] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a waveform generation method for an intelligent reflective surface-assisted safety synaesthesia integrated system based on reinforcement learning.

[0084] A waveform generation system of an intelligent reflective surface auxiliary safety synaesthesia integrated system based on reinforcement learning, comprising:

[0085] The intelligent reflective surface auxiliary safety interawareness integrated system ISAC building module is configured to: build an intelligent reflective surface auxiliary safety interawareness integrated system ISAC;

[0086] The optimization problem building module is configured as follows: based on the known channel state information, the expression of the signal received by the LU and the expression of the received signal of the BS after being reflected by the target object are obtained, and then the signal-to-interference-to-noise ratio or signal-to-noise ratio of the two is obtained, and the eavesdropper's eavesdropping signal-to-interference-to-noise ratio relative to the LU is written; finally, the objective function and constraint conditions of the optimization problem are established, and the optimization parameters are determined;

[0087] The waveform generation module is configured to: introduce the objective function of the RL optimization problem to generate the waveform.

[0088] The beneficial effects of the present invention are as follows:

[0089] 1. The present invention considers a system model that is closer to the real scene. The system designs the ISAC waveform for a congested environment with dense obstacles and shielding objects. The BS has no direct link with the legitimate communication users and the eavesdropper targets, and all links are established through RIS, which makes up for the deficiency of overly idealized system design scenarios in the past.

[0090] 2. The RIS-assisted secure ISAC waveform design in the present invention is a brand-new joint communication perception waveform design. It is not a modification of a single communication waveform or a sensing waveform. Its joint communication perception waveform ensures the performance compromise of both communication and sensing. Compared with waveform designs centered on communication or sensing alone, it has better system performance.

[0091] 3. Formulate the optimization problem and introduce RL to solve the optimization problem, which not only reduces the complexity of calculation, but also solves the high dimensionality and uncertainty of traditional convex optimization. The optimized waveform is more suitable for ISAC system, and the RL algorithm is easier to generalize. Even if the environment changes, the optimization algorithm can obtain the optimal waveform through learning and training. In order to handle continuous actions, the deep deterministic policy gradient algorithm in RL is used.

[0092] In summary, the system of the present invention accurately reflects the real-world scenario architecture of RIS-assisted safety interawareness integration, which can provide a reference for the deployment of power communication access networks. The waveform design method of the present invention can greatly reduce the complexity of node calculations, improve system performance, and achieve a performance tradeoff between communication and sensing. The optimization method based on RL of the present invention is more suitable for solving complex optimization problems, can adapt to environmental changes, and has better generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 It is a structural diagram of an intelligent reflective surface auxiliary safety inter-sensory integrated system;

[0094] Figure 2 It is a schematic diagram of the framework of the deep deterministic policy gradient algorithm; DETAILED DESCRIPTION

[0095] The present invention will be further defined below in conjunction with the accompanying drawings and embodiments, but is not limited thereto.

[0096] Example 1

[0097] A waveform generation method for an intelligent reflective surface assisted safety synaesthesia system based on reinforcement learning, comprising:

[0098] Step 1: Build an intelligent reflective surface auxiliary safety interawareness system ISAC;

[0099] Step 2: Construct the optimization problem; based on the known channel state information, obtain the expression of the signal received by the LU and the expression of the received signal of the BS after being reflected by the target object, and then obtain the signal-to-interference-to-noise ratio or signal-to-noise ratio of the two, and write the eavesdropper's eavesdropping signal-to-interference-to-noise ratio relative to the LU; finally, establish the objective function and constraints of the optimization problem, and determine the optimization parameters;

[0100] Step 3: Introduce RL to solve the objective function of the optimization problem and generate the waveform.

[0101] The generated waveform refers to the signal waveform emitted by the BS. Since the covariance matrix of the information symbol is assumed to be a unit matrix, it is actually a beamforming matrix at the BS. Furthermore, since the system model introduces RIS, the reflection matrix design of RIS is equivalent to performing beamforming at RIS to regulate signal propagation. In order to ensure the performance of the ISAC system, the waveform generated by the present invention is achieved by establishing an optimization problem and then solving the optimization problem. The BS beamforming matrix and the RIS reflection matrix are decision variables of the optimization problem. By solving the optimization problem and obtaining solutions to the two, the required waveform can be obtained.

[0102] Example 2

[0103] The waveform generation method of the intelligent reflective surface auxiliary safety synaesthesia system based on reinforcement learning according to Example 1 is different in that:

[0104] like Figure 1 As shown, the intelligent reflective surface-assisted secure communication perception integrated system based on reinforcement learning includes a BS, multiple single-antenna LUs, a potential eavesdropper target, and a RIS;

[0105] The BS is equipped with M receiving / transmitting antennas arranged as a uniform linear array (ULA) to serve K single-antenna LUs, that is, the BS transmits a transmission signal carrying information through a uniform linear array, establishes an indirect path through the RIS to reach multiple LUs, completes the downlink communication process, and detects point-like targets at the same time; due to the presence of a large number of obstacles and shielding objects in a congested environment, there is no direct link between the BS and the LU, and there is no direct link between the BS and the eavesdropper target. An indirect link is established between the BS and the LU through a RIS equipped with N reflection units. The reflection unit refers to the reflection element on the RIS, which is an adjustable element that can change the phase of the signal, control the propagation direction of the signal, etc. This is more in line with the real scene and can give full play to the role of the RIS. In addition, the target also acts as a potential eavesdropper.

[0106] In the intelligent reflective surface assisted safety communication perception integrated system of the present invention, it is assumed that there are obstacles between the BS and the LU and between the BS and the target object, so there are the following two main paths:

[0107] 1) BS-RIS-LU, but this path will reach multiple different LUs;

[0108] 2) BS-RIS-target-RIS-BS.

[0109] The waveform transmitted by the BS is a joint communication and sensing waveform; it is not designed based on the existing communication waveform or sensing waveform, but a completely new waveform obtained through optimization. Such a waveform can achieve a performance trade-off between communication and sensing.

[0110] The joint radar communication transmission signal X is given by the following formula:

[0111] X=W c s c +W r s r (1);

[0112] Among them, s c ∈C K Represents the information symbol matrix and satisfies C K represents a set of K×1 complex matrices; K represents the number of LUs, E(·) represents the expected operation, and I K represents the K-order identity matrix; Wc∈C M × K represents the corresponding communication beamforming matrix, C M×K represents the set of M×K complex matrices; s r ∈C M It includes M independent radar waveforms and satisfies I M represents the M-order identity matrix; C M represents the set of M×1 complex matrices; W r ∈C M×M represents the radar beamforming matrix; C M×M represents a set of M×M complex matrices;

[0113] For simplicity, the overall symbol vector is defined as The transmit beamforming matrix is

[0114] Finally, the joint radar communication transmission signal X is in the form of:

[0115] X = Ws (2);

[0116] The received signal y of the kth LUk That is, the signal expression received by LU is:

[0117]

[0118] Among them, h r,k ∈C N represents the channel coefficient from RIS to the kth user, G represents the equivalent channel coefficient from BS to RIS, Φ=diag{φ1,φ2,...,φ N} T is the reflection coefficient matrix of RIS and satisfies nk represents the zero-mean additive white Gaussian noise (AWGN) at the kth LU, with a variance of For simplicity, the composite channel coefficient between the BS and the kth user is defined as h k =G H Φh r,k ;definition where w k represents the kth column, w j represents the jth column;

[0119] Therefore, the received signal-to-interference-plus-noise ratio SINR at the kth LU is k That is, the signal-to-interference-noise ratio of the signal received by LU is:

[0120]

[0121] Considering that the BS-target direct link is unavailable, the echo signal y received via the BS-RIS-target-RIS-BS path r That is, the received signal expression of BS is expressed as:

[0122]

[0123] Among them, h r,t is the channel coefficient between RIS and target, n r represents zero-mean additive white Gaussian noise with a variance of α0 depends on the target radar cross section (RCS) and the path loss coefficient of the BS-target link, satisfying For simplicity, the composite channel coefficient between the BS-RIS-target-RIS-BS path is defined as: Therefore, the BS's received signal-to-noise ratio SNR r That is, the signal-to-noise ratio of the BS received signal is:

[0124]

[0125] If the target is regarded as a potential eavesdropper, the signal y received by the eavesdropper ise yes:

[0126]

[0127] Among them, n e represents zero-mean Gaussian white noise with variance For simplicity, the composite channel coefficient between BS and target is defined as: The eavesdropping signal-to-interference-and-noise ratio (SINR) of the kth LU e,k That is, the eavesdropper's signal-to-interference-noise ratio relative to the LU is given by the following formula:

[0128]

[0129] The optimization problem is formulated with the goal of maximizing the radar output signal-to-noise ratio by jointly optimizing the transmit beamforming matrix and the RIS reflection coefficient under the communication performance requirements, secure transmission constraints, transmit power budget, and reflection coefficient constraints of each LU. The optimization problem is as follows:

[0130]

[0131] Among them, Γ k is the signal-to-interference-noise ratio requirement of the kth LU, Γ e,k is the eavesdropping signal-to-interference-to-noise ratio threshold used to achieve the required security transmission performance, P BS represents the transmit power budget; Substituting expressions (1)-(8) into the optimization problem (9) yields equation (10):

[0132]

[0133] Introducing RL to solve the optimization problem, namely, Equation (10), avoids the high dimensionality of variables and the uncertainty of the environment. By setting an agent to interact with the environment, it focuses on training the agent to make decisions by observing the surrounding environment and receiving feedback. The main goal of RL is to optimize the reward function and guide the agent to learn a strategy that determines the action selection in response to the environment state.

[0134] RL is a specialized field in machine learning that focuses on training agents to make decisions by observing their surroundings and receiving feedback. The main goal of RL is to optimize a reward function that guides the agent to learn a policy that determines the choice of actions in response to the state of the environment. The core elements of an RL model include an agent, an environment, and a reward system. In this setting, the agent examines the current state of the environment, selects actions influenced by its existing policy, receives rewards based on the results of its actions, and then adjusts its policy taking into account the rewards received and the new state of the environment. This process is repeated until the agent perfects its policy or a predefined termination condition is met.

[0135] RL is a specific type of machine learning problem in the field of artificial intelligence. In an RL system, the decision maker can observe the environment and take actions based on the observations. After taking the action, it can get a reward. And RL learns how to maximize the reward through interaction with the environment. The agent is the decision maker and learner in the RL system, which can make decisions and receive reward signals. The environment is everything in the reinforcement system except the agent, and it is the object of the agent's interaction.

[0136] The task environment faced by RL can usually be modeled as an MDP, which provides a framework to describe the relationship between the agent and the environment. The core algorithms and core concepts of RL (state value, reward function, etc.) are based on the idea of ​​MDP, and the MDP is solved by continuously approaching the optimal value function or policy function. In the present invention, the deep deterministic policy gradient algorithm used is also based on the MDP framework to solve the optimization problem. Therefore, in order to use the deep deterministic policy gradient algorithm, it is necessary to establish an MDP process. MDP is the most classic and important model in RL. MDP provides a framework to describe the relationship between the agent and the environment. The core algorithms and core concepts of RL (state value, reward function, etc.) are based on the idea of ​​MDP. The deep deterministic policy gradient algorithm we use is also based on MDP. An MDP process includes four key factors, including:

[0137] State: The environment changes state due to the action of the agent. The state s∈S includes all channel state information and phase shift matrix at each RIS:

[0138]

[0139] Among them, s t represents the state at time t, represents the channel coefficient from RIS to the kth LU at time t-1, G t-1 represents the channel coefficient from BS to the RIS at time t-1, represents the channel coefficient from RIS to the target at time t-1, Φ t-1 represents the RIS reflection matrix at time t-1;

[0140] Action: The agent makes decisions based on observations and decides what actions to take on the environment. The action space A includes actions a t , so that after the agent has taken an action, the agent goes from s t To the next state s t+1 Therefore, action a t for:

[0141] a t=[W,Φ] (12)

[0142] Among them, a t represents the action at time t;

[0143] Reward: Reward is the learning goal of the RL system. The agent receives a reward from the environment after taking an action. The goal of RL is to maximize the sum of rewards over a long period of time. The reward is to maximize the radar receiving signal-to-noise ratio of the BS. The reward function is defined as: t =SNR r t ;

[0144] State Transition Probability: In an MDP, the state transition probability defines the probability of transitioning from one state to another given a specific action. If the current state is s t , and the agent is in s t Take action t , then the state transition probability P(s t+1 |s t ,a t ) gives the arrival at the new state s t+1 This probability distribution over possible next states is crucial for planning and decision making in RL.

[0145] Build a deep deterministic policy gradient algorithm to solve the optimization problem; e.g. Figure 2 shown.

[0146] Initialize all neural networks and experience replay, and use offline strategies to generate some experience data;

[0147] For each interaction sequence between the agent and the environment, the environment is initialized to the initial state, actions are performed according to the current strategy, and the actions include exploratory action noise. At the same time, rewards and new states are obtained, and the rewards and states are stored in the experience replay;

[0148] Randomly sample a portion of data from the experience replay and input it into the agent. The current state s is input into the executor network to obtain action A. Both the current state s and action A are input into the critic network to obtain the estimated Q value.

[0149] The new state s' is input into the executor target network to obtain action A'. Both the new state s' and action A' are input into the critic target network to obtain TD target.

[0150] The difference between the obtained TD target and the Q value is the TD error. The TD error is minimized by the gradient descent algorithm to obtain new parameters, which are used to update the critic neural network parameters.

[0151] The policy of the actor network is updated based on the sampled gradients, and new continuous actions are output for the next moment of environmental changes.

[0152] When using a random strategy, both the action space and the state space need to be integrated, while using a deterministic strategy only integrates the state space. If the dimension of the action space is large, the deterministic strategy will have a great advantage.

[0153] In order to explore the complete state and action space, a random strategy is usually required. Therefore, in order to ensure that the deterministic policy gradient algorithm also has a certain degree of exploratory nature, an offline policy learning algorithm is introduced. For both the executor and the critic, a neural network is used for function approximation. After obtaining the gradient, the gradient descent algorithm is used to obtain the parameters. The executor-critic network combines the value-based and policy-based methods, and uses value function approximation at the same time. The executor is an update of the strategy, and the critic is an update of the value. The update of the relevant value can evaluate the value of the executor's action. Experience replay refers to the experience obtained by the executor is collected and placed in a storage space, and then replayed in batches when the parameters are updated.

[0154] The training process of the deep deterministic policy gradient algorithm includes:

[0155] 1) Initialization: Randomly initialize the critic network Q(s, aω) and the executor network μ(sθ), where ω and θ are the weight parameters of the critic network and the executor network, respectively, for approximate estimation of the value function; initialize the target network Q of the critic network and the executor network T and μ T , the parameter is updated to ω T ←ω,θ T ←θ,ω T represents the weight parameter of the critic target network, θ T Represents the weight parameters of the executor target network; initializes the experience playback R;

[0156] 2) Iterative training:

[0157] Outer loop: For each round, first initialize the environment state to s0, and then train the parameters of each time period to update;

[0158] Inner loop: For each time period t, first select a according to the current strategy and exploration noise t =μ(s t |θ)+Ν t , N t Indicates exploration noise, execute a t And get reward t , get the state s for the next time period t+1 , storage tuple (st ,a t ,r t ,s t+1 ) to the experience playback R; sample a portion of data (s) from the experience playback R i ,a i ,r i ,s i+1 );Calculate the timing difference error: y i =r i +γQ T (s i+1 ,μ T (s i+1 |θ T )ω T );

[0159] 3) Training the critic network: The critic network is used to evaluate the quality of the strategy and to calculate the Q value, y i is the time difference target, that is, the target value to be obtained, and Q(s i ,a i |ω) is the value estimated by the critic network, that is, the estimated value. In order to ensure that the estimated value is closer to the target value, the time series difference error between the target value and the estimated value is established. Then, the algorithm is used to minimize the time series difference error. Thus, the critic network parameters obtained make the output value closer to the target value. The required ω is obtained by minimizing the time series difference error. L represents the loss function, which is the average value of the square of the time series difference error:

[0160]

[0161] Where ρ is the number of times sampled from the experience replay;

[0162] Directly use the gradient descent algorithm for minimization:

[0163]

[0164] Among them, α ω is the step size of gradient descent; represents the gradient of the action-value function at time t with respect to ω;

[0165] 4) Train the executor network: Use the gradient descent algorithm to update the executor strategy based on the strategy gradient:

[0166]

[0167] Among them, α θ is the step size of gradient descent; represents the gradient of the action value function at time i with respect to a, Represents the gradient of the action function at time i with respect to θ;

[0168] 5) Update target network parameters:

[0169]

[0170] Where τ represents the network learning rate; then step 2) is performed again. If the error tolerance is met or the maximum number of iterations is reached, the iteration ends; Q(s,a|ω) represents the action value function of the critic network; μ(sθ) represents the action function of the executor network; ω represents the weight parameter of the critic network; θ represents the weight parameter of the executor network; Q T represents the critic target network; μ T represents the actor-target network; ω T represents the weight parameter of the critic target network; θ T represents the weight parameter of the actor target network; t represents exploration noise, ensuring the exploratory nature of the strategy; ρ represents the number of sampling times; y i represents the temporal difference error; L represents the loss function; represents the gradient of the action-value function at time t with respect to ω; Represents the gradient of the action-value function at time i with respect to a; represents the gradient of the action function at time i with respect to θ; τ represents the network learning rate.

[0171] Example 3

[0172] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the waveform generation method of the intelligent reflective surface assisted safety synaesthesia integrated system based on reinforcement learning described in Example 1 or 2 are implemented.

[0173] Example 4

[0174] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the waveform generation method of the intelligent reflective surface assisted safety synaesthesia system based on reinforcement learning described in Example 1 or 2.

[0175] Example 5

[0176] A waveform generation system of an intelligent reflective surface auxiliary safety synaesthesia integrated system based on reinforcement learning, comprising:

[0177] The intelligent reflective surface auxiliary safety interawareness integrated system ISAC building module is configured to: build an intelligent reflective surface auxiliary safety interawareness integrated system ISAC;

[0178] The optimization problem building module is configured as follows: based on the known channel state information, the expression of the signal received by the LU and the expression of the received signal of the BS after being reflected by the target object are obtained, and then the signal-to-interference-to-noise ratio or signal-to-noise ratio of the two is obtained, and the eavesdropper's eavesdropping signal-to-interference-to-noise ratio relative to the LU is written; finally, the objective function and constraint conditions of the optimization problem are established, and the optimization parameters are determined;

[0179] The waveform generation module is configured to: introduce the objective function of the RL optimization problem to generate the waveform.

Claims

1. A waveform generation method for an intelligent reflective surface assisted safety synaesthesia system based on reinforcement learning, characterized in that: include: Step 1: Build an intelligent reflective surface auxiliary safety interawareness system ISAC; Step 2: Construct optimization problem; Based on the known channel state information, we can get the expression of the signal received by LU and the expression of the signal received by BS after being reflected by the target object. Then we can get the signal-to-noise ratio or signal-to-noise ratio of the two, and write down the eavesdropper's signal-to-noise ratio relative to LU. Finally, we can establish the objective function and constraints of the optimization problem, and determine the optimization parameters. Step 3: Introduce RL to solve the objective function of the optimization problem and generate the waveform.

2. According to the waveform generation method of the intelligent reflective surface auxiliary safety synaesthesia system based on reinforcement learning according to claim 1, it is characterized in that: The intelligent reflective surface-assisted secure communication perception integrated system based on reinforcement learning includes a BS, multiple single-antenna LUs, a potential eavesdropper target, and a RIS; The BS is equipped with M receiving / transmitting antennas arranged as a uniform linear array, serving K single-antenna LUs, that is, the BS transmits the transmission signal carrying information through the uniform linear array, establishes an indirect path through the RIS to reach multiple LUs, completes the downlink communication process, and detects point targets at the same time; an indirect link is established between the BS and the LU through a RIS equipped with N reflection units.

3. The waveform generation method of the intelligent reflective surface auxiliary safety synaesthesia system based on reinforcement learning according to claim 2 is characterized in that: The waveform transmitted by the BS is a joint communication sensing waveform; The joint radar communication transmission signal X is given by the following formula: X=W c s c +W r s r (1); Among them, s c ∈C K Represents the information symbol matrix and satisfies represents a set of K×1 complex matrices; K represents the number of LUs, E(·) represents the expected operation, and I K represents the K-order unit matrix; W c ∈C M×K represents the corresponding communication beamforming matrix, C M×K represents the set of M×K complex matrices; s r ∈C M It includes M independent radar waveforms and satisfies I M represents the M-order identity matrix; C M represents the set of M×1 complex matrices; W r ∈C M×M represents the radar beamforming matrix; C M×M represents a set of M×M complex matrices; Define the overall symbolic vector as The transmit beamforming matrix is The form of the joint radar communication transmission signal X is: X = Ws (2); The received signal y of the kth LU k That is, the signal expression received by LU is: Among them, h r,k ∈C N represents the channel coefficient from RIS to the kth user, G represents the equivalent channel coefficient from BS to RIS, Φ=diag{φ1,φ2,...,φ N } T is the reflection coefficient matrix of RIS and satisfies n k represents the zero-mean additive white Gaussian noise at the k-th LU, with variance The composite channel coefficient between the BS and the kth user is defined as h k =G H Φh r,k ;definition where w k represents the kth column, w j represents the jth column; Therefore, the received signal-to-interference-plus-noise ratio SINR at the kth LU is k That is, the signal-to-interference-noise ratio of the signal received by LU is: The echo signal y received through the BS-RIS-target-RIS-BS path r That is, the received signal expression of BS is expressed as: Among them, h r,t is the channel coefficient between RIS and target, n r represents zero-mean additive white Gaussian noise with a variance of α0 depends on the target radar cross section and the path loss coefficient of the BS-target link, satisfying The composite channel coefficient between the BS-RIS-target-RIS-BS path is defined as: Therefore, the BS's received signal-to-noise ratio SNR r That is, the signal-to-noise ratio of the BS received signal is: If the target is regarded as a potential eavesdropper, the signal y received by the eavesdropper is e yes: Among them, n e represents zero-mean Gaussian white noise with variance The composite channel coefficient between BS and target is defined as: The eavesdropping signal-to-interference-and-noise ratio (SINR) of the kth LU e,k That is, the eavesdropper's signal-to-interference-noise ratio relative to the LU is given by the following formula: The optimization problem is formulated with the goal of maximizing the radar output signal-to-noise ratio by jointly optimizing the transmit beamforming matrix and the RIS reflection coefficient under the communication performance requirements, secure transmission constraints, transmit power budget, and reflection coefficient constraints of each LU. The optimization problem is as follows: Among them, Γ k is the signal-to-interference-noise ratio requirement of the kth LU, Γ e,k is the eavesdropping signal-to-interference-to-noise ratio threshold used to achieve the required security transmission performance, P BS represents the transmit power budget; Substituting expressions (1)-(8) into the optimization problem (9) yields equation (10):

4. The waveform generation method of the intelligent reflective surface auxiliary safety synaesthesia system based on reinforcement learning according to claim 2 is characterized in that: Introducing RL to solve the optimization problem, that is, formula (10), establishes an MDP process. An MDP process includes four key factors, including: State: The state s∈S includes all channel state information and phase shift matrices at each RIS: Among them, s t represents the state at time t, represents the channel coefficient from RIS to the kth LU at time t-1, G t-1 represents the channel coefficient from BS to the RIS at time t-1, represents the channel coefficient from RIS to the target at time t-1, Φ t-1 represents the RIS reflection matrix at time t-1; Action: The action space A includes action a t , so that after the agent has taken an action, the agent goes from s t To the next state s t+1 Therefore, action a t for: a t =[W,Φ] (12) Among them, a t represents the action at time t; Reward: The reward is to maximize the radar receiving signal-to-noise ratio of the BS. The reward function is defined as: t =SNR r t ; State transition probability: If the current state is s t , and the agent is in s t Take action t , then the state transition probability P(s t+1 |s t ,a t ) gives the new state s t+1 probability.

5. The waveform generation method of the intelligent reflective surface auxiliary safety synaesthesia system based on reinforcement learning according to claim 4 is characterized in that: Build a deep deterministic policy gradient algorithm to solve the optimization problem; Initialize all neural networks and experience replay, and use offline strategies to generate some experience data; For each interaction sequence between the agent and the environment, the environment is initialized to the initial state, actions are performed according to the current strategy, and the actions include exploratory action noise. At the same time, rewards and new states are obtained, and the rewards and states are stored in the experience replay; Randomly sample a portion of data from the experience replay and input it into the agent. The current state s is input into the executor network to obtain action A. Both the current state s and action A are input into the critic network to obtain the estimated Q value. The new state s' is input into the executor target network to obtain action A'. Both the new state s' and action A' are input into the critic target network to obtain TD target. The difference between the obtained TD target and the Q value is the TD error. The TD error is minimized by the gradient descent algorithm to obtain new parameters, which are used to update the critic neural network parameters. The policy of the actor network is updated based on the sampled gradients, and new continuous actions are output for the next moment of environmental changes.

6. The waveform generation method of the intelligent reflective surface auxiliary safety synaesthesia system based on reinforcement learning according to claim 5 is characterized in that: The training process of the deep deterministic policy gradient algorithm includes: 1) Initialization: Randomly initialize the critic network Q(s,a|ω) and the executor network μ(s|θ), where ω and θ are the weight parameters of the critic network and the executor network, respectively, for approximate estimation of the value function; initialize the target network Q of the critic network and the executor network T and μ T , the parameter is updated to ω T ←ω,θ T ←θ,ω T represents the weight parameter of the critic target network, θ T Represents the weight parameters of the executor target network; initializes the experience playback R; 2) Iterative training: Outer loop: For each round, first initialize the environment state to s0, and then train the parameters of each time period to update; Inner loop: For each time period t, first select a according to the current strategy and exploration noise t =μ(s t |θ)+Ν t , N t Indicates exploration noise, execute a t And get reward t , get the state s for the next time period t+1 , storage tuple (s t ,a t ,r t ,s t+1 ) to the experience playback R; sample a portion of data (s) from the experience playback R i ,a i ,r i ,s i+1 );Calculate the timing difference error: y i =r i +γQ T (s i+1 ,μ T (s i+1 |θ T )ω T ); 3) Train the critic network: y i is the time difference target, Q(s i ,a i |ω) is the value estimated by the critic network, that is, the estimated value, to establish the time series difference error between the target value and the estimated value, and use the algorithm to minimize the time series difference error. Thus, the critic network parameters obtained make the output value closer to the target value, and the required ω is obtained by minimizing the time series difference error. L represents the loss function, which is the average value of the square of the time series difference error: Where ρ is the number of times sampled from the experience replay; Directly use the gradient descent algorithm for minimization: Among them, α ω is the step size of gradient descent; represents the gradient of the action-value function at time t with respect to ω; 4) Train the executor network: Use the gradient descent algorithm to update the executor strategy based on the strategy gradient: Among them, α θ is the step size of gradient descent; represents the gradient of the action value function at time i with respect to a, Represents the gradient of the action function at time i with respect to θ; 5) Update target network parameters: Where τ represents the network learning rate; then step 2) is performed again. If the error tolerance is met or the maximum number of iterations is reached, the iteration ends; Q(s,a|ω) represents the action value function of the critic network; μ(s|θ) represents the action function of the executor network; ω represents the weight parameter of the critic network; θ represents the weight parameter of the executor network; Q T represents the critic target network; μ T represents the actor-target network; ω T represents the weight parameter of the critic target network; θ T represents the weight parameter of the actor target network; t represents the exploration noise; ρ represents the number of sampling times; y i represents the temporal difference error; L represents the loss function; represents the gradient of the action-value function at time t with respect to ω; Represents the gradient of the action-value function at time i with respect to a; represents the gradient of the action function at time i with respect to θ; τ represents the network learning rate.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the waveform generation method of the intelligent reflective surface assisted safety synaesthesia system based on reinforcement learning are implemented as described in any one of claims 1-6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the waveform generation method of the intelligent reflective surface assisted safety synaesthesia system based on reinforcement learning are implemented.

9. A waveform generation system for an intelligent reflective surface assisted safety synaesthesia system based on reinforcement learning, characterized in that: include: The intelligent reflective surface assisted safety interawareness integrated system ISAC building module is configured to: build an intelligent reflective surface assisted safety interawareness integrated system ISAC; The optimization problem building module is configured as follows: based on the known channel state information, the expression of the signal received by the LU and the expression of the received signal of the BS after being reflected by the target object are obtained, and then the signal-to-interference-to-noise ratio or signal-to-noise ratio of the two is obtained, and the eavesdropper's eavesdropping signal-to-interference-to-noise ratio relative to the LU is written; finally, the objective function and constraint conditions of the optimization problem are established, and the optimization parameters are determined; The waveform generation module is configured to: introduce the objective function of the RL optimization problem to generate the waveform.

Images