RIS-assisted unmanned aerial vehicle network-based communication and sensing integrated system and method

Abstract

The invention discloses an RIS-assisted unmanned aerial vehicle network-based communication and sensing integrated system and method. The system comprises a UAV, an MBS module, a user communication association decoupling module and an RIS module. The RIS intelligent reflecting surface and the unmanned aerial vehicle network are applied to the communication and sensing integrated system. In consideration of different service requirements in the system, the process is divided into a communication part and a sensing part. And a deep reinforcement learning algorithm is designed according to the requirements of the system to control the UAV flight path, base station beam forming, user association allocation and RIS phase shift in the system, so that the communication and sensing performance in the communication and sensing integrated system is optimal. According to the invention, while all functions of an original communication and inductance integrated system are reserved, RIS and unmanned aerial vehicle networks are introduced to assist the system, and an alternate iterative optimization method and a deep reinforcement learning algorithm are used to realize scheduling and decision-making of each resource. Compared with the prior art, additional channel gain is provided, and the distribution efficiency of communication sensing resources is improved.

Description

Technical Field

[0001] The present invention relates to a synaesthesia system and method based on RIS-assisted unmanned aerial vehicle network, belonging to the field of communication network optimization and the field of unmanned aerial vehicle technology. Background Art

[0002] With the rapid development of 6G, the integration of communication and perception has been realized in a prototype system, integrating the two functions of communication and sensing to achieve efficient information collection, secure information transmission and processing, and conveniently realize the joint scheduling of communication and perception resources. While wireless networks are performing high-quality communications, they can also obtain perception of target objects and environmental information by analyzing wireless signals, and complete functions such as positioning, ranging, speed measurement, imaging, detection, and identification, thereby improving network efficiency and performance. However, due to the limited perception and communication range and non-line-of-sight (NLOS) limitations, the advantages of the ISAC system may decline. In addition, the communication function and the perception function need to share limited spectrum resources and energy resources, and the needs of the two are usually unbalanced, which will lead to low resource utilization or a function cannot achieve the expected performance. Therefore, we need to propose a better system and optimization solution for the synaesthesia integration scenario.

[0003] For the communication and perception integration scenario, drone networks are an ideal solution. Drone networks are characterized by high flexibility, high mobility, and wide coverage. Drones integrate communication and perception functions to effectively cope with non-direct propagation (NLoS) scenarios in actual networks and can cover areas that traditional networks cannot cover. However, existing drone network-assisted synaesthesia integration mainly focuses on studying perception and communication functions at the same time, which may ignore the asymmetric demand for shared network resources in practice, making it challenging to strike a balance between the two.

[0004] The smart reflector is a metasurface that integrates controllable and low-power analog computing units. By adjusting the absorption, reflection, refraction and phase of the passive reflector unit, the input electromagnetic signal is redirected and guided to the desired direction. On this basis, the amplitude and phase of the reflected echo are used to greatly improve the effective gain of the channel.

[0005] Deep reinforcement learning is an effective method for solving joint optimization problems of multiple variables in complex environments. Deep reinforcement learning is a method that combines deep learning and reinforcement learning. Its goal is to enable machines to learn and improve their decision-making strategies through interaction with the environment. By simulating the interaction between the agent and the environment, the reward signal is used to guide the agent to learn the optimal behavior strategy. In addition, deep reinforcement learning performs well in dealing with complex, dynamic, and uncertain environments, and can cope with various types of problems and tasks. Deep reinforcement learning can handle continuous action spaces, which is very important in many practical problems, such as control systems and robot learning. Therefore, it is feasible and effective to combine deep reinforcement learning with RIS-assisted drone networks to design a synaesthesia integrated system. Summary of the invention

[0006] The purpose of the present invention is to address the defects and shortcomings in the above-mentioned prior art and propose a synaesthesia integrated system and method based on RIS-assisted UAV network. The present invention applies RIS intelligent reflective surface and UAV network to the synaesthesia integrated system. Considering the different service requirements in the system, the process is divided into a communication part and a perception part. And according to the requirements of the system, a deep reinforcement learning algorithm is designed to control the UAV flight trajectory, base station beamforming, user association allocation and RIS phase shift in the system, so that the communication and perception performance in the synaesthesia integrated system is optimized.

[0007] The technical solution adopted by the present invention to solve its technical problem is: a synaesthesia system based on RIS-assisted UAV network, the system includes UAV, MBS module, user communication association decoupling module and RIS module.

[0008] The UAV and MBS modules: UAV and MBS are base stations that integrate communication and sensing functions. With the assistance of RIS, they serve communication users and sensing objects within the system at the same time. The system has a single macro base station MBS and multiple UAVs (UAVs return to the starting point after performing a task in a time slot to maintain and replenish energy for UAVs). In particular, we consider a limited ISAC task cycle T, which consists of multiple time slots. It is assumed that the UAV will return to the initial position in each mission stage to perform trajectory, beamforming, UE association and RIS phase shift design. Note that considering the high path loss in actual scenarios, signals reflected more than twice by RIS are ignored. The transmitted signal of the UAV is expressed as:

[0009]

[0010] where w m ∈C K×1 represents the perceptual beamforming matrix, w k ∈C M×1represents the communication beamforming matrix, C K The vector s in m Indicates Represents the sensing signal, with C M Indicates that Denotes the communication symbol vectors, which are statistically independent of each other, and define w[w m w k ]∈C M×(K+M) represents the general beamforming matrix of the UAV;

[0011] In the system considered, it is assumed that the flight altitude of each drone is set to H and the flight speed is set to V, where 0<V<V max , the flight time of the mission Divided into multiple time periods, each time period is δ t , the horizontal position of UAV n at time t is expressed as Considering the energy consumption limit of the drone, it is assumed that each drone will fly back to the starting position after each duration, so there is L n (T) = L n (0), further, the UAV propulsion energy consumption model is expressed as:

[0012]

[0013] Among them C d is the drag coefficient, is the reference area of ​​the UAV, ι is the z-weight of the UAV, is the efficiency of the motor used by the drone, represents thrust;

[0014] The RIS module: deploys multiple RIS consisting of passive reflection units within the scope of the integrated communication system, which can be placed on the ground or on a building, and effectively reflects the transmitted signal by dynamically adjusting the phase of each RIS unit;

[0015] Assume that each RIS is equipped with an intelligent controller that autonomously adjusts the phase shift of each element, and each RIS is equipped with S = S l ×S w Reflection element, the phase shift matrix of RIS is expressed as:

[0016]

[0017] where θ s is the phase shift of the sth element of RIS. We consider that each RIS element needs to satisfy the following modularity constraint |θ s | 2 =1;

[0018] The parameters of the perception signal are as follows:

[0019] The sensing signal received by BS_n after being reflected by target j can be expressed as

[0020] y j，n =(h i，n ΘP n，i +h j，n )x+n s

[0021] where n s is zero mean and has a covariance of AWGN, P n，i represents the channel matrix from UAV_n to RIS_i, h i，n and h j，n They represent the LoS channel matrix of the UAV-UE link and the LoS channel matrix of the UAV-Target link, respectively, and are expressed as'

[0022]

[0023] where χ i，n and χ j，n is the coefficient mainly affected by the system environment, θ a and θ b denote the angle of the target relative to RIS and BS respectively, so a(θ a ) and b(θ b ) is expressed as

[0024]

[0025] where d R and d B Determined by the minimum number of UAVs and RIS, the perception rate quantifies the information exchange between the detected target and the signal receiver. A higher perception rate indicates that the amount of target information embedded in the received signal is greater. Considering the interference of communication signals, the perceived signal-to-interference-and-noise ratio (SINR) of the perceived target j to the UAV n at time slot t is expressed as:

[0026]

[0027] Among them, H R =h i，n (t)ΦP i，n +h j，n (t), on this basis, at time slot t, the perception rate of target j to UAV_n is expressed as:

[0028]

[0029] use express Wherein the beamforming matrix is ​​represented by W, i=1, ..., K+NM (w_i represents the i-th column);

[0030] Therefore, the total perception rate of target j is expressed as:

[0031]

[0032] The user communication association decoupling module: the user behavior mainly includes initiating an uplink request to the UAV or MBS, and obtaining the requested information through the downlink from the UAV, MBS to the user. In this process, the user is allowed to associate different UAVs and MBSs in the uplink and downlink;

[0033] Similar to the perception signal, for the communication model, the received signal of the kth UE received by UAV_n is expressed as:

[0034] y n，k =(h i，k ΘP n，i +h n，k )x+n k

[0035] The SINR between the kth UE and the nth BS in time slot t is:

[0036]

[0037] Since user association is decoupled and there is only one uplink and downlink in a time slot, the binary variable To represent the association between UE_k and BS_n, it is expressed as:

[0038]

[0039] where DL and UL are distinguished by the symbol (·). Note that in order to ensure that each UE is associated with at most one b in different links (DL or UL), and Need to satisfy and Therefore, the communication rate of UE_k is expressed as:

[0040]

[0041] The SINR of the decoupled method can be used and To calculate, finally, the total communication rate of UEs in UL and DL in time slot t is expressed as:

[0042]

[0043] The present invention also provides an optimization method for a synaesthesia system based on a RIS-assisted UAV network, comprising the following steps:

[0044] The joint optimization problem involving user access (B), UAV trajectory design (L), base station beamforming (ω) and RIS phase shift matrix (θ) is first decomposed into two sub-problems (i beamforming sub-problem (ω); ii user access, UAV trajectory design, base station beamforming and RIS phase shift matrix sub-problem (B, L, θ)). An alternating iterative optimization algorithm is used to fix the variables of one sub-problem and solve the other sub-problem, and this cycle is repeated until the joint optimal solution (B, L, ω, θ) is obtained.

[0045] Step 1: Based on the ISAC network system architecture of the constructed RIS-assisted UAV, a joint optimization problem of UAV trajectory design, user communication association decoupling access, beamforming design and RIS phase shift is proposed. The optimization target is set as the sum of the communication rate of all users and the perception rate of all targets in the entire network system. The problem is expressed as:

[0046]

[0047] in

[0048] In this optimization problem, (1b) constrains the minimum threshold of the signal-to-noise ratio of each sensing target, (1c) ensures that each UAV returns to the starting position after a certain period of time, (1d) ensures that the speed of each UAV is less than the maximum speed in the adjacent time slot, (1e) ensures that the energy consumption of each UAV is lower than the total energy consumption, (1f) represents the phase shift interval of each RIS element, (1g) indicates that the data rate of each UE is greater than the minimum threshold in UL or DL, and (1h) limits the BS power to always be lower than the threshold P within a certain period of time. max .

[0049] Step 2: Decompose the joint optimization problem involving user access (B), UAV trajectory design (L), base station beamforming (ω) and RIS phase shift matrix (θ) into two sub-problems (i beamforming sub-problem (ω); ii user access, UAV trajectory design, base station beamforming and RIS phase shift matrix sub-problem (B, L, θ)). Use the alternating iterative optimization algorithm to fix the variables of one sub-problem and solve the other sub-problem, and repeat this cycle until the joint optimal solution (B, L, ω, θ) is obtained.

[0050] Step 3: Due to the coupling between UE-UAV association and UAV positioning, the optimization problem (P1) containing three variables (UE-UAV association B, trajectory design L, and RIS phase shift θ) is still a nonlinear programming problem with non-convex constraints. To this end, (P1) is decomposed into multiple sub-problems and attempts are made to solve them using iterative alternating optimization.

[0051] Step 4: A distributed MADRL method is proposed, which uses deep neural networks (DNNs) as function approximators to handle the complexity brought by the wide state and action space. The implementation process of this method includes centralized training and distributed execution. Specifically, in the training phase, each agent improves its local strategy by obtaining information from the central controller (i.e., MBS). During the distributed execution step, the agent only uses its own local information to make decisions.

[0052] Further, step 2 of the present invention includes: considering the optimization problem of the beamforming vector in the problem (P1) under the condition of given UE association B, trajectory L and RIS phase shift θ, so the optimization problem is expressed as:

[0053] (P2):

[0054] st(1b),(1g)and(1h)

[0055] After processing by the SDR and SCA algorithms, the problem is simplified to a convex problem with a single optimization variable for beamforming, which is solved using the CVX optimizer.

[0056] Further, step 3 of the present invention includes: considering that (P1) involves UE-UAV association, UAV positioning and RIS phase, the computational complexity of using iterative alternating optimization calculation is Where ι is the number of alternating iterations. The complexity is significantly higher than (P2) and further increases with the increase of the number of drones B, the number of UEs K, and the number of RISs I. In addition, due to the existence of multiple dynamic variables, such as the position L of the drone in different time slots, it becomes challenging to deal with the dynamics of the environment and may lead to local optimal solutions. The present invention adopts deep reinforcement learning to develop an effective method to obtain suboptimal solutions. This method transforms the difficult non-convex problem into a task of maximizing the global reward, so that the agent can continuously interact with the environment to obtain strategies. However, most existing DRL methods focus on single-agent systems. This limited perspective requires extensive global information, which reduces the adaptability to dynamic environments. A distributed multi-agent method based on the DRL principle is designed. First, the problem is converted into POMDP, where each drone acts as an autonomous decision-making agent for iterative learning and experience updating. Then, the MADRL framework is used to obtain the optimal strategy in a decentralized manner, and an enhanced RED-PPO algorithm with intrinsic rewards is proposed to expand the exploration scope and cope with environmental changes. The detailed process of the method is provided below.

[0057] Each UAV is designated as an intelligent agent. By learning and updating experience from the environment, the UE-UAV association strategy, UAV flight trajectory and RIS phase shift are found. However, the behavior of individual agents is affected by the network environment, and this influence is partially visible. Therefore, the joint optimization problem is transformed into a tuple-based POMDP problem.

[0058] in Indicates the environment, Represents a set of actions, represents the reward function, is in state s t Perform operation a t Then transition to the next state s t+1 The probability that the agent The observed partial observations o t ,γ∈[0,1] represents the reward discount factor,

[0059] In the MADRL framework, each agent maximizes the reward by exploring the best strategy in each decision-making process. Therefore, the corresponding reward function is designed with the goal of maximizing the sum of the perception and communication rates in (P1), and the complex optimization goal is simplified to the expected reward. In order to motivate the UAV agent to continuously expand the exploration scope while maintaining the reward, the present invention introduces both external rewards and internal rewards. Therefore, the reward obtained by each agent is expressed as:

[0060]

[0061] in represents the scaling hyperparameter, and the detailed definitions of extrinsic rewards and intrinsic rewards are as follows:

[0062] The external reward is defined by the objective function of (P1). According to the goal of maximizing the sum of the perception rate and the communication rate, the external reward of the agent can be set as the sum of the total communication data rate of all users (uplink and downlink associated) and the total perception rate of the time slot, that is:

[0063]

[0064] In order to motivate each UAV agent to explore a wider area to collect remote sensing information, the location L n (t) and L n Based on the difference in the number of visits to (t+1), an intrinsic reward criterion is designed, namely:

[0065]

[0066] in is the coefficient, C(L n (t)) is the number of occasional states of each drone, and a binary vector is designed and To ensure the normal operation of the system, Indicates the punishment for abnormal drone flight. represents the penalty for the power constraints of all agents, v1 and v2 are weight variables used to measure the importance of the constraints. From the above formula, we can see that the more times a drone passes through or stays in the same position, the less intrinsic reward it gets. Therefore, each drone tends to explore multiple positions to maximize the cumulative reward, thereby expanding its exploration area.

[0067] Furthermore, step 4 of the present invention comprises:

[0068] Centralized training: Each agent needs to control the trajectory, learn UEs association and control RIS phase shift. Each agent stores the state transition experience obtained in the buffer D. In addition, we use experience replay technology to improve training efficiency to achieve the optimal strategy;

[0069] When each event occurs, each agent observes the state s t Get your own observations and will Stored in buffer D, the agent's actor-critic network then inputs the observations of D and generates a probability distribution as output. As training progresses, the evaluation network continuously updates the advantage function, guiding the gradient update at a lower cost;

[0070] In the considered ISAC system, since the MBS has larger computing resources than the UAV, it can more efficiently perform the offline training of the DRL models centrally, and after sufficient training, these models will be deployed in the subsequent decentralized execution step;

[0071] Distributed execution: Each agent uses the trained actor network in each learning step to generate action sequences by observing the environment. Therefore, each drone can adjust its flight trajectory and UE-UAV association to provide better service to related users. Although all drones can be updated simultaneously, drones are not aware of each other's current actions.

[0072] The MADRL-based method includes the following: First, initialize the parameter settings and the actor-critic network. Each training episode is divided into T time slots. In each time slot, agent n selects an action according to its probability density function. Form a joint action sequence To interact with the environment, the agent obtains the perception rate, communication rate, drone position, flight speed and RIS phase shift by observing the state s. Then, the trained Actor network accepts the generated state sequence and obtains rewards by calculating actions. Finally, the agent transforms the tuple Stored in buffer D, the AC network is then trained using the following algorithm;

[0073] Given that actions consist of discrete actions (such as b n (t)) and continuous actions (such as v n (t)). The present invention adopts a proximal policy optimization algorithm for neural network training, which is specially customized for effectively managing continuous action space and deriving policy solutions. The performance of the traditional PPO algorithm may be limited by the dynamic changes of the network environment. The fluctuation of the number of UEs leads to changes in the environment configuration and the adjustment of the proxy tasks. Therefore, a PPO algorithm based on random environment distribution RED-PPO is proposed. This enhancement aims to enhance the robustness of the model, especially when the number of UEs is variable. The specific process of the algorithm is described as follows:

[0074] The training process starts with initializing the necessary parameters, network, and replay memory. Each training episode spans T time slots, during which the training is performed according to the parameters It is worth noting that this training method incorporates uncertainty by considering the probability of changes in environmental states in each time period, such as temporary UE unavailability. Intentionally incorporating different environmental conditions during training enables the agents to develop adaptive capabilities, enabling them to effectively respond to and navigate changes in the environment. This adaptability is particularly significant when environmental conditions are changing and dynamic.

[0075] After the random environment is generated, the agent training procedure is carried out. In each time slot, the agent first generates its own actions. Then all agents form a joint action The agent interacts with the environment, collects the required state information through importance sampling, reduces sampling error, and improves information acquisition efficiency. Then, the agent obtains the reward of the action through calculation and uses the proposed algorithm to train the AC network. In the algorithm, the output of the evaluation network contributes to the loss function of the actuator network. Therefore, we use the sampling experience to establish a composite loss that includes the loss of the actuator network and the loss of the evaluation network. In addition, each agent is based on the old strategy. The sample data collected under the evaluation of the new strategy π θ The expected value of future results is used to guide the new strategy π θ Improvements;

[0076] Finally, through the iterative optimization algorithm, SCA optimization is continuously performed, alternating with MADRL optimization until the target value converges.

[0077] Beneficial effects:

[0078] 1. While retaining all the functions of the original synaesthesia integrated system, the present invention introduces RIS and drone network to assist the system, and uses alternating iterative optimization method and deep reinforcement learning algorithm to realize the scheduling and decision-making of various resources. Compared with previous systems, it provides additional channel gain and improves the allocation efficiency of communication perception resources.

[0079] 2. The present invention adopts an intelligent agent training method based on deep reinforcement learning. By designing the intrinsic rewards and extrinsic rewards of training, the coverage of the communication-perception integrated network is expanded. While ensuring the communication rate of each user and the perception rate of each perception target as well as the perception-communication balance, the overall communication rate and perception rate of the communication-perception integrated network are increased. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 It is a system architecture diagram.

[0081] Figure 2 This is the overall flow chart of the algorithm.

[0082] Figure 3 This is a diagram of the multi-agent deep reinforcement learning framework.

[0083] Figure 4 It is a flowchart of the deep reinforcement learning algorithm.

[0084] Figure 5 This is the trajectory diagram of the UAV in the simulation experiment.

[0085] Figure 6 This is a comparison chart of the iterative results of various algorithms.

[0086] Figure 7 This is a comparison chart of drone activity ranges using different algorithms.

[0087] Figure 8 This is a comparison chart of the robustness of different algorithms in dealing with dynamic environments. DETAILED DESCRIPTION

[0088] like Figure 1 As shown, the present invention provides a synaesthesia system based on RIS-assisted UAV network, which specifically includes:

[0089] UAV and MBS modules: UAV and MBS are base stations that integrate communication and sensing functions. With the assistance of RIS, they simultaneously serve communication users and sensing objects within the system. The system has a single macro base station MBS and multiple UAVs (UAVs return to the starting point after completing a time slot task so that they can be maintained and replenished with energy).

[0090] In particular, we consider a finite ISAC mission period T, which consists of multiple time slots. It is assumed that the UAV returns to the initial position in each mission phase to facilitate trajectory, beamforming, UE association, and RIS phase shift design. Note that considering the high path loss in practical scenarios, signals reflected more than twice by the RIS are ignored. The UAV's transmitted signal is expressed as:

[0091]

[0092] where w m ∈C K×1 represents the perceptual beamforming matrix, w k ∈C M×1 represents the communication beamforming matrix. C K The vector s in m Indicates Represents the sensing signal, with C M Indicates. denotes the communication symbol vectors, which are statistically independent of each other. In short, we define w[w m w k ]∈C M×(K+M) Represents the general beamforming matrix of the UAV.

[0093] In the system considered, we assume that the flight altitude of each drone is set to H and the flight speed is set to V, where 0<V<V max Mission flight time Divided into multiple time periods, each time period is δt The horizontal position of UAV n at time t is expressed as Considering the energy consumption limit of the drone, it is assumed that each drone will fly back to the starting position after each duration, so there is L n (T) = L n (0). Furthermore, the UAV propulsion energy consumption model is expressed as:

[0094]

[0095] Among them C d is the resistance coefficient. is the reference area of ​​the UAV, and ι is the z-weight of the UAV. is the efficiency of the motor used by the drone, Represents thrust.

[0096] User communication association decoupling module: User behavior mainly includes initiating an uplink request to the UAV or MBS, and obtaining the requested information through the downlink from the UAV, MBS to the user. In this process, the user is allowed to associate different UAVs and MBSs in the uplink and downlink (for example, UE1 and UAV1 form an uplink but can form a downlink with UAV2, MBS, etc.), to avoid the deterioration of the link state caused by the movement of the UAV and affect the communication performance.

[0097] Similar to the perceived signal, for the communication model, the received signal of the kth UE received by UAV_n can be expressed as

[0098] y n，k =(h i，k ΘP n，i +h n，k )x+n k

[0099] The SINR between the kth UE and the nth BS in time slot t is

[0100]

[0101] Since user association is decoupled and there is only one uplink and downlink in a time slot, the binary variable To represent the association between UE_k and BS_n, it is expressed as

[0102]

[0103] where DL and UL are distinguished by the symbol (·). Note that in order to ensure that each UE is associated with at most one b in different links (DL or UL), and Need to satisfy and Therefore, the communication rate of UE_k can be expressed as

[0104]

[0105] The SINR of the decoupled method can be used and Finally, the total communication rate of UEs in UL and DL in time slot t is expressed as

[0106]

[0107] RIS module: Multiple RIS consisting of passive reflection units are deployed within the scope of the integrated communication system. They can be placed on the ground or on buildings. By dynamically adjusting the phase of each RIS unit, the transmitted signal can be effectively reflected, improving the link quality of communication and perception, thereby improving the performance of the entire system.

[0108] Assume that each RIS is equipped with an intelligent controller that can autonomously adjust the phase shift of each element, and each RIS is equipped with S = S l ×S w Reflection element. Then the phase shift matrix of RIS can be expressed as:

[0109]

[0110] where θ s is the phase shift of the sth element of RIS. We consider that each RIS element needs to satisfy the following modularity constraint |θ s | 2 =1.

[0111] The parameters of the perception signal are as follows

[0112] The sensing signal received by BS_n after being reflected by target j can be expressed as

[0113] y j，n =(h i，n ΘP n，i +h j，n )x+n s

[0114] where n s is zero mean and has a covariance of AWGN. n，i h represents the channel matrix from UAV_n to RIS_i. i，n and h j，n They represent the LoS channel matrix of the UAV-UE link and the LoS channel matrix of the UAV-Target link, respectively, and are expressed as'

[0115]

[0116] where χ i，n and χ j，n is the coefficient mainly affected by the system environment. a and θ b denote the angle of the target relative to RIS and BS respectively. Therefore, a(θ a ) and b(θ b ) is expressed as

[0117]

[0118] where d R and d B Determined by the minimum number of UAVs and RIS. The sensing rate quantifies the information exchange between the detected target and the signal receiver. A higher sensing rate indicates a greater amount of target information embedded in the received signal. Considering communication signal interference, the sensing signal-to-interference-and-noise ratio (SINR) of the sensing target j to the UAV n at time slot t can be expressed as

[0119]

[0120] Among them, H R =h i，n (t)ΘP i，n +h j，n (t), on this basis, at time slot t, the perception rate of target j to UAV_n can be expressed as

[0121]

[0122] For simplicity, we use express The beamforming matrix is ​​represented by W, i=1, ..., K+NM (w_i represents the i-th column).

[0123] Therefore, the total perception rate of target j can be expressed as

[0124]

[0125] According to the ISAC network system architecture of RIS-assisted UAV, a joint optimization problem of UAV trajectory design, user communication association decoupling access, beamforming design and RIS phase shift is proposed. The optimization target is set as the sum of the communication rate of all users and the perception rate of all targets in the entire network system. The problem can be expressed as

[0126]

[0127] in In this optimization problem, (1b) constrains the minimum threshold of each perceived target signal-to-noise ratio. (1c) Ensures that each UAV returns to its starting position after a certain period of time. (1d) Ensures that the speed of each UAV is less than the maximum speed in the adjacent time slot. (1e) Ensures that the energy consumption of each UAV is less than the total energy consumption. (1f) represents the phase shift interval of each RIS element. (1g) indicates that the data rate of each UE is greater than the minimum threshold in UL or DL. (1h) Limits the BS power to always be lower than the threshold P within a certain period of time. max .

[0128] The overall flow chart of the algorithm of the present invention is as follows: Figure 2 As shown, in general, the present invention involves the joint optimization problem of user access (B), UAV trajectory design (L), base station beamforming (ω) and RIS phase shift matrix (θ), which is first decomposed into two sub-problems (i beamforming sub-problem (ω); ii user access, UAV trajectory design, base station beamforming and RIS phase shift matrix sub-problem (B, L, θ)), and uses an alternating iterative optimization algorithm to fix the variables of one sub-problem and solve the other sub-problem, and repeat this cycle until a joint optimal solution (B, L, ω, θ) is obtained.

[0129] Since the beamforming subproblem is easy to decompose and relatively simple, the communication beamforming ωc and the perception beamforming ωs are processed to give a fixed (B, L, θ), and on this basis, the CVX optimizer is used to optimize the subproblem.

[0130] First, define two variables and To solve the problem by W m and W k The non-convex problem caused by W m >0, rank(W m )≤1 and W k >0, rank(W k )≤1. Considering a H a=mathmTr(aa H ), (P2) can be rewritten as

[0131] (P3):

[0132]

[0133] The objective function in (P3) is non-concave and thus still non-convex. The present invention adopts a method to approximate the non-convex objective function as a convex objective function, allowing the optimal solution to be found using convex optimization techniques. In particular, by exploiting the incremental behavior of the logarithmic function, a lower bound on the objective function is established using an inequality, namely:

[0134]

[0135] After the above steps, the joint problem (P1) can be transformed into

[0136] (P4):

[0137]

[0138] in

[0139]

[0140] For the objective function in (P4), and It's all about W m and W k is a concave function of , so the formula is in the form of concave-minus-concave. Therefore, we use the SCA method to convert W m and W k Replaced by their lower bounds and Where l ≥ 1 is the iteration index. Through the first-order Taylor expansion, and The upper bound of can be expressed as

[0141]

[0142] The function associated with W is and The gradient of

[0143]

[0144] The problem is further optimized by changing the SDR. In addition, the beamforming matrix is ​​subjected to eigenvalue decomposition to obtain the beamforming vector w m and w k To this end, problem P3 can be finally transformed into

[0145] (P5):

[0146]

[0147] Then the solution is iterated and the feasible point is updated until convergence. Since the joint optimization problem of W is convex, the CVX optimizer can be used to solve the problem.

[0148] For the joint optimization problem P6 of user access (B), UAV trajectory design (L) and RIS phase shift matrix (θ), since the problem is non-convex and contains multiple continuous dynamic optimization variables, splitting it into multiple sub-problems for iterative optimization will lead to convergence to the local optimal point or too high complexity.

[0149] (P6):

[0150] To address this problem, the deep reinforcement learning MADRL algorithm is used, such as Figure 3 As shown in the figure, first, each UAV is designated as an intelligent agent, which generates user access association strategies, UAV flight trajectories, and RIS phase shifts by learning and updating experience from the environment; however, the behavior of a single agent is affected by the network environment, and this influence is partially visible. To this end, the joint optimization problem is transformed into a partially observable Markov decision process {S, A, R, P, O}, where S, A, and R are the set of environmental states, the set of actions, and the reward function, respectively. P is the state of the state in state s t Execute action a t Then transition to the next state s t+1 The probability of the partial observation value o observed by the agent from O t .

[0151] In the MADRL framework, each agent maximizes the reward by exploring the best strategy in each decision-making process. Therefore, the corresponding reward function is designed with the goal of maximizing the sum of the perception and communication rates, simplifying the complex optimization goal into the expected reward. In addition, in order to fully motivate the UAV agent to continuously expand the exploration area while maintaining the reward, the present invention introduces extrinsic rewards and intrinsic rewards. The reward R obtained by the agent in each time slot is the extrinsic reward R EX and intrinsic reward R IN The extrinsic reward comes from the optimized objective function, which is the sum of the communication rate of all UEs in the system and all the perception rates; the intrinsic reward is used to motivate each UAV agent to explore a wider area to collect remote sensing information. The difference in the visit count between each UAV location is used to represent the distribution concentration of the UAV location. The more the UAV stays in the same location, the less intrinsic reward it gets. To this end, each UAV tends to explore as many locations as possible to obtain a larger cumulative reward, and the exploration area of ​​the UAV is therefore expanded.

[0152] The algorithm flow chart of deep reinforcement learning training is as follows Figure 4As shown in the paper, a distributed MADRL method is designed and a deep neural network (DNN) is used as a function approximator to cope with the challenges of large state and action spaces. The implementation process of this method includes centralized training and decentralized execution (CTDE). Specifically, in the training phase, the agent optimizes its local strategy by obtaining information from the central controller (MBS). When performing the distributed execution step, the generation agent only uses its own local information to make decisions.

[0153] The specific steps of centralized training and distributed execution are as follows:

[0154] Step 101: System initialization parameters.

[0155] The system first generates policy-related parameters, observation space, action space, and initializes the cache pool by initializing the Markov process and deep reinforcement learning.

[0156] Step 102: Randomize the system environment.

[0157] The performance of the traditional PPO algorithm may be limited by the dynamic changes in the network environment, such as fluctuations in the number of UEs, which leads to changes in the environment configuration and adjustments to the agent's tasks. For this reason, the random environment state algorithm is introduced to incorporate uncertainty by considering the probability of changes in the environment state in each time period, such as temporary UE unavailability. Intentionally incorporating different environmental conditions during the training process enables the agents to have adaptability, allowing them to effectively respond to changes in the environment.

[0158] Step 103: Distributed execution.

[0159] Each UAV agent uses the trained actor network in each learning step to generate a corresponding action sequence based on its own observations. As a result, each drone can adjust its flight trajectory and UE-UAV association to provide better service to the relevant users. In addition, although all drones can be updated simultaneously, the drones are not aware of each other's selected actions.

[0160] In each time slot, the UAV agent selects an action according to the probability density function, and multiple agents form a joint action to interact with the environment. The agent observes the state and obtains the perception rate, communication rate, UAV position, flight speed, and RIS phase shift. The state sequence is then input into the trained Actor network to calculate the reward R obtained. Finally, each agent stores the transition tuple (R, s, a) in the replay memory sequence and then trains the AC network.

[0161] In the MADRL framework, each agent maximizes the reward by exploring the best strategy in each decision-making process. Therefore, the corresponding reward function is designed with the goal of maximizing the sum of the sensor communication rates in (P6), simplifying the complex optimization objective into the expected reward. In order to motivate the UAV agent to continuously expand the exploration scope while maintaining the reward, the present invention introduces both external rewards and internal rewards. Therefore, the reward obtained by each agent can be expressed as

[0162]

[0163] in represents the scaling hyperparameter. The detailed definitions of intrinsic rewards and extrinsic rewards are as follows:

[0164] Extrinsic Rewards

[0165] The external reward originates from the objective function of (P6). According to the goal of maximizing the sum of the perception rate and the communication rate, the external reward of the agent can be set as the sum of the total communication data rate of all users (uplink and downlink associated) and the total perception rate of the time slot, that is:

[0166]

[0167] Intrinsic Rewards

[0168] In order to motivate each UAV agent to explore a wider area to collect remote sensing information, the location L n (t) and L n The difference in the number of reverse visits (t+1) is used to design an intrinsic reward criterion, namely

[0169]

[0170] Where hatδ is the coefficient, C(L n (t)) is the number of occasional states of each drone. A binary vector is designed and To ensure the normal operation of the system, Indicates the punishment for abnormal drone flight. represents the penalty for all agent power constraints. v1 and v2 are weight variables used to measure the importance of constraints. As can be seen from the formula, the more times a drone passes through or stays in the same location, the less intrinsic reward it gets. Therefore, each drone tends to explore multiple locations to maximize the cumulative reward, thereby expanding its exploration area.

[0171] Step 104: Focused Training

[0172] In this process, each agent needs to control the trajectory, learn the association with UEs and control the RIS phase shift. The state transition experience of each agent is stored in the replay buffer D. In addition, the experience replay technique is used to improve the training stability to obtain the optimal strategy.

[0173] During each event, each agent obtains its own observations by observing the system state and stores them into a buffer D. Then, the agent's actor-critic network takes the observations in D as input and outputs a probability distribution. Specifically, the Actor DNN is responsible for generating action sequences by optimizing the objective function. The DNN is used in the actuator network to parameterize the policy function and generate a policy that follows a Gaussian distribution.

[0174] As the training progresses, the evaluation network continuously updates the advantage function to achieve better results. In the considered ISAC system, since the MBS has larger computing resources than the UAV, it can more efficiently perform offline training of the DRL model in a centralized manner. After sufficient training, these models can be used in the subsequent distributed execution stage.

[0175] Step 105: Generate results.

[0176] Finally, the flight trajectory of each agent, the associated access of UE-UAV, and the phase shift matrix adjustment of each RIS are generated according to the strategy.

[0177] The present invention performs simulation on a dedicated experimental platform and verifies the performance of the proposed optimization method by changing experimental parameters, which specifically includes the following:

[0178] 1. Simulation data

[0179] In the simulation, we considered a RIS-assisted UAV ISAC network equipped with a base station, 4 UAVs, 4 RIS, 10 sensing targets and 20 UEs randomly distributed in the base station coverage area with a radius of 600m. All UAVs fly at an altitude of 200m and each UAV is equipped with 6 antennas. For the training model, the actor and critic networks of each agent have 2 fully connected layers with 40 neurons and a learning rate of 0.0001. The discount factor is 0.999. The attenuation factors are 1.44544 and 199.526.

[0180] 2. Comparison of simulation results

[0181] exist Figure 5In the figure, the four drones are set to the initial positions (100, 100), (-100, -100), (-100, 100) and (100, -100), and all fly at an altitude of 200 meters. It can be seen that in this case, each drone can fly without collision and can return to its initial position after completing a task.

[0182] The RED-PPO method is compared with two different reinforcement learning methods (Vanilla-PG, TRPO, PPO) in terms of obtaining rewards.

[0183] Figure 6 The rewards obtained using four different algorithms are compared. Experimental results show that the proposed algorithm achieves a monotonically increasing global reward and converges after almost 3000 iterations. Compared with the other three algorithms, the proposed algorithm has higher rewards and faster convergence speed. This shows the advantages of the proposed RED-PPO method in training actor-critic networks and optimizing agent policies. Furthermore, the effectiveness of the algorithm is studied from the perspectives of expanding the detection range of drones and enhancing robustness.

[0184] A heat map is used to describe the exploration frequency of the UAV throughout the policy selection process. Figure 7 Comparison of the flight area of ​​UAV 1 under the original PPO and RED-PPO. The brighter the color, the more likely the UAV is to explore the area in actual flight. Compared with the original PPO, RED-PPO shows that the UAV can explore a wider area. The reason is that the intrinsic reward encourages greater exploration to obtain more rewards, proving the effectiveness of the algorithm in expanding the exploration range of the UAV.

[0185] In addition, the robustness of the system using different numbers of UEs in different environments is also evaluated. Figure 8 As shown in Figure 2, the average rewards of RED-based PPO, namely RED-PPO and the proposed PPO, are always higher than those of the original PPO. This indicates that introducing random environments during training can improve the model's ability to adapt to changing environments under the considered network settings.

[0186] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A synaesthesia system based on RIS-assisted drone network, characterized in that: The system includes a UAV, an MBS module, a user communication association decoupling module and a RIS module; The UAV and MBS modules: UAV and MBS are base stations that integrate communication and sensing functions. With the assistance of RIS, they serve communication users and sensing objects within the system at the same time. The system has a single macro base station MBS and multiple UAVs (UAVs return to the starting point after performing a task in a time slot to maintain and replenish energy for UAVs). In particular, we consider a limited ISAC task cycle T, which consists of multiple time slots. It is assumed that the UAV will return to the initial position in each mission stage to perform trajectory, beamforming, UE association and RIS phase shift design. Note that considering the high path loss in actual scenarios, signals reflected more than twice by RIS are ignored. The transmitted signal of the UAV is expressed as: where w m ∈C K×1 represents the perceptual beamforming matrix, w k ∈C M×1 represents the communication beamforming matrix, C K The vector s in m Indicates Represents the sensing signal, with C M Indicates that Denotes the communication symbol vectors, which are statistically independent of each other, and define w[w m w k ]∈C M×(K+M) represents the general beamforming matrix of the UAV; In the system considered, it is assumed that the flight altitude of each drone is set to H and the flight speed is set to V, where 0<V<V max , the flight time of the mission Divided into multiple time periods, each time period is δ t , the horizontal position of UAV n at time t is expressed as Considering the energy consumption limit of the drone, it is assumed that each drone will fly back to the starting position after each duration, so there is L n (T) = L n (0), further, the UAV propulsion energy consumption model is expressed as: Among them C d is the drag coefficient, is the reference area of ​​the UAV, ι is the z-weight of the UAV, is the efficiency of the motor used by the drone, represents thrust; The RIS module: deploys multiple RIS consisting of passive reflection units within the scope of the integrated communication system, which can be placed on the ground or on a building, and effectively reflects the transmitted signal by dynamically adjusting the phase of each RIS unit; Assume that each RIS is equipped with an intelligent controller that autonomously adjusts the phase shift of each element, and each RIS is equipped with S = S l ×S w Reflection element, the phase shift matrix of RIS is expressed as: where θ s is the phase shift of the sth element of RIS. We consider that each RIS element needs to satisfy the following modularity constraint |θ s | 2 =1; The parameters of the perception signal are as follows: The sensing signal received by BS_n after being reflected by target j can be expressed as the j，n =(h i，n ΘP n，i + h j，n )x+n s where n s is zero mean and has a covariance of AWGN, P n，i represents the channel matrix from UAV_n to RIS_i, h i，n and h j，n They represent the LoS channel matrix of the UAV-UE link and the LoS channel matrix of the UAV-Target link, respectively, and are expressed as' where χ i，n and χ j，n is the coefficient mainly affected by the system environment, θ a and θ b denote the angle of the target relative to RIS and BS respectively, so a(θ a ) and b(θ b ) is expressed as where d R and d B Determined by the minimum number of UAVs and RIS, the perception rate quantifies the information exchange between the detected target and the signal receiver. A higher perception rate indicates that the amount of target information embedded in the received signal is greater. Considering the interference of communication signals, the perceived signal-to-interference-and-noise ratio (SINR) of the perceived target j to the UAV n at time slot t is expressed as: Among them, H R =h i，n (t)ΦP i，n +h j，n (t), on this basis, at time slot t, the perception rate of target j to UAV_n is expressed as: use express Wherein the beamforming matrix is ​​represented by W, i=1, ..., K+NM (w_i represents the i-th column); Therefore, the total perception rate of target j is expressed as: The user communication association decoupling module: the user behavior mainly includes initiating an uplink request to the UAV or MBS, and obtaining the requested information through the downlink from the UAV, MBS to the user. In this process, the user is allowed to associate different UAVs and MBSs in the uplink and downlink; Similar to the perception signal, for the communication model, the received signal of the kth UE received by UAV_n is expressed as: the n，k =(h i，k ΘP n，i + h n，k )x+n k The SINR between the kth UE and the nth BS in time slot t is: Since user association is decoupled and there is only one uplink and downlink in a time slot, the binary variable To represent the association between UE_k and BS_n, it is expressed as: where DL and UL are distinguished by the symbol (·). Note that in order to ensure that each UE is associated with at most one b in different links (DL or UL), and Need to satisfy and Therefore, the communication rate of UE_k is expressed as: The SINR of the decoupled method can be used and To calculate, finally, the total communication rate of UEs in UL and DL in time slot t is expressed as:

2. An optimization method for a synaesthesia system based on a RIS-assisted UAV network, characterized in that: The method comprises the following steps: Step 1: Based on the ISAC network system architecture of the constructed RIS-assisted UAV, a joint optimization problem of UAV trajectory design, user communication association decoupling access, beamforming design and RIS phase shift is proposed. The optimization target is set as the sum of the communication rate of all users and the perception rate of all targets in the entire network system. The problem is expressed as: in In this optimization problem, (1b) constrains the minimum threshold of the signal-to-noise ratio of each sensing target, (1c) ensures that each UAV returns to the starting position after a certain period of time, (1d) ensures that the speed of each UAV is less than the maximum speed in the adjacent time slot, (1e) ensures that the energy consumption of each UAV is lower than the total energy consumption, (1f) represents the phase shift interval of each RIS element, (1g) indicates that the data rate of each UE is greater than the minimum threshold in UL or DL, and (1h) limits the BS power to always be lower than the threshold P within a certain period of time. max ; Step 2: Decompose the joint optimization problem involving user access (B), UAV trajectory design (L), base station beamforming (ω) and RIS phase shift matrix (θ) into two sub-problems (i beamforming sub-problem (ω); ii user access, UAV trajectory design, base station beamforming and RIS phase shift matrix sub-problem (B, L, θ)). Use the alternating iterative optimization algorithm to fix the variables of one sub-problem and solve the other sub-problem, and repeat this process until the joint optimal solution (B, L, ω, θ) is obtained. Step 3: Due to the coupling between UE-UAV association and UAV positioning, the optimization problem (P1) containing three variables (UE-UAV association B, trajectory design L, and RIS phase shift θ) is still a nonlinear programming problem with non-convex constraints. To this end, (P1) is decomposed into multiple sub-problems and attempts are made to solve them using iterative alternating optimization. Step 4: A distributed MADRL method is proposed, which uses deep neural networks (DNNs) as function approximators to handle the complexity brought by the wide state and action space. The implementation process of this method includes centralized training and distributed execution. Specifically, in the training phase, each agent improves its local strategy by obtaining information from the central controller (i.e., MBS). During the distributed execution step, the agent only uses its own local information to make decisions.

3. The optimization method of the synaesthesia system based on RIS-assisted UAV network according to claim 2 is characterized in that: The step 2 includes: considering the optimization problem of the beamforming vector in the (P1) problem under the given UE association B, trajectory L and RIS phase shift θ, so the optimization problem is expressed as: (P2)： st(1b),(lg)and(1h) After processing by the SDR and SCA algorithms, the problem is simplified to a convex problem with a single optimization variable for beamforming, which is solved using the CVX optimizer.

4. The optimization method of the synaesthesia system based on RIS-assisted UAV network according to claim 2 is characterized in that: The step 3 includes: considering that (P1) involves UE-UAV association, UAV positioning and RIS phase, the computational complexity of iterative alternating optimization calculation is Where ι is the number of alternating iterations. The complexity is significantly higher than (P2) and further increases with the increase of the number of drones B, the number of UEs K, and the number of RIS I. In addition, due to the existence of multiple dynamic variables, such as the position L of the drone in different time slots, it becomes challenging to deal with the dynamics of the environment and may lead to local optimal solutions. Deep reinforcement learning is used to develop an effective method to obtain suboptimal solutions. This method transforms the difficult non-convex problem into a task of maximizing the global reward, enabling the agent to continuously interact with the environment to obtain strategies. However, most existing DRL methods focus on single-agent systems. This limited perspective requires extensive global information, which reduces the adaptability to dynamic environments. A distributed multi-agent method based on the DRL principle is designed. First, the problem is converted into POMDP, where each drone acts as an autonomous decision-making agent for iterative learning and experience updating. Then, the MADRL framework is used to obtain the optimal strategy in a decentralized manner, and an enhanced RED-PPO algorithm with intrinsic rewards is proposed to expand the exploration scope and cope with environmental changes. The detailed process of the method is provided below. Each UAV is designated as an intelligent agent. By learning and updating experience from the environment, the UE-UAV association strategy, UAV flight trajectory and RIS phase shift are found. However, the behavior of individual agents is affected by the network environment, and this influence is partially visible. Therefore, the joint optimization problem is transformed into a tuple-based POMDP problem. in Indicates the environment, Represents a set of actions, represents the reward function, is in state s t Perform operation a t Then transition to the next state s t+1 The probability that the agent The observed partial observations o t ,γ∈[0,1] represents the reward discount factor, In the MADRL framework, each agent maximizes the reward by exploring the best strategy in each decision-making process. Therefore, the corresponding reward function is designed to maximize the sum of the perception and communication rates in (P1), and the complex optimization objective is simplified to the expected reward. In order to motivate the UAV agent to continuously expand the exploration scope while maintaining the reward, external rewards and internal rewards are introduced at the same time. Therefore, the reward obtained by each agent is expressed as: in represents the scaling hyperparameter, and the detailed definitions of extrinsic rewards and intrinsic rewards are as follows: The external reward is defined by the objective function of (P1). According to the goal of maximizing the sum of the perception rate and the communication rate, the external reward of the agent can be set as the sum of the total communication data rate of all users (uplink and downlink associated) and the total perception rate of the time slot, that is: In order to motivate each UAV agent to explore a wider area to collect remote sensing information, the location L n (t) and L n Based on the difference in the number of visits to (t+1), an intrinsic reward criterion is designed, namely: in is the coefficient, C(L n (t)) is the number of occasional states of each drone, and a binary vector is designed and To ensure the normal operation of the system, Indicates the punishment for abnormal drone flight. represents the penalty for the power constraints of all agents, v1 and v2 are weight variables used to measure the importance of the constraints. From the above formula, we can see that the more times a drone passes through or stays in the same position, the less intrinsic reward it gets. Therefore, each drone tends to explore multiple positions to maximize the cumulative reward, thereby expanding its exploration area.

5. The optimization method of the synaesthesia system based on RIS-assisted UAV network according to claim 2 is characterized in that: The step 4 comprises: Centralized training: Each agent needs to control the trajectory, learn UEs association and control RIS phase shift. Each agent stores the state transition experience obtained in the buffer D. In addition, we use experience replay technology to improve training efficiency to achieve the optimal strategy; When each event occurs, each agent observes the state s t Get your own observations and will Stored in buffer D, the agent's actor-critic network then inputs the observations of D and generates a probability distribution as output. As training progresses, the evaluation network continuously updates the advantage function, guiding the gradient update at a lower cost; In the considered ISAC system, since the MBS has larger computing resources than the UAV, it can more efficiently perform the offline training of the DRL models centrally, and after sufficient training, these models will be deployed in the subsequent decentralized execution step; Distributed execution: Each agent uses the trained actor network in each learning step to generate action sequences by observing the environment. Therefore, each drone can adjust its flight trajectory and UE-UAV association to provide better service to related users. Although all drones can be updated simultaneously, drones are not aware of each other's current actions. The MADRL-based method includes the following: First, initialize the parameter settings and the actor-critic network. Each training episode is divided into T time slots. In each time slot, agent n selects an action according to its probability density function. Form a joint action sequence To interact with the environment, the agent obtains the perception rate, communication rate, drone position, flight speed and RIS phase shift by observing the state s. Then, the trained Actor network accepts the generated state sequence and obtains rewards by calculating actions. Finally, the agent transforms the tuple Stored in buffer D, the AC network is then trained using the following algorithm; Given that actions consist of discrete actions (such as b n (t)) and continuous actions (such as u n (t)) is composed of a proximal policy optimization algorithm for neural network training, which is specially tailored for effectively managing continuous action space and deriving policy solutions. The performance of the traditional PPO algorithm may be limited by the dynamic changes of the network environment. The fluctuation of the number of UEs leads to changes in the environment configuration and the adjustment of the proxy tasks. Therefore, a PPO algorithm based on random environment distribution RED-PPO is proposed. This enhancement aims to enhance the robustness of the model, especially when the number of UEs is variable. The specific process of the algorithm is described as follows: The training process starts with initializing the necessary parameters, network, and replay memory. Each training episode spans T time slots, during which the training is performed according to the parameters It is worth noting that this training method incorporates uncertainty by considering the probability of changes in environmental states in each time period, such as temporary UE unavailability. Intentionally incorporating different environmental conditions during training enables the agents to develop adaptive capabilities, enabling them to effectively respond to and navigate changes in the environment. This adaptability is particularly significant when environmental conditions are changing and dynamic. After the random environment is generated, the agent training procedure is carried out. In each time slot, the agent first generates its own actions. Then all agents form a joint action The agent interacts with the environment and collects the required state information through importance sampling, which reduces sampling error and improves information acquisition efficiency. Then, the agent obtains the reward of the action through calculation and trains the AC network using the proposed algorithm. In the algorithm, the output of the evaluation network contributes to the loss function of the actuator network. Therefore, we use the sampling experience to establish a composite loss that includes the loss of the actuator network and the loss of the evaluation network. In addition, each agent performs a The sample data collected under the evaluation of the new strategy π θ The expected value of future results is used to guide the new strategy π θ Improvements; Finally, through the iterative optimization algorithm, SCA optimization is continuously performed, alternating with MADRL optimization until the target value converges.

Images