Resource allocation method based on internal-external circulation and hierarchical reinforcement learning

Abstract

The invention relates to the technical field of wireless communication, intelligent traffic networks and unmanned aerial vehicle cooperative communication, and discloses a resource allocation method based on internal-external circulation and hierarchical reinforcement learning. The resource allocation method is suitable for a multi-link cooperative communication system comprising a base station BS, a reconfigurable intelligent surface RIS, M groups of V2V communication vehicle pairs, W vehicles needing RIS auxiliary communication, an unmanned aerial vehicle UAV0 and K cooperative unmanned aerial vehicles UAVs, and comprises the following steps: under the condition that link constraint conditions are met, realizing system performance optimization and fairness maximization as optimization targets; the resource allocation problem for constructing the channel model of the multi-link cooperative communication system is as follows: under the link constraint condition, the fair allocation of the transmission rate of each link is realized by using a maximized fair index Jain's as a target function; and solving an optimization problem of the constructed channel model based on a resource allocation optimization solving process of internal-external circulation and deep reinforcement learning to obtain a system global performance optimal solution meeting a link constraint condition.

Description

Technical Field

[0001] The present invention relates to the technical fields of wireless communications, intelligent transportation networks and unmanned aerial vehicle cooperative communications, and in particular to a resource allocation method based on inner-outer loops and hierarchical reinforcement learning. Background Art

[0002] In modern wireless communication networks, with the rapid development of intelligent transportation and drone technology, vehicle-to-everything (V2X) and drone networks (UAVNetwork) are gradually becoming important components supporting smart cities and intelligent transportation systems. However, these networks face multiple challenges in actual deployment, such as signal fading in non-line-of-sight (NLOS) environments, link instability, and performance bottlenecks caused by uneven resource allocation. Especially in V2X communications, due to interference from buildings, trees, and dynamic traffic scenes, the direct path between the base station (BS) and the vehicle is often blocked, and the communication quality is severely limited. UAV networks, on the other hand, need to face the challenges of high dynamics and interference complexity in collaborative links.

[0003] In recent years, reconfigurable smart surface (RIS) has become an ideal solution to communication problems in NLOS environments as a revolutionary technology due to its low power consumption, low cost and high flexibility. RIS consists of a large number of reflective units with adjustable phase shift and amplitude, which can intelligently manipulate the propagation path of the incident signal to enhance the target signal or suppress interference. With the assistance of RIS, the base station can provide reliable communication services for NLOS vehicles by optimizing the reflected signal, while drones can also use RIS to improve the performance of cooperative communication links. In addition, the introduction of RIS can significantly reduce the consumption of network resources and improve the coverage of the system. However, how to optimize resource allocation and link performance is still an urgent problem to be solved in V2X and drone communications. Summary of the invention

[0004] In order to overcome the above technical defects, the present invention provides a resource allocation method based on inner-outer loop and hierarchical reinforcement learning, which is characterized by being applicable to a multi-link cooperative communication system including: a base station BS, a reconfigurable intelligent surface RIS, M groups of V2V communication vehicle pairs, W vehicles requiring RIS-assisted communication, a UAV0 and K cooperative UAVs;

[0005] Resource allocation methods include:

[0006] S1. Under the condition of satisfying link constraints, with the optimization goal of achieving systematic optimization and maximizing fairness, the resource allocation problem of constructing the channel model of the multi-link cooperative communication system is: Under the link constraints, by maximizing Jain's fairness index as the objective function, the fair allocation of the transmission rate of each link is achieved, where each link includes BS-RIS-V link, UAV0-UAV link and V2V link;

[0007] Jain's fairness index is denoted as J(t), and its objective function is expressed as follows:

[0008] Link constraints include link capacity constraints, power constraints, spectrum resource constraints, RIS configuration constraints, and signal-to-interference-plus-noise ratio (SINR) constraints.

[0009] S2. Based on the inner-outer loop and deep reinforcement learning resource allocation optimization solution process, the optimization problem of the constructed channel model is solved to obtain the optimal solution for the system global performance that meets the link constraints.

[0010] More specifically, the link capacity constraint is expressed as:

[0011] 0≤R i (t)≤C i , Among them, C i represents the maximum capacity of link i in time slot t;

[0012] The power constraint is expressed as:

[0013] P BS (t)≤P BS,max

[0014] Among them, P BS (t) and They represent the transmission power of the base station, UAV and vehicle at time slot t respectively;

[0015] The spectrum resource constraint is expressed as:

[0016] Among them, B i (t) represents the allocated bandwidth of link i in time slot t, B total is the total spectrum resource of the system;

[0017] RIS configuration constraints are expressed as:

[0018] θ n (t)∈[0,2π), 0≤ρ n(t)≤1,

[0019] Among them, θ n (t) and ρ n (t) are the phase and amplitude of the nth RIS unit at time slot t, respectively;

[0020] The signal-to-interference-plus-noise ratio (SINR) constraint is expressed as:

[0021] SINR i (t)≥γ min , Among them, γ min It is the minimum SINR threshold that meets the communication quality requirements.

[0022] More specifically, the resource allocation optimization solution process based on inner-outer loop and deep reinforcement learning in S2 includes:

[0023] S2.1. Initialization parameters: Initialize the meta-parameters φ in the channel model = {φ RB ,φ Power ,φ RIS} and hyperparameters, φ RB The initial parameter corresponding to the resource block allocation, φ Power The initial parameters corresponding to the power allocation, φ RIS Corresponding to the initial parameters configured by RIS, the hyperparameters include the inner loop learning rate, the outer loop learning rate, and the task batch size;

[0024] S2.2, Task definition and decomposition: According to the objective function of maximizing Jain's fairness index, the resource allocation problem is decomposed into three subtasks: resource block RB allocation target, power allocation target and RIS configuration optimization target;

[0025] The resource block allocation goal is to optimize the resource block allocation strategy. Maximize the transmission rate to meet the link capacity constraints and spectrum resource constraints; the power allocation goal is: optimize the power allocation B(t) = {B i (t)}, maximize the total system throughput under the condition of satisfying power constraints and signal-to-interference-and-noise ratio (SINR) constraints; RIS configuration optimization goal: optimize the RIS reflection coefficient Θ(t), improve the link signal-to-interference-and-noise ratio SINR, and satisfy the RIS configuration constraints;

[0026] S2.3, inner loop update: For each subtask of resource block RB allocation target, power allocation target and RIS configuration optimization target, the following task-level optimization operations are performed:

[0027] S2.3.1, Randomly sample a small batch of data for a task;

[0028] S2.3.2. Based on the objective function and constraints of the current task, the specific update of the task parameters is performed by gradient descent as follows: in, For the task τ i The loss function of

[0029] S2.4, outer loop update: Summarize the gradient information of all tasks, calculate the meta-gradient φ and update the meta-parameters using the following calculation formula: in, It is the task loss after the inner loop update;

[0030] S2.5, Iterate until convergence: Repeat S2.2 to S2.4 for iterative calculation until any of the following convergence conditions is met:

[0031] The total fairness index, i.e., Jain's fairness index J(t), reaches a preset threshold, or the value of the objective function maximizing Jain's fairness index tends to be stable;

[0032] S2.6. Output optimization results: After the training is completed, the optimal strategy for each subtask is obtained, that is, the optimal resource block allocation strategy B * (t), optimal power allocation strategy P * (t), and the optimal RIS configuration Θ * (t).

[0033] The present invention adopts the above technical solution to achieve the following technical effects:

[0034] 1. Meta-learning enables the system to quickly adapt to new tasks. Hierarchical reinforcement learning further decomposes the global optimization problem into multiple subtasks (such as resource block allocation, power allocation, and RIS configuration) for separate optimization, thereby greatly improving the algorithm's computational efficiency and resource utilization. While maximizing the overall system performance, it also takes into account resource allocation fairness in multi-user and multi-link scenarios.

[0035] 2. The target design of the present invention responds to the core requirements of intelligent transportation networks for both efficiency and fairness, and provides an innovative solution that is in line with the 6G vision.

[0036] 3. The optimization framework proposed in this invention not only focuses on throughput optimization, but also comprehensively considers multi-dimensional performance indicators such as power allocation, RIS configuration, SINR and QoS, and constructs a multi-objective optimization model that can provide balanced improvement of performance and adaptability under multiple scenario requirements.

[0037] 4. This system has the ability to adapt to a variety of scenarios: it performs well in complex scenarios such as highly obstructed urban roads, mountainous areas, and remote uninhabited areas, providing a highly adaptable and innovative solution for intelligent transportation, drone collaborative networks, and emergency communications.

[0038] 5. Balancing resource efficiency and fairness: By optimizing resource utilization and improving fairness among users, this study achieves the dual goals of optimizing resource allocation and improving user experience, providing a win-win solution with practical application value for network operators and end users.

[0039] 6. The present invention proposes a number of innovations in system design, target construction and algorithm methods, and especially provides new technical ideas and theoretical support for resource allocation and fairness optimization problems in complex communication environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a model structure diagram of the multi-link cooperative communication system in the present invention;

[0041] Figure 2 A schematic diagram of a flow chart of a resource allocation method based on inner-outer loop and hierarchical reinforcement learning in the present invention;

[0042] Figure 3 are the optimization problem related parameters used in the simulation and analysis of the present invention;

[0043] Figure 4 are algorithm-related parameters used in the simulation and analysis of the present invention;

[0044] Figure 5 are the communication environment related parameters used in the simulation and analysis of the present invention;

[0045] Figure 6 A curve relationship diagram between the fairness index and the signal to interference noise ratio obtained by simulation analysis of the present invention;

[0046] Figure 7 This is a curve relationship diagram between the cumulative probability and the fairness index obtained by simulation analysis of the present invention. DETAILED DESCRIPTION

[0047] The present invention is further described in detail below in conjunction with specific embodiments, but the protection scope of the present invention is not limited thereto.

[0048] This paper combines deep reinforcement learning (DRL) technology with an event trigger mechanism to propose an innovative solution. Specifically, an event trigger mechanism is introduced to dynamically detect changes in system status and ensure that the optimization algorithm is updated only when necessary, thereby significantly reducing computational overhead and improving system efficiency. At the same time, combining the advantages of the PPO (Proximal Policy Optimization) algorithm and the SAC (Soft Actor-Critic) algorithm, the performance and stability of multi-link resource allocation are further improved through a joint optimization strategy.

[0049] In the multi-link cooperative communication system of the present invention, RIS is carried on the UAV to dynamically adapt to the needs of complex scenarios and play a key role in the multi-link communication between the base station and the vehicle and the UAV. By jointly optimizing the power allocation, RIS phase adjustment and UAV trajectory through the DRL algorithm, the present invention can effectively improve the channel quality of the communication link and achieve a significant improvement in system performance. At the same time, by maximizing the Jain's fairness index of the system, a more balanced resource allocation between different users or links is ensured. The innovation of this study is not only reflected in the combination of RIS and UAVs, but also in the collaborative design of event triggers and advanced optimization algorithms, which provides a new technical path for solving complex optimization problems in V2X and UAV networks.

[0050] First, the following terms are explained:

[0051] The PPO algorithm is a reinforcement learning algorithm based on policy gradient, which is used to optimize the policy network to maximize the cumulative reward. Its core idea is to update the policy by estimating the policy gradient to maximize the cumulative reward. However, traditional policy gradient algorithms may have problems such as unstable training and slow convergence when updating policies. The emergence of the PPO algorithm is to solve these problems. It can more effectively use sample data for policy updates and improve training efficiency and stability.

[0052] The SAC algorithm, an advanced model-free reinforcement learning algorithm, is mainly used for decision-making problems in continuous action spaces. In the field of reinforcement learning, the agent needs to learn the optimal strategy in the environment to maximize the cumulative reward. Traditional algorithms may face challenges when dealing with continuous action spaces. For example, in the fields of robot control and autonomous driving, the action space is continuous, such as the adjustment of the robot's joint angles or the acceleration and steering of the car. The emergence of the SAC algorithm provides an effective solution to these problems, and it also introduces the concept of entropy regularization, which enables the agent to achieve a better balance between exploring the environment and utilizing existing knowledge.

[0053] Secondly, in order to facilitate those skilled in the art to understand the technical solution in the present invention, the system structure of the multi-link cooperative communication system constructed in the present invention is described below, as follows:

[0054] The multi-link cooperative communication system of the present invention is a complex vehicle-to-everything (V2X) communication system that includes multiple communication links. Figure 1 As shown in the figure, this system integrates three key communication links: the BS-RIS-V link between the base station and the vehicle, the UAV-UAV link between drones, and the V2V link between vehicles. This multi-link cooperative communication model not only enhances the communication quality of the V2X network, but also significantly improves the network reliability in dynamic traffic and high occlusion scenarios, reflecting the collaborative gain in complex communication scenarios.

[0055] Below Figure 1 The system descriptions of the system components, system links, RIS model, etc. of the system shown are as follows:

[0056] 1) The system consists of: base station BS, reconfigurable intelligent surface RIS, M groups of V2V communication vehicle pairs, W vehicles that need RIS-assisted communication, UAV0, and K cooperative UAVs, as follows:

[0057] Base station BS: The only base station in the system, responsible for exchanging data with vehicles that need to communicate.

[0058] Reconfigurable smart surface RIS: It consists of N reflection units and can enhance channel quality or suppress interference by adjusting the phase shift and reflection coefficient of each unit. It is mainly used to optimize signal propagation of communication links.

[0059] M groups of V2V communication vehicle pairs: Each group of vehicles conducts short-distance, low-latency direct communication through V2V links to exchange local traffic information or collaborative driving information.

[0060] W vehicles that need RIS-assisted communication: Due to obstacles, W vehicles cannot communicate directly with the base station, but communicate through RIS reflection links.

[0061] UAV0: Equipped with RIS, it is responsible for providing services to non-line-of-sight NLOS vehicles through reflection links and achieving communication by bypassing obstacles.

[0062] K collaborative UAVs: collaborate with UAV0 to extend the system coverage through inter-UAV communication links, supporting dynamic deployment of the system and efficient data transmission.

[0063] 2) System links include: BS-RIS-V link, UAV0-UAV link, V2V link, as follows:

[0064] BS-RIS-V link: The base station provides communication services to W obscured vehicles through the RIS carried on UAV0. RIS enhances signal quality by dynamically adjusting the phase shift and amplitude of the reflector unit.

[0065] UAV0-UAV link: UAV0 forms a collaborative network with the surrounding K drones, using this link to transmit control information and data to further expand the communication coverage.

[0066] V2V link: A direct communication link between M pairs of vehicles for exchanging real-time information, ensuring low latency and high reliability of vehicle-to-vehicle communication.

[0067] 3) RIS model: In this system, the reconfigurable smart surface (RIS) significantly improves the channel quality through beamforming and path optimization. RIS consists of N reflection units, and the reflection coefficient of each reflection unit is defined as:

[0068]

[0069] Where: θ n is the phase offset of the nth reflection unit, and its value range is [0, 2π); ρ n is the reflection amplitude coefficient of the nth reflection unit, and its value range is [0, 1].

[0070] By adjusting Θ n RIS can realize intelligent signal reflection and optimize system performance, such as maximizing signal reception strength or improving signal-to-interference-to-noise ratio (SINR). The system goal is to dynamically optimize Θ based on real-time channel state information (CSI). n Configuration.

[0071] The following is a mathematical modeling of the BS-RIS-V link in the system, including the analysis of the received signal, signal-to-interference-noise ratio (SINR), and transmission rate. Assuming that the system adopts a discrete timing model, the channel statistical characteristics remain unchanged in each time slot.

[0072] Received signal: For the BS-RIS-V link reflected by RIS, the wth vehicle V w The signal received at time slot t is expressed as:

[0073]

[0074] In formula (2), represents the channel gain from BS to the nth RIS unit; Represents the distance from the nth RIS unit to the vehicle V w The channel gain of x BS (t) represents the signal sent by the base station; Indicates noise, obey

[0075] Signal to Interference and Noise Ratio SINR: The signal to interference and noise ratio SINR of the wth BS-RIS-V link in time slot t is expressed as:

[0076]

[0077] In formula (3), I intra (t) represents the interference of the BS-RIS-V links in the same group, and its expression is:

[0078]

[0079] I UAV (t) represents the interference from other UAVs, and its expression is:

[0080]

[0081] I V2V (t) represents the interference from the V2V link, which is expressed as:

[0082]

[0083] Transmission rate: The transmission rate of the wth BS-RIS-V link in time slot t is expressed as:

[0084]

[0085] In formula (7), B represents the bandwidth of the link.

[0086] Based on the channel model of the multi-link cooperative communication system described above, the resource allocation method based on inner-outer loop and hierarchical reinforcement learning in the present invention is applicable to a multi-link cooperative communication system including a base station BS, a reconfigurable intelligent surface RIS, M groups of V2V communication vehicle pairs, W vehicles requiring RIS-assisted communication, a UAV0, and K cooperative UAVs, and the specific scheme thereof includes:

[0087] S1. Under the condition of satisfying link constraints, the resource allocation problem of constructing the channel model of the multi-link cooperative communication system with the optimization goal of achieving system performance optimization and fairness maximization is as follows: Under the link constraints, by maximizing the fairness index Jain's as the objective function, the fair allocation of the transmission rate of each link is achieved;

[0088] Specifically, Jain's fairness index is denoted as J(t), and its objective function is expressed as:

[0089]

[0090] in, is the transmission power set of the base station, UAV and vehicle at time slot t;

[0091] B(t)={B i (t)} is the set of spectrum bandwidths allocated to each link i at time slot t;

[0092] Θ(t)={θ n (t),ρ n (t)} is the reflection configuration set of RIS at time slot t, where θ n (t) and ρ n (t) are the phase and amplitude of the nth reflection unit respectively;

[0093] R i (t) is the transmission rate of the i-th link in time slot t;

[0094] R i (t) = B i (t))log2(1+SINR i (t)) B i (t) is the allocated bandwidth of the ith link in time slot t.

[0095] More specifically, the link constraint conditions include link capacity constraint, power constraint, spectrum resource constraint, RIS configuration constraint and signal-to-interference-plus-noise ratio (SINR) constraint.

[0096] The link capacity constraint is expressed as:

[0097] 0≤R i (t)≤C i , Among them, C i represents the maximum capacity of link i in time slot t;

[0098] The power constraint is expressed as:

[0099]

[0100] Among them, P BS (t) and They represent the transmission power of the base station, UAV and vehicle at time slot t respectively;

[0101] The spectrum resource constraint is expressed as:

[0102] Among them, B i (t) represents the allocated bandwidth of link i in time slot t, B total is the total spectrum resource of the system;

[0103] RIS configuration constraints are expressed as:

[0104] θ n (t)∈[0,2π),0≤ρ n (t)≤1,

[0105] Among them, θ n (t) and ρ n (t) are the phase and amplitude of the nth RIS unit at time slot t, respectively;

[0106] The signal to interference plus noise ratio (SINR) constraint is expressed as:

[0107] SINR i (t)≥γ min , Among them, γ min It is the minimum SINR threshold that meets the communication quality requirements.

[0108] The above constraints ensure the feasibility of the system in terms of link capacity, power, spectrum resources, RIS configuration and communication quality, maximize the fair distribution of transmission rate, and improve the overall system performance.

[0109] S2. Based on the inner-outer loop and deep reinforcement learning resource allocation optimization solution process, the optimization problem of the constructed channel model is solved to obtain the optimal solution for the system global performance that meets the link constraints.

[0110] The above solution process includes the following steps:

[0111] S2.1. Initialization parameters: Initialize the meta-parameters in the channel model, which correspond to the initial parameters of resource block allocation, power allocation and RIS configuration, and set the following hyper-parameters: inner loop learning rate, outer loop learning rate and task batch size.

[0112] The above meta-parameters are φ={φ RB ,φ Power ,φ RIS},φ RB The initial parameter corresponding to the resource block allocation, φ Power The initial parameters corresponding to the power allocation, φ RIS Corresponding to the initial parameters of RIS configuration.

[0113] S2.2, Task definition and decomposition: According to the objective function of maximizing Jain's fairness index, the resource allocation problem is decomposed into three subtasks: resource block RB allocation target, power allocation target and RIS configuration optimization target. Each subtask has independent objectives and constraints:

[0114] The resource block allocation goal is: optimize the resource block allocation strategy Maximize the transmission rate and meet the link capacity constraints and spectrum resource constraints. The power allocation goal is: optimize the power allocation B(t) = {B i (t)}, maximizing the total system throughput under the condition of satisfying the power constraint and signal-to-interference-and-noise ratio (SINR) constraint. RIS configuration optimization goal: optimize the RIS reflection coefficient Θ(t), improve the link signal-to-interference-and-noise ratio SINR, and satisfy the RIS configuration constraints.

[0115] S2.3, inner loop update, i.e. task level optimization: for each subtask of resource block RB allocation target, power allocation target and RIS configuration optimization target, the following task level optimization operations are performed:

[0116] S2.3.1, Randomly sample a small batch of data for a task;

[0117] S2.3.2. Based on the objective function and constraints of the current task, the specific update of the task parameters is performed by gradient descent as follows: in, For the task The loss function of .

[0118] S2.4, outer loop update: Summarize the gradient information of all tasks, calculate the meta-gradient φ and update the meta-parameters. The calculation formula is: in, It is the task loss after the inner loop update.

[0119] S2.5, Iterate until convergence: Repeat S2.2 to S2.4 for iterative calculation until any of the following convergence conditions is met: the total fairness index, i.e., Jain's fairness index J(t), reaches a preset threshold, or the objective function The value tends to be stable without significant change.

[0120] S2.6. Output optimization results: After the training is completed, the optimal strategy for each subtask is obtained, that is, the optimal resource block allocation strategy B * (t), optimal power allocation strategy P * (t), and the optimal RIS configuration Θ * (t).

[0121] Finally, the optimal solution of the system's global performance that meets the constraints is obtained.

[0122] Finally, in order to verify the technical effect of the technical solution of the present invention, Figure 3 The optimization problem related parameters shown in Figure 4 The algorithm-related parameters shown in Figure 5The communication environment related parameters shown in the simulation and result analysis are obtained as follows Figure 6 and Figure 7 The simulation results are shown.

[0123] Figure 6 The variation trend of the fairness index J(t) with the increase of signal-to-interference-to-noise ratio (SNIR) is shown. It can be seen that as the SNIR increases, the fairness index gradually increases, indicating that the fairness of resource allocation is significantly improved under better channel conditions. From the comparison of algorithms, the joint inner-outer loop deep reinforcement learning method shows higher fairness under different SNIR conditions, indicating that it is more effective in optimizing resource allocation; while the fairness of the pure outer loop method is lower, but it is more stable than the pure inner loop method; the pure inner loop method has limited effect under low SNIR conditions and poor fairness. Therefore, the joint inner-outer loop deep reinforcement learning method provides better resource allocation fairness under various channel conditions and is the most balanced choice.

[0124] Figure 7 The relationship between the cumulative probability and the fairness index is shown. As can be seen from the figure, the fairness of the joint inner-outer loop deep reinforcement learning method is mainly concentrated in the higher range, indicating that it can provide higher resource allocation fairness in most cases and adapt to different channel conditions. In contrast, the fairness of the pure outer loop method is widely distributed, indicating that its resource allocation effect is greatly affected by channel conditions and has poor stability. The fairness of the pure inner loop method is mainly concentrated in the lower range, showing that it is insufficiently fair in most cases, especially when the channel is poor. Therefore, the joint inner-outer loop deep reinforcement learning method is more stable in resource allocation and is suitable for application scenarios with high requirements for fairness.

[0125] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. A resource allocation method based on inner-outer loop and hierarchical reinforcement learning, characterized in that: Applicable to a multi-link cooperative communication system including: a base station BS, a reconfigurable intelligent surface RIS, M groups of V2V communication vehicle pairs, W vehicles requiring RIS-assisted communication, a UAV0 and K cooperative UAVs; The resource allocation method comprises: S1. Under the condition of satisfying link constraints, with the optimization goal of achieving systematic optimization and maximizing fairness, the resource allocation problem of constructing the channel model of the multi-link cooperative communication system is: Under the link constraints, by maximizing Jain's fairness index as the objective function, the fair allocation of the transmission rate of each link is achieved, where each link includes BS-RIS-V link, UAV0-UAV link and V2V link; Jain's fairness index is denoted as J(t), and its objective function is expressed as follows: Link constraints include link capacity constraints, power constraints, spectrum resource constraints, RIS configuration constraints, and signal-to-interference-plus-noise ratio (SINR) constraints. S2. Based on the inner-outer loop and deep reinforcement learning, the resource allocation optimization solution process solves the optimization problem of the constructed channel model and obtains the optimal solution for the system global performance that meets the link constraints.

2. The resource allocation method based on inner-outer loop and hierarchical reinforcement learning according to claim 1, characterized in that: The link capacity constraint is expressed as: 0≤R i (t)≤C i , Among them, C i represents the maximum capacity of link i in time slot t; The power constraint is expressed as: P BS (t)≤P BS,max , Among them, P BS (t) and They represent the transmission power of the base station, UAV and vehicle at time slot t respectively; The spectrum resource constraint is expressed as: Among them, B i (t) represents the allocated bandwidth of link i in time slot t, B total is the total spectrum resource of the system; RIS configuration constraints are expressed as: i n (t)∈[0,2π),0≤ρ n (t)≤1, Among them, θ n (t) and ρ n (t) are the phase and amplitude of the nth RIS unit at time slot t, respectively; The signal-to-interference-plus-noise ratio (SINR) constraint is expressed as: SINR i (t)≥γ min Among them, γ min It is the minimum SINR threshold that meets the communication quality requirements.

3. The resource allocation method based on inner-outer loop and hierarchical reinforcement learning according to claim 1 or 2, characterized in that: The resource allocation optimization solution process based on inner-outer loop and deep reinforcement learning in S2 includes: S2.

1. Initialization parameters: Initialize the meta-parameters φ in the channel model = {φ RB ,φ Power ,φ RIS } and hyperparameters, φ RB The initial parameter corresponding to the resource block allocation, φ Power The initial parameters corresponding to the power allocation, φ RIS Corresponding to the initial parameters configured by RIS, the hyperparameters include the inner loop learning rate, the outer loop learning rate, and the task batch size; S2.2, Task definition and decomposition: According to the objective function of maximizing Jain's fairness index, the resource allocation problem is decomposed into three subtasks: resource block RB allocation target, power allocation target and RIS configuration optimization target; The resource block allocation goal is to optimize the resource block allocation strategy. Maximize the transmission rate to meet the link capacity constraints and spectrum resource constraints; the power allocation goal is: optimize the power allocation B(t) = {B i (t)}, maximize the total system throughput under the condition of satisfying power constraints and signal-to-interference-and-noise ratio (SINR) constraints; RIS configuration optimization goal: optimize the RIS reflection coefficient Θ(t), improve the link signal-to-interference-and-noise ratio SINR, and satisfy the RIS configuration constraints; S2.3, inner loop update: For each subtask of resource block RB allocation target, power allocation target and RIS configuration optimization target, the following task-level optimization operations are performed: S2.3.1, Randomly sample a small batch of data for a task; S2.3.

2. Based on the objective function and constraints of the current task, the specific update of the task parameters is performed by gradient descent as follows: in, For the task τ i The loss function of S2.4, outer loop update: Summarize the gradient information of all tasks, calculate the meta-gradient φ and update the meta-parameters using the following calculation formula: in, It is the task loss after the inner loop update; S2.5, Iterate until convergence: Repeat S2.2 to S2.4 for iterative calculation until any of the following convergence conditions is met: The total fairness index, i.e., Jain's fairness index J(t), reaches a preset threshold, or the value of the objective function maximizing Jain's fairness index tends to be stable; S2.

6. Output optimization results: After the training is completed, the optimal strategy for each subtask is obtained, that is, the optimal resource block allocation strategy B * (t), optimal power allocation strategy P * (t), and the optimal RIS configuration Θ * (t).

Images