Access control and resource allocation method for space-air-ground integrated network
By adopting the combination method of multi-objective deep reinforcement learning and greedy algorithms in the integrated air-space and earth network, optimizing access control and channel allocation, the problem of resource allocation and switching overhead in the existing technology is solved, and more efficient resource utilization is achieved and user experience quality.
Patent Information
- Application Number
- CN202411034154.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-07-30
AI Technical Summary
The prior art lacks effective access control and resource allocation methods in the integrated air-space and earth network, and cannot simultaneously optimize the user's service rate quality, service delay quality and system switching overhead, and ignores the demand differences under different services.
A method based on multi-objective deep reinforcement learning is proposed. By transforming access control and channel allocation problems into multi-objective Markov decision-making process, multi-objective deep Q learning algorithm is adopted, combined with greedy algorithms, and the total switching overhead of the system is optimized. The total service rate quality of users and the total service delay quality realize dynamic allocation and switching decisions of resources.
It improves the efficiency of communication resource utilization of the aerospace and earth integrated network, meets different business needs, reduces system overhead, and achieves a more efficient user experience quality.
Smart Images

Figure CN119031385A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of communication technology, and in particular to an air-space-ground integrated network access control and resource allocation method based on multi-objective deep reinforcement learning. Background Art
[0002] As an integration of satellite systems, air networks and ground communications, the space-ground integrated network has become an emerging network architecture in the past few years and has attracted extensive research interest in academia. Compared with traditional ground networks or satellite networks, the dynamic changes in the topology of the space-ground integrated network will lead to frequent triggering of switching, which greatly increases the signaling overhead and makes it difficult to obtain the best performance. Therefore, it is of great significance to perform effective access control in the space-ground integrated network.
[0003] With the increasing development and popularity of mobile devices, many emerging multimedia services have been spawned, such as voice recognition and autonomous driving. Therefore, the integrated air-ground-air network not only needs to provide users with global seamless communication coverage, but also needs to provide users with various service support. In real scenarios, various actual services and applications have different requirements in terms of throughput, latency, etc. The integrated air-ground-air network needs to efficiently allocate resources according to user needs, thereby improving the user's service experience quality.
[0004] In the space-ground integrated network architecture, reference [1] introduced high-altitude platforms to cover hot spots. In this context, in order to maximize throughput, a dynamic multi-beam joint resource optimization method based on low-orbit satellite motion prediction was proposed. The Lagrangian dual method and Karush-Kuhn-Tucker (KKT) condition were used to find the optimal solution. Reference [2] proposed a dynamic channel reservation strategy for a multi-service low-orbit satellite communication system based on a deep Q network, and expressed the dynamic channel reservation problem of multiple services as a reinforcement learning task. However, the channel allocation scheme of this scheme only includes the minimum bandwidth and the maximum bandwidth, and it is not possible to flexibly allocate channels. Reference [3] proposed a switching method using the adaptive learning rate DQN framework (DQN-ALrM), which uses the DQN algorithm to solve the optimization target, and uses the ALrM method to intelligently adjust the learning rate to optimize the convergence speed, switching rate, call failure rate and multi-index service quality (QoS) performance. Reference [4] points out that dynamic resource allocation is a key technology to improve network performance in resource-limited satellite systems. It proposes a framework based on deep reinforcement learning and uses a system environment-based image tensor reconstruction method to extract the spatiotemporal characteristics of traffic. The effectiveness of the algorithm in time-varying scenarios is verified. Reference [5] proposes a user-driven deep reinforcement learning scheme, in which a centralized agent deployed at the backhaul end is responsible for training the parameters of the deep Q network, so that each user can make its own access decision independently according to the trained parameters to improve the long-term throughput of the system and avoid frequent switching between non-ground nodes. However, the above-mentioned references only consider the switching problem or channel allocation problem under the integrated space-ground network architecture, and lack research that considers access control and resource allocation at the same time. At the same time, existing research ignores the different needs of different services and lacks research on switching and channel allocation under multi-service classification.
[0005] Different from the existing work, (1) the present invention considers a scenario under an integrated air-ground network, where various services generated by users can be transmitted and processed under the satellite beam and drone covering the user. The downlink of the satellite beam and the drone is divided equally, and different bandwidths can be allocated according to the user's service needs during the transmission process. (2) The present invention simultaneously considers the needs of different services, the user's service rate quality, service delay quality and the system's switching overhead, and optimizes the three objectives for a long time to improve the user's service experience and reduce system overhead. (3) The present invention proposes two algorithms to solve the proposed system and optimization objectives. The first is a solution based on a greedy algorithm, which obtains the total optimization objective function by weighted summation of the optimization objectives, and then adopts the local optimal strategy under the current state when each user makes access control and channel allocation decisions, so as to approximately obtain the global optimal solution; the second is a solution based on multi-objective deep reinforcement learning, which first converts the access control and channel allocation problems into a multi-objective Markov decision process, and then proposes a multi-objective deep Q learning algorithm through a deep reinforcement learning algorithm to solve the proposed optimization problem.
[0006] [1].Y.Li, N.Deng and W.Zhou, "A Hierarchical Approach to ResourceAllocation in Extensible Multi-Layer LEO-MSS," in IEEEAccess, vol.8, pp.18522-18537, 2020.
[0007] [2].Z.Li,Z.Xie and
[0008] [3].J.Yang, Z.Xiao, H.Cui, J.Zhao, G.Jiang andZ.Han, "DQN-ALrM-BasedIntelligent Handover Method for Satellite-Ground Integrated Network," in IEEETransactions on Cognitive Communications and Networking, vol.9, no.4, pp.977-990, Aug.2023.
[0009] [4].
[0010] [5].Y.Cao, S.-Y.Lien and Y.-C.Liang, "Deep Reinforcement Learning ForMulti-User Access Control in Non-Terrestrial Networks," in IEEE Transactionson Communications, vol.69, no.3, pp.1605-1619, March 2021. Summary of the invention
[0011] The problem to be solved by the present invention is the access control and resource allocation method in the air-ground integrated network. In the method proposed by the present invention, according to the received signal strength of the user and the service status information of the user, the three objectives of the total switching overhead of the system, the total service rate quality of the user and the total service delay quality are optimized, and the optimization strategy of access control and channel allocation is given, so as to maximize the communication resource utilization efficiency of the air-ground integrated network.
[0012] To solve the above problems, the present invention adopts the following technical solutions:
[0013] Step 1: Air-space-ground integrated network system model based on low-orbit satellites and drones;
[0014] The space-ground integrated network system is composed of low-orbit satellites, hovering drones and ground mobile users. It communicates through frequency division multiplexing, and models the switching overhead, service rate quality and service delay quality, and introduces three objective functions respectively. Finally, the user's access object and the channel resources allocated to the user are taken as the optimization objects, and the model is modeled as a multi-objective optimization problem.
[0015] Step 2: Framework of decision-making algorithm;
[0016] First, before each time slot begins, users can independently obtain the channel quality between satellites and drones. The access status and channel allocation status of satellites and drones need to be uniformly issued by the dispatch center. Then, users make access decisions and channel allocation decisions based on the information obtained. Finally, users send the decision content to satellites or drones to achieve reallocation of communication resources.
[0017] Step 3: Multi-objective optimization strategy for access control and channel allocation of the air-ground integrated network system;
[0018] For the proposed multi-objective optimization problem, two algorithms are proposed to solve it.
[0019] The first algorithm is a greedy algorithm solution. By weighting and summing the optimization objectives, the total optimization objective function can be obtained. In each access control and channel allocation decision, the local optimal choice under the current state is adopted. This algorithm has low complexity and can still achieve better performance when the system computing power is insufficient.
[0020] The second algorithm is a solution based on a multi-objective deep reinforcement learning algorithm. This algorithm requires a large amount of computing resources for training. When the system has sufficient computing power, this algorithm can be used to obtain the global optimal solution. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 The system model includes a schematic diagram of the structure of multi-beam satellites, UAVs, and users;
[0022] Figure 2 Schematic diagram of the principle of multi-objective deep reinforcement learning. DETAILED DESCRIPTION
[0023] The present invention is further described below.
[0024] Step 1: Air-space-ground integrated network system model based on low-orbit satellites and drones
[0025] Figure 1The system model of the present invention is shown, considering an air-ground integrated network system consisting of a multi-beam LEO satellite, multiple hovering drones and a large number of mobile user terminals. The network architecture consists of three layers: space layer, air layer and ground layer.
[0026] Space layer: The space layer provides extensive network coverage for mobile users on the ground through S2G links in the form of LEO satellites. The height of satellite ο is denoted as a constant h o , then the position of satellite ο can be expressed as (x o (t),y o (t),h o ).
[0027] Aerial layer: The aerial layer contains a group of drones, Each drone Network operators can flexibly deploy drones to hotspots with high service demand, and provide wireless transmission services to mobile users through A2G links. Since drones are closer to mobile users, the service quality of mobile users is significantly improved. The position of drone i is represented by (x i (t),y i (t),h i (t)), whose service range is denoted as R i .
[0028] Ground layer: The ground layer consists of a large number of mobile users. The location of the mobile user is represented by (x m (t),y m (t), 0). Then, the distances between user m and satellite ο and drone i can be expressed as
[0029]
[0030] In this network scenario, the problem that needs to be solved is that when a user has multiple available networks during the switching process, he needs to make a correct switching decision to access the most appropriate network.
[0031] Step 1.1 Transfer model
[0032] The integrated air-ground-space network system completes the coverage of ground areas by satellites and drones through frequency reuse, assuming that a transmission channel can be reused by different downlinks in different time slots, while multiple users must use different channels to access the same satellite or the same drone in the same time slot.
[0033] During the transmission process, the total bandwidth of the downlink of beam k and UAV i is denoted as B k and Bi , and the downlink bandwidth is divided into 10 sub-bandwidths. The downlink sub-bandwidth resources allocated by beam k to user m at time t are The downlink sub-bandwidth resource allocated by drone i to user m at time t is share, and
[0034] Since LEO satellites move very fast relative to the ground, the channel conditions between users and satellites change rapidly. Therefore, considering that the downlink from satellite to user is a time-varying dynamic channel, without considering other transmission losses such as rain attenuation, the channel condition between users and satellites is changed rapidly. To represent the channel gain between user m and satellite ο at time t:
[0035]
[0036] Among them, G S is the satellite transmitting antenna gain, G M is the antenna gain received by the user, represents the free space loss between the user and the satellite, which can be further expressed as:
[0037]
[0038] According to Shannon's formula, the downlink data rate of user m accessing beam k at time t can be expressed as:
[0039]
[0040] in, represents the power allocated to user m by satellite ο at time t, σ 2 is Gaussian white noise.
[0041] For drones, use To represent the channel gain between user m and drone i at time t:
[0042]
[0043] Among them, G U is the antenna gain of the UAV transmission, represents the free space loss between the user and the drone, which can be further expressed as:
[0044]
[0045] According to Shannon's formula, the downlink data rate of user m accessing drone i at time t can be expressed as:
[0046]
[0047] in, represents the power allocated by UAV i to user m at time t.
[0048] Step 1.2 Switching Overhead
[0049] Obviously, a user may be covered by both satellites and drones at time t, but can only access one network at most, while the same satellite or drone can be connected to multiple users at the same time.
[0050] Introduction To represent the access status of user m at time t: When , it means that user m accesses beam k; when , it means that user m has accessed drone i. Indicates that user m has access to the satellite, using Indicates that user m has accessed the drone. Finally, Indicates which satellite beam user m accesses, using Indicates which drone user m has connected to.
[0051] use represents the switching overhead between beams within the LEO satellite, and is expressed as represents the switching overhead between LEO satellite and UAV, and represents the switching cost between drones. Then the switching cost of user m at time t can be expressed as
[0052]
[0053] Step 1.3 Service Quality of Experience (QoE)
[0054] Consider a set of service requests , each user's service request It can be represented by a set as s = {r min ,r max ,d min ,d max ,p}, where r min To meet the minimum data rate requirement of service s, r max is the data rate when service s achieves the best user experience, d max is the maximum tolerable delay of service s, d min is the delay when service s achieves the best user experience, and p is the data packet size of the service content.
[0055] According to the downlink data rate of user m at time t , when it receives service s, the downlink transmission time of user m can be expressed as
[0056]
[0057] Thus, the service rate quality of user m at time t can be obtained and quality of service delay They are
[0058]
[0059] Step 1.4 Problem Modeling
[0060] In order to make full use of the channel resources of satellites and drones to provide more diverse services for more users, the proposed dynamic switching algorithm aims to maximize the quality of service rate and service delay and minimize the total switching overhead. By satisfying the channel resource constraints, the channel resource allocation is completed dynamically and the correct switching decision is made. Therefore, the corresponding dynamic switching and channel resource optimization problem model is established as follows:
[0061]
[0062] Constraints 1 and 2: Each user can only select one satellite beam or drone to access;
[0063] Constraint 3: The sum of the channel resources allocated to each user cannot exceed the total bandwidth of the beam;
[0064] Constraint 4: The sum of the channel resources allocated to each user cannot exceed the total bandwidth of the drone.
[0065] Step 3: Multi-objective optimization strategy for access control and channel allocation of the air-ground integrated network system;
[0066] Since the optimization goal is an optimization and mixed integer programming problem that considers multiple objectives at the same time, this problem is NP-hard. Due to the uncertainty brought by the dynamic changes of existing users, the random arrival of services and the randomness of the wireless communication environment, it is necessary to find an effective optimization strategy from the perspective of the long-term benefits of the system. Therefore, the present invention proposes two decision algorithms, which can be selected according to the computing power of the decision-making end.
[0067] Step 3.1: Optimization strategy based on greedy algorithm
[0068] In order to balance multiple conflicting optimization objectives and obtain the access control and channel allocation optimization strategy of the air-ground integrated network system, an optimization strategy based on the greedy algorithm is proposed. The fundamental idea of the greedy algorithm is to gradually obtain the local optimal solution and approximate the global optimal solution through the local optimal solution. It is a simple strategy for solving optimization problems.
[0069] In order to balance all optimization objectives, assume Ω = [ω 1 ,ω 2 ,ω 3 ] is the weight vector of all optimization objectives, and the total optimization objective function of user m at time t is expressed as
[0070]
[0071] Therefore, the system's different preferences for various optimization objectives can be achieved by simply changing the weight vector Ω.
[0072] In this solution, for the user set All users in the , according to the order of the user queue, select one user at a time to make a decision. All nodes that the user may access are recorded as ID∈{SAT 1 ,…,SAT k ,UAV 1 ,…,UAV i}, record the number of channels that all users may be assigned as ω∈{0,1,2,…,10}, and then calculate all possible total objective function values of user m at time t Finally, according to the greedy strategy, the access node and the number of channel allocations when the total optimization objective function value is the largest are selected, that is,
[0073]
[0074] Next, lock the channel resources allocated to user m so that subsequent users cannot choose the channel allocated to user m. Repeat the above operation until all users have made their decisions, and you can get the decision set of all users. .
[0075] The greedy algorithm has low computational complexity and requires few computing resources, and can quickly obtain suboptimal decisions when the network has low computing resources.
[0076] Step 3.2: Solution based on multi-objective deep reinforcement learning algorithm
[0077] In order to solve the problem that the optimization strategy based on the greedy algorithm cannot effectively obtain the global optimal solution, this scheme transforms the optimization problem into a multi-objective Markov decision process (MOMDP), and proposes a multi-objective deep Q learning algorithm to solve it. Figure 2 shown.
[0078] Step 3.2.1 Multi-objective Markov decision process;
[0079] MOMDP with triples To represent, where S, A, are the state space, action space, and reward vector, respectively.
[0080] (1) State space S: Let s m (t)∈S represents the state of user m at time t, which includes the following:
[0081] 1) User m corresponds to the channel quality of each beam and drone at the current time t, that is, .in, is the signal-to-noise ratio between the satellite or UAV and user m at time t.
[0082] 2) The channel resource allocation status of each beam and UAV at the last moment t-1, that is,
[0083]
[0084] Therefore, in s m There are (k+i)×2 elements in (t), that is,
[0085]
[0086] (2) Action space A: User m dynamically adjusts the satellite beam or drone it accesses at each moment, that is, selects one from the k+i satellite beams and drones as a target, and then allocates channel resources to all beams and drones. Therefore, let a m (t)∈A represents the action of user m at time t, which is defined as:
[0087]
[0088] (3) Rewards :set up represents the reward obtained by user m at time t. The three dimensions of the reward vector correspond to the three objectives of the optimization problem, namely in The reward functions representing service rate quality, service delay quality and switching overhead respectively;
[0089] 1) Service rate quality: In order to maximize the transmission rate of all users, the service rate quality reward of user m at time t is defined as:
[0090]
[0091] 2) Service delay quality: In order to minimize the communication delay of all users, the service delay quality reward of user m at time t is defined as:
[0092]
[0093] 3) Switching penalty: In order to minimize the frequent switching caused by improving user experience, the switching cost penalty of user m at time t is defined as:
[0094]
[0095] If the user accesses the same beam or drone as the previous moment, the switching cost at that moment is 0; if the user switches between different beams within the satellite, the switching cost at that moment is ; If the user switches between the satellite and the drone, the switching cost at that moment is ; If the user switches between drones, the switching cost at that moment is .
[0096] Step 3.2.2 Multi-objective deep Q-learning algorithm;
[0097] The multi-objective deep Q learning algorithm is an evolution of the single-objective deep Q learning algorithm, which is used to optimize multiple conflicting objectives. In this algorithm, MODQN builds a DQN network for each optimization objective. When the agent interacts with the environment, the environment gives the agent a reward vector corresponding to each objective, that is, Since the goals in the optimization problem are conflicting, MODQN needs to find a strategy that maximizes the total benefits of these optimization goals. In the present invention, for each optimization goal, an action value function Q(s t ,a t ) to indicate that in state s t Next action a t Therefore, for all the action values of the optimization objective, we can use an action value vector To unify the representation, so as to perform weighted summation of the action values under all optimization objectives according to the weight vector Ω of all optimization objectives, that is,
[0098]
[0099] The agent then greedily chooses the action value function Q(s t ,a t ) The action that reaches the maximum value obtains the current state s t Optimal access control and channel allocation decisions.
[0100]
[0101] In this algorithm, the Q value of each optimization target is estimated by the evaluation network and the target network, and the parameters of the network are represented as θ and θ respectively. -The evaluation network is used to select the action with the largest action value function, and the target network is used to calculate the value function of the selected action after the action is selected. Among them, the parameters θ of the evaluation network are updated after each decision step, and the parameters θ of the target network are updated after each decision step. - Updated regularly at a specific time. The iterative formula for evaluating the parameters θ of the network is as follows:
[0102]
[0103] The loss function of the network is calculated as mean square error and is defined as follows:
[0104]
[0105] Among them, y t is the time difference target, calculated as follows:
[0106]
Claims
1. A method for access control and resource allocation of an air-ground integrated network, characterized in that: According to the user's received signal strength and the user's service status information, the three objectives of the system's total switching overhead, the user's total service rate quality and the total service delay quality are optimized, and the optimization strategy of access control and channel allocation is given to maximize the communication resource utilization efficiency of the air-ground integrated network; the specific steps are as follows: Step 1: Air-space-ground integrated network system model based on low-orbit satellites and drones; The space-ground integrated network system is composed of low-orbit satellites, hovering drones and ground mobile users. It communicates through frequency division multiplexing, and models the switching overhead, service rate quality and service delay quality. Three objective functions are introduced respectively. Finally, the user's access object and the channel resources allocated to the user are taken as the optimization objects, and the model is modeled as a multi-objective optimization problem. Step 2: Framework of decision-making algorithm; First, before each time slot begins, users autonomously obtain the channel quality between the satellite and the drone. The access status and channel allocation status of the satellite and drone need to be uniformly issued by the dispatch center. Then, the user makes access decisions and channel allocation decisions based on the information obtained; finally, the user sends the decision content to the satellite or drone to achieve the reallocation of communication resources; Step 3: Multi-objective optimization strategy for access control and channel allocation of the air-ground integrated network system; For the proposed multi-objective optimization problem, two algorithms are proposed to solve it; The first algorithm is a greedy algorithm solution, which obtains the total optimization objective function by weighted summation of optimization objectives. In each access control and channel allocation decision, the local optimal choice under the current state is adopted. The algorithm has low complexity and can still achieve better performance when the system computing power is insufficient. The second algorithm is a solution based on a multi-objective deep reinforcement learning algorithm. This algorithm requires a lot of computing resources for training. When the system has sufficient computing power, this algorithm can obtain the global optimal solution.
2. The access control and resource allocation method of an air-ground integrated network according to claim 1, characterized in that: In step 1, consider an air-ground integrated network system consisting of one multi-beam LEO satellite, multiple hovering drones, and a large number of mobile user terminals, which consists of three layers: space layer, air layer, and ground layer; Space layer: The space layer provides extensive network coverage for mobile users on the ground through S2G links in the form of LEO satellites. The height of satellite ο is recorded as a constant h o , then the position of satellite ο is expressed as (x o (t),y o (t),h o ); Aerial layer: The aerial layer contains a group of drones, Each drone The network operator flexibly deploys it to the hotspots of service demand and provides wireless transmission services to mobile users through A2G links. The position of drone i is represented by (x i (t),y i (t),h i (t)), whose service range is denoted as R i ; Ground layer: The ground layer consists of a large number of mobile users. The location of the mobile user is recorded as (x m (t),y m (t),0); the distances between user m and satellite ο and drone i are respectively expressed as 3. The access control and resource allocation method of an air-ground integrated network according to claim 2, characterized in that: Step 1 includes: step 1.1 transmission model; The integrated air-ground network system completes the coverage of the ground area by satellites and drones through frequency reuse, assuming that a transmission channel can be reused by different downlinks in different time slots, while multiple users must use different channels to access the same satellite or the same drone in the same time slot; During the transmission process, the total bandwidth of the downlink of beam k and UAV i is denoted as B k and B i , and the downlink bandwidth is divided into 10 sub-bandwidths. The downlink sub-bandwidth resources allocated by beam k to user m at time t are The downlink sub-bandwidth resource allocated by drone i to user m at time t is share, and Since LEO satellites move very fast relative to the ground, the channel conditions between users and satellites change rapidly. Considering that the downlink from satellite to user is a time-varying dynamic channel, without considering the rain attenuation transmission loss, the channel condition between users and satellites changes rapidly. To represent the channel gain between user m and satellite ο at time t: Among them, G S is the satellite transmitting antenna gain, G M is the antenna gain received by the user, represents the free space loss between the user and the satellite, expressed as: According to Shannon's formula, the downlink data rate of user m accessing beam k at time t is expressed as: in, represents the power allocated to user m by satellite ο at time t, σ 2 is Gaussian white noise; For drones, use To represent the channel gain between user m and drone i at time t: Among them, G U is the antenna gain of the UAV transmission, represents the free space loss between the user and the drone, which is further expressed as: According to Shannon's formula, the downlink data rate of user m accessing drone i at time t is expressed as: in, represents the power allocated to user m by drone i at time t; Step 1.2 Switching overhead; Obviously, a user may be covered by both satellites and drones at time t, but can only access one network at most, while the same satellite or drone is connected to multiple users at the same time; To represent the access status of user m at time t: When , it means that user m accesses beam k; when , it means that user m has accessed drone i; at the same time, Indicates that user m has access to the satellite, using Indicates that user m has access to the drone; finally, use Indicates which satellite beam user m accesses, using Indicates which drone user m has connected to; use represents the switching overhead between beams within the LEO satellite, and is represents the switching overhead between LEO satellite and UAV, and represents the switching cost between drones; then the switching cost of user m at time t is expressed as Step 1.3: Quality of experience (QoE) Consider a set of service requests Service requests per user It can be represented by a set as s = {r min ,r max ,d min ,d max ,p}, where rmin is the minimum data rate requirement for service s, r max is the data rate when service s achieves the best user experience, d max is the maximum tolerable delay of service s, d min is the delay when service s achieves the best user experience, and p is the data packet size of the service content; According to the downlink data rate of user m at time t When it receives service s, the downlink transmission time of user m is expressed as Thus, the service rate quality of user m at time t is obtained and quality of service delay They are Step 1.4 Problem modeling; The corresponding dynamic switching and channel resource optimization problem model is established as follows: Constraints 1 and 2: Each user can only select one satellite beam or drone to access; Constraint 3: The sum of the channel resources allocated to each user cannot exceed the total bandwidth of the beam; Constraint 4: The sum of the channel resources allocated to each user cannot exceed the total bandwidth of the drone.
4. The access control and resource allocation method of an air-ground integrated network according to claim 1, characterized in that: The optimization strategy based on the greedy algorithm in step 3 is described as follows: In order to balance all optimization objectives, assuming that Ω = [ω1, ω2, ω3] is the weight vector of all optimization objectives, the total optimization objective function of user m at time t is expressed as Only by changing the weight vector Ω, the system can achieve different preferences for various optimization objectives; For user collections According to the order of the user queue, one user is selected at a time for decision making; all nodes that the user may access are recorded as ID∈{SAT1,…,SAT k ,UAV1,…,UAV i }, record the number of channels that all users may be assigned as ω∈{0,1,2,…,10}, and then calculate all possible total objective function values of user m at time t Finally, according to the greedy strategy, the access nodes and channel allocation number when the total optimization objective function value is the largest are selected, that is, Lock the channel resources allocated to user m so that subsequent users cannot choose the channel allocated to user m; repeat the above operation until all users have made their decisions, and you can get the decision set of all users. The greedy algorithm has low computational complexity, requires few computing resources, and can quickly obtain suboptimal decisions when the network has low computing resources.
5. The access control and resource allocation method of an air-ground integrated network according to claim 1, characterized in that: The multi-objective deep reinforcement learning algorithm in step 3 is described as follows: In order to solve the problem that the optimization strategy based on the greedy algorithm cannot effectively obtain the global optimal solution, the optimization problem is transformed into a multi-objective Markov decision process MOMDP, and a multi-objective deep Q learning algorithm is proposed to solve it; Step 3.2.1 Multi-objective Markov decision process; MOMDP with triples To represent, where S, A, They are state space, action space and reward vector respectively; (1) State space S: Let s m (t)∈S represents the state of user m at time t, where Includes the following: 1) User m corresponds to the channel quality of each beam and drone at the current time t, that is, in, is the signal-to-noise ratio between the satellite or UAV and user m at time t; 2) The channel resource allocation status of each beam and UAV at the last moment t-1, that is, In s m There are (k+i)×2 elements in (t), that is, (2) Action space A: User m dynamically adjusts the satellite beam or UAV to which it accesses at each moment, that is, selects one from the k+i satellite beams and UAVs as a target, and then allocates the channel resources of all beams and UAVs; let a m (t)∈A represents the action of user m at time t, which is defined as: (3) Rewards set up represents the reward obtained by user m at time t. The three dimensions of the reward vector correspond to the three objectives of the optimization problem, namely in The reward functions representing service rate quality, service delay quality and switching overhead respectively; 1) Service rate quality: To improve the transmission rate of all users, the service rate quality reward of user m at time t is defined as: 2) Service delay quality: To reduce the communication delay of all users, the service delay quality reward of user m at time t is defined as: 3) Switching penalty: To reduce frequent switching due to improving user experience, the switching cost penalty of user m at time t is defined as: If the user accesses the same beam or drone as the previous moment, the switching cost at that moment is 0; if the user switches between different beams within the satellite, the switching cost at that moment is If the user switches between the satellite and the drone, the switching cost at that moment is If the user switches between drones, the switching cost at that moment is 6. The access control and resource allocation method of an air-ground integrated network according to claim 1, characterized in that: The multi-objective deep Q learning algorithm in step 3 is described as follows: MODQN constructs a DQN network for each optimization target. When the agent interacts with the environment, the environment gives the agent a reward vector corresponding to each target, that is, For each optimization objective, an action-value function Q(s t ,a t ) represents the expected discounted reward of action at in state st; For all action values of the optimization objective, use an action value vector To unify the representation, so as to perform weighted summation of the action values under all optimization objectives according to the weight vector Ω of all optimization objectives, that is, The agent then greedily chooses the action value function Q(s t ,a t ) The action that reaches the maximum value obtains the current state s t Optimal access control and channel allocation decisions; The Q value of each optimization target is estimated by the evaluation network and the target network, and the parameters of the network are denoted as θ and θ respectively. - The evaluation network is used to select the action with the largest action value function, and the target network is used to calculate the value function of the selected action after the action is selected; the parameters θ of the evaluation network are updated after each decision step, and the parameters θ of the target network are updated after each decision step. - Updated regularly at specific times; the iterative formula for evaluating the network parameters θ is as follows: The loss function of the network is calculated as mean square error and is defined as follows: L(θ t )=E[(y t -Q(s t ,a t ;θ t )) 2 ] Among them, y t is the time difference target, calculated as follows:
Citation Information
Patent Citations
Network resource optimization method and device for space-air-ground integrated network
CN117674958A
Method and device for accessing Internet of Things terminal of space-ground convergence network
CN117750402A
Satellite-ground interconnection scheduling method based on improved analytic hierarchy process
CN118337265A
Cited By
Unmanned aerial vehicle trajectory optimization and task offloading method facing air-space-ground integrated network
CN119907048A
Space-time ripple effect suppression method of sky-ground integrated network
CN119997034A
Air-space-ground integrated network architecture based on cellular-free network and network switching method
CN120416964A
Space-air-ground integrated network architecture based on non-cellular network and network switching method
CN120416964B
Intelligent vehicle-oriented space-ground integrated network channel dynamic optimization method
CN120614617A
