Star-ground fusion network spectrum sharing method based on branch parameterized deep Q network

Abstract

The application designs a star-ground fusion network spectrum sharing method based on a branch parameterized deep Q network, taking a target LEO satellite and its service coverage area in a preset resource scheduling period as a basic resource scheduling unit; each basic resource scheduling unit includes a target LEO satellite, a ground base station located in the service coverage area of the target LEO satellite, and a frequency domain continuous physical resource block (PRB) shared by the target LEO satellite and the ground base station; the target LEO satellite and the ground base station perform joint PRB selection and power allocation in a discrete time slot; the downlink transmission rate calculation of the satellite network and the ground network is included, a Stackelberg game target is constructed, and the observation, action and reward functions of the leader and follower agents are constructed; the leader agent and the follower agent are constructed; the double time scale mechanism is used to train the agent, and the trained leader agent and follower agent are used to perform online spectrum sharing resource allocation.

Description

Technical Field

[0001] This invention belongs to the field of resource allocation technology in wireless communication, and specifically relates to a joint resource block scheduling and power allocation method suitable for satellite-ground converged networks. Background Technology

[0002] With the development of 6th Generation (6G) wireless communication systems, Satellite-Terrestrial Integrated Networks (STINs) are considered an important network architecture for achieving seamless global coverage. Low Earth Orbit (LEO) satellite constellations can effectively extend the coverage of communication networks. However, with the large-scale deployment of LEO satellites and the continuous growth in wireless service demand, spectrum resources are becoming increasingly scarce. To improve spectrum utilization, satellite and terrestrial networks typically operate collaboratively in the same frequency band through spectrum sharing, resulting in severe co-channel interference between satellites and ground stations. This makes resource allocation in a spectrum-sharing environment a critical issue affecting system performance. Currently, existing research on spectrum sharing resource allocation in STINs mainly employs centralized optimization methods, game theory-based methods, and Deep Reinforcement Learning (DRL) methods for solving the problem.

[0003] One type of resource allocation method combines centralized optimization and game theory. Centralized optimization methods achieve better system performance by collecting global Channel State Information (CSI) for unified resource scheduling. However, this method requires frequent exchange of global CSI, resulting in significant signaling overhead and computational complexity. To reduce reliance on global information, some studies model the resource allocation process as a Stackelberg game, achieving resource coordination through policy interactions between leaders and followers. However, existing methods typically rely on iterative optimization solvers, making real-time resource scheduling difficult to achieve in dynamic satellite-to-ground environments.

[0004] Another type of resource allocation method is the DRL-based intelligent decision-making method. This type of method can autonomously learn resource allocation strategies through interaction with the environment, exhibiting good environmental adaptability. However, the joint physical resource block (PRB) scheduling and power allocation problem in satellite-ground fusion networks involves both discrete PRB selection and continuous power control, essentially belonging to a hybrid action optimization problem. For this type of problem, parameterized deep Q-networks (P-DQN) can jointly handle discrete and continuous actions, thus becoming an effective solution method. However, as the number of PRBs increases, the number of discrete action combinations grows exponentially, leading to an explosion in the action space dimension, making existing P-DQN methods difficult to directly apply to large-scale STIN scenarios. Furthermore, existing DRL-based Stackelberg game solution methods typically employ a single-timescale training mechanism, with leaders and followers updating their strategies synchronously, which easily leads to instability in the training process and difficulty in converging to a stable Stackelberg equilibrium.

[0005] To address the aforementioned problems, this invention proposes a spectrum sharing resource allocation method for satellite-terrestrial fusion networks based on Branching Parameterized Deep Q-Network (BPDQN). This method effectively solves the resource optimization problem caused by joint PRB scheduling and power allocation in spectrum-sharing environments. While reducing resource scheduling decision overhead, it improves resource allocation efficiency and system performance, and is suitable for large-scale dynamic satellite-terrestrial fusion network scenarios. Summary of the Invention

[0006] Technical Problem: This invention proposes a spectrum sharing method for satellite-terrestrial fusion networks based on branched parameterized deep Q-networks. This method utilizes Stackelberg game theory to establish a master-slave decision-making mechanism between the satellite and terrestrial networks, transforming the centralized resource allocation problem into a distributed resource allocation problem. By combining a branched action structure with a parameterized deep Q-network to construct the BPDQN method, it achieves efficient solution for large-scale mixed action spaces. Simultaneously, a dual-timescale training mechanism is employed, enabling leaders and followers to learn strategies according to the hierarchical decision-making order of the Stackelberg game, thereby improving the convergence stability and resource allocation performance of the method.

[0007] Technical Solution: The present invention provides a spectrum sharing method for a satellite-ground fusion network based on a branch-parameterized deep Q-network, comprising: using a target LEO satellite and its service coverage area within a preset resource scheduling period as a basic resource scheduling unit; each basic resource scheduling unit includes a target LEO satellite and satellites located within the service coverage area of ​​the target LEO satellite. A ground base station and the target LEO satellite and Shared by terrestrial base stations A frequency-domain contiguous physical resource block (PRB); the target LEO satellite and ground base station in Perform joint PRB selection and power allocation within discrete time slots; let the set Represents the set of resource scheduling time slots; let the set Let the set of ground base stations be represented; let the set be represented by the set of ground base stations. Represents a shared PRB set; where This represents the total number of resource scheduling slots. This represents the total number of ground base stations and the total number of available PRBs in the system. In any time slot Within the satellite network, the resource allocation strategy is composed of both the PRB selection vector and the power allocation vector; let This represents the PRB selection vector for the satellite. This represents the satellite's power allocation vector; where, elements in Indicates the satellite in Occupancy status on each PRB; Indicates satellite occupancy of PRB ; This indicates that the satellite is not occupying the PRB. ; Indicates the satellite in The power transmitted on each PRB, with a value range of [value missing]. , Indicates the satellite in The maximum transmit power on each PRB depends on the actual design parameters of the satellite; when hour, Since each basic resource scheduling unit corresponds to one target LEO satellite, the PRB selection matrix and power allocation matrix of the satellite network are denoted as follows: , ; For any ground base station ,make Indicates the first PRB selection vectors for each ground base station; let Indicates the first The power allocation vector for each ground base station; where... elements in Indicates ground base station In the Occupancy status on each PRB; Indicates ground base station PRB usage ; Indicates ground base station PRB not occupied ; Indicates ground base station In the The power transmitted on each PRB, with a value range of [value missing]. , Indicates ground base station In the The maximum transmit power on each PRB depends on the actual design parameters of the ground base station; when hour, The ground base station's resources are scheduled by a ground network control node. Let the ground network's PRB selection matrix and power allocation matrix be respectively... , .

[0008] Specifically, the steps include the following: Step 1: Calculation of downlink transmission rates implemented by satellite and terrestrial networks. This step includes: 1.1 Calculation of achievable downlink transmission rate in satellite networks 1.2 Calculation of downlink transmission rate achievable by terrestrial network The second step involves constructing a Stackelberg game objective. The satellite network, acting as the leader, first performs PRB selection and power allocation. The terrestrial network, acting as the follower, observes the satellite network's resource usage and interference status. Its control nodes then perform PRB selection and power allocation for each terrestrial base station. The satellite network updates its own resource configuration based on the terrestrial network's response, thus forming a distributed spectrum sharing process under the Stackelberg game. 2.1 Constructing the Leader Satellite Network Utility Function 2.2 Constructing the follower ground network utility function, Step 3: Construct the observation, action, and reward functions for the leader and follower agents; this step includes the following process: 3.1 Building a Leader Satellite Network Intelligent Agent in Time Slots Observations , 3.2 Constructing Follower Ground Network Intelligent Agents in Time Slots Observations , 3.3, Constructing Leader and Follower Intelligent Agents in Time Slots The action, 3.4 Constructing reward functions for leader and follower agents, Step 4: Construct the leader agent and follower agents; this step includes: 4.1 Building the Leader Intelligent Agent 4.2 Constructing the follower agent, Step 5: Train the agent using a dual-timescale mechanism. This step includes: 5.1 Initialize training configuration and network weights. 5.2, Implementing outer-layer decisions of the leader's intelligence system. 5.3, Perform follower smart body inner response update. 5.4 Determine the follower agent's response. 5.5, Perform leader intelligence outer layer weight update. 5.6 Repeat the training until the required number of training rounds is reached. 5.7 Output the trained leader agent and follower agents , Step 6: Utilize the trained leader agent and follower agents Perform online spectrum sharing resource allocation.

[0009] in, The downlink transmission rate calculation implemented by the satellite network in step 1.1 is specifically as follows: 1.1a, in time slot The satellite in Downlink transmission rate on each PRB It is given by the following formula: (1), in, This represents the bandwidth of a single PRB, and its specific value depends on the subcarrier spacing and the number of subcarriers contained in a single PRB. This represents the noise power on a single PRB, obtained from channel estimation. Indicates in time slot Satellite to satellite user terminal In the Channel power gain on each PRB; Indicates in time slot Ground base station To satellite user terminal In the Channel power gain on each PRB and All were obtained through channel estimation; 1.1b, in time slot Downlink transmission rate achievable by satellite networks We obtain it from the following formula: (2), Step 1.2, the calculation of the downlink transmission rate achievable by the terrestrial network, is specifically as follows: 1.2a, in time slot Ground base station In the Downlink transmission rate on each PRB We obtain it from the following formula: (3), in, Indicates in time slot Ground base station To its service ground user terminal In the Channel power gain on each PRB; Indicates in time slot Satellite to ground base station Ground user terminals for services In the Channel power gain on each PRB; Indicates in time slot Ground base station to ground base station Ground user terminals for services In the Channel power gain on each PRB , and All were obtained through channel estimation; 1.2b, in time slot Downlink transmission rate achieved by terrestrial networks We obtain it from the following formula: (4).

[0010] Step 2.1, constructing the leader satellite network utility function, specifically involves: make Indicates that the leader is The average utility of resource scheduling within a discrete time slot is calculated using the following formula: (5), in, This indicates taking the expected function. This represents the satellite's unit data transmission revenue coefficient. This represents the cost per unit power of a satellite. This represents the cost coefficient per unit of PRB occupied by the satellite. This indicates the fixed operating cost of the satellite; , , and The specific value is determined by the network operator. Step 2.2, constructing the follower ground network utility function, specifically involves: make Indicates followers The average utility of resource scheduling within a discrete time slot is calculated using the following formula: (6), in, This represents the average speed requirement for terrestrial networks. The weights represent the degree to which terrestrial network speed requirements are met. This represents the unit power cost coefficient of terrestrial networks. This represents the cost coefficient per unit of PRB occupied by the terrestrial network; , and The specific value can be determined by the operator.

[0011] Step 3.1 involves constructing the leader satellite network intelligent agent in time slots. Observations : (7), in, This represents the large-scale fading power gain matrix from satellite to satellite user terminal; This represents the large-scale fading power gain matrix from satellite to ground user terminal; This represents the PRB selection matrix for a time slot on a satellite network; This represents the power allocation matrix for a time slot on a satellite network. Step 3.2 involves constructing the follower ground network agent in time slots. Observations : (8), in, This represents the channel power gain matrix from the ground base station to the ground user terminal; This represents the interference matrix between satellites and ground user terminals. This represents the PRB selection matrix for the ground network in the previous time slot; This represents the power allocation matrix of the ground network in the previous time slot; Step 3.3 involves constructing leader and follower agents in time slots. The action, The actions of the leader agent are recorded as follows: The actions of the follower agent are denoted as: ; In step 3.4, the reward functions for the leader and follower agents are constructed. make The reward function for the leader agent consists of satellite network data service profits, system spectral efficiency, and penalties for unmet rate demands from the terrestrial network. (9), in, Indicates the profit weighting of satellite network data services; Indicates the system spectral efficiency weight; This indicates the penalty weight for the proportion of terrestrial networks failing to meet demand; , and The specific value is determined by the operator; BW is the total system bandwidth.

[0012] make The reward function for the follower agent is composed of the corresponding follower utility function: (10).

[0013] Step 4.1 involves constructing the leader intelligent agent. , 4.1a, Constructing the observation feature vector of the leader agent ; in each time slot The leader agent constructed in step 3.1 of the third step will be used for observation. By network weight The leader shares the observation feature extraction layer, mapping it to the observation feature vector of the leader agent. , serving as the observation input for each branch of the leader agent; 4.1b, construct parameter networks and Q networks for each PRB branch of the leader agent; treat each PRB as a branch. For the leader agent, which includes one satellite, it is necessary to... Each PRB is selected and power is allocated, thus constituting The number of parallel branches; in the... In each leader agent branch, the observed feature vector of the leader agent is input. The weight is parameter network Output the continuous power allocation variable of the current branch. Subsequently, the observed feature vector of the leader agent will be... and the continuous power allocation variable of the current branch output The weight input to the current branch is In a Q-network, evaluate the discrete occupancy indicator variable of the current branch. Action value ; thereby obtaining the leader intelligent agent The optimal action pair for each branch ,in , All branches execute the above operations in parallel to obtain the optimal action pairs for each branch of the leader agent. The result is the selection and power allocation of the shared PRB by the current time slot leader agent based on the input observations; Step 4.2 involves constructing the follower agent. , 4.2a, Construct the observation feature vector of the follower agent ; in each time slot The follower agent constructed in step 3.2 of the third step will be observed. By network weight The followers share the observation feature extraction layer, which maps it to the observation feature vector of the follower agent. , serving as the observation input for each branch of the follower agent; 4.2b, construct parameter networks and Q networks for each PRB branch of the follower agent; treat each PRB as a branch, for the follower agent, including... Each ground base station needs to... Each PRB is selected and power is allocated, thus constituting The number of parallel branches; in the... In each follower agent branch, the observed feature vector of the follower agent is input. The weight is parameter network Output the continuous power allocation variable of the current branch. Subsequently, the observed feature vectors of the follower agents are... and the continuous power allocation variable of the current branch output The weight input to the current branch is In a Q-network, evaluate the discrete occupancy indicator variable of the current branch. Action value ; thereby obtaining the follower agent's first The optimal action pair for each branch ,in , All branches execute the above operations in parallel to obtain the optimal action pairs for each branch of the follower agent. The result is the selection and power allocation of the shared PRB by the current time slot follower agent based on the input observations.

[0014] Step 5.1 involves initializing the training configuration and network weights; make This represents the total number of training rounds, the specific value of which depends on the agent's convergence requirements and available training resources; let... This represents the number of inner-layer updates in the follower agent, and its specific value depends on the degree of policy updates and computational resources of the follower agent in each training round; let and These represent the experience cache of the leader agent and the experience cache of the follower agent, respectively. make and These represent the learning rates of the leader agent and the follower agent, respectively. The specific values ​​depend on the parameter update magnitude and training convergence stability requirements of the respective agents. make and These represent the discount factors for the leader agent and the follower agent, respectively, with specific values ​​depending on the degree to which the respective agents weigh current and future rewards. make and These represent the soft update coefficients of the target network weights for the leader agent and the follower agent, respectively. The specific values ​​depend on the tracking speed and training stability requirements of the corresponding target network for the current network parameters. make and These represent the number of samples used by the leader agent and the follower agent for each update, respectively. The specific values ​​depend on the number of available samples in the corresponding experience buffer, the stability of parameter updates, and computing resources. For the leader agent, let This indicates that they share the network weights of the observation feature extraction layer. Indicate its The parameters of each branch are the network weights. Indicate its The Q-value network weights of each branch; for the follower agent, they include... One branch, making , and These represent the shared observation feature extraction layer network weights, the parametric network weights of each branch, and the Q-network weights of each branch, respectively. Step 5.2 involves executing the outermost layer decision of the leader's intelligent system. In the Training intervals The satellite network acquires the channel information of the current time slot and the resource allocation results of the previous time slot, and constructs the leader agent's observation according to step 3.1 in the third step. Treat it as the leader intelligent agent Input, leader intelligent agent Output Action This action is performed by the follower agent. The input conditions for the response; Step 5.3 involves updating the internal response of the follower's smart body. In a given leader action Afterwards, the follower agents implement Second inner layer update; in the current time slot For the first Second inner layer update The set of values ​​is Perform the following operations; 5.3a. Based on the channel power gain from each ground base station to the ground user terminal, the interference caused by the target LEO satellite to the ground user terminal, and the resource allocation results of the last inner-layer update on the ground network, the ground network constructs the observation of the follower agent according to step 3.2 in step 3. Treat it as a follower intelligent agent The input, the follower agent Output Action ;in, Indicates the first Ground network PRB selection matrix during the next inner layer update Indicates the first Ground network power allocation matrix during the next inner layer update; 5.3b, follower agents receive rewards. and the next observation and the sample Store in the follower agent's experience cache ; 5.3c, from Extraction A set of samples constitutes a mini-batch training sample for the follower agent; for each sample, the target Q-value is calculated. The target Q-value employs a dual-depth Q-network, which includes online network and target network structure calculations to mitigate Q-value overestimation. Its form is as follows: (11), in, , and These represent the weights of the target network in the shared observation feature extraction layer and the weights of the first layer, respectively. Q-target network weights for each branch; 5.3d, Unfreezing Follower Agent Shared Observation Feature Extraction Layer Online Network Weights and the online network weights of each branch Q Freeze follower agent parameters and online network weights Update by minimizing the time difference error and The Q-value loss function for the follower agent is: (12) in, Indicates the experience cache Calculate the expected value of the sample drawn from the sample; 5.3e, Freezing the online network weights of the follower agent's shared observation feature extraction layer. and the online network weights of each branch Q Unfreeze follower agent parameters online network weights Update by minimizing the parameter network loss function The loss function for the follower agent parameter network is: (13) in, This indicates that the frozen follower agents share the network weights of the observation feature extraction layer. The frozen follower agent is the first The Q-network weights of each branch are frozen during the parameter network update phase. and This prevents the exploration gradient of the power parameter from backpropagating to the shared observation feature extraction layer, thereby reducing gradient interference. 5.3f, performs soft updates to the target network weights of the follower agent: (14) (15) (16) in, , and These represent the target network weights corresponding to the shared observation feature extraction layer, the Q-network of each branch, and the parameter network of each branch of the follower agent, respectively; the weights of the target network are updated using the same soft update coefficient. ; Step 5.4 involves determining the follower agent's response. Finish After the last inner update, the follower action obtained from the last inner update is used as the follower response. ,in Indicates followers in time slots Actions of the leader The response results.

[0015] Step 5.5 involves updating the outer layer weights of the leader's intelligent system. 5.5a, the leader agent responds based on the responses of the follower agents. Receive rewards and the next observation and empirical samples Store in the leader agent's experience cache ; 5.5b, from Extraction A small batch of samples is used to form the leader agent's training samples, and the leader agent's objective Q-value is calculated. : (17) in, , and These represent the weights of the target network in the shared observation feature extraction layer for the leader agent and the weights of the first layer, respectively. Q-target network weights for each branch; 5.5c, Unfreezing the leader agent's shared observation feature extraction layer online network weights and the online network weights of each branch Q Freeze the leader agent parameters and online network weights Update by minimizing the Q-value loss of the leader agent. and : (18) in, Indicates the experience cache Calculate the expected value of the sample drawn from the sample; 5.5d, freezing the online network weights of the leader agent's shared observation feature extraction layer. and the online network weights of each branch Q Unfreeze leader agent parameters online network weights Update by minimizing the parameter network loss function The network loss function for the leader agent parameters is: (19) in, This represents the shared observation feature extraction layer network weights of the frozen leader agent. The leader agent after being frozen is the first... Q-network weights of each branch; Version 5.5e performs soft updates to the network weights of each objective agent of the leader agent: (20) (twenty one), (twenty two), in, , and These represent the target network weights corresponding to the shared observation feature extraction layer, the Q-network of each branch, and the parameter network of each branch of the leader agent, respectively; the weights of the above target networks are updated using the same soft update coefficient. ; 5.6 Repeat the training until the required number of training rounds is reached. Repeat steps (5.2)-(5.5) for each time slot in each round of training until completion. Round training; because the follower agent executes under each leader agent's action. The leader agent performs an outer update after the follower agent completes the inner update, while the follower agent performs an outer update after the follower agent completes the inner update. Therefore, the follower agent is in a fast timescale update and the leader agent is in a slow timescale update. This dual timescale training mechanism allows the follower agent to first approximate the optimal response to the leader agent's actions, and then the leader agent updates its strategy based on the response. This conforms to the master-follower decision-making process of Stackelberg game and is conducive to obtaining a stable Stackelberg equilibrium result. 5.7 Output the trained leader agent and follower agents This includes the shared observation feature extraction layer for each agent, the post-training weights of each branch parameter network and each branch Q network, which are used for subsequent online spectrum sharing resource allocation.

[0016] The sixth step is specifically as follows: Step 6.1: Execute the leader agent's online resource allocation decision. 6.1a, in online resource scheduling slots The target LEO satellite acquires the channel information of the current time slot and the resource allocation results of the previous time slot, and constructs the observation of the leader agent according to step 3.1 in step 3. ; 6.1b, will observe Input the trained leader agent The leader agent first utilizes a shared observation feature extraction layer to extract observations. The mapping is done to an observation feature vector; subsequently, each PRB branch outputs the discrete occupancy indicator variable and continuous power allocation variable of the corresponding PRB based on the observation feature vector, and the results are combined to obtain the leader action. ; 6.1c, Actions of the Leader Feasibility mapping is performed, mapping each PRB occupancy indicator variable to 0 or 1, and limiting each power allocation variable to the maximum transmit power range of the corresponding PRB; when the occupancy indicator variable of a certain PRB is 0, the transmit power corresponding to that PRB is set to 0; based on the mapped branch actions, the target LEO satellite in the time slot is constructed respectively. PRB selection matrix and power allocation matrix ; 6.2 Execute online response decisions for the follower agent. 6.2a After the target LEO satellite completes its leader decision, each ground base station will aggregate the spectrum sensing results and channel information to the ground network control node. Based on the channel power gain from each ground base station to the ground user terminal, the interference caused by the target LEO satellite to the ground user terminal, and the resource allocation results of the previous time slot on the ground network, the ground network control node will construct the follower agent's observation according to step 3.2 in step 3. ; 6.2b, will observe Input the trained follower agent The follower agent first utilizes a shared observation feature extraction layer to extract observations. The data is mapped to an observation feature vector. Subsequently, each "Ground Base Station-PRB" branch outputs the discrete occupancy indicator variable and continuous power allocation variable of the corresponding ground base station on the corresponding PRB based on the observation feature vector, and combines them to obtain the follower action. ; 6.2c, actions for followers Feasibility mapping is performed, mapping each PRB occupancy indicator variable to 0 or 1, and limiting each power allocation variable to the maximum transmit power range of the corresponding ground base station; when a ground base station does not select the corresponding PRB, the transmit power of that ground base station on that PRB is set to 0. Based on the mapped branch actions, the ground network is constructed in the time slots. PRB selection matrix and power allocation matrix ; 6.3, Perform satellite-to-ground network resource configuration. 6.3a, The feasibility-mapped leader actions are sent to the target LEO satellite, and the feasibility-mapped follower actions are sent to all ground base stations; the target LEO satellite and all ground base stations, according to the corresponding PRB selection results and power allocation results, in the time slots... Execute downlink data transmission; 6.3b, After the current time slot transmission is completed, construct a new time slot based on the updated environment. Leader observation and follower observation ; 6.3c, In each subsequent online resource scheduling time slot, steps 6.1-6.3 are repeated, so that the leader agent makes the satellite network resource allocation decision first, and then the follower agent makes the ground network resource allocation response given the leader's action, thereby realizing dynamic PRB scheduling and power allocation according to the master-slave decision order of the Stackelberg game. 6.4 Output the online spectrum sharing resource allocation results. 6.4a, Output the PRB selection matrix and power allocation matrix of the target LEO satellite in each resource scheduling time slot, as well as the PRB selection matrix and power allocation matrix of each ground base station; 6.4b, Based on the transmission rate calculation method described in the first step, calculate the downlink transmission rate that the satellite network and the terrestrial network can achieve after adopting the resource configuration results, and obtain the system spectrum efficiency, satellite network data service profit and the degree to which the terrestrial network rate demand is met. 6.4c, During online resource scheduling, the leader agent and follower agents By using the trained network weights for forward inference, it is not necessary to re-execute iterative optimization or network training in each time slot, thereby reducing the online computational complexity and decision latency of spectrum sharing resource allocation.

[0017] Beneficial Effects: The BPDQN-based satellite-terrestrial fusion network spectrum sharing resource allocation method proposed in this invention has the following advantages: It models the spectrum sharing process between the satellite and terrestrial networks as a Stackelberg game, reducing the global channel state information interaction overhead required for centralized resource allocation through distributed decision-making by the satellite network leader and the terrestrial network followers; simultaneously, it jointly optimizes PRB selection and power allocation, and utilizes BPDQN to decompose the mixed action space corresponding to multiple PRBs into parallel branches, reducing the solution complexity of high-dimensional resource allocation problems. Furthermore, this invention employs an asymmetric network weight update and dual-timescale training mechanism, which can reduce gradient interference between discrete PRB selection learning and continuous power parameter learning, enabling the terrestrial network to form responses at a fast timescale and the satellite network to update its strategy at a slow timescale, thereby improving training stability and resource allocation convergence. Through the above design, this invention can achieve coordinated spectrum sharing between satellite and terrestrial networks while balancing satellite service profits, the proportion of terrestrial service demand satisfaction, system spectrum efficiency, and resource consumption costs. Attached Figure Description

[0018] Figure 1 This is a diagram of the BPDQN methodology framework.

[0019] Figure 2 It is a flowchart for training intelligent agents using a dual timescale mechanism. Detailed Implementation

[0020] A target LEO satellite and its service coverage area within a preset resource scheduling period are used as a basic resource scheduling unit. Each basic resource scheduling unit includes one target LEO satellite and the service coverage area of ​​that target LEO satellite. A ground base station and the target LEO satellite and Shared by terrestrial base stations A frequency-domain continuous PRB. The target LEO satellite and ground base station are in Perform joint PRB selection and power allocation within discrete time slots. Let the set Represents the set of resource scheduling time slots; let the set Let the set of ground base stations be represented; let the set be represented by the set of ground base stations. This indicates a shared PRB set. This represents the total number of resource scheduling slots. This indicates the total number of ground base stations. This indicates the total number of PRBs available in the system. , and The specific value depends on the coverage area, service scale, and wireless resource configuration of the satellite-ground converged network.

[0021] In any time slot Within the network, the satellite network resource allocation strategy is composed of both the PRB selection vector and the power allocation vector. This represents the PRB selection vector for the satellite. This represents the satellite's power allocation vector. Wherein, elements in Indicates the satellite in Occupancy status on each PRB; Indicates satellite occupancy of PRB ; This indicates that the satellite is not occupying the PRB. ; Indicates the satellite in The power transmitted on each PRB, with a value range of [value missing]. , Indicates the satellite in The maximum transmit power per PRB (the specific value depends on the actual design parameters of the satellite); when hour, Since each basic resource scheduling unit corresponds to one target LEO satellite, the PRB selection matrix and power allocation matrix of the satellite network are denoted as follows: , .

[0022] Similarly, for any ground base station ,make Indicates the first PRB selection vectors for each ground base station; let Indicates the first The power allocation vector for each ground base station. elements in Indicates ground base station In the Occupancy status on each PRB; Indicates ground base station PRB usage ; Indicates ground base station PRB not occupied ; Indicates ground base station In the The power transmitted on each PRB, with a value range of [value missing]. , Indicates ground base station In the The maximum transmit power on each PRB (the specific value depends on the actual design parameters of the ground base station); when hour, The terrestrial base station's resource scheduling is handled by a terrestrial network control node. Let the terrestrial network's PRB selection matrix and power allocation matrix be respectively... , .

[0023] To illustrate the technical solution of this invention, this embodiment uses a basic resource scheduling unit as an example. The basic resource scheduling unit includes a target LEO satellite, four ground base stations located within the service coverage area of ​​the target LEO satellite, and 30 consecutive frequency domain PRBs shared by the target LEO satellite and the four ground base stations. The target LEO satellite and the ground base stations perform joint PRB selection and power allocation within 20 discrete time slots. Using a single target LEO satellite to constitute a basic resource scheduling unit is only used to illustrate the resource allocation process within a single resource scheduling unit and does not constitute a limitation on the number of satellites applicable to this invention.

[0024] Step 1: Calculation of achievable downlink transmission rates for satellite and terrestrial networks. This step includes the following steps: (1) Calculation of downlink transmission rate achievable by satellite network.

[0025] (a) in time slot The satellite in Downlink transmission rate on each PRB It is given by the following formula: (1), in, This represents the bandwidth of a single PRB (the specific value depends on the subcarrier spacing and the number of subcarriers contained in a single PRB). The noise power on a single PRB can be estimated from the channel. Indicates in time slot Satellite to satellite user terminal In the Channel power gain on each PRB; Indicates in time slot Ground base station To satellite user terminal In the Channel power gain on each PRB and All of these were obtained through channel estimation.

[0026] (b) in time slots Downlink transmission rate achievable by satellite networks We obtain it from the following formula: (2), (2) Calculation of downlink transmission rate achievable by terrestrial network.

[0027] (a) in time slot Ground base station In the Downlink transmission rate on each PRB We obtain it from the following formula: (3), in, Indicates in time slot Ground base station To its service ground user terminal In the Channel power gain on each PRB; Indicates in time slot Satellite to ground base station Ground user terminals for services In the Channel power gain on each PRB; Indicates in time slot Ground base station to ground base station Ground user terminals for services In the Channel power gain on each PRB , and All of these were obtained through channel estimation.

[0028] (b) in time slots Downlink transmission rate achievable by terrestrial networks We obtain it from the following formula:

[0029] (4).

[0030] Step 2: Construct the Stackelberg game objective. The satellite network, as the leader, first performs PRB selection and power allocation; the terrestrial network, as the follower, after observing the satellite network's resource usage and interference status, has its control nodes perform PRB selection and power allocation for each terrestrial base station. The satellite network updates its own resource configuration based on the terrestrial network's response, thus forming a distributed spectrum sharing process under the Stackelberg game. This step includes the following process: (1) Construct the leader (satellite network) utility function.

[0031] make Indicates that the leader is The average utility of resource scheduling within a discrete time slot is calculated using the following formula: (5), in, This indicates taking the expected function. This represents the satellite's unit data transmission revenue coefficient. This represents the cost per unit power of a satellite. This represents the cost coefficient per unit of PRB occupied by the satellite. This indicates the fixed operating cost of the satellite. , , and The specific value can be determined by the network operator.

[0032] (2) Construct the follower (ground network) utility function.

[0033] make Indicates followers The average utility of resource scheduling within a discrete time slot is calculated using the following formula: (6), in, This represents the average speed requirement of the terrestrial network (the specific value depends on the system design). The weights represent the degree to which terrestrial network speed requirements are met. This represents the unit power cost coefficient of terrestrial networks. This represents the cost coefficient per unit of PRB occupied by the terrestrial network. , and The specific value can be determined by the operator.

[0034] Step 3: Construct the observation, action, and reward functions for the leader and follower agents. This step includes the following process: (1) Constructing leader (satellite network) agents in time slots Observations : (7), in, This represents the large-scale fading power gain matrix from satellite to satellite user terminal; This represents the large-scale fading power gain matrix from satellite to ground user terminal; This represents the PRB selection matrix for a time slot on a satellite network; This represents the power allocation matrix for a time slot on a satellite network.

[0035] (2) Constructing follower (ground network) agents in time slots Observations : (8), in, This represents the channel power gain matrix from the ground base station to the ground user terminal; This represents the interference matrix between satellites and ground user terminals. This represents the PRB selection matrix for the ground network in the previous time slot; This represents the power allocation matrix of the ground network in the previous time slot.

[0036] (3) Construct leader and follower agents in time slots The action.

[0037] The actions of the leader agent are recorded as follows: The actions of the follower agent are denoted as: .

[0038] (4) Construct reward functions for leader and follower agents.

[0039] make The reward function for the leader agent consists of satellite network data service profits, system spectral efficiency, and penalties for unmet rate demands from the terrestrial network. (9), in, Indicates the profit weighting of satellite network data services; Indicates the system spectral efficiency weight; This indicates the penalty weight for the proportion of ground network failure to meet demand. , and The specific value can be determined by the operator.

[0040] make The reward function for the follower agent is composed of the corresponding follower utility function: (10). The steps described above are explained in detail below: In this example, the satellite orbital altitude is set to 600 km, the equivalent isotropic radiated power density of the satellite is 28 dBW / MHz, the total transmit power of each ground base station is 46 dBm, and the noise power spectral density is set to -174 dBm / Hz. The gains of the ground base station and satellite antennas are set to 18 dBi and 24 dBi, respectively, while the antenna gains of both the satellite user terminal and the ground user terminal are set to 0 dBi. The Rice factor is set to 15 dB, and the data throughput that the ground network needs to guarantee is... Set to 80 Mbps. Satellite network unit data transmission revenue coefficient. Set as Satellite unit power cost coefficient Set to 0.1, satellite unit PRB occupancy cost coefficient Set at 0.05, satellite fixed operating cost Set to 1, satellite network data service profit weight. Set to 1, system spectral efficiency weight The penalty weight is set to 0.5 for the proportion of ground network failure to meet demand. Set to 100; terrestrial network requirements meet proportional weights. Set to 20, the unit power cost coefficient for terrestrial networks. Set to 0.01, the cost coefficient per unit PRB for terrestrial network occupancy. Set to 0.05.

[0041] Step 4: Building the Leader Intelligent Agent and follower agents The BPDQN methodology framework is shown in the attached diagram in the specification. Figure 1 As shown, this step includes the following process: (1) Constructing a leader intelligent agent , (a) Constructing the observation feature vector of the leader agent In each time slot The leader agent constructed in step (1) of the third step will be observed. By network weight The leader shares the observation feature extraction layer, mapping it to the observation feature vector of the leader agent. , serving as the observation input for each branch of the leader agent.

[0042] (b) Construct parameter networks and Q networks for each PRB branch of the leader agent. Treat each PRB as a branch; for the leader agent, which contains one satellite, it is necessary to... Each PRB is selected and power is allocated, thus constituting The first parallel branch. In each leader agent branch, the observed feature vector of the leader agent is input. The weight is parameter network Output the continuous power allocation variable of the current branch. Subsequently, the observed feature vector of the leader agent is... and the continuous power allocation variable of the current branch output The weight input to the current branch is In a Q-network, evaluate the discrete occupancy indicator variable of the current branch. Action value This is how the leader agent is obtained. The optimal action pair for each branch ,in , All branches execute the above operations in parallel to obtain the optimal action pairs for each branch of the leader agent. The result is the selection and power allocation of the shared PRB by the current time slot leader agent based on the input observations.

[0043] (2) Constructing follower agents , (a) Constructing the observation feature vector of the follower agent In each time slot The follower agent constructed in step (2) of the third step will observe By network weight The followers share the observation feature extraction layer, which maps it to the observation feature vector of the follower agent. , serving as the observation input for each branch of the follower agent.

[0044] (b) Construct parameter networks and Q-networks for each PRB branch of the follower agent. Treat each PRB as a branch; for the follower agent, it includes... Each ground base station needs to... Each PRB is selected and power is allocated, thus constituting The first parallel branch. In each follower agent branch, the observed feature vector of the follower agent is input. The weight is parameter network Output the continuous power allocation variable of the current branch. Subsequently, the observed feature vectors of the follower agents are... and the continuous power allocation variable of the current branch output The weight input to the current branch is In a Q-network, evaluate the discrete occupancy indicator variable of the current branch. Action value This is how the follower agent is obtained. The optimal action pair for each branch ,in , All branches execute the above operations in parallel to obtain the optimal action pairs for each branch of the follower agent. The result is the selection and power allocation of the shared PRB by the current time slot follower agent based on the input observations.

[0045] Step 5: Train the agent using a dual-timescale mechanism. See the attached diagram in the instruction manual for the flowchart. Figure 2 As shown. This step includes the following process: (1) Initialize the training configuration and network weights.

[0046] make Let represent the total number of training rounds (the specific value depends on the agent's convergence requirements and available training resources); This represents the number of inner-layer update steps of the follower agent (the specific value depends on the degree of policy update and computational resources of the follower agent in each round of training); let and These represent the experience cache of the leader agent and the experience cache of the follower agent, respectively.

[0047] make and Let these represent the learning rates of the leader agent and the follower agent, respectively (the specific values ​​depend on the parameter update magnitude and training convergence stability requirements of the respective agents); and Let represent the discount factors for the leader agent and the follower agent, respectively (the specific values ​​depend on the respective agents' trade-offs between current and future rewards); let and Let represent the soft update coefficients of the target network weights for the leader agent and follower agent, respectively (the specific values ​​depend on the target network's tracking speed and training stability requirements for the current network parameters); let and These represent the number of samples used by the leader agent and the follower agent for each update (the specific value depends on the number of available samples in the corresponding experience buffer, the stability of parameter updates, and computing resources).

[0048] For the leader agent, let This indicates that they share the network weights of the observation feature extraction layer. Indicate its The parameters of each branch are the network weights. Indicate its The Q-value network weights of each branch. Similarly, for the follower agent, the network weights include... One branch, making , and These represent the shared observation feature extraction layer network weights, the parameter network weights of each branch, and the Q-network weights of each branch, respectively.

[0049] (2) Implement the leader’s intelligent outer layer decision.

[0050] In the Training intervals The satellite network acquires the channel information of the current time slot and the resource allocation results of the previous time slot, and constructs the observation of the leader agent according to step (1) in the third step. Treat it as the leader intelligent agent Input, leader intelligent agent Output Action This action is performed by the follower agent. The input conditions for the response.

[0051] (3) Execute follower smart body internal response update.

[0052] In a given leader action Afterwards, the follower agents implement Second inner layer update. In the current time slot. For the first Second inner layer update The set of values ​​is Perform the following operations.

[0053] (a) Based on the channel power gain from each ground base station to the ground user terminal, the interference caused by the target LEO satellite to the ground user terminal, and the resource configuration results of the last inner layer update on the ground network, the ground network constructs the observation of the follower agent according to step (2) in the third step. Treat it as a follower intelligent agent The input, the follower agent Output Action .in, Indicates the first Ground network PRB selection matrix during the next inner layer update Indicates the first Ground network power allocation matrix during the next inner layer update.

[0054] (b) Follower agents receive rewards and the next observation and the sample Store in the follower agent's experience cache .

[0055] (c) From Extraction A set of samples constitutes a mini-batch training sample for the follower agent. For each sample, the target Q-value is calculated. The target Q-value is calculated using a dual-depth Q-network structure (including an online network and a target network) to mitigate Q-value overestimation. Its form is as follows: (11), in, , and These represent the weights of the target network in the shared observation feature extraction layer and the weights of the first layer, respectively. Q-target network weights for each branch.

[0056] (d) Unfreezing the online network weights of the follower agent's shared observation feature extraction layer and the online network weights of each branch Q Freeze follower agent parameters and online network weights Update by minimizing the time difference error and The Q-value loss function for the follower agent is: (12) in, Indicates the experience cache Calculate the expected value of the sample drawn from the sample.

[0057] (e) Freeze the online network weights of the shared observation feature extraction layer for follower agents. and the online network weights of each branch Q Unfreeze follower agent parameters online network weights Update by minimizing the parameter network loss function The loss function for the follower agent parameter network is: (13) in, This indicates that the frozen follower agents share the network weights of the observation feature extraction layer. The frozen follower agent is the first The Q-network weights of each branch are frozen during the parameter network update phase. and This prevents the exploration gradient of the power parameter from backpropagating to the shared observation feature extraction layer, thereby reducing gradient interference.

[0058] (f) Perform soft updates to the target network weights of the follower agent: (14) (15) (16) in, , and These represent the target network weights corresponding to the shared observation feature extraction layer, the Q-network of each branch, and the parameter network of each branch of the follower agent, respectively. The target network weights are updated using the same soft update coefficient. .

[0059] (4) Determine the follower agent's response.

[0060] Finish After the last inner update, the follower action obtained from the last inner update will be used as the follower response: .in Indicates followers in time slots Actions of the leader The response results.

[0061] (5) Execute leader intelligence outer layer weight update.

[0062] (a) The leader agent responds based on the follower agent's response Receive rewards and the next observation and empirical samples Store in the leader agent's experience cache .

[0063] (b) From Extraction A set of samples constitutes a mini-batch training sample for the leader agent. The objective Q-value of the leader agent is calculated. : (17) in, , and These represent the weights of the target network in the shared observation feature extraction layer for the leader agent and the weights of the first layer, respectively. Q-target network weights for each branch.

[0064] (c) Unfreezing the leader agent's shared observation feature extraction layer online network weights and the online network weights of each branch Q Freeze the leader agent parameters and online network weights Update by minimizing the Q-value loss of the leader agent. and : (18) in, Indicates the experience cache Calculate the expected value of the sample drawn from the sample.

[0065] (d) Freeze the online network weights of the leader agent's shared observation feature extraction layer. and the online network weights of each branch Q Unfreeze leader agent parameters online network weights Update by minimizing the parameter network loss function The network loss function for the leader agent parameters is: (19)

[0066] in, This represents the shared observation feature extraction layer network weights of the frozen leader agent. The leader agent after being frozen is the first... Q-network weights for each branch.

[0067] (e) Perform soft updates to the network weights of each objective of the leader agent: (20) (twenty one), (twenty two), in, , and These represent the target network weights corresponding to the shared observation feature extraction layer, the Q-network of each branch, and the parameter network of each branch of the leader agent, respectively. The same soft update coefficient is used for updating the weights of the target network. .

[0068] (6) Repeat the training until the required number of training rounds is reached.

[0069] Repeat steps (2)-(5) for each time slot in each round of training until completion. Round training. Because the follower agent performs actions under each leader agent's action. The leader agent performs an outer update after the follower agent completes its inner-layer response. Therefore, the follower agent updates on a fast timescale, while the leader agent updates on a slow timescale. This dual-timescale training mechanism allows the follower agent to first approximate the optimal response to the leader agent's actions, and then the leader agent updates its policy based on that response. This aligns with the master-slave decision-making process in Stackelberg games and is conducive to obtaining stable Stackelberg equilibrium results.

[0070] (7) Output the trained leader agent and follower agents This includes the shared observation feature extraction layer for each agent, the post-training weights of each branch parameter network and each branch Q network, which are used for subsequent online spectrum sharing resource allocation.

[0071] Step 6: Utilize the trained leader agent and follower agents Perform online spectrum sharing resource allocation. This step includes the following procedures: (1) Execute the leader agent's online resource allocation decision.

[0072] (a) During online resource scheduling slots The target LEO satellite acquires the channel information of the current time slot and the resource allocation results of the previous time slot, and constructs the leader agent's observation according to step (1) in the third step. .

[0073] (b) Observation Input the trained leader agent The leader agent first utilizes a shared observation feature extraction layer to extract observations. The mapping is done to an observation feature vector; subsequently, each PRB branch outputs the discrete occupancy indicator variable and continuous power allocation variable of the corresponding PRB based on the observation feature vector, and the results are combined to obtain the leader action. .

[0074] (c) Actions of the leader Perform feasibility mapping. Map each PRB occupancy indicator variable to 0 or 1, and limit each power allocation variable to the maximum transmit power range of the corresponding PRB; when the occupancy indicator variable of a certain PRB is 0, set the transmit power of the corresponding PRB to 0. Based on the mapped branch actions, construct the target LEO satellite in the time slot. PRB selection matrix and power allocation matrix .

[0075] (2) Execute the online response decision of the follower agent.

[0076] (a) After the target LEO satellite completes its leader decision, each ground base station will aggregate the spectrum sensing results and channel information to the ground network control node. Based on the channel power gain from each ground base station to the ground user terminal, the interference caused by the target LEO satellite to the ground user terminal, and the resource allocation results of the previous time slot on the ground network, the ground network control node will construct the observation of the follower agent according to step (2) in step three. .

[0077] (b) Observation Input the trained follower agent The follower agent first utilizes a shared observation feature extraction layer to extract observations. The data is mapped to an observation feature vector. Subsequently, each "Ground Base Station-PRB" branch outputs the discrete occupancy indicator variable and continuous power allocation variable of the corresponding ground base station on the corresponding PRB based on the observation feature vector, and combines them to obtain the follower action. .

[0078] (c) Actions to followers Perform feasibility mapping. Map each PRB occupancy indicator variable to 0 or 1, and limit each power allocation variable to the maximum transmit power range of the corresponding terrestrial base station; when a terrestrial base station does not select a corresponding PRB, set the transmit power of that terrestrial base station on that PRB to 0. Based on the mapped branch actions, construct the terrestrial network in time slots. PRB selection matrix and power allocation matrix .

[0079] (3) Perform satellite-to-ground network resource configuration.

[0080] (a) The feasibility-mapped leader actions are sent to the target LEO satellite, and the feasibility-mapped follower actions are sent to each ground station. The target LEO satellite and each ground station, according to the corresponding PRB selection results and power allocation results, send the actions in the time slots. Perform downlink data transmission.

[0081] (b) After the current time slot transmission is completed, a new time slot is constructed based on the updated environment. Leader observation and follower observation .

[0082] (c) In each subsequent online resource scheduling time slot, (1)-(3) are executed repeatedly, so that the leader agent makes a satellite network resource allocation decision first, and then the follower agent makes a ground network resource allocation response under the given leader action, thereby realizing dynamic PRB scheduling and power allocation in accordance with the master-slave decision order of the Stackelberg game.

[0083] (4) Output the online spectrum sharing resource allocation results.

[0084] (a) Output the PRB selection matrix and power allocation matrix of the target LEO satellite in each resource scheduling time slot, as well as the PRB selection matrix and power allocation matrix of each ground base station.

[0085] (b) Based on the transmission rate calculation method described in the first step, calculate the downlink transmission rate that the satellite network and the terrestrial network can achieve after adopting the resource configuration result, and obtain the system spectrum efficiency, satellite network data service profit and the degree to which the terrestrial network rate demand is met.

[0086] (c) During online resource scheduling, the leader agent and follower agents By using the trained network weights for forward inference, it is not necessary to re-execute iterative optimization or network training in each time slot, thereby reducing the online computational complexity and decision latency of spectrum sharing resource allocation.

[0087] The above is explained in detail below: Considering the balance between training time and convergence effect, the total number of training rounds... Set to 2000. In the dual-timescale framework: followers need to converge quickly to their optimal response given a leader's action, therefore the inner loop updates the number of steps. The learning rate is set to 5. The follower agent is configured for rapid local adaptation: a large learning rate is used. Set to 0.001, a smaller discount factor. Setting it to 0.5 provides a relatively large soft update coefficient. Set to 0.01, and a larger sampling size. The value is set to 256. Accordingly, the leader focuses on evaluating the long-term strategic impact given the best follower response, thus adopting a conservative update configuration: using a smaller learning rate. Set to 0.0001, a relatively large discount factor. Set to 0.9, a relatively small soft update coefficient. Set to 0.005, and a smaller sampling size. Set to 128.

[0088] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A spectrum sharing method for satellite-ground fusion networks based on branch-parameterized deep Q-networks, characterized in that, A target LEO satellite and its service coverage area within a preset resource scheduling period are used as a basic resource scheduling unit; each basic resource scheduling unit includes a target LEO satellite and the service coverage area of ​​that target LEO satellite. A ground base station and the target LEO satellite and Shared by terrestrial base stations A frequency-domain contiguous physical resource block (PRB); the target LEO satellite and ground base station in Perform joint PRB selection and power allocation within discrete time slots; let the set Represents the set of resource scheduling time slots; let the set Let the set of ground base stations be represented; let the set be represented by the set of ground base stations. Represents a shared PRB set; where This represents the total number of resource scheduling slots. This indicates the total number of ground base stations. This indicates the total number of PRBs available in the system; In any time slot Within the satellite network, the resource allocation strategy is composed of both the PRB selection vector and the power allocation vector; let This represents the PRB selection vector for the satellite. This represents the satellite's power allocation vector; where, elements in Indicates the satellite in Occupancy status on each PRB; Indicates satellite occupancy of PRB ; This indicates that the satellite is not occupying the PRB. ; Indicates the satellite in The power emitted on each PRB, with a value range of [value missing]. , Indicates the satellite in The maximum transmit power on each PRB depends on the actual design parameters of the satellite; when hour, Since each basic resource scheduling unit corresponds to one target LEO satellite, the PRB selection matrix and power allocation matrix of the satellite network are denoted as follows: , ; For any ground base station ,make Indicates the first PRB selection vectors for each ground base station; let Indicates the first The power allocation vector for each ground base station; where... elements in Indicates ground base station In the Occupancy status on each PRB; Indicates ground base station PRB usage ; Indicates ground base station PRB not occupied ; Indicates ground base station In the The power emitted on each PRB, with a value range of [value missing]. , Indicates ground base station In the The maximum transmit power on each PRB depends on the actual design parameters of the ground base station; when hour, The ground base station's resources are scheduled by a ground network control node. Let the ground network's PRB selection matrix and power allocation matrix be respectively... , .

2. The spectrum sharing method for satellite-ground fusion networks based on branch-parameterized deep Q-networks according to claim 1, characterized in that, Specifically, the steps include the following: Step 1: Calculation of downlink transmission rates implemented by satellite and terrestrial networks. This step includes: 1.1 Calculation of achievable downlink transmission rate in satellite networks 1.2 Calculation of downlink transmission rate achievable by terrestrial network The second step involves constructing a Stackelberg game objective. The satellite network, acting as the leader, first performs PRB selection and power allocation. The terrestrial network, acting as the follower, observes the satellite network's resource usage and interference status. Its control nodes then perform PRB selection and power allocation for each terrestrial base station. The satellite network updates its own resource configuration based on the terrestrial network's response, thus forming a distributed spectrum sharing process under the Stackelberg game. 2.1 Constructing the Leader Satellite Network Utility Function 2.2 Constructing the follower ground network utility function, Step 3: Construct the observation, action, and reward functions for the leader and follower agents; this step includes the following process: 3.1 Building a Leader Satellite Network Intelligent Agent in Time Slots Observations , 3.2 Constructing Follower Ground Network Intelligent Agents in Time Slots Observations , 3.3, Constructing Leader and Follower Intelligent Agents in Time Slots The action, 3.4 Constructing reward functions for leader and follower agents, Step 4: Construct the leader agent and follower agents; this step includes: 4.1 Building the Leader Intelligent Agent 4.2 Constructing the follower agent, Step 5: Train the agent using a dual-timescale mechanism. This step includes: 5.1 Initialize training configuration and network weights. 5.2, Implementing outer-layer decisions of the leader's intelligence system. 5.3, Perform follower smart body inner response update. 5.4 Determine the follower agent's response. 5.5, Perform leader intelligence outer layer weight update. 5.6 Repeat the training until the required number of training rounds is reached. 5.7 Output the trained leader agent and follower agents , Step 6: Utilize the trained leader agent and follower agents Perform online spectrum sharing resource allocation.

3. The spectrum sharing method for satellite-ground fusion networks based on branch-parameterized deep Q-networks according to claim 1, characterized in that, The downlink transmission rate calculation implemented by the satellite network in step 1.1 is specifically as follows: 1.1a, in time slot The satellite in Downlink transmission rate on each PRB It is given by the following formula: （1）， in, This represents the bandwidth of a single PRB, and its specific value depends on the subcarrier spacing and the number of subcarriers contained in a single PRB. This represents the noise power on a single PRB, obtained from channel estimation. Indicates in time slot Satellite to satellite user terminal In the Channel power gain on each PRB; Indicates in time slot Ground base station To satellite user terminal In the Channel power gain on each PRB and All were obtained through channel estimation; 1.1b, in time slot Downlink transmission rate achievable by satellite networks We obtain it from the following formula: （2）， Step 1.2, the calculation of the downlink transmission rate achievable by the terrestrial network, is specifically as follows: 1.2a, in time slot Ground base station In the Downlink transmission rate on each PRB We obtain it from the following formula: （3）， in, Indicates in time slot Ground base station To its ground user terminal In the Channel power gain on each PRB; Indicates in time slot Satellite to ground base station Ground user terminals for services In the Channel power gain on each PRB; Indicates in time slot Ground base station to ground base station Ground user terminals for services In the Channel power gain on each PRB , and All were obtained through channel estimation; 1.2b, in time slot Downlink transmission rate achieved by terrestrial networks We obtain it from the following formula: （4）。 4. The spectrum sharing method for satellite-ground fusion networks based on branch-parameterized deep Q-networks according to claim 1, characterized in that, Step 2.1, constructing the leader satellite network utility function, specifically involves: make Indicates that the leader is The average utility of resource scheduling within a discrete time slot is calculated using the following formula: （5）， in, This indicates taking the expected function. This represents the satellite's unit data transmission revenue coefficient. This represents the cost per unit power of a satellite. This represents the cost coefficient per unit of PRB occupied by the satellite. This indicates the fixed operating cost of the satellite; , , and The specific value is determined by the network operator. Step 2.2, constructing the follower ground network utility function, specifically involves: make Indicates followers The average utility of resource scheduling within a discrete time slot is calculated using the following formula: （6）， in, This represents the average speed requirement for terrestrial networks. The weights represent the degree to which terrestrial network speed requirements are met. This represents the unit power cost coefficient of terrestrial networks. This represents the cost coefficient per unit of PRB occupied by the terrestrial network; , and The specific value can be determined by the operator.

5. The spectrum sharing method for satellite-ground fusion networks based on branch-parameterized deep Q-networks according to claim 1, characterized in that, Step 3.1 involves constructing the leader satellite network intelligent agent in time slots. Observations : （7）， in, This represents the large-scale fading power gain matrix from satellite to satellite user terminal; This represents the large-scale fading power gain matrix from satellite to ground user terminal; This represents the PRB selection matrix for a time slot on a satellite network; This represents the power allocation matrix for a time slot on a satellite network. Step 3.2 involves constructing the follower ground network agent in time slots. Observations : （8）， in, This represents the channel power gain matrix from the ground base station to the ground user terminal; This represents the interference matrix between satellites and ground user terminals. This represents the PRB selection matrix for the ground network in the previous time slot; This represents the power allocation matrix of the ground network in the previous time slot; Step 3.3 involves constructing leader and follower agents in time slots. The action, The actions of the leader agent are recorded as follows: The actions of the follower agent are denoted as: ; In step 3.4, the reward functions for the leader and follower agents are constructed. make The reward function for the leader agent consists of satellite network data service profits, system spectral efficiency, and penalties for unmet rate demands from the terrestrial network. （9）， in, Indicates the profit weighting of satellite network data services; Indicates the system spectral efficiency weight; This indicates the penalty weight for the proportion of terrestrial networks failing to meet demand; , and The specific value is determined by the operator; BW is the total system bandwidth. make The reward function for the follower agent is composed of the corresponding follower utility function: （10）。 6. The spectrum sharing method for satellite-ground fusion networks based on branch-parameterized deep Q-networks according to claim 1, characterized in that, Step 4.1 involves constructing the leader intelligent agent. , 4.1a, Constructing the observation feature vector of the leader agent ; in each time slot The leader agent constructed in step 3.1 of the third step will be used for observation. By network weight The leader shares the observation feature extraction layer, mapping it to the observation feature vector of the leader agent. , serving as the observation input for each branch of the leader agent; 4.1b, construct parameter networks and Q networks for each PRB branch of the leader agent; treat each PRB as a branch. For the leader agent, which includes one satellite, it is necessary to... Each PRB is selected and power is allocated, thus constituting One parallel branch; In the In each leader agent branch, the observed feature vector of the leader agent is input. The weight is parameter network Output the continuous power allocation variable of the current branch. Subsequently, the observed feature vector of the leader agent will be... and the continuous power allocation variable of the current branch output The weight input to the current branch is In a Q-network, evaluate the discrete occupancy indicator variable of the current branch. Action value ; thereby obtaining the leader intelligent agent The optimal action pair for each branch ,in , All branches execute the above operations in parallel to obtain the optimal action pairs for each branch of the leader agent. The result is the selection and power allocation of the shared PRB by the current time slot leader agent based on the input observations; Step 4.2 involves constructing the follower agent. , 4.2a, Construct the observation feature vector of the follower agent ; in each time slot The follower agent constructed in step 3.2 of the third step will be observed. By network weight The followers share the observation feature extraction layer, which maps it to the observation feature vector of the follower agent. , serving as the observation input for each branch of the follower agent; 4.2b, construct parameter networks and Q networks for each PRB branch of the follower agent; treat each PRB as a branch, for the follower agent, including... Each ground base station needs to... Each PRB is selected and power is allocated, thus constituting The first parallel branch; in the second In each follower agent branch, the observed feature vector of the follower agent is input. The weight is parameter network Output the continuous power allocation variable of the current branch. Subsequently, the observed feature vectors of the follower agents are... and the continuous power allocation variable of the current branch output The weight input to the current branch is In a Q-network, evaluate the discrete occupancy indicator variable of the current branch. Action value ; thereby obtaining the follower agent's first The optimal action pair for each branch ,in , All branches execute the above operations in parallel to obtain the optimal action pairs for each branch of the follower agent. The result is the selection and power allocation of the shared PRB by the current time slot follower agent based on the input observations.

7. The spectrum sharing method for satellite-ground fusion networks based on branch-parameterized deep Q-networks according to claim 1, characterized in that, Step 5.1 involves initializing the training configuration and network weights; make This represents the total number of training rounds, the specific value of which depends on the agent's convergence requirements and available training resources; let... This represents the number of inner-layer updates in the follower agent, and its specific value depends on the degree of policy updates and computational resources of the follower agent in each training round; let and These represent the experience cache of the leader agent and the experience cache of the follower agent, respectively. make and These represent the learning rates of the leader agent and the follower agent, respectively. The specific values ​​depend on the parameter update magnitude and training convergence stability requirements of the respective agents. make and These represent the discount factors for the leader agent and the follower agent, respectively, with specific values ​​depending on the degree to which the respective agents weigh current and future rewards. make and These represent the soft update coefficients of the target network weights for the leader agent and the follower agent, respectively. The specific values ​​depend on the tracking speed and training stability requirements of the corresponding target network for the current network parameters. make and These represent the number of samples used by the leader agent and the follower agent for each update, respectively. The specific values ​​depend on the number of available samples in the corresponding experience buffer, the stability of parameter updates, and computing resources. For the leader agent, let This indicates that they share the network weights of the observation feature extraction layer. Indicate its The parameters of each branch are the network weights. Indicate its The Q-value network weights of each branch; for the follower agent, they include... One branch, making , and These represent the shared observation feature extraction layer network weights, the parametric network weights of each branch, and the Q-network weights of each branch, respectively. Step 5.2 involves executing the outermost layer decision of the leader's intelligent system. In the Training intervals The satellite network acquires the channel information of the current time slot and the resource allocation results of the previous time slot, and constructs the leader agent's observation according to step 3.1 in the third step. Treat it as the leader intelligent agent Input, leader intelligent agent Output Action This action is performed by the follower agent. The input conditions for the response; Step 5.3 involves updating the internal response of the follower's smart body. In a given leader action Afterwards, the follower agents implement Second inner layer update; in the current time slot For the first Second inner layer update The set of values ​​is Perform the following operations; 5.3a. Based on the channel power gain from each ground base station to the ground user terminal, the interference caused by the target LEO satellite to the ground user terminal, and the resource allocation results of the last inner-layer update on the ground network, the ground network constructs the observation of the follower agent according to step 3.2 in step 3. Treat it as a follower intelligent agent The input, the follower agent Output Action ;in, Indicates the first Ground network PRB selection matrix during the next inner layer update Indicates the first Ground network power allocation matrix during the next inner layer update; 5.3b, follower agents receive rewards. and the next observation and the sample Store in the follower agent's experience cache ; 5.3c, from Extraction A set of samples constitutes a mini-batch training sample for the follower agent; for each sample, the target Q-value is calculated. The target Q-value employs a dual-depth Q-network, which includes online network and target network structure calculations to mitigate Q-value overestimation. Its form is as follows: （11）， in, , and These represent the weights of the target network in the shared observation feature extraction layer and the weights of the first layer, respectively. Q-target network weights for each branch; 5.3d, Unfreezing Follower Agent Shared Observation Feature Extraction Layer Online Network Weights and the online network weights of each branch Q Freeze follower agent parameters and online network weights Update by minimizing the time difference error and The Q-value loss function for the follower agent is: （12）， in, Indicates the experience cache Calculate the expected value of the sample drawn from the sample. 5.3e, Freezing the online network weights of the follower agent's shared observation feature extraction layer. and the online network weights of each branch Q Unfreeze follower agent parameters online network weights Update by minimizing the parameter network loss function The loss function for the follower agent parameter network is: （13）， in, This indicates that the frozen follower agents share the network weights of the observation feature extraction layer. The frozen follower agent is the first The Q-network weights of each branch are frozen during the parameter network update phase. and This prevents the exploration gradient of the power parameter from backpropagating to the shared observation feature extraction layer, thereby reducing gradient interference. 5.3f, performs soft updates to the target network weights of the follower agent: （14）， （15）， （16）， in, , and These represent the target network weights corresponding to the shared observation feature extraction layer, the Q-network of each branch, and the parameter network of each branch of the follower agent, respectively; the weights of the target network are updated using the same soft update coefficient. ; Step 5.4 involves determining the follower agent's response. Finish After the last inner update, the follower action obtained from the last inner update is used as the follower response. ,in Indicates followers in time slots Actions of the leader The response results.

8. The spectrum sharing method for satellite-ground fusion networks based on branch-parameterized deep Q-networks according to claim 1, characterized in that, Step 5.5 involves updating the outer layer weights of the leader's intelligent system. 5.5a, the leader agent responds based on the responses of the follower agents. Receive rewards and the next observation and empirical samples Store in the leader agent's experience cache ; 5.5b, from Extraction A small batch of samples is used to form the leader agent's training samples, and the leader agent's objective Q-value is calculated. : （17）， in, , and These represent the weights of the target network in the shared observation feature extraction layer for the leader agent and the weights of the first layer, respectively. Q-target network weights for each branch; 5.5c, Unfreezing the leader agent's shared observation feature extraction layer online network weights and the online network weights of each branch Q Freeze the leader agent parameters and online network weights Update by minimizing the Q-value loss of the leader agent. and : （18）， in, Indicates the experience cache Calculate the expected value of the sample drawn from the sample. 5.5d, freezing the online network weights of the leader agent's shared observation feature extraction layer. and the online network weights of each branch Q Unfreeze leader agent parameters online network weights Update by minimizing the parameter network loss function The network loss function for the leader agent parameters is: （19）， in, This represents the shared observation feature extraction layer network weights of the frozen leader agent. The leader agent after being frozen is the first... Q-network weights of each branch; Version 5.5e performs soft updates to the network weights of each objective agent of the leader agent: （20）， （21）， （22）， in, , and These represent the target network weights corresponding to the shared observation feature extraction layer, the Q-network of each branch, and the parameter network of each branch of the leader agent, respectively; the weights of the above target networks are updated using the same soft update coefficient. ; 5.6 Repeat the training until the required number of training rounds is reached. Repeat steps (5.2)-(5.5) for each time slot in each round of training until completion. Round training; because the follower agent executes under each leader agent's action. The leader agent performs an outer update after the follower agent completes the inner update, while the follower agent performs an outer update after the follower agent completes the inner update. Therefore, the follower agent is in a fast timescale update and the leader agent is in a slow timescale update. This dual timescale training mechanism allows the follower agent to first approximate the optimal response to the leader agent's actions, and then the leader agent updates its strategy based on the response. This conforms to the master-follower decision-making process of Stackelberg game and is conducive to obtaining a stable Stackelberg equilibrium result. 5.7 Output the trained leader agent and follower agents This includes the shared observation feature extraction layer for each agent, the post-training weights of each branch parameter network and each branch Q network, which are used for subsequent online spectrum sharing resource allocation.

9. The spectrum sharing method for satellite-ground fusion networks based on branch-parameterized deep Q-networks according to claim 1, characterized in that, The sixth step is specifically as follows: Step 6.1: Execute the leader agent's online resource allocation decision. 6.1a, in online resource scheduling slots The target LEO satellite acquires the channel information of the current time slot and the resource allocation results of the previous time slot, and constructs the observation of the leader agent according to step 3.1 in step 3. ; 6.1b, will observe Input the trained leader agent The leader agent first utilizes a shared observation feature extraction layer to extract observations. The mapping is done to an observation feature vector; subsequently, each PRB branch outputs the discrete occupancy indicator variable and continuous power allocation variable of the corresponding PRB based on the observation feature vector, and the results are combined to obtain the leader action. ; 6.1c, Actions of the Leader Feasibility mapping is performed, mapping each PRB occupancy indicator variable to 0 or 1, and limiting each power allocation variable to the maximum transmit power range of the corresponding PRB; when the occupancy indicator variable of a certain PRB is 0, the transmit power corresponding to that PRB is set to 0; based on the mapped branch actions, the target LEO satellite in the time slot is constructed respectively. PRB selection matrix and power allocation matrix ; 6.2 Execute online response decisions for the follower agent. 6.2a After the target LEO satellite completes its leader decision, each ground base station will aggregate the spectrum sensing results and channel information to the ground network control node. Based on the channel power gain from each ground base station to the ground user terminal, the interference caused by the target LEO satellite to the ground user terminal, and the resource allocation results of the previous time slot on the ground network, the ground network control node will construct the follower agent's observation according to step 3.2 in step 3. ; 6.2b, will observe Input the trained follower agent The follower agent first utilizes a shared observation feature extraction layer to extract observations. The data is mapped to an observation feature vector. Subsequently, each "Ground Base Station-PRB" branch outputs the discrete occupancy indicator variable and continuous power allocation variable of the corresponding ground base station on the corresponding PRB based on the observation feature vector, and combines them to obtain the follower action. ; 6.2c, actions for followers Feasibility mapping is performed, mapping each PRB occupancy indicator variable to 0 or 1, and limiting each power allocation variable to the maximum transmit power range of the corresponding ground base station; when a ground base station does not select the corresponding PRB, the transmit power of that ground base station on that PRB is set to 0. Based on the mapped branch actions, the ground network is constructed in the time slots. PRB selection matrix and power allocation matrix ; 6.3, Perform satellite-to-ground network resource configuration. 6.3a, The feasibility-mapped leader actions are sent to the target LEO satellite, and the feasibility-mapped follower actions are sent to all ground base stations; the target LEO satellite and all ground base stations, according to the corresponding PRB selection results and power allocation results, in the time slots... Execute downlink data transmission; 6.3b, After the current time slot transmission is completed, construct a new time slot based on the updated environment. Leader observation and follower observation ; 6.3c, In each subsequent online resource scheduling time slot, steps 6.1-6.3 are repeated, so that the leader agent makes the satellite network resource allocation decision first, and then the follower agent makes the ground network resource allocation response given the leader's action, thereby realizing dynamic PRB scheduling and power allocation according to the master-slave decision order of the Stackelberg game. 6.4 Output the online spectrum sharing resource allocation results. 6.4a, Output the PRB selection matrix and power allocation matrix of the target LEO satellite in each resource scheduling time slot, as well as the PRB selection matrix and power allocation matrix of each ground base station; 6.4b, Based on the transmission rate calculation method described in the first step, calculate the downlink transmission rate that the satellite network and the terrestrial network can achieve after adopting the resource configuration results, and obtain the system spectrum efficiency, satellite network data service profit and the degree to which the terrestrial network rate demand is met. 6.4c, During online resource scheduling, the leader agent and follower agents By using the trained network weights for forward inference, it is not necessary to re-execute iterative optimization or network training in each time slot, thereby reducing the online computational complexity and decision latency of spectrum sharing resource allocation.

Images