Near-zero-power-consumption beam-hopping satellite communication system resource allocation method based on NOMA (Non-Orthogonal Multiple Access)

Abstract

The invention provides a near-zero power consumption beam-hopping satellite communication system resource allocation method based on NOMA, and belongs to the technical field of beam-hopping satellite communication systems. Users in the same beam are divided into time delay sensitive users and time delay tolerant users, a hopping beam time slot is divided into a data transmission stage and an energy collection stage through a time division multiplexing mechanism, and radio frequency energy generated by the time delay sensitive users is used for supplying power to the time delay tolerant users; the method comprises the following steps of: establishing a data transmission rate and power distribution joint optimization model by taking the minimization of the difference between the total transmitting power and the user request flow of a beam hopping satellite and the minimization of the delay sensitive user service interruption probability as targets, and constructing a deep reinforcement learning framework in combination with a multi-dimensional optimization space; deploying the deep reinforcement learning network framework in a satellite-ground cooperative manner, and performing offline training and verification of the model at a gateway station; and uploading the trained model parameters to a beam hopping satellite, and dynamically adjusting a beam hopping pattern, a power distribution coefficient and a time slot division parameter according to real-time service requirements.

Description

Technical Field

[0001] The present invention relates to the technical field of hopping beam satellite communication systems, and particularly to a resource allocation method for a near-zero power consumption hopping beam satellite communication system based on NOMA. Background Art

[0002] The next-generation high-throughput hopping beam satellite system adopts multi-point beam and frequency reuse technologies to construct a space-based backbone network to achieve global mobile communication services. However, its fixed power and spectrum allocation methods are difficult to adapt to the growth of the number of served users and non-uniform service demands, resulting in low resource utilization. In addition, the inter-beam interference caused by large-scale user access exacerbates the energy consumption of served users, further restricting the efficient allocation of resources.

[0003] The hopping beam technology dynamically hops between multiple beam positions in a time-division multiplexing manner and uses an on-board phased array to allocate resources in four dimensions of time, space, power, and bandwidth as needed. The non-orthogonal multiple access technology (NOMA) serves multiple users simultaneously on the same time-frequency resource and performs non-orthogonal allocation in the power domain. While improving the spectral efficiency, it can more flexibly meet different communication requirements. Existing solutions transform non-convex programming into a convex optimization problem through heuristic algorithms, and optimize the power and beam illumination patterns for each hopping beam time slot, improving the flexibility of traditional fixed allocation.

[0004] The existing NOMA-based hopping beam satellite systems do not fully consider the interference problem between beams and the large user energy consumption overhead caused by the interference. In the NOMA hopping beam system, resource allocation needs to consider multiple dimensions such as the time scheduling of beam hopping, NOMA user grouping and power allocation, and frequency resource management simultaneously, resulting in an exponential growth of the solution space of the optimization problem. Especially when the system scale expands, such as a low-earth orbit hopping beam satellite constellation needs to serve hundreds of beams and thousands of users simultaneously, traditional optimization algorithms often cannot meet the real-time requirements due to excessive computational complexity. The actual hopping beam satellite system needs to optimize multiple objectives such as spectral efficiency, energy efficiency, user fairness, and service delay simultaneously. Traditional methods use weighted summation to transform multiple objectives into a single objective, but the selection of weights lacks theoretical guidance, and fixed weights are difficult to adapt to dynamic environments.

[0005] Therefore, a resource allocation method for a near-zero power consumption hopping beam satellite communication system based on NOMA is needed. Summary of the Invention

[0006] In view of this, the present invention provides a resource allocation method for a near-zero power consumption hopping beam satellite communication system based on NOMA. By constructing a multi-dimensional resource joint optimization model and an intelligent decision-making mechanism based on deep reinforcement learning, an efficient matching of on-board resources and user request traffic in a high-throughput hopping beam satellite system is achieved.

[0007] For this purpose, the present invention provides the following technical solutions: A resource allocation method for a near-zero power consumption hopping beam satellite communication system based on NOMA, comprising: Dividing the users within the same beam into delay-sensitive users and delay-tolerant users, dividing the hopping beam time slots into a data transmission stage and an energy harvesting stage through a time division multiplexing mechanism, and using the radio frequency energy generated by the delay-sensitive users to supply power to the delay-tolerant users, thereby constructing a near-zero power consumption hopping beam satellite communication system; Taking the minimization of the difference between the total transmission power of the hopping beam satellite and the user request traffic and the lowest service interruption probability of the delay-sensitive users as the objectives, establishing a joint optimization model for data transmission rate and power allocation; Based on the joint optimization model of data transmission rate and power allocation and combined with a multi-dimensional optimization space, constructing a deep reinforcement learning framework; Deploying the deep reinforcement learning network framework in a satellite-ground collaborative manner, performing offline training and verification of the joint optimization model of data transmission rate and power allocation at the gateway station using historical state and service demand data; uploading the trained model parameters to the hopping beam satellite; the intelligent agent on the hopping beam receives the model parameters and dynamically adjusts the beam hopping pattern, power allocation coefficient and time slot division parameters according to the real-time service demand, so as to realize the adaptive matching scheduling of on-board resources and beam traffic.

[0008] Furthermore, a digital model for receiving signals of the hopping beam satellite is constructed to represent the total transmission power of the hopping beam satellite and the user request traffic; The digital model:

[0009] Among them, , represents the hopping beam time slot, represents the received signal of the hopping beam satellite under beam b within the hopping beam time slot , represents the access state of the th DTU, represents the access state of the th DSU; M represents the number of delay-sensitive users under beam b, N represents the number of delay-tolerant users under beam b, and S represents the number of illuminated beams in the snapshot; represents 's transmission power, represents Transmission power; is Transmission signal of is Transmission signal of is the beam of service of Channel gain; is the beam of service of Communication band.

[0010] Furthermore, the joint optimization model of data transmission rate and power allocation further includes: Taking that the beam satellite meets the power requirements of delay-tolerant user transmissions while giving priority to ensuring the transmission requirements of delay-sensitive user services as a constraint condition.

[0011] Furthermore, the deep reinforcement learning framework includes: Modeling the hopping beam satellite as an agent responsible for learning and decision-making; Modeling the data packets to be sent to each beam cell within the coverage range received by the hopping beam satellite as the environment; Modeling the problem of which beam cell each working beam is scheduled to in different hopping beam time slots as the interaction process between the agent and the environment; Taking the number of data packets to be transmitted with different remaining delay tolerances stored in each beam queue in the hopping beam time slot as the state; Taking the beam cell covered by the working beam selection in the current hopping beam time slot as the action; Taking the difference between the number of packets processed in the current hopping beam time slot and the number of packets that have timed out and failed as the reward.

[0012] Furthermore, the Q-network structure of the deep reinforcement learning framework includes: a 5-layer convolutional neural network; Each layer of the convolutional neural network includes: 2 convolutional layers, 1 Flatten layer, and 2 fully connected layers; and the ReLU activation function is used.

[0013] Furthermore, the space-ground collaborative deployment of the deep reinforcement learning network framework further includes: Uploading the Q-network parameters after training convergence to the on-board processing unit through the TT&C link using the differential update mechanism.

[0014] Furthermore, the dynamic adjustment of the beam hopping pattern, power allocation coefficient, and time slot division parameters according to real-time service requirements includes: Receiving requests from the beam cells in each hopping beam time slot and updating the state; Normalizing the updated state; The agent determines an action through the policy network and calculates the Q-value corresponding to the action through the Q network; Adopt The policy selects the action with the largest Q-value with a probability of as the action for the current hopping beam time slot; Parse the action to obtain the beam hopping pattern, power allocation coefficient, and time slot division parameters.

[0015] Furthermore, training the Q network of the deep reinforcement learning framework through the deep deterministic policy gradient algorithm includes: Use the Adam method to minimize the loss function; The loss function:

[0016] where represents the output of the current network, represents the parameters of the main network, represents the label value, represents the state, represents the action.

[0017] Furthermore, the joint optimization model of data transmission rate and power allocation:

[0018]

[0019] where is the normalization constant, is the weighting value, is the power allocated to the DSU by beam b in the hopping beam time slot ; is the power provided by the energy storage device, is the actual power demand of the DTU in the hopping beam time slot, represents the total transmission power of the hopping beam satellite in the current hopping beam time slot T; represents the requested traffic of all DTUs in the snapshot of the hopping beam time slot T; represents the requested traffic of all DSUs in the snapshot of the hopping beam time slot T; represents the normalization constant, set according to the system scale; represents the user under beam b in the hopping beam time slot the service interruption cost, represents the user the interruption indication function; the number of beams of the hopping beam satellite is ; is the energy used by the DTU for data transmission; For the hopping beam time slot The remaining energy in the energy storage device at the start; , Indicates that beam b is activated within the hopping beam time slot Indicates that beam b is not activated within the hopping beam time slot Indicates that beam b is not activated within the hopping beam time slot Indicates that beam b is not activated within the hopping beam time slot

[0020] Advantages and positive effects of the present invention: This method realizes the efficient dynamic allocation of resources in a high-throughput hopping beam satellite system through multi-dimensional resource joint optimization and an intelligent decision-making mechanism based on deep reinforcement learning.

[0021] Construct a joint optimization model based on NOMA: Divide the hopping beam time slot into two stages of data transmission and energy harvesting through a time division multiplexing mechanism, and use the radio frequency energy generated by delay-sensitive users to power delay-tolerant users, improving energy utilization efficiency while ensuring service continuity. Through power domain multiplexing of NOMA technology, multiple users can be served within the same beam, effectively improving the resource utilization rate under non-uniform traffic distribution; the traditional hopping beam system has difficulty in taking into account both delay-sensitive services and energy-constrained users.

[0022] Implement intelligent scheduling based on PDQN: Combine the advantages of DQN for handling discrete actions and DDPG for handling continuous actions, and achieve joint optimization of beam scheduling and power allocation through a hierarchical decision-making network, improving the adaptability of the system in a dynamic environment; solve the problem of insufficient adaptability of traditional optimization algorithms in complex hopping beam satellite communication environments.

[0023] This method enables lightweight deployment through satellite-ground collaboration: Complete offline training of the model through a ground gateway station, and perform lightweight inference on the on-board platform, effectively reducing the on-board computing burden while ensuring decision-making real-time performance; solve the contradiction between the limited on-board computing power and the real-time requirement of services. Description of the Drawings

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0025] Figure 1 Flowchart of the resource allocation method for a near-zero power consumption hopping beam satellite communication system based on NOMA in an embodiment of the present invention; Figure 2 Structural diagram of a near-zero power consumption hopping beam satellite communication system in an embodiment of the present invention; Figure 3 Schematic diagram of zero-power network energy harvesting for sidelink communication in the embodiments of the present invention; Figure 4 Schematic diagram of the deep reinforcement learning framework structure in the embodiments of the present invention; Figure 5 PDQN network structure diagram in the embodiments of the present invention; Figure 6 Flowchart of online generation of beam scheduling and traffic allocation strategies in the embodiments of the present invention. Detailed implementation manners

[0026] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those clearly listed steps or units, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0028] The present invention provides a resource allocation method for a near-zero power consumption hopping beam satellite communication system based on NOMA. By implementing power domain multiplexing based on NOMA technology, multiple users can be served within the same beam. Through a time division multiplexing mechanism, the hopping beam time slot is divided into two phases: data transmission and energy harvesting. The radio frequency energy generated by the delay-sensitive users is used to power the delay-tolerant users, thereby constructing a self-sustaining communication network. A multi-dimensional optimization space including beam hopping patterns, power allocation coefficients, and hopping beam time slot division parameters is established. Combining the advantages of the Deep Q-Network (DQN) for handling discrete actions and the Deep Deterministic Policy Gradient (DDPG) algorithm for handling continuous actions, an intelligent scheduling algorithm based on PDQN is designed to achieve dynamic matching of on-board resources and beam traffic through a hierarchical decision-making network. At the same time, a lightweight satellite-ground collaborative deployment scheme is adopted, and through an offline training-online inference architecture, the on-board computing load is effectively reduced while ensuring the real-time nature of decision-making.

[0029] Embodiment Combined with Figure 1 As shown, the steps of this method include: In this embodiment, a hopping beam satellite and multiple ground users form a near-zero power consumption hopping beam satellite communication system, and the structure is as Figure 2 shown.

[0030] The coverage area of the hopping beam satellite is geographically partitioned to obtain multiple beam cells; one beam can only illuminate one beam cell in each hopping beam time slot; in each hopping beam time slot, several beam cells are selected as a beam snapshot for irradiation, and after one hopping beam cycle, it is ensured that all beam cells are covered at least once. Each beam cell is provided with an energy storage device for storing the radio frequency energy generated by the delay-sensitive users collected by the delay-tolerant users in the beam cell through the environment.

[0031] To improve data transmission performance, the hopping beam satellite can generate multiple beams and simultaneously transmit data to different ground regions. Define the number of beams of the hopping beam satellite as , the total transmit power of the hopping beam satellite as , and the total system bandwidth as .

[0032] Each beam cell serves N delay-sensitive users (DSUs) and M delay-tolerant users (DTUs) simultaneously. Since the priority of DSUs is higher, the satellite gives priority to serving DSUs and provides services to DTUs on the premise of ensuring the service of DSUs.

[0033] Step 1: Divide the hopping beam time slot into two phases of data transmission and energy harvesting through a time-division multiplexing mechanism. Utilize the radio frequency energy generated by delay-sensitive users to power delay-tolerant users and construct a self-sustaining communication network as a near-zero power consumption communication network. Based on the near-zero power consumption communication network, build a power model for the communication within the beam cell and quantify the energy harvesting and distribution process of the hopping beam satellite communication system.

[0034] Under each beam, the delay-tolerant user harvests the radio frequency energy generated by the delay-sensitive user in the environment and stores it in the energy storage device within each beam cell. A zero-power consumption system based on sidelink communication is adopted for its own communication, including: Step 1-1: Divide each hopping beam time slot into and two phases. Among them, represents the time-division parameter. During the time, the DTU uses the remaining energy from the previous hopping beam time slot in the energy storage device to transmit data; during the remaining , the DTU harvests radio frequency energy from the DSU to charge the energy storage device. First, provide energy for the data transmission of the DTU within a hopping beam time slot, and then perform the charging of the storage device to ensure that the DTU service requirements in the next hopping beam time slot can be met.

[0035] Figure 3 This is the block diagram of the zero-power consumption network energy harvesting scheme based on sidelink communication in this embodiment: and The signals transmitted respectively reach and through channels with channel gains of . After that, the power splitter distributes the energy harvesting and data transmission phases according to the signal indication; the harvested energy is used for the communication between the data transmission phase and the hopping beam satellite. and represent the noises in the signal transmission and energy harvesting links respectively.

[0036] Step 1-2: Model the signal received by the hopping beam satellite under beam b within the hopping beam time slot :

[0037] Among them, , represents the signal received by the hopping beam satellite under beam b, represents the access state of the th DTU, represents the access state of the th DSU; Indicates the transmission power of Indicates the transmission power of; Is the transmitted signal of Is the transmitted signal of; Is the service beam of the channel gain of; Is the service beam of the communication frequency band of.

[0038] Steps 1-3: When the DTU collecting radio frequency energy from the DSU does not cause additional energy consumption of the DSU, based on the model of the received signal of the hopping beam satellite under beam b within the hopping beam time slot determine the energy collected under beam b within the hopping beam time slot which is expressed by the formula as:

[0039] where, is the energy collected under beam b within the hopping beam time slot ; is the energy collection efficiency; Is between the channel gain.

[0040] Steps 1-4: Determine the remaining energy of the energy storage device in beam b at the end of the hopping beam time slot The calculation formula is:

[0041] where, is the remaining energy in the energy storage device at the start of the hopping beam time slot ; is the remaining energy in the energy storage device at the end of the hopping beam time slot ; is the energy used by the DTU for data transmission; represents the upper limit of the energy storage of the energy storage device.

[0042] Steps 1-5: Use the remaining energy of the energy storage device for data transmission. Under beam b within the hopping beam time slot use the energy in the energy storage device to provide energy for the DTU data transmission, then the power provided by the energy storage device to the DTU is:

[0043] Among them, represents the power provided by the energy storage device in beam b for DTU data transmission.

[0044] Step 2: Construct a joint optimization model of data transmission rate and power allocation with the goal of minimizing the difference between the total transmission power of the hopping beam satellite and the user-requested traffic. After the hopping beam satellite collects the demands of users in the service area, a joint optimization model of data transmission rate and power allocation is constructed with the goal of minimizing the difference between the total transmission power of the hopping beam satellite and the user-requested traffic, taking into account the service interruption probability of delay-sensitive users and the minimum power guarantee of delay-tolerant users.

[0045] Step 2-1: Determine the hopping beam time slot within which the signal-to-noise ratio of the served user in beam

[0046] Among them, is the signal-to-noise ratio of the served user in beam during the hopping beam time slot T; is the channel gain between the beam serving the user and the user ; is the power allocation coefficient of the user ; is the power allocated by the hopping beam satellite to beam ; is the in-beam interference of the served user with other users in the same beam; is the inter-beam interference of the served user with other active beams; is the Gaussian white noise; the in-beam interference and inter-beam interference are calculated as follows:

[0047]

[0048] Among them, represents the in-beam interference, represents the inter-beam interference; represents whether the adjacent beam is active in the snapshot within the hopping beam time slot ; represents being active, represents not being active.

[0049] Step 2-2: Determine the hopping beam time slot In the snapshot of, the beam The data transmission rate of the served user is calculated as follows:

[0050] Among them, Indicates the served user Data transmission rate; Is the bandwidth of the beam ; , Indicates that the beam b is activated within the hopping beam time slot , Indicates that the beam b is not activated within the hopping beam time slot, and there is , , Indicates the set of beams to be activated in the snapshot within the hopping beam time slot T, and S is the number of illuminated beams in the snapshot.

[0051] Step 2-3: Determine the hopping beam time slot The traffic provided by the hopping beam satellite to each served user under beam b within is calculated as follows:

[0052] Among them, Indicates the hopping beam time slot The total traffic provided by the hopping beam satellite to the served user in the beam .

[0053] Step 2-4: Determine the total traffic provided by the hopping beam satellite to all served users under beam b and the total transmit power of the hopping beam satellite within the hopping beam time slot :

[0054]

[0055] Among them, Indicates the total traffic provided by the hopping beam satellite to beam b; Indicates the total transmit power of the hopping beam satellite.

[0056] Step 2-5: Determine the requested traffic of the DTU and DSU within the hopping beam time slot The calculation formula is:

[0057]

[0058] Among them, the total traffic demand of all DTUs under beam b within the hopping beam time slot T is , and the total traffic demand of all DSU under beam b within the hopping beam time slot T is ; represents the requested traffic of all DTUs in the snapshot of the hopping beam time slot T; represents the requested traffic of all DSU in the snapshot of the hopping beam time slot T.

[0059] Step 2-6: Taking the matching of the traffic provided by beam b within the hopping beam time slot T with the user-requested traffic of the cell and the lowest service interruption probability of DSU users as the optimization objective, which is expressed by the formula:

[0060]

[0061] Among them, represents the normalization constant, which is set according to the system scale; represents user under beam b in the hopping beam time slot The service interruption cost, represents user The interruption indication function.

[0062]

[0063] In the formula, represents the interruption threshold of the signal-to-noise ratio, When represents the normal communication between the hopping beam satellite and user , otherwise it represents communication interruption.

[0064] Step 2-7: Determine the constraint conditions. Within the hopping beam time slot , while giving priority to ensuring the service transmission requirements of DSU, the beam satellite meets the power requirements of DTU transmission. The constraint condition formula is expressed as:

[0065] Among them, is the power allocated to DSU by beam b within the hopping beam time slot ; is the power provided by the energy storage device, is the actual power requirement of DTU within the hopping beam time slot, represents the total transmission power of the hopping beam satellite in the current hopping beam time slot T.

[0066] The calculation formula for the power allocated to DSU under beam b within the hopping beam time slot T is:

[0067] Among them, is the hopping beam time slot The power allocated by the hopping beam satellite to the beam within.

[0068] Step 2-8: Aiming at minimizing the difference between the total transmission power of the hopping beam satellite and the user-requested traffic, combined with the user power demand constraint, a joint optimization model of data transmission rate and power allocation is constructed:

[0069] Converted to:

[0070]

[0071] Among them, is the normalization constant, is the weighting value.

[0072] Model the hopping beam satellite system as a discrete-time event system, driven by new service arrival events, and convert the hopping problem in the hopping beam satellite system into a sequential decision-making problem.

[0073] Combining the advantages of the deep Q-network in dealing with discrete action spaces and the advantages of the deep deterministic policy gradient algorithm in dealing with continuous action spaces, a beam scheduling and power allocation framework based on the Parameterized Deep Q-Network (PDQN) is constructed: Model the hopping beam satellite as an agent responsible for learning and decision-making; Model the data packets to be sent received by the hopping beam satellite to each beam cell within the coverage area as the environment; Since the essence of beam scheduling in the hopping beam satellite system is to allocate the hopping beam time slot resources of the hop, the problem of which beam cell each working beam is scheduled to in different hopping beam time slots is modeled as the interaction process between the agent and the environment; And through continuous training and learning of the PDQN, the long-term maximum benefit is achieved.

[0074] Taking the joint optimization model of data transmission rate and power allocation as the objective function, construct a deep reinforcement learning framework to realize the offline training of the Q-network and the online generation of beam scheduling and power allocation strategies.

[0075] Design the relevant parameters for model training; then, through the strong computing and processing capabilities of the gateway station, conduct offline training and verification on the joint optimization model of data transmission rate and power allocation; finally, upload the trained model parameters to the hopping beam satellite through the TT&C link, perform online inference and decision-making operations on the satellite, and complete the real-time scheduling of on-satellite resources and beam traffic according to different service QoS requirements.

[0076] Step 3: Taking the joint optimization model of data transmission rate and power allocation as the objective function, construct a deep reinforcement learning framework, the structure of which is as Figure 4 shown; Step 3-1: Determine the state; The remaining delay tolerances of different types of service data packets are different, and it is the maximum delay tolerance among all service types. For each hopping beam time slot experienced, the remaining delay tolerance is decreased by 1. Therefore, the state of the hopping beam time slot is defined as:

[0077] where the matrix represents the number of packets to be transmitted with different remaining delay tolerances stored in each beam queue of the hopping beam time slot.

[0078] Step 3-2: Determine the action; Take the beam cell covered by the illuminating beam in the current hopping beam time slot as the action:

[0079] where represents the output action; indicates whether beam b is activated within the hopping beam time slot T.

[0080] Step 3-3: Determine the reward; The optimization objective of the objective function is to minimize the difference between the total transmission power of the hopping beam satellite and the user-requested traffic. Therefore, take the difference between the number of packets processed in the current hopping beam time slot and the number of packets that time out and fail as the reward:

[0081] where represents the reward function; represents the total number of packets processed by the system after selecting the action in the hopping beam time slot, and is the total number of packets that time out and fail in the current hopping beam time slot.

[0082] Step 3-4: Determine the network structure; As Figure 5As shown, in this embodiment, the Q network serves as the core decision-making module and adopts a 5-layer convolutional neural network structure, including 2 convolutional layers, 1 Flatten layer, and 2 fully connected layers, and uses the ReLU activation function. This design efficiently extracts the deep features of the state matrix through the convolutional layers and then realizes the accurate mapping from the state to the action value through the fully connected layers. The policy network serves as an auxiliary module, and its output is only used to guide the exploration direction of the Q network or provide initial action suggestions and does not directly participate in the final decision. When the two work together, the gradient update of the policy network is based on the optimization objective of the Q network to ensure the consistency of its auxiliary function with the main network.

[0083] Step 3-5: Introduce an experience pool to store the quadruples generated during the interaction between the agent and the environment , so as to break the correlation between data during the neural network training process and enable the neural network to learn from independently and identically distributed data.

[0084] Step 3-6: Introduce a target network to update the main network. The target network has the same structure as the main network but different parameters. The parameters of the target network are a slow copy of the main network parameters, that is, the parameters of the target network are copied and updated from the main network every G steps to reduce the correlation between the target Q value and the current Q value.

[0085] Step 3-7: Determine the loss function:

[0086] Among them, represents the output of the current network, represents the output of the target network, represents the parameters of the main network, represents the parameters of the target network; is the label value:

[0087] Among them, γ is the discount factor of future rewards.

[0088] Step 4: Train the PDQN through the deep deterministic policy gradient algorithm; Step 4-1: Initialize the scene parameters of the hopping beam satellite system.

[0089] Step 4-2: Randomly initialize the online Q network and the target network , and empty the experience pool.

[0090] Step 4-3: Set the number of training epochs and the number of hopping beam time slots per epoch, and start the training process.

[0091] Step 4-4: Initialize the state of the Q network for this epoch , Action and Reward .

[0092] In each hopping beam time slot , the agent determines the action through the policy network and calculates the Q-value corresponding to the action through the Q-network. Based on the policy ( -greedy policy), with a probability of , the action with the maximum Q-value is selected.

[0093]

[0094] Among them, represents the greedy factor, is the power allocation coefficient; is the hopping beam selection pattern coefficient; is the time-sharing parameter, which is used to divide a hopping beam time slot to determine the ratio of data transmission time to energy harvesting time.

[0095] Step 4-5: The agent executes the action , and the state changes from to , and calculates the reward value of this action :

[0096] Among them, , is the resource allocation satisfaction, ; is the traffic gap of each beam, = - ; is a constant used to standardize .

[0097] Step 4-6: Store the quadruple in the experience pool. If the experience pool is full, discard the earliest quadruple.

[0098] Step 4-7: Select a batch of experience samples from the experience pool, and calculate the loss function and label value through Step 3-7.

[0099] Step 4-8: Update the network parameters using the Adam method. The target network parameters are updated from the Q-network every G steps.

[0100] Step 5: Deploy the satellite-ground cooperative deep reinforcement learning network framework. The flowchart is as shown in Figure 6As shown; execute the training process of step 4 until the Q-network converges; then execute the online decision-making phase to obtain the beam scheduling and power allocation strategy. The specific steps are as follows: Step 5-1: The TT&C link uploads the Q-network parameters and the target network parameters from the gateway station to the on-board agent. The parameter upload adopts a differential update mechanism, and only the differences from the previous version are transmitted to reduce the uplink resource occupancy.

[0101] Step 5-2: The on-board agent makes an online decision based on the trained network for the current beam queue status at each moment, and generates the action for the current hopping beam time slot; 1) The on-board agent loads the received model parameters; initializes the environmental state, agent action, and experience pool.

[0102] 2) And establish an initial state synchronization with each beam cell to obtain the current beam queue status at each moment.

[0103] 3) Initialize the state matrix to record the number of data packets with different remaining delay tolerances for each beam.

[0104] 4) Start the online decision-making. In each hopping beam time slot , accept the requests from the beam cells, update the state matrix , and perform normalization processing on the state data.

[0105] 5) Input the normalized state into the Q-network to obtain the Q-value estimates of possible actions . Adopt the policy to select the action for the current hopping beam time slot .

[0106] Step 5-3: Parse the action from to decode the beam cells to be activated in the snapshot of the T-hop beam time slot:

[0107] Among them, represents the set of beams to be activated, represents whether the beam is activated, and . Calculate the transmit power of beam b according to the power allocation coefficient :

[0108] Among them, represents the power allocation coefficient of beam b. Set according to the time-sharing parameter Time slot structure of time slot beam b.

[0109] Step 5-4: Go to 4) to start the next iteration decision.

[0110] The resource allocation of the near-zero power consumption hopping beam satellite communication system is output through the deep learning framework, realizing the matching of the traffic provided by the hopping beam satellite with the user-requested traffic and the lowest DSU user service interruption probability.

[0111] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A resource allocation method for a near-zero power consumption hopping beam satellite communication system based on NOMA, characterized in that, Including: Dividing the users within the same beam into delay-sensitive users and delay-tolerant users, dividing the hopping beam time slots into a data transmission phase and an energy harvesting phase through a time-division multiplexing mechanism, and using the radio frequency energy generated by the delay-sensitive users to power the delay-tolerant users, to construct a near-zero power consumption hopping beam satellite communication system; Taking the minimization of the difference between the total transmission power of the hopping beam satellite and the user request traffic and the lowest service interruption probability of the delay-sensitive users as the objectives, to establish a joint optimization model of data transmission rate and power allocation; Based on the joint optimization model of data transmission rate and power allocation and combined with a multi-dimensional optimization space, to construct a deep reinforcement learning framework; Deploying the deep reinforcement learning network framework in a satellite-ground collaborative manner, and performing offline training and verification of the joint optimization model of data transmission rate and power allocation at the gateway station using historical state and service demand data; Uploading the trained model parameters to the hopping beam satellite; the agent on the hopping beam receives the model parameters, and dynamically adjusts the beam hopping pattern, power allocation coefficient and time slot division parameters according to the real-time service demand, to achieve the adaptive matching scheduling of on-board resources and beam traffic.

2. The method according to claim 1, wherein Constructing a digital model of the received signal of the hopping beam satellite to represent the total transmission power of the hopping beam satellite and the user request traffic; The digital model: Among them, , represents the hopping beam time slot, represents the hopping beam time slot The received signal of the hopping beam satellite under beam b within the hopping beam time slot, represents the access status of the DTU, represents the access status of the DSU; M represents the number of delay-sensitive users under beam b, N represents the number of delay-tolerant users under beam b, and S represents the number of illuminated beams in the snapshot; represents the transmission power of, represents the transmission power of; is the transmitted signal of, is the transmitted signal of; is the channel gain of the beam serving ; is the communication band of the beam serving .

3. The method according to claim 1, wherein The joint optimization model of data transmission rate and power allocation further includes: Taking that the beam satellite satisfies the power demand for the transmission of the delay-tolerant users while giving priority to ensuring the service transmission requirements of the delay-sensitive users as the constraint condition.

4. The method according to claim 1, characterized in that, The deep reinforcement learning framework includes: Modeling the hopping beam satellite as an agent, responsible for learning and decision-making; Modeling the to-be-transmitted data packets received by the hopping beam satellite and destined for each beam cell within the coverage area as the environment; Modeling the problem of which beam cell each working beam is scheduled to in different hopping beam time slots as the interaction process between the agent and the environment; Taking the number of to-be-transmitted data packets with different remaining delay tolerances stored in each beam queue in the hopping beam time slot as the state; Taking the beam cell covered by the selection of the working beam in the current hopping beam time slot as the action; Taking the difference between the number of packets processed in the current hopping beam time slot and the number of packets that have timed out and failed as the reward.

5. The method according to claim 4, wherein The Q network structure of the deep reinforcement learning framework includes: a 5-layer convolutional neural network; Each layer of the convolutional neural network includes: 2 convolutional layers, 1 Flatten layer, 2 fully connected layers; and the ReLU activation function is used.

6. The method according to claim 4, wherein The satellite-ground collaborative deployment of the deep reinforcement learning network framework further includes: Uploading the Q network parameters after training convergence to the on-board processing unit through the TT&C link, using a differential update mechanism.

7. The method according to claim 1, characterized in that, The dynamically adjusting the beam hopping pattern, power allocation coefficient and time slot division parameters according to the real-time service demand includes: Receiving requests from the beam cells in each hopping beam time slot and updating the state; Performing normalization processing on the updated state; The agent determines the action through the policy network and calculates the Q value corresponding to the action through the Q network; Adopt the strategy to select the action with the largest Q value with a probability of, as the action for the current hopping beam time slot; Parsing the action to obtain the beam hopping pattern, power allocation coefficient and time slot division parameters.

8. The method according to claim 4, wherein Training the Q network of the deep reinforcement learning framework through the deep deterministic policy gradient algorithm includes: Using the Adam method to minimize the loss function; The loss function: Among them, represents the current network output, represents the parameters of the main network, represents the label value, represents the state, represents the action.

9. The method according to claim 1, characterized in that, The joint optimization model of data transmission rate and power allocation: Among them, is the normalization constant, is the weighting value, is the hopping beam time slot The power allocated to the DSU by inner beam b; The power provided by the energy storage device, is the actual power demand of the DTU within the hopping beam time slot, represents the total transmission power of the hopping beam satellite in the current hopping beam time slot T; represents the requested traffic of all DTUs in the snapshot of hopping beam time slot T; represents the requested traffic of all DSUs in the snapshot of hopping beam time slot T; represents the normalization constant, set according to the system scale; represents the user in beam b hopping beam time slot The service interruption cost below, represents the user The interruption indication function of; the number of beams of the hopping beam satellite is ; is the energy used by the DTU for data transmission; is the hopping beam time slot The remaining energy in the energy storage device at the start; , represents that inner beam b is activated within the hopping beam time slot , represents that inner beam b is not activated within the hopping beam time slot .

Images