An Online Resource Joint Allocation Method for Satellite Internet of Things Based on Meta-Reinforcement Learning

Abstract

A method for jointly allocating online resources for satellite Internet of Things based on meta-reinforcement learning belongs to the field of satellite communications. A joint allocation model for uplink channels and power resources is established; the joint allocation problem for uplink resources is modeled as an MDP process; the parameters of the online channel selection and power allocation model are trained using a meta-reinforcement learning method; whenever a newly deployed LEO satellite is launched, the trained meta-reinforcement learning parameters are distributed to the LEO satellite; each LEO satellite fine-tunes the Q-value network until convergence, and uses the deep reinforcement learning model parameters migrated to the local real space environment to perform joint allocation of online channels and power resources. The present invention uses meta-learning to train model parameters with strong generalization capabilities, which helps to improve the on-board convergence speed in the actual deployment of LEO constellations, while reducing the computational resource overhead required for training, and reducing the number of interactions with the actual system required for training, thereby extending the normal service time of LEO satellites.

Description

Technical Field

[0001] The present invention belongs to the field of satellite communications and relates to a satellite Internet of Things uplink resource joint allocation method based on meta-reinforcement learning. Background Art

[0002] With the development of technologies such as smart homes, digital healthcare, and the Internet of Vehicles, the Internet of Things (IoT) industry is rapidly expanding and its applications are becoming increasingly prevalent. The global demand for seamless connectivity of all things is becoming increasingly prominent. However, the IoT, which relies on terrestrial networks, is limited by the geographical environment in which base stations are deployed, making it difficult to provide data transmission services for IoT terminals in remote or non-terrestrial areas for data collection, disaster monitoring, wildlife tracking, and other purposes. Thanks to the wide-area coverage capabilities of satellites, the satellite IoT is gaining increasing attention from researchers. The resource allocation problem for the satellite IoT requires addressing the following difficulties:

[0003] (1) Large-scale terminal connections. Compared with the terrestrial IoT, the satellite IoT has a wider coverage area and needs to support large-scale terminal connections. In addition, the data transmission of various terminals is highly bursty, causing the number of terminals that the network needs to serve to change dynamically.

[0004] (2) Terminal energy is limited. Compared to traditional user terminals, satellite IoT terminals located in remote areas are usually powered by solar cells. The batteries have a rated number of cycles, and their lifespan is inversely proportional to the average energy consumption of data transmission. Therefore, in the power control problem of IoT terminals, it is necessary to balance the relationship between energy consumption and transmission performance.

[0005] (3) Highly dynamic. Satellite IoT is mostly based on LEO constellations, where topology, links, and node status are dynamically changing. Furthermore, channel quality is sensitive to weather conditions and fluctuates widely. Therefore, the dynamic and unpredictable channel quality poses challenges to resource allocation.

[0006] Existing resource allocation methods based on modeling optimization face the following shortcomings in addressing the above challenges:

[0007] (1) Traditional modeling optimization mechanisms require a complete understanding of the wireless environment and are typically based on specific channel model assumptions or use accurate global CSI as a known condition for the algorithm. For channel modeling methods, when the real-world channel differs from the predefined channel model due to interference or weather changes, the performance of the resource allocation scheme may be severely degraded. For methods that assume known channel quality, accurate and complete real-time channel status data is difficult to obtain due to the long latency in satellite networks and the frequent occurrence of link interruptions.

[0008] (2) The large-scale characteristics of satellite IoT will make resource allocation, a non-convex optimization problem, increasingly complex. Most non-convex optimization methods, such as metaheuristic methods, require multiple rounds of iterations, occupy precious computing resources on board, and have poor real-time performance.

[0009] (3) Traditional optimization mechanisms only consider the current moment. In the process of dynamic changes in network status, once the environment changes, re-modeling and iterative optimization are required, which has high modeling and computational costs.

[0010] Deep reinforcement learning (DRL), a sequential decision-making method that does not rely on precise prior modeling, can make real-time adjustments to dynamic channel quality changes and optimize the long-term benefits of sequential decisions. Therefore, it is considered an effective approach to addressing the technical challenges of resource allocation in satellite physical networks. However, satellite network resource allocation methods using deep reinforcement learning have the following shortcomings:

[0011] (1) The satellite IoT uplink requires joint optimization of the terminal’s channel and transmission power. However, the existing satellite IoT resource allocation method based on deep reinforcement learning only focuses on the single problem of spectrum or power allocation, which makes it difficult to achieve the optimal effect.

[0012] (2) Existing methods pay little attention to actual deployment issues. If training is conducted on the ground, the difference between the ground simulation environment and the real space environment will lead to a decrease in algorithm accuracy. If training is conducted entirely on board, each satellite in a large-scale LEO constellation will train its own model, which is computationally expensive. In addition, during the cold start period when the algorithm training converges, the LEO satellite cannot normally provide IoT data transmission services, resulting in a huge waste of resources.

[0013] (3) The state features of existing satellite network resource allocation methods based on deep reinforcement learning often focus on user geographic location information, contain less contextual information, and are sparsely organized, resulting in high state feature dimensions, more neural network parameters, and limited perceptible information. Summary of the Invention

[0014] The present invention provides a satellite Internet of Things online resource joint allocation method based on meta-reinforcement learning. By using meta-learning to train model parameters with strong generalization ability and using them as initialization parameters, it helps to improve the on-board convergence speed in the actual deployment of LEO constellations, reduce the computing resource overhead required for training, and reduce the number of interactions with the actual system required for training, thereby extending the normal service time of LEO satellites.

[0015] A satellite Internet of Things online resource joint allocation method based on meta-reinforcement learning includes the following steps:

[0016] Step 1: Build an approximate model for channel quality, terminal data transmission and other required environmental information based on historical data, thereby establishing a joint allocation model for satellite IoT uplink channel and power resources;

[0017] Step 2: Model the joint uplink resource allocation problem at each LEO satellite as an MDP process;

[0018] Step 3: Use a centralized training approach to train the online channel selection and power allocation model parameters through a meta-reinforcement learning method;

[0019] Step 4: During the actual LEO constellation operation, whenever a newly deployed LEO satellite is launched, the meta-reinforcement learning parameters trained in step 3 are distributed to this LEO satellite;

[0020] Step 5: Each LEO satellite migrates the model based on the actual onboard environment and fine-tunes the Q value network until convergence.

[0021] Step 6: Each LEO satellite uses the deep reinforcement learning model parameters migrated to the local real space environment to perform online channel and power resource joint allocation.

[0022] The present invention adopts the above technical solution and has the following advantages compared with the prior art:

[0023] (1) Joint decision-making on power and spectrum, two scarce resources in satellite IoT; at the same time, considering that terminal power resources are very valuable in IoT scenarios, energy utilization and transmission success rate are taken as optimization targets to optimize long-term comprehensive performance.

[0024] (2) In order to solve the problem that the state features in the existing methods of satellite resource allocation based on deep reinforcement learning contain less contextual information and are sparsely organized, which is not conducive to the rapid convergence of deep reinforcement learning, the feature organization method is improved. It not only effectively compresses the state information dimension, but also increases the contextual information contained in the state without increasing the communication cost, thereby improving the convergence effect.

[0025] (3) The attention mechanism is used to improve the Q-value neural network structure in deep reinforcement learning, so that the value network learns to focus on the channel, thereby improving the strategy optimization ability of deep reinforcement learning.

[0026] (4) A practical deployment scheme for a joint resource allocation method based on meta-reinforcement learning is proposed. By using meta-learning to train model parameters with strong generalization capabilities and using them as initialization parameters, it helps to improve the on-board convergence speed in the actual deployment of LEO constellations. On the one hand, it reduces the computing resource overhead required for training; on the other hand, it reduces the number of interactions with the actual system required for training, thereby extending the normal service time of LEO satellites. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Schematic diagram of the application scenario of the present invention, namely, the satellite Internet of Things scenario;

[0028] Figure 2 This is a schematic diagram of the overall framework of the online resource allocation method based on deep reinforcement learning of the present invention;

[0029] Figure 3 This is a diagram of the Q-value neural network structure improved based on the attention mechanism;

[0030] Figure 4 This is a schematic diagram of the impact of deployment training time on satellite available service time. DETAILED DESCRIPTION

[0031] A satellite Internet of Things online resource joint allocation method based on meta-reinforcement learning includes the following steps:

[0032] Step 1: Build an approximate model for channel quality, terminal data transmission and other required environmental information based on historical data, thereby establishing a joint allocation model for satellite IoT uplink channel and power resources;

[0033] Step 2: Model the joint uplink resource allocation problem at each LEO satellite as an MDP process;

[0034] Step 3: Use a centralized training approach to train the online channel selection and power allocation model parameters through a meta-reinforcement learning method;

[0035] Step 4: During the actual LEO constellation operation, whenever a newly deployed LEO satellite is launched, the meta-reinforcement learning parameters trained in step 3 are distributed to this LEO satellite;

[0036] Step 5: Each LEO satellite migrates the model based on the actual onboard environment and fine-tunes the Q value network until convergence.

[0037] Step 6: Each LEO satellite uses the deep reinforcement learning model parameters migrated to the local real space environment to perform online channel and power resource joint allocation.

[0038] The technical solution of the present invention is further illustrated below with reference to the accompanying drawings and specific embodiments.

[0039] Step 1: Build an approximate model for the required environmental information such as channel quality and terminal data transmission based on historical data, thereby establishing a joint allocation model of satellite IoT uplink channel and power resources.

[0040] like Figure 1As shown: Each multi-beam LEO satellite provides data transmission services to remote IoT terminal devices such as forest fire sensors, meteorological and hydrological sensors, and crop sensors within the coverage area of its beam. Each LEO satellite shares the spectrum of the Ka band. The present invention does not study the network access selection problem, so it is assumed that all IoT terminals access their nearest satellite. Therefore, for a terminal in a certain area, it can be approximately considered that there is only one satellite above it. Assuming that the terminal remains stationary, the satellite only has a transparent forwarding function. There are multiple antennas on the satellite to receive uplink data sent by IoT terminals within each beam range. Let K and N represent the terminal u respectively. k and beam b n The number of terminals and beam sets can be expressed as and and The relationship between the terminal and the beam can be expressed as x k =[x k,1 ,x k,2 ,…x k,N ]x k,n ∈{0,1},x k,n represents the kth terminal u k Located in the coverage area of the nth beam. The maximum forwarding power of a single satellite is P total , the maximum forwarding power of a single beam is P B .

[0041] The overall spectrum resources of the system are divided into 1 control channel and M data transmission channels. k The overall transmission process of the burst data transmission request requires the following three steps: (1) Terminal u k The amount of data you need to transmit D k The satellite resource control module transmits the information to the satellite resource control module through the control channel; (2) The satellite resource control module determines the terminal u based on the existing resources, the channel and power conditions of the terminal being served. k Transmission power p k,S and sending channel c k,S and transmits the resource allocation result to terminal u through the control channel k ; (3) Finally, terminal u k In the sending channel c k,S The transmission power p k,S The data is sent until completion and then the resources are released. The present invention focuses on the channel allocation and power control decision in the second step of the above process.

[0042] For channel allocation, the data transmission channel set is expressed as And the bandwidth of each channel is B. The channel allocation problem is to select In the example, a suitable channel is selected for the current transmission request sensor, and its allocation result can be expressed as w k =[wk,1 ,w k,2 ,L w k,M ] T ,w k,m ∈{0,1}, where w k,m =1 means channel c m Assigned to terminal u k The channel allocation result of all sensors in the system can be expressed as W = [w1,w2,…,w M ],

[0043] For power control, let P k Indicates sensor u k The maximum transmission power of , although the power is a continuous variable, it is usually quantified in the satellite network resource allocation problem. The present invention quantifies the power as N p gears, so the power control problem is to solve the problem of set ={P k / N p ,2P k / N p ,…,(N p -1)P k / N p ,P k} is the terminal u k Select the transmission power, and its allocation scheme can be expressed as Indicates terminal u k Select the nth P The power of the gear is used as the transmission power, and the transmission power is p k,S =n P ·P k / N p , the power distribution result of all sensors is expressed as P = [p1, p2, ..., p K ],

[0044] For satellite IoT terminals, the overall link gain can be expressed as Among them G k Indicates the terminal's transmit antenna gain. represents the satellite receiving antenna gain, where For terminal u k With satellite beam b n The angle between them. is the channel gain, which can be expressed as Among them PL k Represents u k The free space propagation loss, It represents time-varying small-scale fading, including multipath fading and atmospheric attenuation losses caused by factors such as clouds and rain.

[0045] According to the above definition, terminal u k The data transmission signal-to-noise ratio can be expressed as

[0046]

[0047] Among them G k,b represents the link gain between the terminal and the satellite, σ represents the noise power, b k For terminal u k Beam, c k For terminal u k The allocated uplink channel; j is the terminal index, used to communicate with u k Distinguish; To ensure terminal u k The transmission quality, its signal-to-noise ratio should exceed the minimum threshold δ th , namely SINR k ≥δ th Otherwise, no resources will be allocated for this transmission and the terminal request will be rejected. Finally, according to Shannon's formula, the transmission rate is expressed as C k =Blog2(1+SINR k ).

[0048] At time t, if there is a terminal u k If data needs to be transmitted, a joint resource allocation algorithm needs to be run to obtain a channel and power allocation solution. and

[0049] Resource allocation in satellite IoT uplink scenarios requires utilizing limited resources while taking into account the following two objectives: on the one hand, the success probability of IoT terminal data requests needs to be maximized; on the other hand, due to the scarcity of terminal energy, the terminal power utilization needs to be optimized. express Whether the data transmission request is successful depends on whether the signal-to-noise ratio meets the threshold requirement. Terminal u k The power utilization rate can be expressed as in For terminal u k The transmission rate, For terminal u k The uplink joint resource allocation problem can be expressed as:

[0050] opt.

[0051]

[0052] st

[0053]

[0054]

[0055]

[0056]

[0057] The optimization objective in this optimization problem and Respectively represent the maximization of long-term transmission success rate and power utilization; the first constraint indicates that the sum of the total uplink power of all terminals does not exceed the satellite limit. Since the satellite adopts a forwarding mechanism, the total forwarding power limits the sum of the uplink power of all terminals, where represents the sum of all terminal uplink powers; the second constraint limits the sum of all terminal uplink powers in each beam, where Indicates the upper limit of the uplink power of all terminals in each beam, x k,n Indicates the ownership of each terminal to the beam. If x k,n =1 then terminal u k Belongs to beam b n ; In the third constraint, represents the set of terminals that are already transmitting. This constraint indicates that the resources allocated to the newly generated terminal data transmission request at any time should not cause the signal-to-noise ratio of the terminals already in service to fall below the threshold, where For terminal u j The last two constraints on the transmission rate at time t indicate that only one channel and one power level are allocated to the terminal.

[0058] Step 2: Model the joint uplink resource allocation problem at each LEO satellite as an MDP process.

[0059] This paper models the online resource allocation problem as a Markov decision process. First, we introduce the online resource allocation framework. Then, we elaborate on the key aspects of the Markov decision process, including the action space, state information, immediate rewards, and neural network architecture.

[0060] 1. Overall framework

[0061] To address the issue of intermittent transmission requests from large-scale terminals, this paper employs an online resource allocation framework, reframing the channel allocation and power control problems in the optimization model as a sequential decision problem driven by data transmission requests. The specific mechanism involves sequentially allocating resources to the newly generated request sequence at each time slice. Compared to resource allocation mechanisms based on persistent connections, the advantages of online resource allocation in satellite IoT uplinks are as follows:

[0062] (1) IoT data transmission is characterized by short bursts, and the number of terminal nodes served in the system changes dynamically. The online allocation method is highly scalable to changes in the number of terminals.

[0063] (2) It only focuses on the newly generated requests at the current moment and allocates resources to them, while ignoring the massive number of other dormant terminals and not changing the resource allocation of terminals that are transmitting data. In practice, it can effectively reduce the dimension of the action space and address the difficulties in connecting large-scale terminals in the satellite Internet of Things.

[0064] The overall framework for solving this Markov decision problem using deep reinforcement learning is as follows: Figure 2 As shown in the figure, the satellite resource control unit is used as the intelligent agent, and the channel quality, the channel occupancy of the existing terminals in the data transmission state, the transmission power and the geographical location information are used as the dynamic environment to be observed. t Action a that determines the channel and power allocation t , a t Influence the environment to make it s t+1 , and the agent receives immediate benefit r from environmental feedback t , the experience gained from this process t ,a t ,r t ,s t+1} is put into the experience pool for the Q-value network to iteratively learn the strategy of maximizing long-term returns. After the Q-value network converges, the system's long-term power utilization and transmission success rate can be improved.

[0065] 2-state characteristics

[0066] Status information t Contains environmental information related to the current requested channel allocation and power control decision, specifically including the current requested transmission terminal u k Information Allocated resource terminal set Terminal information Channel quality information Csi On , channel allocation matrix W and transmit power matrix P, the state information can be expressed as The satellite resource control unit can sense the channel quality information at time t-1 through the uplink signal of the terminal that is transmitting data. There is no additional communication overhead required to obtain this information.

[0067] In order to improve the efficiency of the neural network's perception of state information, the following steps are taken to convert the above state information into tensor form:

[0068] Step 1: Since interference only occurs between terminals on the same frequency, the terminals are divided into M groups according to the channels.

[0069] Step 2: Channel c m The corresponding set of terminals, according to the current terminal u k Sort by distance and take the top K N Terminals are considered The set of neighboring terminals most relevant to co-channel interference For each neighbor terminal u j , and arrange its related contextual state information into a vector in Represents the neighbor terminal u j and distance, Represents the neighbor terminal u at the previous moment j The transmission power, D k Represents the amount of data that the terminal needs to transmit. Represents the channel quality at the last moment, Represents the transmission rate at the last moment, Indicates the energy utilization rate at the last moment. N The state information of neighbor terminals is arranged into a matrix It contains the channel c at time t m The terminal information, channel quality information and channel and power allocation scheme that are currently transmitting data are collected.

[0070] Step 3: The state information matrix constructed for each channel Arrange as a tensor As the state input of the Q-value neural network.

[0071] 3 Action Space

[0072] Since the online allocation framework allocates resources to newly generated data transmission requests in sequence, each decision focuses only on the terminal currently requesting data transmission, rather than allocating resources to all terminals. Therefore, the action space is the set of possible resource allocation solutions for the current terminal.

[0073] For terminal u k For uplink data transmission, the agent needs to allocate channels and decide the transmission power. This joint decision action a t Can be expressed as a t ={CA t,PC t}, where CA t and PC t They represent the channel allocation action and power control decision at time t respectively. The action space is M×N p , that is, the Q value network outputs M×N p dimensional vector, representing the estimated long-term benefit of each action.

[0074] 4. Immediate benefits

[0075] In satellite IoT scenarios, the fewer IoT terminals a satellite serves, the lower the potential for co-channel interference and the higher the terminal energy efficiency. However, at this point, the system is far from reaching a high load, and the satellite still has a large amount of idle channels and power resources, resulting in a waste of satellite forwarding resources. When the number of terminals served by the satellite reaches a maximum, to utilize limited spectrum resources to carry more terminal transmission requests, the co-channel reuse ratio must be increased, resulting in greater co-channel interference and ultimately lower terminal power utilization.

[0076] In designing immediate benefits t When and energy utilization targets The present invention converts the current request success indicator Change in overall resource utilization caused by the current request The weighted sum of is the benefit, that is Where α1 and α2 are normalization parameters, Represents the current terminal u k The change in resource utilization caused by the resource allocation scheme can be expressed as in Represents terminal u k The energy utilization rate at time t is: For terminal u j Energy utilization rate at time t-1, For terminal u j Energy utilization rate at time t, j is the terminal index, used to compare with u k distinguish; For the terminal u k After assigning services, the terminal group that is transmitting data The sum of the energy utilization rate reduction caused by the same frequency interference. The immediate benefit of this method is r t Equivalent to optimizing the weighted objective function

[0077] 5 Q-value network structure based on Attention module

[0078] like Figure 3As shown in FIG, the present invention uses a convolutional neural network to perceive state features and uses the attention mechanism of the channel attention module (CAM) to enable the Q value network to make more accurate judgments. Figure 3 The network structure shown in the figure takes the input environment information tensor and passes through the two convolutional layers to perceive the features. The CAM module then learns the importance of the feature parts corresponding to each channel, assigns different weights, and extracts more critical and important information. The information features perceived by the high-level layer are then projected into the decision space through the flattening layer and the fully connected layer to obtain the decision vector. Figure 3 As shown in the lower part, after the feature tensor is transformed, transposed, and dot-producted, the self-attention matrix X is obtained, which represents the importance of each channel. The self-attention matrix X is then multiplied with the original tensor matrix to obtain the feature tensor E after weighting the importance of each channel. This matrix serves as the input of the subsequent fully connected layer to make channel and power allocation decisions. This allows the Q-value network to pay more attention to channels that are less occupied by existing sensors and ignore channels that are more occupied and therefore have more severe co-channel interference.

[0079] The core idea of the CAM module employed in this invention is to learn the importance relationship between the feature parts corresponding to each feature channel and assign them different weights. During the state feature construction of this invention, the matrix corresponding to each channel represents the relevant information of the served terminals transmitting data on each channel. Therefore, using this attention module allows the agent to learn the importance relationship between channels. For example, if channel c0 carries a large number of transmitting terminals and has a high transmission power, then the co-channel interference on channel c0 is large, and the channel corresponding to its tensor should be ignored. However, if the co-channel interference on channel c1 is small, allowing the terminal to meet transmission requirements with lower power, then the channel corresponding to channel c1 should be considered an important feature channel and given attention.

[0080] Step 3: Use a centralized training approach to train the online channel selection and power allocation model parameters through meta-reinforcement learning.

[0081] The training process is as follows:

[0082] The online resource allocation method based on deep reinforcement learning in step 2 faces the following two problems in the actual deployment of LEO satellite IoT constellations:

[0083] (1) Training cost. If the DRL method is trained from the very beginning on the satellite, the terminal and the system will have to bear the loss of decision-making errors during the initial observation and exploration of the network, resulting in a waste of system resources. If the training is conducted on the ground, there will be differences between the ground simulation environment and the actual environment on the satellite, which will cause the accuracy of the algorithm to decrease after deployment, affecting the optimization of resource utilization;

[0084] (2) In the actual deployment of large-scale LEO constellations, the computational cost of training models for all LEO satellites is high. In the future, satellites in large-scale LEO constellations will continue to fail and be scrapped, and new satellites will be deployed. The deployment process of new satellites will face the consumption of satellite available service time caused by the interaction and training of intelligent algorithms with the environment. Figure 4 As shown in FIG, the longer the deployment training phase takes, the more system resources are wasted during this phase, which reduces the available service time of the satellite.

[0085] Low learning efficiency is a major challenge facing reinforcement learning, often requiring tens of thousands of training samples for effective convergence. To address this issue, transfer reinforcement learning proposes to leverage useful knowledge from previous tasks to solve new tasks. This allows an agent to quickly learn an optimal strategy in a new target domain, leveraging only the learning achievements from the source domain and using a small amount of available data.

[0086] The meta-learning method is a typical transfer learning method. Its core is to improve the generalization ability of the model in various tasks by learning general knowledge, thereby reducing the number of iterations required for model training and improving training efficiency when initializing meta-learning parameters.

[0087] This paper utilizes a meta-reinforcement learning approach to train initialization parameters with strong generalization capabilities, thereby reducing training overhead during LEO constellation deployment, thereby extending LEO satellite service time and reducing the computational resources required for training. The basic idea is to treat the allocation of local online resources to each LEO satellite as multiple tasks. By averaging the loss functions of each task, the average loss function is used to derive the gradient descent direction of the meta-learning parameters, resulting in meta-learning parameters with strong generalization capabilities across tasks. This avoids optimizing performance for a particular task while degrading performance for other tasks. The specific process is as follows:

[0088] The DRL-CAPC method is trained according to the classic DRL training mechanism, that is, based on the Bellman equation, the estimated value of the Q value is calculated through the target network Q′ Right now The mean square error between the Q-value estimate and the Q-network prediction is where r t is the immediate benefit, γ is the discount factor, s t is the state information at time t, s t+1 is the state information at time t+1, a t is the channel allocation and power control decision at time t, a t+1 It is the channel allocation and power control decision at time t+1.

[0089] The core of meta-reinforcement learning lies in averaging the losses from all first-stage deployed LEO satellites as the loss function for updating meta-parameters. This average loss is used to train the meta-parameter Q-value network via gradient descent, updating the parameters and the target network Q′. The training process is as follows.

[0090] Step 1 Parameter initialization: Let t = 0, initialize the meta-learning value network Q parameters θ and the target network Q′ parameters θ′, and determine the training parameters such as the ε greedy exploration probability, discount factor γ, optimization period T, batch size, number of observations, and network update step size G.

[0091] Step 2 Each LEO satellite collects the terminal's newly generated requests and releases the end transmission resources. The terminal data transmission requests newly generated in [t-1, t) are put into the collection Remove the terminal that ends the transmission And delete its channel occupation. Go to Step 3, otherwise go to Step 6.

[0092] Step 3: Allocation decision. For each terminal request, the state information s is organized according to the characteristics of the state characteristics part. t Organized as tensor As the input of the Q value network, we get the action value function vector Determine action a based on the ε greedy mechanism t , interacting with the environment according to Calculate r t , and sense environmental changes t+1 , this experience t ,a t ,r t ,s t+1}Put it into the experience pool.

[0093] Step 4. Training parameters. Sample a batch of samples from the experience pool according to L(θ)=E[(r j +γmax(Q′(s t+1 ,a t+1 |θ′))-Q(s t ,a t |θ)) 2 ]Calculate the least squares loss, average the losses at all LEO satellites, get the average loss and use it to train the parameters θ of the network Q.

[0094] Step 5: Update the target network. If the update step size G is reached, set θ′ = θ′.

[0095] Step 6: If t=T, end and return the network Q parameter θ; otherwise, set t=t+1 and go to Step 2.

[0096] Step 5: Each LEO satellite performs model migration based on the actual onboard environment and fine-tunes the Q-value network until convergence. The Q-value neural network parameters with strong generalization ability obtained by meta-reinforcement learning need to be migrated to the local environment of the newly deployed LEO satellite. The specific mechanism is as follows:

[0097] Because the parameters of the first few convolutional layers of the Q-value neural network can learn the impact of various state features on resource allocation, this learned meta-knowledge remains valid in the local onboard environment. Therefore, as shown in Algorithm 2, by fixing the first few convolutional layers, the domain knowledge of the resource allocation problem learned by the agent in the simulation environment can be retained. By fine-tuning the final fully connected layer, the Q-value network can quickly adapt to the real onboard environment of the currently deployed satellite's target domain. Leveraging the strong generalization capabilities of the meta-learning parameters trained using the meta-reinforcement learning mechanism in step 3, this Q-value neural network can effectively adapt to the onboard environment of a newly deployed satellite with only a small amount of interaction with the real onboard environment and fine-tuning of the fully connected layers, improving the convergence efficiency of onboard training and reducing the consumption of onboard computing resources.

[0098]

[0099] Step 6: Each LEO satellite uses the deep reinforcement learning model parameters migrated to the local real space environment to perform online channel and power resource joint allocation. After the meta-reinforcement learning parameters in step 5 have been fine-tuned and converged, it means that the model has been migrated to the local area. The LEO satellite can use this Q value network to observe the environment and allocate channels and transmission power to IoT terminals within the coverage area, as shown in Algorithm 3.

[0100]

[0101] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes based on the technical solutions and inventive concepts of the present invention within the technical scope disclosed by the present invention without departing from the spirit and scope of the present invention and should be covered by the scope of protection of the present invention.

Claims

1. A satellite Internet of Things online resource joint allocation method based on meta-reinforcement learning, characterized by The steps include: Step 1: Build an approximate model for channel quality and the environment required for terminal data transmission based on historical data, and establish a joint allocation model for uplink channel and power resources at the LEO satellite; Step 2: Model the joint uplink resource allocation problem at each LEO satellite as an MDP process; Step 3: Use a centralized training approach to train the online channel selection and power allocation model parameters through a meta-reinforcement learning method; The specific process is: The DRL-CAPC method is trained according to the classic DRL training mechanism, that is, based on the Bellman equation, the estimated value of the Q value is calculated through the target network Q′ Right now The mean square error between the Q-value estimate and the Q-network prediction is L(θ)=E[(r t +γmax(Q′(s t+1 ,a t+1 |θ′))-Q(s t ,a t |θ)) 2 ], where r t is the immediate benefit, γ is the discount factor, s t is the state information at time t, s t+1 is the state information at time t+1, a t is the channel allocation and power control decision at time t, a t+1 The channel allocation and power control decision at time t+1; The losses at all the first batch of deployed LEO satellites are averaged and used as the loss function for updating the meta-parameters. The meta-parameter Q value network is trained using the average loss through the gradient descent method, the parameters are updated, and the target network Q′ is updated. The training process is as follows: Step 1 Parameter initialization: Let t = 0, initialize the meta-learning value network Q parameter θ and the target network Q′ parameter θ′, and determine the training parameters: ε greedy exploration probability, discount factor γ, optimization cycle T, batch size, number of observations, and network update step size G; Step 2 Each LEO satellite collects the newly generated terminal requests, releases the end transmission resources, and puts the newly generated terminal data transmission requests in [t-1, t) into the collection Remove the terminal that ends the transmission And delete its channel occupation; if Go to Step 3, otherwise go to Step 6; Step 3: Allocation decision. For each terminal request, the state information s is organized according to the characteristics of the state characteristics part. t Organized as tensor As the input of the Q value network, we get the action value function vector Determine the joint decision action a based on the ε greedy mechanism t , interacting with the environment according to Calculate r t , and sense environmental changes t+1 , this experience t ,a t ,r t ,s t+1 }Put into experience pool; Step 4: training parameters. Sampling a batch of samples from the experience pool according to L(θ)=E[(r j +γmax(Q′(s t+1 ,a t+1 |θ′))-Q(s t ,a t |θ)) 2 ]Calculate the least squares loss, average the losses at all LEO satellites, get the average loss and use it to train the parameters θ of the network Q; Step 5: Update the target network; if the update step size G is reached, set θ′ = θ′; Step 6: If t = T, end and return the network Q parameter θ, otherwise set t = t + 1 and go to Step 2; Step 4: During the actual LEO constellation operation, whenever a newly deployed LEO satellite is launched, the meta-reinforcement learning parameters trained in step 3 are distributed to this LEO satellite; Step 5: Each LEO satellite migrates the model based on the actual onboard environment and fine-tunes the Q value network until convergence; Step 6: Each LEO satellite uses the deep reinforcement learning model parameters migrated to the local real space environment to perform online channel and power resource joint allocation.

2. The satellite Internet of Things online resource joint allocation method based on meta-reinforcement learning according to claim 1 is characterized in that In step 1 above, a joint allocation model of uplink channel and power resources at the LEO satellite is established. The specific process is as follows: Step 1.1 analyzes the satellite IoT scenario and link transmission capabilities, as follows: Each multi-beam LEO satellite provides data transmission services to remote IoT terminal devices within its beam coverage. Let K and N represent the terminal u, k and beam b n The number of terminals and beam sets are represented by and and express; In the satellite physical network resource allocation problem based on LEO constellation, for channel allocation, the data transmission channel set is expressed as And the bandwidth of each channel is B; the channel allocation problem is to select In the example, a suitable channel is selected for the current transmission request sensor, and the allocation result is expressed as w k =[w k,1 ,w k,2 ,…w k,M ] T ,w k,m ∈{0,1}, where w k,m =1 means channel c m Assigned to terminal u k ; The channel allocation results of all terminals in the system are expressed as For power control, let P k Indicates terminal u k The maximum transmission power is quantified as N p gears, so the power control problem is to solve the problem of set ={P k / N p ,2·P k / N p ,…,(N p -1)·P k / N p ,P k } is the terminal u k Select the transmit power, and its allocation scheme is expressed as Indicates terminal u k Select the nth P The power of the gear is used as the transmission power, and the transmission power is p k,S =n P ·P k / N p , the power distribution results of all sensors are expressed as Terminal u k The data transmission signal-to-noise ratio is expressed as Among them G k,b represents the link gain between the terminal and the satellite, σ represents the noise power, b k For terminal u k Beam, c k For terminal u k The allocated uplink channel; j is the terminal index, used to communicate with u k Distinguish; To ensure terminal u k The transmission quality, its signal-to-noise ratio should exceed the minimum threshold δ th , namely SINR k ≥δ th Otherwise, no resources will be allocated for this transmission and the terminal request will be rejected. Finally, according to Shannon's formula, the transmission rate is expressed as C k =Blog2(1+SINR k ); Step 1.2 models the online resource allocation problem as follows: At time t, if there is a terminal u k If data needs to be transmitted, a joint resource allocation algorithm needs to be run to obtain a channel and power allocation solution. and make Indicates terminal u k Whether the data transmission request is successful depends on whether the signal-to-noise ratio meets the threshold requirement. Terminal u k The power utilization rate is expressed as in For terminal u k The transmission rate, For terminal u k The uplink joint resource allocation problem is expressed as: The optimization objective in this optimization problem and Respectively represent the maximization of long-term transmission success rate and power utilization; the first constraint indicates that the sum of the total uplink power of all terminals does not exceed the satellite limit. Since the satellite adopts a forwarding mechanism, the total forwarding power limits the sum of the uplink power of all terminals, where represents the sum of all terminal uplink powers; the second constraint limits the sum of all terminal uplink powers in each beam, where Indicates the upper limit of the uplink power of all terminals in each beam, x k,n Indicates the ownership of each terminal to the beam. If x k,n =1 then terminal u k Belongs to beam b n ; In the third constraint, represents the set of terminals that are already transmitting. This constraint indicates that the resources allocated to the newly generated terminal data transmission request at any time should not cause the signal-to-noise ratio of the terminals already in service to fall below the threshold, where For terminal u j The transmission rate at time t; the last two constraints indicate that only one channel and one power level are allocated to the terminal.

3. The satellite Internet of Things online resource joint allocation method based on meta-reinforcement learning according to claim 2 is characterized in that In step 2 above, the joint uplink resource allocation problem at each LEO satellite is modeled as an MDP process, specifically including the design of state characteristics, action space, immediate benefits, and neural network structure: Step 2.1 Status Information t Contains environmental information related to the current requested channel allocation and power control decision, specifically including the current requested transmission terminal u k Information Allocated resource terminal set Terminal information Channel quality information Csi On , channel allocation matrix W and transmit power matrix P, state information representation The satellite resource control unit senses the channel quality information at time t-1 through the uplink signal of the terminal that is transmitting data. Obtaining this information does not require additional communication overhead; Step 2.2 Action Space: For terminal u k For uplink data transmission, the agent needs to allocate channels and decide the transmission power. This joint decision action a t Represented as a t ={CA t ,PC t }, where CA t and PC t They represent the channel allocation action and power control decision at time t respectively; the action a represented by the one-hot encoding form t M×N p dimensional vector, that is, the Q value network output M×N p dimensional vector, representing the estimated long-term benefit of each action; Step 2.3 Immediate Benefits: Designing Immediate Benefits t When maximizing the long-term transmission success rate, full consideration should be given to the goal of and power utilization targets The current request success indicator Change in overall resource utilization caused by the current request The weighted sum of Where α1 and α2 are normalization parameters, Represents the current terminal u k The resource allocation scheme causes the change of resource utilization, which is expressed as in Represents terminal u k The energy utilization rate at time t is: For terminal u j Energy utilization rate at time t-1, For terminal u j Energy utilization rate at time t, j is the terminal index, used to compare with terminal u k distinguish; For the terminal u k After assigning services, the terminal group that is transmitting data Causes co-frequency interference, resulting in the sum of the energy utilization rate reduction; immediate benefit r t Equivalent to optimizing the weighted objective function Step 2.4 Q-value network structure based on the Attention module: A convolutional neural network is used to perceive state features, and the Q-value network makes more accurate judgments with the help of the attention mechanism of the channel attention module. The channel attention module learns the importance relationship of the feature parts corresponding to each channel and assigns different weights.

4. The satellite Internet of Things online resource joint allocation method based on meta-reinforcement learning according to claim 3 is characterized in that In step 5 above, each LEO satellite performs model migration based on the actual onboard environment and fine-tunes the Q value network until convergence. The specific process is as follows: Copy the trained Q value network Parameter θ; interact with the local real environment, and only fine-tune the last two fully connected layers of the Q value network until convergence is obtained where θ i ′ is the Q value network neuron parameter migrated to the local environment, and the Q value network migrated to the local real environment is obtained.

5. The satellite Internet of Things online resource joint allocation method based on meta-reinforcement learning according to claim 4 is characterized in that In the above step 6, each LEO satellite uses the deep reinforcement learning model parameters migrated to the local real space environment to perform online channel and power resource joint allocation. The process is as follows: According to the feature organization step of the state feature part, the state information is organized into Organized as tensor The tensor features As the input of the Q value network, we get the action value function vector Get the channel allocation and power control decision of this terminal

Images