Traffic shaping method for Internet of Things of low earth orbit satellites

Abstract

The invention relates to a low-orbit satellite internet of things traffic shaping method, and belongs to the technical field of satellite mobile communication. The method comprises the following steps: S1, modeling a low-orbit satellite Internet of Things system; s2, a periodic arrival model and a Poisson arrival model are adopted to simulate periodic traffic and event-driven traffic respectively, and a random network calculation theory is utilized to describe a traffic arrival process; s3, describing the dynamic characteristics of the satellite wireless communication channel by adopting a Gilbert-Elicott model, and describing a channel service process by utilizing a random network calculation theory; s4, establishing a satellite transponder structure model; and S5, modeling a traffic shaping problem into a Markov decision process, solving a Markov decision process optimization problem by adopting a TD3 algorithm, and learning an optimal bandwidth allocation and token generation rate strategy through an intelligent agent. According to the invention, the service quality of different businesses can be ensured while the bandwidth utilization rate is obviously improved.

Description

Technical Field

[0001] The invention belongs to the technical field of satellite mobile communications and relates to a low-orbit satellite Internet of Things traffic shaping method. Background Art

[0002] In recent years, with the rapid development of Internet of Things technology and the continuous improvement of satellite communication technology, satellite Internet of Things is being widely used as an important means of communication. Satellite Internet of Things has the advantages of crossing geographical boundaries, wide coverage, stable and reliable communication, and has become an effective way to connect remote areas and mobile devices. However, compared with its huge potential value, satellite Internet of Things faces a series of challenges that limit its development and application scope in practical applications.

[0003] First, the huge number of IoT devices and the large amount of data transmission, coupled with their distribution in different geographical locations and environmental conditions, have led to serious network congestion problems in satellite IoT. Secondly, due to the diversity of IoT application scenarios, different applications have different requirements for communication. Satellite IoT needs to be able to provide different service quality guarantees according to the needs of different application scenarios to meet the diverse needs of users. Finally, due to limited satellite communication resources, satellite IoT often faces insufficient bandwidth during data transmission, which will limit the data transmission rate and affect the communication quality and user experience. At the same time, due to the particularity of low-orbit satellite IoT, such as the particularity of signal propagation paths, dynamic changes in network topology, and limited satellite resources, ensuring the stability and reliability of communication quality has become a technical challenge.

[0004] Traffic Shaping is a network management technology used to control the rate and distribution of network traffic, solve the congestion problem caused by network traffic bursts, and smooth the peaks and valleys of data transmission to ensure the rational use of network resources and optimize network performance. Through traffic shaping technology to control network traffic, some non-critical business traffic that maliciously seizes bandwidth resources can be restricted in the actual network environment, ensuring the transmission of critical business traffic, ensuring the fairness of bandwidth use, preventing congestion in the entire network, and improving the QoS of the network. Therefore, under the premise that the bandwidth of low-orbit satellite Internet of Things is limited and network congestion is prone to occur, the low-orbit satellite Internet of Things network traffic shaping technology has important research value. Summary of the invention

[0005] In view of this, the object of the present invention is to provide a low-orbit satellite Internet of Things traffic shaping method.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A low-orbit satellite Internet of Things traffic shaping method specifically comprises the following steps:

[0008] S1: Modeling the low-orbit satellite IoT system, including multi-beam satellite access nodes, gateway stations, ground networks, satellite networks, and IoT devices;

[0009] S2: The periodic arrival model and Poisson arrival model are used to simulate periodic traffic and event-driven traffic respectively, and the random network calculus theory is used to describe the traffic arrival process;

[0010] S3: The Gilbert-Elliott model is used to describe the dynamic characteristics of satellite wireless communication channels, and the stochastic network calculus theory is used to describe the channel service process;

[0011] S4: Establish a satellite transponder structure model and set up independent token buckets for different service levels to control traffic output, and use random network calculus theory to derive the upper bound of packet delay to ensure service quality;

[0012] S5: The traffic shaping problem is modeled as a Markov decision process, and the state, action and reward functions are defined. The TD3 algorithm is used to solve the Markov decision process optimization problem, and the optimal bandwidth allocation and token generation rate strategy are learned through the intelligent agent.

[0013] Furthermore, in S1, the LEO-IoT scenario considered in this paper includes LEO multi-beam satellite access nodes, gateways, ground networks, satellite networks, and IoT devices. In the LEO multi-beam satellite system, the entire satellite geographical coverage area is divided into several cells, similar to the ground cellular system, and the satellite uses multiple narrow beams to cover the cells. IoT devices within the coverage of a single beam can transmit data packets through the satellite uplink of the beam, and the data packets of multiple beams converge at the onboard receiver. According to the type of service, the packet classifier diverts data packets of different service types to the corresponding onboard buffer queue, and then forwards them to the satellite network or the ground network through the shaper. Among them, the gateway is the key node connecting the satellite downlink and the ground network.

[0014] Furthermore, S2 specifically includes: using a periodic arrival model and a Poisson arrival model to simulate periodic traffic and event-driven traffic respectively, and using random network calculus theory to describe the traffic arrival process;

[0015] The traffic arrival model targets the two main types of traffic in the low-orbit satellite IoT system: periodic traffic and event-driven traffic. The periodic arrival model and Poisson arrival model are used to simulate them respectively, and the traffic arrival process is described using the random network calculus theory. The random network calculus theory uses the affine arrival curve α(t) = σ(θ) + ρ(θ)·t to describe the packet arrival process, and combines it with the service curve to analyze the network performance. By modeling the arrival process and the service process as a (σ, ρ) constraint model, the upper bound of the packet delay can be derived and the service quality can be evaluated, where σ is the burst tolerance and ρ is the arrival rate.

[0016] The periodic arrival model is suitable for simulating periodic traffic, such as the scenario where sensors collect and upload data periodically. The model assumes that devices generate fixed-size packets at fixed time intervals. For example, assuming that a device generates σ units of data every τ seconds, the arrival process can be described by (σ A ,ρ A ) constraint model. We can get the arrival process (σ A ,ρ A ) Constrain model parameters:

[0017] σ A (θ)=σ,

[0018] The Poisson arrival model is suitable for simulating event-driven traffic, such as data reporting triggered by sensors detecting abnormal conditions. The model assumes that the packet arrival process follows a Poisson process with an average arrival rate of λ. Assuming that the packet size is 1ν bytes, we can get the arrival process (σ A ,ρ A ) Constrain model parameters:

[0019] σ A (θ)=0,

[0020] Furthermore, S3 specifically includes: using the Gilbert-Elliott model to describe the dynamic characteristics of the satellite wireless communication channel, and using the random network calculus theory to describe the channel service process;

[0021] The Gilbert-Elliott model is used to describe the dynamic characteristics of satellite wireless communication channels, and the channel service process is described using the theory of stochastic network calculus. Gilbert-Elliott is a classic Markov chain model used to describe channels with two states, such as a "good" state and a "bad" state. The model simulates the switching of channel states through state transition probabilities. For example, the probability of switching from a "good" state to a "bad" state is p1, and the probability of switching from a "bad" state to a "good" state is p2. The steady-state distribution of channel states can be calculated using the following formula:

[0022] π=[p1 / (p1+p2),p2 / (p1+p2)] T

[0023] When the channel is in a "good" state, it transmits data with the allocated bandwidth b; when the channel is in a "bad" state, it cannot transmit data. The channel service process can be described using the random network calculus theory, and the arrival process (σ S ,ρ S ) Constrain model parameters:

[0024] σ s (θ)=0

[0025]

[0026] Further, in S4, a satellite transponder structure model is established, and independent token buckets are set for different service levels to control traffic output, and the upper bound of data packet delay is derived using random network calculus theory to ensure service quality, which specifically includes the following steps:

[0027] Satellite transponders set up independent token buckets for different service levels to control traffic output. When using a multi-service token bucket shaper to shape multi-service data flows, the outflow rate of each service flow is controlled by the token bucket, and the parameter settings of the token bucket itself determine the rate of the output traffic and the bandwidth occupancy ratio.

[0028] Assume that the total output bandwidth of the satellite transponder (i.e. the total amount of data that the satellite can process and forward) is B out (t), in the time interval [t,t+τ b ], the output data volume transmitted by the on-board transponder is R out (t), where τ b is the minimum working interval of the beam, obviously R out (t)≤B out (t)·τ b The model contains M different service levels (represented by set M), and sets a separate token bucket for each service level. According to the standard, IoT traffic can be divided into four categories: control data traffic, audio, video, and best effort, so let M = 4. The average data transmission rate allowed by token bucket i IR(t) = [ir1(t), ir2(t), ir3(t), ir4(t)] represents that BS i is the capacity of token bucket i, which is used to indicate the maximum burst size of traffic.

[0029] At time t, the length of the queue is L(t) = [l1(t), l2(t), l3(t), l4(t)], then the consumption of tokens in the token bucket is TK(t) = [tk1(t), tk2(t), tk3(t), tk4(t)]. For the i-th token bucket, τ b The number of tokens consumed (or the amount of data transferred) can be expressed as follows:

[0030]

[0031] Then, during this period [t,t+τ b ], the output bandwidth utilization of the satellite transponder can be expressed as During this period, the number of tokens in the token bucket changes as follows:

[0032]

[0033] The forwarder forwards the data packet in a first-in-first-out (FIFO) manner. b ], the length of the i-th queue of the onboard queue changes as follows:

[0034]

[0035] Where label∈{0,1}, label=1 indicates that the service category matches the sequence number of the on-board queue.

[0036] The data packets in each beam go through the service process of the channel separately. Then, the data packets of all beams are gathered at the on-board receiver. Then, the data packets enter the on-board queue through the classifier and reach the hierarchical token bucket traffic shaper. Then, the data packets leave the satellite access node through the on-board transponder. The arrival of data packets in the N beams of the satellite is independent, and the arrival process of data packets in each beam is determined by After the channel service, all packets are aggregated to the onboard traffic shaper. Therefore, the arrival flow of the onboard traffic shaper represents the aggregate flow of all outgoing beam channel service processes. Network calculus theory provides a convenient and efficient model for the aggregate arrival process, and the superposition of m arrival processes is also Constraints, parameters are as follows:

[0037]

[0038] Finally, if the target flow A(s, t) and the serving process S(s, t) are independent, then given a random delay d≥0, the probability of a violation of the target flow at any time t≥0 is as follows:

[0039]

[0040] Furthermore, in S5, the traffic shaping problem is modeled as a Markov decision process, and the state, action and reward functions are defined. The TD3 algorithm is used to solve the Markov decision process optimization problem, and the optimal bandwidth allocation and token generation rate strategy are learned by the intelligent agent. Specifically, the following steps are included:

[0041] S51: Constructing traffic shaping optimization model;

[0042] S52: define state space S, action space A and reward function R;

[0043] S53: Choose a suitable reinforcement learning algorithm, such as the double-delayed deep deterministic policy gradient (TD3) algorithm. The TD3 algorithm has the characteristics of dual learning, noise addition, and delayed update, and can effectively solve the continuous action space and exploration-exploitation balance problems.

[0044] S54: Design the structure of the Actor and Critic networks. The Actor network is responsible for selecting actions based on the current state, and the Critic network is responsible for evaluating the potential value of the state-action pair. Use the TargetNetwork to help update the weights of the Actor and Critic networks to improve training stability.

[0045] S55: Initialize system parameters and replay buffer, perform actions in the environment, and collect experience data. Randomly sample a batch of data from the replay buffer, and use the gradient descent algorithm to update the weights of the behavior network and the evaluation network and update the weights of the target network so that it gradually approaches the weights of the main network. Repeat the above steps until the convergence condition is met or the maximum number of training steps is reached.

[0046] S56: Evaluate the performance of the recursive satellite traffic shaping method, analyze indicators such as bandwidth utilization and delayed satisfaction rate, and verify the optimization effect of the model.

[0047] The beneficial effects of the present invention are as follows: the present invention simultaneously takes into account the different delay requirements of various service levels, the scarcity of on-board peer resources and the uncertainty of satellite export bandwidth caused by the high dynamics of satellite topology, and improves the utilization rate of satellite export bandwidth resources in low-orbit satellite Internet of Things traffic shaping by transforming the optimization problem of resource allocation into a Markov decision process, while ensuring the delay requirements of various services.

[0048] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0050] Figure 1 This is a flow chart of the low-orbit satellite traffic shaping method of the present invention;

[0051] Figure 2 This is a schematic diagram of a low-orbit satellite Internet of Things scenario applicable to the present invention;

[0052] Figure 3 This is a schematic diagram of the structure of a transponder on a low-orbit satellite in the present invention;

[0053] Figure 4 The figure is a flow chart of the low-orbit satellite Internet of Things traffic shaping algorithm in the present invention. DETAILED DESCRIPTION

[0054] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0055] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0056] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0057] See also Figure 1 to Figure 4, the present invention provides a low-orbit satellite Internet of Things traffic shaping method, which specifically includes the following steps:

[0058] S1: Modeling the low-orbit satellite IoT system, including multi-beam satellite access nodes, gateway stations, ground networks, satellite networks, and IoT devices;

[0059] Figure 2 The schematic diagram of the low-orbit satellite IoT scenario applicable to the present invention is shown in FIG. The multi-beam satellite access node divides the satellite coverage area into multiple cells, similar to the terrestrial cellular network. Each cell is covered by a beam, providing uplink connection for IoT devices within the range. The onboard receiver is responsible for receiving data packets from multiple beams and performing preliminary processing. The packet classifier diverts data packets to different onboard buffer queues according to the service type (e.g., control data, audio, video, best effort). The onboard buffer queue stores data packets of different service types, waiting for subsequent processing and forwarding. The shaper performs traffic shaping on the data packets in the onboard buffer queue to ensure that the data packets are transmitted according to the preset rate and burst limit. The satellite network transmits the shaped data packets to the destination, which may be a ground network or other satellite. The ground network is connected to the satellite network to provide IoT devices with the ability to connect to the Internet. IoT devices are distributed in the satellite coverage area and are connected to the Internet through the satellite network for data transmission. As a key node connecting the satellite downlink and the ground network, the gateway station is responsible for receiving, processing and forwarding data packets.

[0060] S2: The periodic arrival model and Poisson arrival model are used to simulate periodic traffic and event-driven traffic respectively, and the random network calculus theory is used to describe the traffic arrival process;

[0061] In the working environment of IoT devices, periodic updates of location or repeated monitoring of characteristic parameters usually lead to periodic business traffic. This type of periodic traffic is generated by the device sending data at fixed time intervals, such as regularly reporting location or environmental parameters. In addition, there is also event-driven traffic. Event-driven traffic refers to the device sending data when a specific event occurs. This type of traffic does not have a fixed time interval, but is determined by the triggering frequency of the event. This type of traffic is usually triggered when it exceeds or falls below the set threshold of the sensor. This means that when the sensor detects an abnormal situation, such as the temperature exceeds the preset range, or the humidity is below a certain critical value, data will be sent immediately. In addition, event-driven traffic also includes information transmitted when necessary emergency maintenance is required. That is, when the device needs emergency maintenance or fails, the device will immediately send a maintenance request or fault report.

[0062] The traffic arrival model targets the two main types of traffic in the low-orbit satellite IoT system: periodic traffic and event-driven traffic. The periodic arrival model and Poisson arrival model are used to simulate them respectively, and the traffic arrival process is described using the random network calculus theory. The random network calculus theory uses the affine arrival curve α(t) = σ(θ) + ρ(θ)·t to describe the packet arrival process, and combines it with the service curve to analyze the network performance. By modeling the arrival process and the service process as a (σ, ρ) constraint model, the upper bound of the packet delay can be derived and the service quality can be evaluated, where σ is the burst tolerance and ρ is the arrival rate.

[0063] The periodic arrival model is suitable for simulating periodic traffic, such as the scenario where sensors collect and upload data periodically. This model assumes that devices generate fixed-size packets at fixed time intervals. Assume that the device τ +nτ, n=0,1,2,...} generates a data volume of σ units. Where τ represents the length of the minimum moment, U τ Uniformly distributed on the interval [0,1], U τ It is known that the continuous periodic arrival process A(t) satisfies the following for any t≥0 and θ≥0:

[0064]

[0065] In the discrete time model, we can assume that events occur at fixed discrete time points, such as t = nτ, where n is an integer. Therefore, time t is replaced by nτ. Assume that at each discrete time step n, the source generates a workload of σ units. Redefining the generating function and rate parameters, here is the formula for the discrete time model:

[0066]

[0067] For such a discrete periodic arrival process, we know:

[0068]

[0069] The right side of the above formula can be further simplified to:

[0070]

[0071] The right side of the above equation is not a function of θ, which means that the random arrival curve obtained from the above equation is is insensitive to the choice of θ when determining the boundary function. This means that α(n) is indeed a deterministic arrival curve for a periodic source. At the same time, A(n) is also subject to (σ A ,ρ A ) constrained arrival process, where

[0072] σ A (θ)=σ,

[0073] Therefore, assuming that the device generates σ units of data every τ seconds, the arrival process can be explained by the random network calculus theory (σ A ,ρ A ) constraint model. We can get the arrival process (σ A ,ρ A ) Constrain model parameters:

[0074] σ A (θ)=σ,

[0075] The Poisson arrival model is suitable for simulating event-driven traffic, such as data reporting triggered by sensors detecting abnormal conditions. The model assumes that the packet arrival process follows a Poisson process. It is generally believed that the counting process of packets arriving at the system is a Poisson process with an average arrival rate of λ. Given a packet size of 1 / ν, the moment generating function of the arrival process is:

[0076]

[0077] When θ≥0, A(t) is subject to (σ A ,ρ A ) constrained arrival process, we can get the arrival process (σ A ,ρ A ) Constrain model parameters:

[0078] σ A (θ)=0,

[0079] S3: The Gilbert-Elliott model is used to describe the dynamic characteristics of satellite wireless communication channels, and the stochastic network calculus theory is used to describe the channel service process;

[0080] Consider a traffic approximation model for an arrival process A(t), which is described by an irreducible Markov chain with two states. The two states of the Markov chain are state 1 (off / bad) and state 2 (on / good). The transitions between states have specific rates, the rate from state 2 to state 1 is p1, and the rate from state 1 to state 2 is p2. In this model, when the Markov chain is in state 1, no arrival events occur, that is, the arrival rate is zero. When the Markov chain is in state 2, the arrival process is a constant rate process. For the special case of an arrival process with a peak rate b, a closed form solution for the spectral radius is provided:

[0081]

[0082] Therefore, the moment generating function of the arrival process is subject to (σ A ,ρ A ) constraints, where for θ≥0:

[0083] σ(θ)=0

[0084]

[0085] The burstiness of the arrival process can be determined by the burstiness parameter T=1 / p1+1 / p2.

[0086] S4: Establish a satellite transponder structure model and set up independent token buckets for different service levels to control traffic output, and use random network calculus theory to derive the upper bound of packet delay to ensure service quality;

[0087] Figure 3 The schematic diagram of the structure of the onboard transponder of the low-orbit satellite in the present invention. The satellite transponder sets independent token buckets for different service levels to control the flow output. When the multi-service token bucket shaper is used to shape the multi-service data flow, the outflow rate of each service flow is controlled by the token bucket, and the parameter setting of the token bucket itself determines the rate of the output flow and the bandwidth occupancy ratio.

[0088] Assume that the total output bandwidth of the satellite transponder (i.e. the total amount of data that the satellite can process and forward) is B out (t), in the time interval [t,t+τ b ], the output data volume transmitted by the on-board transponder is R out (t), where τ b is the minimum working interval of the beam, obviously R out (t)≤B out (t)·τ b The model contains M different service levels (represented by set M), and sets a separate token bucket for each service level. According to the standard, IoT traffic can be divided into four categories: control data traffic, audio, video, and best effort, so let M = 4. The average data transmission rate allowed by token bucket i IR(t) = [ir1(t), ir2(t), ir3(t), ir4(t)] represents that BS i is the capacity of token bucket i, which is used to indicate the maximum burst size of traffic.

[0089] At time t, the length of the queue is L(t) = [l1(t), l2(t), l3(t), l4(t)], then the consumption of tokens in the token bucket is TK(t) = [tk1(t), tk2(t), tk3(t), tk4(t)]. For the i-th token bucket, τ bThe number of tokens consumed (or the amount of data transferred) can be expressed as follows:

[0090]

[0091] Then, during this period [t,t+τ b ], the output bandwidth utilization of the satellite transponder can be expressed as During this period, the number of tokens in the token bucket changes as follows:

[0092]

[0093] The forwarder forwards the data packet in a first-in-first-out (FIFO) manner. b ], the length of the i-th queue of the onboard queue changes as follows:

[0094]

[0095] Where label∈{0,1}, label=1 indicates that the service category matches the sequence number of the on-board queue.

[0096] The data packets in each beam go through the service process of the channel separately. Then, the data packets of all beams are gathered at the on-board receiver. Then, the data packets enter the on-board queue through the classifier and reach the hierarchical token bucket traffic shaper. Then, the data packets leave the satellite access node through the on-board transponder. The arrival of data packets in the N beams of the satellite is independent, and the arrival process of data packets in each beam is determined by After the channel service, all packets are aggregated to the onboard traffic shaper. Therefore, the arrival flow of the onboard traffic shaper represents the aggregate flow of all outgoing beam channel service processes. Network calculus theory provides a convenient and efficient model for the aggregate arrival process, and the superposition of m arrival processes is also Constraints, parameters are as follows:

[0097]

[0098] Finally, if the target flow A(s, t) and the serving process S(s, t) are independent, then given a random delay d≥0, the probability of a violation of the target flow at any time t≥0 is as follows:

[0099]

[0100] S5: The traffic shaping problem is modeled as a Markov decision process, and the state, action and reward functions are defined. The TD3 algorithm is used to solve the Markov decision process optimization problem, and the optimal bandwidth allocation and token generation rate strategy are learned through the intelligent agent.

[0101] In the on-board traffic shaper, it is crucial to make token allocation and uplink bandwidth allocation decisions. It consists of two main aspects, including controlling the number of uplink packets and weighted forwarding of queue data flows. An optimization problem is constructed in which the on-board traffic shaper acts as an intelligent agent responsible for making bandwidth and token allocation decisions {B(t), IR(t)}, t∈T. Therefore, the optimization problem can be formulated as follows:

[0102]

[0103] The goal of this problem is to maximize the maximum utilization of the onboard export bandwidth by adjusting {B(t), IR(t)}, t∈T. The constraints express the preferences of the traffic shaper as follows. In constraint C1, the allocated beam bandwidth b i The sum of (t) does not exceed the total uplink bandwidth B up In constraint C2, the number of packets forwarded by the token bucket does not exceed the on-board forwarding capacity R out (t) (output bandwidth). Constraint C3 indicates that the queue length does not exceed the maximum queue length L max The delay of the ith queue in constraint C4 does not exceed the maximum delay tolerance d i .

[0104] S4 describes the probability upper bound of the target flow A(s, t) violating the constraint at any time t≥0 given a random delay d≥0. However, this probability upper bound is an uncertain condition. In order to integrate it into the constraint condition C4, the probability upper bound needs to be converted into a definite constraint condition. One method is to set the probability upper bound to be less than a certain threshold ε (for example, ε=0.01), that is, The optimization problem P can be modified to P1:

[0105]

[0106] Markov Decision Process (MDP) is a mathematical framework for describing dynamic systems with randomness and decision-making properties. It is based on Markov chain theory and describes the probability of transitioning to another state from the current state. A Markov decision process is defined as a tuple (S, A, T, R, S θ ,γ), where S is the state set, A is the action set, T is the transition probability set, R is the reward signal, S θ is the initial state distribution, γ∈[0,1] is the discount factor.

[0107] State s(t): The system state should include the current beam channel state H(t), the on-board queue length L(t), the number of tokens in the token bucket TK(t) and the output bandwidth Bout (t). Therefore, the state s(t) of the MDP can be expressed as:

[0108] s(t)=(H(t),L(t),TK(t),B out (t))

[0109] Action a(t): After observing the environment, the agent determines the action a(t) to be performed in the state by obtaining the corresponding state s(t). According to the bandwidth utilization maximization problem defined, the decision combination should be a set of uplink bandwidth allocation B(t) and token generation rate allocation IR(t) that satisfies constraints C1 to C4. Therefore, action a(t) can be defined as:

[0110] a(t)=(B(t),IR(t))

[0111] Reward r(t): The reward r(t) is the feedback that the agent receives immediately after taking action a(t). For the defined bandwidth utilization maximization problem, the reward r(t) can be expressed as:

[0112]

[0113] Both s(t) and a(t) contain continuous variables, so the state space and action space of the above MDP are infinite. The TD3 algorithm can solve the infinite state-action space problem by separating the action selection and action evaluation processes. TD3 uses two independent evaluation networks Q1 and Q2 to estimate the Q value and correct the overestimation of the Q value in DDPG, while DDPG has only one behavior network and one evaluation network. The main behavior network is a policy-based network responsible for action selection, denoted as a=π(s|φ), where φ represents the weight of the main behavior network. The main evaluation network is a value function-based network responsible for evaluating the potential value of state-action pairs, denoted as Q(s,a|θ i ) i＝1,2 , where θ i represents the weight of the i-th main evaluation network. Both the main behavior network and the main evaluation network have corresponding target networks to help weight update: the target behavior network has weight φ′ and the target evaluation network has weight θ i ′.

[0114] Figure 4 The figure is a flow chart of the traffic shaping algorithm for low-orbit satellite IoT in the present invention. During the training process, the weights of each network will undergo multiple updates. First, before each weight update, a batch of data (s, a, r, s′) of size N is randomly sampled from the replay buffer to update the evaluation network and the behavior network. The main evaluation network is updated by minimizing the error between the evaluation value and the target value. Its loss function is expressed as:

[0115]

[0116] Use a gradient descent algorithm (such as the Adam optimizer) to update the evaluation network parameters:

[0117]

[0118] Where η is the learning rate. Each time the weight θ is updated i Previously, the minimum value of Q1 and Q2 was selected as the estimated value, and then the y value was calculated using this value. Then, the weight update direction of the main behavior network is to obtain the action selection policy with the maximum performance target and the maximum expected evaluation network value. The updated main evaluation network Q1 or Q2 is used to calculate the gradient of the policy and update the parameters of the main behavior network.

[0119]

[0120] The gradient descent algorithm is also used to update the main behavioral network parameters:

[0121]

[0122] Finally, in order to stabilize the training process, the Polyak averaging method is used to update the target network weights so that its parameters gradually approach the parameters of the main network:

[0123] θ′ i ←τ Q θ i +(1-τ Q )θ i

[0124] φ′←τ π φ+(1-τ π )φ′

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A low-orbit satellite Internet of Things traffic shaping method, characterized by: Includes the following step: S1: Modeling the low-orbit satellite IoT system, including multi-beam satellite access nodes, gateway stations, ground networks, satellite networks, and IoT devices; S2: The periodic arrival model and Poisson arrival model are used to simulate periodic traffic and event-driven traffic respectively, and the random network calculus theory is used to describe the traffic arrival process; S3: The Gilbert-Elliott model is used to describe the dynamic characteristics of satellite wireless communication channels, and the stochastic network calculus theory is used to describe the channel service process; S4: Establish a satellite transponder structure model and set up independent token buckets for different service levels to control traffic output, and use random network calculus theory to derive the upper bound of packet delay to ensure service quality; S5: The traffic shaping problem is modeled as a Markov decision process, and the state, action and reward functions are defined. The TD3 algorithm is used to solve the Markov decision process optimization problem, and the optimal bandwidth allocation and token generation rate strategy are learned through the intelligent agent.

2. A low-orbit satellite Internet of Things traffic shaping method according to claim 1, characterized in that: In S1, the constructed low-orbit satellite Internet of Things scenario includes multi-beam satellite access nodes, gateway stations, ground networks, satellite networks and Internet of Things devices.

3. A low-orbit satellite Internet of Things traffic shaping method according to claim 1, characterized in that: In S2, the periodic arrival model is suitable for simulating periodic traffic, including scenarios where sensors periodically collect and upload data; the Poisson arrival model is suitable for simulating event-driven traffic, including data reporting triggered when sensors detect abnormal conditions.

4. A low-orbit satellite Internet of Things traffic shaping method according to claim 1, characterized in that: The S3 specifically includes: The Gilbert-Elliot model is used to describe a channel with two states, including a "good" state and a "bad" state, and simulates the switching of channel states through state transition probabilities.

5. A low-orbit satellite Internet of Things traffic shaping method according to claim 1, characterized in that: The S4 specifically includes: The data packets in each beam go through the service process of the channel separately; Data packets from all beams are aggregated to the onboard receiver; The data packet enters the on-board queue through the classifier and reaches the hierarchical token bucket traffic shaper; The data packet leaves the satellite access node through the onboard transponder; the arrival of data packets in the N satellite beams is independent, and the arrival process of data packets in each beam is determined by = represents; after the channel service, all data packets are aggregated to the on-board traffic shaper; the arrival flow of the on-board traffic shaper represents the aggregate flow of all outgoing beam channel service processes; the superposition of m arrival processes is Constraints, parameters are as follows: If the target flow A(s, t) and the service process S(s, t) are independent, given a random delay d ≥ 0, the probability of violation of the target flow at any time t ≥ 0 is as follows:

6. A low-orbit satellite Internet of Things traffic shaping method according to claim 1, characterized in that: In S5, constructing the traffic shaping problem optimization model specifically includes the following steps: In the on-board traffic shaper, the number of uplink packets and the weighted forwarding of queue data flows are controlled. An optimization problem is constructed in which the on-board traffic shaper acts as an intelligent agent responsible for making bandwidth and token allocation decisions {B(t), IR(t)}, t∈T. The optimization problem is expressed as: The goal of this problem is to maximize the maximum utilization of the onboard export bandwidth by adjusting {B(t), IR(t)}, t∈T; the constraints express the preferences of the traffic shaper. In constraint C1, the allocated beam bandwidth b i The sum of (t) does not exceed the total uplink bandwidth B up ; In constraint C2, the number of packets forwarded by the token bucket does not exceed the on-board forwarding capacity R out (t), that is, the output bandwidth; constraint C3 means that the length of the queue does not exceed the maximum queue length L max ; The delay of the i-th queue in constraint C4 does not exceed the maximum delay tolerance d i .

7. A low-orbit satellite Internet of Things traffic shaping method according to claim 1, characterized in that: In S5, the traffic shaping problem is modeled as a Markov decision process, the state includes the current beam channel state, the on-board queue length, the number of tokens in the token bucket and the output bandwidth, the action includes uplink bandwidth allocation and token generation rate allocation, and the reward is bandwidth utilization.

8. A low-orbit satellite Internet of Things traffic shaping method according to claim 1, characterized in that: In S5, the TD3 algorithm is used to solve the Markov decision process optimization problem, including defining the state space, action space and reward function, designing the behavior network and evaluation network, and using the target network to help update the network weights.

9. A low-orbit satellite Internet of Things traffic shaping method according to claim 1, characterized in that: In S5, the intelligent agent learns the optimal bandwidth allocation and token generation rate strategy through a reinforcement learning algorithm to maximize the on-board export bandwidth utilization.

10. A low-orbit satellite Internet of Things traffic shaping method according to claim 1, characterized in that: In S5, the probability constraint condition is converted into a definite constraint condition to control the queue delay not to exceed the maximum delay tolerance.

Images