LEO internet of things random access method based on deep reinforcement learning

Abstract

The application provides a LEO Internet of Things random access method based on deep reinforcement learning, comprising the following steps: step S1, setting the device distribution, power level and optional degree set of the kth beam in the LEO Internet of Things; step S2, building a deep deterministic policy gradient network; step S3, the LEO satellite moves to the kth beam, and the user covered by the beam selects a degree from the optional degree set to perform random access according to the current state through an online Actor network; step S4, designing a two-step iterative SIC receiver at the receiving end, and the receiver comprises intra-slot SIC and inter-slot SIC; and step S5, randomly sampling experience data from an experience replay pool to train and update the deep deterministic policy gradient network. The application can help the satellite to adjust the access protocol according to the instantaneous overload condition, so as to improve the overall average throughput.

Description

Technical Field

[0001] This invention relates to a LEO (Lead-Oriented Internet of Things) random access method based on deep reinforcement learning. Background Technology

[0002] With the rapid growth in the scale of devices in large-scale machine-type communication scenarios, supporting various IoT devices has become one of the main tasks of future wireless communication systems. Therefore, the demand for integrated satellite-terrestrial networks is increasing, and Low Earth Orbit (LEO) satellites are considered a significant extension of terrestrial networks to achieve wider coverage. However, with the movement of LEO satellites and the uneven distribution of large-scale terrestrial devices in real-world scenarios, the number of devices covered by LEO satellites will vary greatly. Furthermore, the orbits of LEO satellites are irreversible, meaning that if a device cannot access an LEO satellite, it is difficult for it to regain contact with that satellite in a short period. Therefore, designing an effective SA (Standalone) protocol to adjust the access strategy based on transient overload is crucial to ensuring a high probability of successful access. Summary of the Invention

[0003] This invention proposes a LEO (Low-Oriented Oriented Satellite) IoT random access method based on deep reinforcement learning, which can help satellites adjust their access protocols according to instantaneous overload conditions, thereby improving the overall average throughput.

[0004] This invention is achieved through the following technical solution:

[0005] The LEO (Local Internet of Things) random access method based on deep reinforcement learning includes the following steps:

[0006] Step S1: Set the device distribution of the k-th beam in the LEO IoT, classify the devices in the k-th beam into L sets and configure different power levels, and set the selectability set of devices in each LEO satellite beam. The device with the lth power level is a type l device;

[0007] Step S2: Construct a deep deterministic policy gradient network, which includes a target Actor network, an online Actor network, a target Critic network, and an online Critic network;

[0008] Step S3: The LEO satellite moves to the k-th beam. Users covered by this beam randomly access the network by selecting a degree from the set of possible degrees based on their current status through the online Actor network.

[0009] Step S4: Design a two-step iterative SIC receiver at the receiving end. This receiver includes intra-slot SIC and inter-slot SIC. The decoding steps include:

[0010] Step S41: Within a time slot, SIC utilizes power diversity to recover colliding device data packets within a single time slot;

[0011] Step S42: For type l device in the k-th beam, if the device does not transmit in the subframe corresponding to the (k-1)-th beam, then the device's other... A copy of a data packet is eliminated from the corresponding time slot in the subframe corresponding to the k-th beam through inter-slot SIC. If the device transmits a data packet in the subframe corresponding to the (k-1)-th beam and the data packet is not successfully decoded, then the copy of the data packet in the subframe corresponding to the (k-2)-th beam is also eliminated through inter-slot SIC. The received data from multiple subframes is jointly decoded through coupling.

[0012] Step S43: Iterate through steps S41 and S42 multiple times until there are no data packets that can be successfully decoded in the SIC phase within the time slot, thus completing the data decoding. The number of users who successfully accessed each type of device is weighted and used as the throughput.

[0013] Step S44: Store the current state of the k-th beam, the selected degree, the current throughput, and the state of the next beam as experience data in the experience playback pool;

[0014] Step S5: Randomly sample empirical data from the empirical replay pool to train and update the deep deterministic policy gradient network;

[0015] Where L, l, and k are all integers, and l = 1, 2, ..., L.

[0016] Furthermore, in step S1, the devices are distributed as follows: in, This refers to the device representing the Lth set within the k-th beam. This represents the number of devices in the Lth set within the kth beam. This refers to a device that shares the Lth set with the k-th beam and the (k-1)-th beam. This refers to the device that shares the Lth set with the kth beam and the (k+1)th beam.

[0017] Furthermore, in step S1, the target signal to interference noise ratios Γ1,Γ2,…,Γ are respectively set for the L device sets of the k-th beam. L Then the power rating of type l equipment is Where N0 is the noise power at the base station.

[0018] Furthermore, in the deep deterministic policy gradient network of step S2, the target Actor network and the online Actor network are three-layer fully connected neural networks with the same structure, and the target Critic network and the online Critic network are two-layer fully connected neural networks with the same structure.

[0019] Furthermore, step S41 specifically involves: if the signal-to-interference-noise ratio decoded by the type l device is greater than the target signal-to-interference-noise ratio Γ l If so, the data packets of type l devices in that time slot can be successfully decoded, the power level of the data packets is detected sequentially until multiple data packets with the same power are found, and the data packets that have not collided are immediately recovered.

[0020] Furthermore, in step S2, the target Actor network is represented as μ′(s|φ μ′ The online Actor network is represented as μ(s|φ). μ The target Critic network is represented as Q′(s,a′|φ). Q′ The online Critic network is represented as Q(s,a|φ). Q ), φ μ′ φ μ φ Q′ φ Q The weight parameters of each network are represented sequentially. The input states of the target Critic network and the online Critic network are defined as follows: in The residual matrix of the forward V-1 frames, For the new transmission matrix of the current beam, Given the device distribution of the current beam, the action space is defined as the set of selectivity, and the network reward is set as a weighted value of the average number of devices of each type successfully accessing each time slot in the beam.

[0021] Furthermore, step S5 specifically includes:

[0022] Step S51: In the online Critic network, by inputting the current state s i and current action a i To obtain the action value function Q, and then use the action value function Q to calculate the loss function. And based on the standard backpropagation method, the following is obtained: For φ Q gradient And it is optimized and updated to obtain the updated φ Q ,in, To find the mean squared error, y i The target value for the Critic network;

[0023] Step S52: The policy of the online Critic network is updated according to the deterministic policy gradient to obtain the updated φ. μ ;

[0024] Step S53, obtain φ Q and φ μ Then, the parameters of the target Actor network are updated to φ. Q′ ←τφ Q +(1-τ)φ Q′ The parameters of the target Critic network are updated to φ μ′ ←τφ μ +(1-τ)φ μ′ , where τ are the update parameters for the target Actor network and the target Critic network.

[0025] The present invention has the following beneficial effects:

[0026] 1. This invention relies on reinforcement learning algorithms to introduce power diversity among devices and assign separate transmit power levels to different sets of devices. At the transmission end, a random access optimization strategy based on the DDPG network is designed to adjust the degree distribution of different types of user devices in real time. At the receiving end, a two-step iterative SIC receiver is designed to jointly decode the received data from multiple subframes through coupling. This enables real-time adjustment of the degree distribution settings of different types of devices based on LEO satellite IoT access overload, thereby adjusting the RA protocol and quickly adapting to drastic changes in the number of access users. This makes the system more intelligent and effectively utilizes the power diversity of heterogeneous devices to optimize the degree distribution of multi-power-level heterogeneous devices, achieving better throughput performance. Attached Figure Description

[0027] The present invention will now be described in further detail with reference to the accompanying drawings.

[0028] Figure 1 This is a flowchart of the present invention.

[0029] Figure 2 This is a schematic diagram of the two-step iterative SIC receiver of the present invention.

[0030] Figure 3 This is a schematic diagram of the deep deterministic policy gradient network of the present invention.

[0031] Figure 4 This is a performance simulation diagram of the overall average throughput of the present invention.

[0032] Figure 5 This is a performance simulation diagram illustrating the success probability of various types of devices accessing the device according to the present invention.

[0033] Figure 6This is a simulation diagram comparing the average throughput of the present invention and existing technologies under different device densities. Detailed Implementation

[0034] The LEO IoT access method based on deep reinforcement learning relies on an unlicensed LEO satellite IoT system, where LEO satellites act as base stations, providing services to a large number of devices. It is assumed that synchronization between the satellite and ground devices is guaranteed through a hemispherical information-assisted algorithm. LEO satellites provide services to devices using multiple beams, and the service order of these beams is related to the satellite's trajectory. Different beams employ time-division multiplexing, where each beam is allocated one subframe for device data packet transmission. Two beams with adjacent sequence numbers share a portion of the area, and their allocated subframes are consecutive but do not overlap; these two beams are called adjacent beams. In this embodiment, each beam contains three types of devices, with an average total of 70. Each beam is allocated 50 time slots, assuming that the number of devices with the lowest transmission power is the largest, consistent with the common device distribution in real communication systems. Adjacent beams share approximately 30% of the area of ​​a single beam. To optimize the degree distribution for each type of device, the maximum selectivity is set to 8.

[0035] The LEO IoT access method based on deep reinforcement learning includes the following steps:

[0036] Step S1: Set the device distribution of the k-th beam of the LEO satellite in the LEO IoT, classify the devices in the k-th beam into L sets and configure different power levels, and set the selectability set of devices in each beam of the LEO satellite. The device with the lth power level is a type l device;

[0037] Specifically, the equipment is distributed as follows: in, This refers to the device representing the Lth set within the k-th beam. This refers to a device that shares the Lth set with the k-th beam and the (k-1)-th beam. This refers to the device that shares the Lth set with the kth beam and the (k+1)th beam; This represents the number of devices in the Lth set within the kth beam. This represents the number of devices sharing the Lth set with the k-th beam and the (k-1)-th beam. This represents the number of devices sharing the Lth set with the kth beam and the (k+1)th beam; in this embodiment, L = 3. The mean is 70;

[0038] If the L sets of the k-th beam are respectively assigned corresponding target signal to interference noise ratios Γ1, Γ2, Γ3, then the power level of type l equipment is: They are classified into Type 1, Type 2, and Type 3 devices according to their power levels from highest to lowest, where N0 is the noise power at the base station.

[0039] Step S2: Construct a Deep Deterministic Policy Gradient Network (DDPG network), such as... Figure 3 As shown, the network includes a target Actor network μ′(s|φ) μ′ ), online Actor network μ(s|φ μ ), Target Critic Network Q′(s,a′|φ Q′ ) and the online Critic network Q(s,a|φ Q ), the target Actor network μ′(s|φ μ′ ) and online Actor network μ(s|φ μ The target Critic network Q′(s,a′|φ) is a three-layer fully connected neural network with the same structure. Q′ ) and the online Critic network Q(s,a|φ Q ) are two-layer fully connected neural networks with the same structure, φ μ′ φ μ φ Q′ φ Q The weight parameters of each network are represented sequentially; the input states of the target Critic network and the online Critic network are defined as follows: in This is the residual matrix of the previous V-1 frames from the current frame. For the new transmission matrix of the current beam, Let a′ represent the device distribution of the current beam, and a′ represent the target Actor network μ′(s|φ). μ′ The action selected, denoted as 'a', is the online Actor network μ(s|φ). μ The selected action, the action space is defined as the set of selectability [1,2,8], and the network reward is set as the weighted value of the number of each type of device successfully accessed per time slot in the beam. This weighted value is set based on experience. In this embodiment, the weighted value ratio of type 1 devices to type 3 devices is [2.2:1.53:1].

[0040] Step S3: The LEO satellite moves to the k-th beam. Users covered by this beam randomly access the network by selecting a degree from the set of possible degrees through the online Actor network according to their current state. The state consists of three parts: the first part is the residual matrix of the forward v-1 beam stored in the receiver, the second part is the new transmission matrix of the current beam, and the last part is the device distribution vector. The current state is the state within the k-th beam time.

[0041] Step S4: Design a two-step iterative SiC receiver at the receiving end, such as... Figure 2 As shown, the receiver includes intra-slot SIC and inter-slot SIC, and the decoding steps include:

[0042] Step S41: Within a time slot, SIC utilizes power diversity to recover colliding device data packets within a single time slot; specifically, if the signal-to-interference-to-noise ratio decoded by type l device is greater than the set target signal-to-interference-to-noise ratio Γ... l If so, the data packets of type l devices in that time slot can be successfully decoded, the power level of the data packets is detected sequentially until multiple data packets with the same power are found, and the non-collision data packets are immediately recovered;

[0043] Step S42: For type l device in the k-th beam, if the device does not transmit in the subframe corresponding to the (k-1)-th beam, then the device's other... A copy of a data packet is eliminated from the corresponding time slot in the subframe corresponding to the k-th beam through inter-slot SIC. If the device transmits a data packet in the subframe corresponding to the (k-1)-th beam and the data packet is not successfully decoded, then the copy of the data packet in the subframe corresponding to the (k-2)-th beam is also eliminated through inter-slot SIC. The received data from multiple subframes is jointly decoded through coupling.

[0044] Step S43: Iterate through steps S41 and S42 multiple times until there are no data packets that can be successfully decoded in the SIC phase within the time slot, thus completing the data decoding. The number of users who successfully accessed the device for each type of device is weighted and used as the throughput. Successful decoding of user data is defined as successful access for that user.

[0045] Step S44: Store the current state of the k-th beam, the selected degree, the current throughput, and the state of the next beam as experience data in the experience playback pool;

[0046] Step S5: Randomly sample empirical data of batch size 32 from the empirical replay pool to train and update the deep deterministic policy gradient network, specifically as follows:

[0047] Step S51: In the online Critic network, by inputting the current state s i and current action ai To obtain the action value function Q, and then use the action value function Q to calculate the loss function. And based on the standard backpropagation method, the following is obtained: For φ Q gradient And it is optimized and updated to obtain the updated φ Q ,in, To find the mean squared error, y i The target value for the Critic network;

[0048] Step S52: The policy of the online Critic network is updated according to the deterministic policy gradient to obtain the updated φ. μ ;

[0049] Step S53, obtain φ Q and φ μ Then, the parameters of the target Actor network are updated to φ. Q′ ←τφ Q +(1-τ)φ Q′ The parameters of the target Critic network are updated to φ μ′ ←τφ μ +(1-τ)φ μ′ Where “←” indicates update, and τ is the update parameter of the target Actor network and the target Critic network. In this embodiment, τ = 0.001; where L, l, and k are all integers, and l = 1, 2, ..., L.

[0050] Figure 4 This is a performance simulation diagram of the overall average throughput. The value for each point is obtained by counting and normalizing the throughput of 200 beams. It can be seen that during the training phase from 0 to 50 rounds, the probability of successful device access increases slowly, while in the phase from 50 to 100 rounds, the system's average throughput performance is significantly improved. Figure 5 The performance simulation diagram for the access success probability of each type of device is shown. It can be seen that the three types of devices have a higher access success probability after the training phase, and the access success probability of these three types of devices tends to stabilize around the 600th round. Figure 6 This is a simulation diagram comparing the average throughput of the present invention (NOMA-IRSA) with existing technologies (DRL-NSA, fixed degree 2 scheme, fixed degree 3 scheme, and random degree scheme) under different device densities. It can be seen that the present invention has significant performance gains under different device densities. Moreover, when the device density is greater than 95, the present invention can maintain better throughput performance compared to the low efficiency of other schemes, and can also more effectively cope with high device load scenarios.

[0051] The above description is merely a preferred embodiment of the present invention and should not be construed as limiting the scope of the present invention. All equivalent changes and modifications made in accordance with the scope of the patent application and the contents of the specification of the present invention should still fall within the scope of the patent of the present invention.

Claims

1. A LEO (Lead-Oriented Object) IoT random access method based on deep reinforcement learning, characterized in that: Includes the following steps: Step S1: Configure the LEO IoT system. The equipment distribution of the beam, and the beam The equipment for each beam is classified as follows: Each set is configured with different power levels, and the selectability set of devices in each beam of the LEO satellite is set. , No. For devices of each power level Type of equipment; Step S2: Construct a deep deterministic policy gradient network, which includes a target Actor network, an online Actor network, a target Critic network, and an online Critic network; Step S3, LEO satellite moves to the... A beam, and users covered by the beam randomly access the network by selecting a degree from the set of options based on their current state through the online Actor network; Step S4: Design a two-step iterative SIC receiver at the receiving end. This receiver includes intra-slot SIC and inter-slot SIC. The decoding steps include: Step S41: Within a time slot, SIC utilizes power diversity to recover colliding device data packets within a single time slot; Step S42, for the first In each beam Type of device, if the device is not in the first If the signal is transmitted in a subframe corresponding to a beam, then the other beams of the device... The packet copy, transmitted from the SIC between time slots, is... Eliminate in the corresponding time slot of the subframe corresponding to each beam, if the device in the If a data packet is transmitted in the subframe corresponding to a beam and that data packet is not successfully decoded, then the device in the [number]th beam... The data packet copies in the subframe corresponding to each beam are also eliminated through inter-slot SIC, and the received data from multiple subframes are jointly decoded through coupling. Step S43: Iterate through steps S41 and S42 multiple times until there are no data packets that can be successfully decoded in the SIC phase within the time slot, thus completing the data decoding. The number of users who successfully accessed each type of device is weighted and used as the throughput. Step S44, the first The current state of each beam, the selected degree, the current throughput, and the state of the next beam are stored as experience data in the experience playback pool; Step S5: Randomly sample empirical data from the empirical replay pool to train and update the deep deterministic policy gradient network; in, , , All are integers. ; In step S1, the devices are distributed as follows: ,in, , , , Indicates the first The first of the beams A collection of devices, Indicates the first The first of the beams The number of devices in a set Indicates the first The beam and the first The first beam shared A collection of devices, Indicates the first The beam and the first The first beam shared A collection of devices.

2. The LEO IoT random access method based on deep reinforcement learning according to claim 1, characterized in that: In step S1, for the first beam Each device set is configured with a corresponding target signal and interference-to-noise ratio. ,but The power rating of this type of equipment is ,in, This represents the noise power at the base station.

3. The LEO IoT random access method based on deep reinforcement learning according to claim 1 or 2, characterized in that: In the deep deterministic policy gradient network of step S2, the target Actor network and the online Actor network are three-layer fully connected neural networks with the same structure, and the target Critic network and the online Critic network are two-layer fully connected neural networks with the same structure.

4. The LEO IoT random access method based on deep reinforcement learning according to claim 1 or 2, characterized in that: Specifically, step S41 involves: if The signal-to-interference-to-noise ratio of the device decoding type is greater than the target signal-to-interference-to-noise ratio. Then in that time slot Data packets from the device of this type can be successfully decoded. The power level of the data packets is checked sequentially until multiple data packets with the same power are found, and the uncollision data packets are immediately recovered.

5. The LEO IoT random access method based on deep reinforcement learning according to claim 1 or 2, characterized in that: In step S2, the target Actor network is represented as follows: Online Actor networks are represented as The target Critic network is represented as The online Critic network is represented as , , , , The weight parameters of each network are represented sequentially. The input states of the target Critic network and the online Critic network are defined as follows: ,in Forward The residual matrix of the frame, For the new transmission matrix of the current beam, Given the device distribution of the current beam, the action space is defined as the set of selectivity, and the network reward is set as a weighted value of the average number of devices of each type successfully accessing each time slot in the beam.

6. The LEO IoT random access method based on deep reinforcement learning according to claim 5, characterized in that: Step S5 specifically includes: Step S51: In the online Critic network, input the current status. and current action To obtain the action value function Using action value function Calculate the loss function And based on the standard backpropagation method, obtain against gradient And optimize and update it to obtain the updated version. ,in, , To find the mean square error, The target value for the Critic network; Step S52: The policy of the online Critic network is updated according to the deterministic policy gradient to obtain the updated policy. ; Step S53, Obtain and Then, the parameters of the target Actor network are updated to... The parameters of the target Critic network are updated to ,in, Update the parameters for the target Actor network and the target Critic network.

Images