Wireless network time slot allocation method based on near-end strategy optimization

Abstract

According to the wireless network time slot allocation method based on near-end strategy optimization, the bandwidth, the signal-to-noise ratio, the service load, the packet loss rate and the bit error rate of each node are collected to serve as parameters of reinforcement learning, and reasonable allocation of time slots is guaranteed. The method comprises the following steps of: firstly, comprehensively considering the characteristics of bandwidth, signal-to-noise ratio, packet loss rate and bit error rate, modeling networking and link establishment time slot allocation as a Markov decision process, then, setting a reward function by taking maximization of throughput rate and maximum propagation delay as optimization targets, and finally, solving the problem of networking and link establishment time slot allocation by adopting a near-end strategy optimization method. According to the method, the time slots of each node are reasonably distributed, so that the performance indexes such as time delay and throughput of the network are improved while the reliable transmission of data is ensured.

Description

Technical Field

[0001] This invention belongs to the field of wireless networking, specifically a wireless network time slot allocation method based on near-end strategy optimization. Background Technology

[0002] Maritime wireless networks require efficient time slot allocation to achieve low latency and high reliability. When a node's transmission bandwidth is fixed, increased service load coupled with insufficient allocated time slots can lead to data queuing and increased latency. Continued increases in service load can even cause buffer overflows, resulting in packet loss. Conversely, time slots are wasted when service load is low. Different nodes may have significantly different service loads, making it difficult to allocate time slots reasonably based on actual node loads, easily leading to uneven allocation. Since time slot allocation is fixed within a certain timeframe, it's impossible to adjust the allocation strategy promptly when channel quality changes. If a node is transmitting data with high bandwidth, a decrease in the link signal-to-noise ratio (SNR) may result in poor data transmission quality, but transmission must still be carried out according to the predetermined time slots. In this case, the transmission bandwidth can be reduced to improve the link SNR and ensure reliable data transmission. Similar to the SNR issue, under the current time slot allocation, if interference or other factors increase the bit error rate (BER) of a particular time slot, the time slot can be skipped or a different channel can be used to avoid interference and reduce the BER.

[0003] Therefore, in the face of rapid changes in information such as node transmission bandwidth, signal-to-noise ratio, service load, packet loss rate, and bit error rate, how to reasonably allocate time slots for each node to ensure reliable data transmission while improving network performance indicators such as latency and throughput is a technical problem that current swarm networks urgently need to solve. Summary of the Invention

[0004] This invention proposes a wireless network time slot allocation method based on near-end strategy optimization. It addresses the rapid changes in information such as transmission bandwidth, signal-to-noise ratio, service load, packet loss rate, and bit error rate of each node in the wireless network, and rationally allocates time slots for each node to ensure reliable data transmission while improving network performance indicators such as latency and throughput.

[0005] The technical solution to achieve the objective of this invention is: a wireless network time slot allocation method based on near-end policy optimization, comprising:

[0006] Step 1: Initialize policy network parameters buffer size The total number of time slots N per unit time is determined, and based on the wireless network topology, the bandwidth capacity of each node in the wireless network and the service information of each link are collected to form a network topology feature matrix; at the same time, the total number of time slots N is evenly divided according to the number of nodes to form an initial time slot table.

[0007] Step 2: Randomly select a node from the wireless network as the current node, and construct the state space of the current node at the current time k based on the network topology feature matrix;

[0008] Step 3: Input the state space of the current node at the current time k into the policy network, calculate the normalized probability of the number of time slots allocated to the current node, and form the action space;

[0009] Step 4: Select a number of time slots to be allocated in the action space according to the existing strategy, and save the normalized probability of selecting that number of time slots in the action space.

[0010] Step 5: Update the time slot table according to the number of time slots selected by this node, and receive the reward. Save the current state parameters, number of time slots, selection probability, and reward function to the buffer, proceed to the next state, return to step 4, and continue until the buffer is reached. Internal data is full After grouping, proceed to the next step;

[0011] Step 6: Calculate the state value and advantage estimate at each time step, calculate the ratio of the probabilities of the new and old policies and the loss value, then calculate the objective function, and update the policy network parameters by maximizing the objective function;

[0012] Step 7: Jump back to step 6 and update n times consecutively. After obtaining the latest network parameters, clear the buffer. The state is updated to the next state, and the process returns to step 2 until all nodes in the wireless network topology have been traversed.

[0013] Compared with the prior art, the significant advantages of this invention are: the invention rationally allocates time slots for each node to ensure reliable data transmission while improving network performance indicators such as latency and throughput.

[0014] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0015] Figure 1 This is a flowchart of a wireless network time slot allocation method based on near-end policy optimization.

[0016] Figure 2 This is a wireless network topology diagram. Detailed Implementation

[0017] A wireless network time slot allocation method based on near-end policy optimization is proposed. This method collects bandwidth, signal-to-noise ratio (SNR), service load, packet loss rate, and bit error rate (BER) of each node as parameters for reinforcement learning to ensure reasonable time slot allocation. First, considering the characteristics of bandwidth, SNR, packet loss rate, and BER, the network connection establishment time slot allocation is modeled as a Markov decision process. Then, a reward function is set with the optimization objectives of maximizing throughput and maximizing propagation delay. Finally, a near-end policy optimization method is used to solve the network connection establishment time slot allocation problem.

[0018] The proximal policy optimization algorithm can reuse sampled data to improve training speed. The algorithm consists of a policy network and an evaluation network. The policy network outputs the mean of actions; the evaluation network outputs the value of the state. The policy network is a neural network that outputs the probability distribution of actions. It takes the current state as input and outputs the probability of the number of time slots assigned to the current node. The network structure includes an input layer, several hidden layers, and an output layer. The policy network consists of three fully connected layers, each with 256 neurons. The ReLU activation function is used to add non-linearity and improve the neural network's expressive power. The output layer outputs the mean of actions, generating the probability distribution of output actions through the Softmax function. The evaluation network takes the current state as input and outputs the value of the state. The network structure includes an input layer, several hidden layers, and an output layer. The evaluation network also consists of four fully connected layers, each with 256 neurons. Unlike the policy network, the evaluation network outputs only one value, representing the value estimate of the current state.

[0019] A wireless network time slot allocation method based on near-end policy optimization includes:

[0020] Step 1: Initialize policy network parameters , buffer Size is The total number of time slots N per unit time is determined, and based on the wireless network topology, the bandwidth capacity of each node in the wireless network, the service load of each link, the signal-to-noise ratio, the packet loss rate, the bit error rate, and the current number of time slots are collected to form a network topology feature matrix; at the same time, the total number of time slots N is evenly divided according to the number of nodes to form an initial time slot table.

[0021] Step 2: Randomly select a node from the wireless network as the current node, and construct the state space of the current node at the current time k based on the network topology feature matrix. Includes remaining network time slots Remaining available data capacity for transmission Signal-to-noise ratio Packet loss rate Bit error rate ,Right now This state is then used as input to the policy network;

[0022] Step 3: The policy network receives the current state. As input, the normalized probability of the number of time slots allocated to the current node is calculated, forming the action space. (N is the total number of time slots per unit time).

[0023] Step 4: Based on existing strategies In the action space Select a number of time slots to be allocated and store them in the action space. The normalized probability of selecting that number of time slots is:

[0024] Step 5: Update the time slot table according to the number of time slots selected for this node, and receive a reward. This includes data such as current state parameters, number of time slots, selection probability, and reward function. Save to buffer Enter the next state Proceed to step 4 and continue execution until the buffer is reached. Internal data is full Follow up with step 6.

[0025] Step 6: Wait for the buffer to be full. After grouping, calculate the state value at each time step. and advantage estimation The specific formula is as follows: ; Calculate the ratio of the probabilities of the new and old strategies. Loss value Then calculate the objective function. By maximizing Update network parameters.

[0026] In a further embodiment, the ratio of the probabilities of the new and old strategies is specifically as follows:

[0027]

[0028] The strategy here is updated n times consecutively, using the previous K sets of data (K>n). The old strategy is the strategy before the update, and the new strategy is the strategy after the update. This process is repeated n times. The probability here is the probability of randomly selecting a certain number of time slots within N time slots.

[0029] In a further embodiment, the loss value is specifically:

[0030] In a further embodiment, the objective function is specifically:

[0031]

[0032] Step 7: Jump back to step 6 and update n times consecutively to obtain the latest network parameters. Clear the buffer afterwards The state is updated to the next state. Return to step 2 until all nodes in the wireless network topology have been traversed.

[0033] State space:

[0034] Remaining network time slots Within a network time slot scheduling cycle, the total available time slot resources within the network are fixed. Therefore, each node cannot exceed the total time slot resources when requesting time slot resources.

[0035] Remaining data capacity that can be sent The remaining transmittable data capacity has a significant impact on the link. A larger remaining transmittable data capacity means a relatively smaller demand for time slots. Therefore, calculating the remaining transmittable data capacity of each node has a significant impact on the rational allocation of time slots across the entire network.

[0036] = (Bandwidth capacity * Number of time slots) - Total service load

[0037] Signal-to-noise ratio Signal-to-noise ratio (SNR) is an important parameter for judging the quality of a link. A high-quality link can provide reliability for data transmission, thereby reducing the retransmission of data services and avoiding unnecessary waste of time slot resources.

[0038] Packet loss rate Packet loss rate directly affects the normal operation of network applications. Even if network transmission is not interrupted, users may still be unable to access data due to a high packet loss rate. In addition, excessive network load will increase the probability of packet loss.

[0039] Bit error rate The bit error rate (BER) directly affects the reliability and transmission rate of a communication system. An excessively high BER can significantly impact the reliability of transmitted data.

[0040] In summary, the state space at each time step can be represented as follows:

[0041] Action Space: This paper aims to allocate the number of time slots to each node in a reasonable manner. Therefore, the action space should be the normalized probability of the number of time slots allocated to each node. , where N represents the number of time slots.

[0042] Reward Function: The optimization objectives of this paper are maximum throughput and maximum propagation delay. Definition For the first The reward for the maximum throughput at a given moment. The number of bytes or bits transmitted per unit of time, in bytes per second (B / s).

[0043] In addition, the maximum propagation delay should be considered, and rewards should be given. Defined as the maximum propagation delay generated when a node transmits data.

[0044]

[0045] Generally, increasing bandwidth allows for the transmission of more data per unit time, but this leads to a decrease in the signal-to-noise ratio and a risk of increased packet loss. To improve link reliability and avoid simply increasing bandwidth at the expense of higher packet loss, the reward function should include a performance evaluation based on packet loss rate. If the packet loss rate increases, points are deducted accordingly. Definition The penalty for an increased packet loss rate is that points will be deducted if the packet loss rate exceeds 30%.

[0046]

[0047] In summary, the total reward function for the k-th time slot is:

[0048]

[0049] Example

[0050] This invention provides a wireless network time slot allocation method based on near-end policy optimization. The invention will be further described below with reference to the accompanying drawings. The network topology is as follows: Figure 2 As shown, it includes 10 nodes and 14 full-duplex links. The numbers next to the node boxes represent bandwidth capacity, and the solid lines represent the service load of the links. The bandwidth capacity of each node is randomly selected from 50 to 400 Mbit / s, and the service load between each link is randomly selected from [100, 10000]. The number of time slot resources per node is... Each time slot is 10ms; the network objective function is evaluated based on the weight values. = -0.5, the weight value of the entropy in the policy model. = -0.01, the attenuation factor for the standard deviation of the probability distribution of action values. = 0.9995, simulation count 5000, buffer Size = 128, number of consecutive network updates = 8, cutting factor = 0.2, calculate the discount factor for the expected reward. =0.97, learning rate of policy network and evaluation network = 0.001. The specific implementation process of the network time slot allocation method for each node is as follows:

[0051] 1. Initialize network parameters Equal to a Gaussian distribution with a mean of 0 and a standard deviation of 0.01, buffer zone The size is 128;

[0052] 2. Collect the underlying topology feature matrix of each node in the network, including bandwidth capacity, service load of each link, signal-to-noise ratio, packet loss rate, bit error rate, and current number of time slots;

[0053] 3. The first node follows the existing strategy. Construct the state space at the current moment. This includes the remaining network time slots, the remaining data capacity that the current node can send, the signal-to-noise ratio, the packet loss rate, and the bit error rate;

[0054] 4. By Select Action The normalized probability of the number of time slots allocated to each node. (N=100), and save the action probabilities.

[0055] 5. Perform the time slot table adjustment action and proceed to the next state. Receive rewards

[0056] 6. Current Data Save to buffer

[0057] 7. After the buffer contains 128 sets of data, calculate the expected reward. and advantage estimation

[0058] 8. Perform 8 consecutive updates, with the state value obtained by the evaluation network each time. And calculate the objective function By maximizing Update network parameters

[0059] 9. Update network parameters and clear the buffer. Update status

[0060] 10. Proceed to the next node operation.

Claims

1. A wireless network time slot allocation method based on near-end policy optimization, characterized in that, include: Step 1: Initialize policy network parameters buffer size The total number of time slots N per unit time is determined, and based on the wireless network topology, the bandwidth capacity of each node in the wireless network and the service information of each link are collected to form a network topology feature matrix; at the same time, the total number of time slots N is evenly divided according to the number of nodes to form an initial time slot table. Step 2: Randomly select a node from the wireless network as the current node, and construct the state space of the current node at the current time k based on the network topology feature matrix; Step 3: Input the state space of the current node at the current time k into the policy network, calculate the normalized probability of the number of time slots allocated to the current node, and form the action space; Step 4: Select a number of time slots to be allocated in the action space according to the existing strategy, and save the normalized probability of selecting that number of time slots in the action space. Step 5: Update the time slot table according to the number of time slots selected by this node, and receive the reward. Save the current state parameters, number of time slots, selection probability, and reward function to the buffer, proceed to the next state, return to step 4, and so on until the buffer is reached. Internal data is full After grouping, proceed to the next step; Step 6: Calculate the state value and advantage estimate at each time step, calculate the ratio of the probabilities of the new and old policies and the loss value, then calculate the objective function, and update the policy network parameters by maximizing the objective function; Step 7: Jump back to step 6 and update n times consecutively. After obtaining the latest network parameters, clear the buffer. The state is updated to the next state, and the process returns to step 2 until all nodes in the wireless network topology have been traversed.

2. The wireless network time slot allocation method based on near-end policy optimization according to claim 1, characterized in that, The service information for each link includes load, signal-to-noise ratio, packet loss rate, bit error rate, and the current number of timeslots.

3. The wireless network time slot allocation method based on near-end policy optimization according to claim 1, characterized in that, The state space of the current node at time k contains the remaining time slots of the network. Remaining available data capacity for transmission Signal-to-noise ratio Packet loss rate Bit error rate ,Right now .

4. The wireless network time slot allocation method based on near-end policy optimization according to claim 1, characterized in that, The reward obtained in step 5 is as follows: ； In the formula, For the first The reward for the maximum throughput at a given moment. The maximum propagation delay that occurs when a node transmits data. Penalty for increased packet loss rate , This indicates the packet loss rate.

5. The wireless network time slot allocation method based on near-end policy optimization according to claim 1, characterized in that, The advantage estimate at each moment is as follows: ； in ; In the formula, Indicates the first The reward received at each moment The discount factor representing the expected reward. Represents the time at time K The value of a state Indicated by the evaluation network The value of a state To reward expected value.

6. The wireless network time slot allocation method based on near-end policy optimization according to claim 1, characterized in that, The ratio of the probabilities of the new and old strategies is as follows: ； In the formula, This represents the probability of randomly selecting a certain number of time slots out of N time slots after the strategy update. This represents the probability of randomly selecting a certain number of time slots from N time slots before the strategy update.

7. The wireless network time slot allocation method based on near-end policy optimization according to claim 1, characterized in that, The specific loss value is as follows: ； In the formula, The ratio of the probabilities of the new and old strategies. For the advantage estimation at each moment, The function is Set with 1- For the upper bound, 1+ As a constraint on the lower bound, The cutoff constant is... This indicates that the expected value of the expression within the square brackets is to be calculated.

8. The wireless network time slot allocation method based on near-end policy optimization according to claim 1, characterized in that, The objective function is as follows: ； In the formula, This is the loss value. It is the entropy of the policy network. It evaluates the network objective function. and It is a constant.

Images