A CSMA Optimization Method Based on Deep Reinforcement Learning

Abstract

The present invention discloses a CSMA optimization method based on deep reinforcement learning, aiming at the unfairness of the binary exponential backoff scheme in the current slot CSMA protocol and the problem that the CSMA transmission capacity drops sharply when the network scale is large. The present invention divides data into time slots in a manner similar to slotted ALOHA, and a single node uses deep reinforcement learning to decide whether to participate in the channel competition in this time slot, enabling the network to intelligently allocate channels according to the actual network situation, avoiding the unfairness caused by the higher transmission probability of the nodes that have sent successfully in slotted CSMA, and at the same time, when the network scale is large, the nodes can intelligently reduce the probability of participating in the channel competition, thereby avoiding the sharp drop in performance caused by a large number of data collisions.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technologies, and particularly to an optimized CSMA method based on deep reinforcement learning. Background Art

[0002] Reinforcement learning is a way for an agent to learn by "trial and error". The agent obtains rewards through interaction with the environment to guide its behavior. The goal is to maximize the rewards obtained by the agent. Reinforcement learning is different from supervised learning in connectionist learning, mainly in terms of the reinforcement signal. In reinforcement learning, the reinforcement signal provided by the environment is an evaluation of the quality of the generated action (usually a scalar signal), rather than telling the reinforcement learning system RLS (reinforcement learning system) how to generate the correct action. Since the information provided by the external environment is very limited, the RLS has to learn from its own experiences. In this way, the RLS obtains knowledge in an action-evaluation environment and improves its action plan to adapt to the environment.

[0003] Deep learning has strong perception ability but lacks certain decision-making ability; while reinforcement learning has decision-making ability but is helpless with perception problems. Therefore, combining the two can complement each other's advantages and provide a solution to the perception and decision-making problems of complex systems. Deep reinforcement learning combines the perception ability of deep learning and the decision-making ability of reinforcement learning, and can directly control according to the input image. It is an artificial intelligence method closer to the human thinking mode.

[0004] Carrier Sense Multiple Access (CSMA) is a protocol that allows multiple devices to send signals on the same channel. The devices in it listen to whether other devices are busy and only send when the line is idle. CSMA / CA (Carrier Sense Multiple Access with Collision Avoidance) is an improvement of CSMA in the case of wireless networks and has been widely used in the field of wireless communication.

[0005] In a CSMA / CD system, the station starts to execute the back-off algorithm after a collision occurs. For a CSMA / CA system, when a station wants to send a packet, it first listens to the state of the channel. If the channel is idle and remains idle after DIFS, the station starts to send information. If the channel is busy, it keeps listening to the idle time of the channel for more than DIFS. When the channel finally becomes idle, the station uses the binary back-off algorithm and enters the back-off state to avoid collisions. Because there is no collision detection mechanism, the station has to execute the back-off algorithm when the channel changes from busy to idle.

[0006] When the network scale is large and the network load is heavy, the number of nodes participating in channel competition at the same time increases, which will lead to a large number of data collisions, resulting in a large number of data retransmissions, and thus the transmission performance is greatly reduced. In addition, CSMA also has a serious problem of insufficient fairness. For example, in CSMA / CA, the binary backoff algorithm is used to avoid collisions. Since the node that sent successfully last time will obtain a smaller backoff window, the probability of the node that sent successfully last time competing for and obtaining the channel is greater. This way of channel allocation is seriously unfair. Summary of the Invention

[0007] Object of the Invention: Aiming at the problems existing in the above background technology, the present invention provides an optimized CSMA method based on deep reinforcement learning. Data is divided into time slots in a manner similar to slotted ALOHA. Each individual device serves as an agent, obtains environmental information, and decides whether to participate in the channel competition of a single transmission time slot in the way of deep reinforcement learning, so that the network can intelligently allocate channels according to the actual network situation, and intelligently reduce its own frequency of participating in channel competition when the transmission load in the network is large, thus avoiding a large number of data collisions that occur in traditional CSMA, improving the system performance and robustness, and at the same time improving the fairness among devices.

[0008] Technical Solution: To achieve the above object, the technical solution adopted by the present invention is as follows:

[0009] An optimized CSMA method based on deep reinforcement learning, comprising the following steps:

[0010] Step S1: Each individual node serves as an agent for deep reinforcement learning, and completes the initialization of the learning model parameters; obtain the time slot length DataSlotTime and the period T;

[0011] Step S2: When a certain node has frame information to be sent, the node selects the actions to be taken in the subsequent n time slots DataSlotTime through deep reinforcement learning, and represents the actions as an array Action of length n, where Action[i] represents that the node will participate in the channel competition in the subsequent i-th time slot; when the node has no frame information to be sent, the node remains in the receiving information state within this time slot, and records the environmental information until there is frame information to be sent.

[0012] The specific steps for the node to send frame information include:

[0013] Step S3.1: Initialize the counter SlotCount = 0;

[0014] Step S3.2: When Action[SlotCount] is true, it means the node participates in the competition of this time slot, and then the node attempts to send a frame to the destination node; when Action[SlotCount] is not true, it continues to wait for the DataSlotTime; after completing the sending or waiting operation, the counter is incremented by one;

[0015] Step S3.3: Repeat Step S3.2 until SlotCount = n;

[0016] Furthermore, in Step S2, each node is used as an agent for deep reinforcement learning to execute the deep reinforcement learning algorithm with T = n × DataSlotTime as the basic operation time unit; then the reinforcement learning algorithm model in Step S2 is expressed as follows:

[0017] State t =(NN t , NI t , P t , SEND t-1 ), t = 1, 2, 3,...

[0018] where State t represents the state set, and t represents the number of cycles; NN t represents the number of two-hop neighbors of the node; NI t represents the two-hop neighbor situation information of the node, which is a two-dimensional matrix in structure; P t represents the number of frames waiting to be sent when sending the load; SEND t-1 represents the number of data packets successfully sent by this node in the previous cycle;

[0019] Each node independently maintains the two-hop neighbor situation information, updates its own situation table through the beacon frames broadcast by one-hop neighbor nodes, and broadcasts its own information and the maintained situation table to one-hop neighbors; where the situation information includes the node sending load P n , the number of neighbor nodes NN n these 2 values, and NI t is represented as a 2×NN t matrix.

[0020] The reward function in the reinforcement learning algorithm model in Step S2 is expressed as follows:

[0021]

[0022] where

[0023]

[0024] r iIt represents the reward value of the i-th time slot in the previous action cycle T. The received correct data frame means receiving a frame that has passed the CRC check by any destination node.

[0025] Furthermore, sending a frame in step S3.2 means sending a frame using the CSMA / CA method. The specific steps include:

[0026] Step S3.2.1: The node performs backoff using the binary exponential backoff algorithm and sets the minimum backoff window CW min = 2 2 = 4, and the maximum backoff window CW min = 2 5 = 32, and the maximum number of repetitions is 4 times. When the repetition exceeds 4 times, it means the backoff fails. When the backoff fails or the remaining time of the DataSlotTime time slot is not enough to complete the transmission of the frame, then wait until the end of the time slot and execute step 3.3;

[0027] Step S3.2.2: When the node successfully competes for the channel, it sends a frame message. The frame message includes a data frame and a beacon frame. The data frame is obtained by the network layer and is used to complete data interaction between nodes. The beacon frame is periodically generated by the MAC layer and is used to complete information interaction and time synchronization between nodes. When the frame message sent is a data frame, it is sent using the CSMA / CA method. When the frame message sent is a beacon frame, it is sent using the direct broadcast method. After the transmission is completed, wait for the acknowledgment frame ACK replied by the receiving end and record the transmission result. When the transmission fails, cancel the repeated transmission and perform a retransmission operation in the subsequent time slots.

[0028] Beneficial effects:

[0029] The CSMA optimization method based on deep reinforcement learning provided by the present invention determines whether to participate in the channel competition of a single transmission time slot by perceiving the environment and in the way of deep reinforcement learning, intelligently adjusts the frequency of the device participating in the channel competition, and intelligently performs channel allocation according to the actual network situation. When the transmission load of the system is small, the channel can be fully utilized. At the same time, when the transmission load in the network is large, it can avoid a large number of data collisions that occur in traditional CSMA, avoid a sharp decline in system performance, and enhance system robustness. At the same time, the learning algorithm can adjust the probability of nodes participating in the competition according to the actual load situation of each node in the network, so that nodes with a larger load obtain more transmission opportunities, and at the same time avoid some nodes always obtaining the channel, improving the fairness and real-time performance of the network. Description of the drawings

[0030] Figure 1 is the flowchart of the CSMA optimization method based on deep reinforcement learning provided by the present invention;

[0031] Figure 2It is a schematic diagram of the deep reinforcement learning process of the agent provided by the present invention;

[0032] Figure 3 It is a schematic diagram of the competition situation of multiple nodes after optimization in the embodiment of the present invention. Specific embodiments

[0033] The present invention will be further described below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative work shall fall within the protection scope of the present invention.

[0034] As Figure 1 shown is a typical optimization method flowchart when the present invention is applied to a wireless ad hoc network. After a node enters the network, it first waits to receive beacon frames from other nodes. If a beacon frame is received, time synchronization and slot length correction are performed. Otherwise, the time stamp and slot length DataSlotTime and period T are initialized with default values, and thus the node completes network entry. The node scheduling method after network entry includes the following steps:

[0035] Step S1: A single node serves as an agent for deep reinforcement learning to complete the initialization of the learning model parameters; obtain the slot length DataSlotTime and period T.

[0036] Step S2: When a certain node has frame information to be sent, the node selects the actions to be taken in the subsequent n slots DataSlotTime through deep reinforcement learning, and represents the actions as an array Action of length n, where Action[i] represents that the node participates in channel competition in the subsequent i-th slot; when the node has no frame information to be sent, the node remains in the receiving information state in this slot and records the environmental information until there is frame information to be sent.

[0037] The frame information sent by the above nodes includes data frames and beacon frames. Among them, the data frames are sent by the network layer for data interaction between nodes; the beacon frames are periodically generated by the MAC layer for information interaction and time synchronization between nodes. Table 1 below shows some control fields in the beacon frame and their meanings.

[0038] Table 1 Some control fields in the beacon frame and their meanings.

[0039] Field Meaning Tick Timestamp maintained by the source device node NumSlot Number of time slots per cycle DataSlotTime Length of a single time slot NeighborNum Number of one-hop neighbors of the source device NodePayload Payload sent by the source device NeighborInfo One-hop neighbor information of the source device, represented as a two-dimensional know

[0040] In the above embodiment, each node is used as an agent for deep reinforcement learning to execute the deep reinforcement learning algorithm, with T = n × DataSlotTime as the basic running time unit; then the reinforcement learning algorithm model is expressed as follows:

[0041] State t =(NN t , NI t , P t , SEND t-1 ), t = 1, 2, 3,...

[0042] where State t represents the state set, and t represents the number of cycles; NN t represents the number of two-hop neighbors of the node; NI t represents the two-hop neighbor situation information of the node, which is a two-dimensional matrix in structure; P t represents the number of frames waiting to be sent when sending the load;

[0043] SEND t-1 represents the number of data packets successfully sent by this node in the previous cycle;

[0044] Each node independently maintains the two-hop neighbor situation information, updates its own situation table through the beacon frames broadcast by one-hop neighbor nodes, and broadcasts its own information and the maintained situation table to one-hop neighbors; among them, the situation information includes the node sending load P n , the number of neighbor nodes NN n these two values, then NI t is represented as a 2×NN t matrix.

[0045] The reward function in the reinforcement learning algorithm model is expressed as follows:

[0046]

[0047] where

[0048]

[0049] r i represents the reward value at the i-th time slot in the previous action cycle T. Receiving a correct data frame means receiving a data frame or situation frame that passes the CRC check of any destination node.

[0050] Such as Figure 1As shown, after the device initialization is completed, the device enters the waiting state; the device is divided into time slots of length DataSlotTime in the time domain. If there is no frame to be sent, it waits to receive a data frame in the current time slot until the time slot ends. When the time slot ends, the node status information is updated. The frames waiting to be sent include data frames from the upper layer and beacon frames generated by the MAC layer at regular intervals; if there is a data frame to be sent, the device runs the CSMA optimization learning algorithm based on deep reinforcement learning. The DQN network outputs the Q function, and the device selects the best action Action for the subsequent n time slots based on the Q function with an ∈-greedy strategy. Then, the device executes the actions for the subsequent n time slots using the node sending data steps in Table 3; after the actions are executed, the device checks again whether there are frames to be sent and repeats this step.

[0051] The DQN network learning algorithm is specifically shown in Table 2 below:

[0052] Table 2 CSMA Optimization Learning Algorithm Based on Deep Reinforcement Learning

[0053]

[0054] Step S3. The specific steps for the node to send frame information include:

[0055] Step S3.1. Initialize the counter SlotCount = 0;

[0056] Step S3.2. When Action[SlotCount] is true, it means the node participates in the competition of this time slot, and the node tries to send a frame to the destination node; when Action[SlotCount] is not true, it continues to wait for the DataSlotTime time; after the sending or waiting operation is completed, the counter is incremented by one;

[0057] The above "sending a frame" means that the node sends a frame in the CSMA / CA manner, and the specific method is shown in Table 3 below:

[0058] Table 3 Node Sending Data Steps

[0059]

[0060] Step S3.2.1. The node performs a backoff using the binary exponential backoff algorithm, defining the minimum backoff window CW min = 2 2 = 4, the maximum backoff window CW min = 2 5 = 32, and the maximum number of repetitions is 4 times, that is, repeating more than 4 times means the backoff fails; when the backoff fails or the remaining time of the DataSlotTime time slot is not enough to complete the sending of the frame, then wait until the time slot ends and execute Step S3.3;

[0061] Step S3.2.2: When a node successfully competes for the channel, send a frame message. When the frame message to be sent is a data frame, it is sent using the CSMA / CA method; when the frame message to be sent is a beacon frame, it is sent using the direct broadcast method; after sending, wait for the acknowledgment frame ACK replied by the receiving end and record the sending result; when the sending fails, cancel the repeated sending and perform a retransmission operation in the subsequent time slot.

[0062] Step S3.3: Repeat Step S3.2 until SlotCount = n.

[0063] Figure 3 The channel competition situation in the case of 3 nodes is shown. It can be seen that the network can adjust its own sending frequency in a timely manner according to the channel situation so that the system can operate with the best performance.

[0064] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A CSMA optimization method based on deep reinforcement learning, characterized in that, It includes the following steps: Step S1: A single node serves as an agent for deep reinforcement learning to complete the initialization of learning model parameters; obtain the slot length DataSloTime and the period T; Step S2: When a certain node has frame information to be sent, the node selects the actions to be taken in the subsequent n time slots DataSlotTime through deep reinforcement learning, and represents the actions as an array Action of length n, where Action[i] represents that the node participates in the channel competition in the subsequent i-th time slot; when the node has no frame information to be sent, the node remains in the receiving information state within this time slot and records the environmental information until there is frame information to be sent; Step S3: The specific steps for the node to send frame information include: Step S3.1: Initialize the counter SlotCount = 0; Step S3.2: When Action[SlotCount] is true, it means the node participates in the competition in this time slot, and the node attempts to send a frame to the destination node; when Action[SlotCount] is not true, continue to wait for the DataSloTime time; after completing the sending or waiting operation, increment the counter by one; Step S3.3: Repeat Step S3.2 until SlotCount = n; In Step S2, each node is used as an agent for deep reinforcement learning to execute the deep reinforcement learning algorithm, with T = n × DataSlotTime as the basic operation time unit; then the reinforcement learning algorithm model in Step S2 is expressed as follows: State t =(NN t , NI t , P t , SEND t-1 ), t = 1, 2, 3,... Among them, State t represents the state set, and t represents the number of cycles; NN t represents the number of two-hop neighbors of the node; NI t represents the two-hop neighbor situation information of the node, which is a two-dimensional matrix in structure; P t represents the number of frames waiting to be sent when sending the load; SEND t-1 represents the number of data packets successfully sent by this node in the previous cycle; Each node independently maintains the situation information of its two-hop neighbors, updates its own situation table through the beacon frames broadcast by its one-hop neighbor nodes, and broadcasts its own information and the maintained situation table to its one-hop neighbors; where the situation information includes the node transmission load P n , the number of neighbor nodes NN n These two values, NI t are represented as a 2×NN t matrix.

2. The CSMA optimization method based on deep reinforcement learning according to claim 1, characterized in that The reward function in the reinforcement learning algorithm model in Step S2 is expressed as follows: Where r i represents the reward value of the i-th time slot in the previous action cycle T, and the received correct data frame represents receiving a frame that has passed the CRC check of any destination node.

3. A CSMA optimization method based on deep reinforcement learning according to claim 1, characterized in that Sending a frame in Step S3.2 means sending a frame using the CSMA / CA method; The specific steps include: Step S3.2.

1. The node performs backoff using the binary exponential backoff algorithm and sets the minimum backoff window CW min = 2 2 = 4, the maximum backoff window CW min = 2 5 = 32, the maximum number of repetitions is 4 times. When the repetition exceeds 4 times, it means the backoff fails; when the backoff fails or the remaining time in the DataSlotTime time slot is not enough to complete the transmission of the frame, wait until the end of the time slot and then execute step 3.3; Step S3.2.2: When the node successfully competes for the channel, send a frame information; the frame information includes a data frame and a beacon frame; the data frame is obtained by the network layer and is used to complete data interaction between nodes; the beacon frame is periodically generated by the MAC layer and is used to complete information interaction and time synchronization between nodes; when the frame information sent is a data frame, it is sent using the CSMA / CA method; when the frame information sent is a beacon frame, it is sent using the direct broadcast method, and wait for the acknowledgment frame ACK replied by the receiving end and record the sending result after sending; when the sending fails, cancel the repeated sending and perform retransmission operations in subsequent time slots.

Images