Dynamic random access communication method based on deep reinforcement learning
By adopting a dynamic random access communication method based on deep reinforcement learning in the Internet of Things scenario, dynamically adjusting the number of packets sent and time slots, the packet collision problem in the traditional random access protocol is solved, and communication efficiency and reliability are improved.
Patent Information
- Application Number
- CN202411887358.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-20
- Publication Date
- 2025-03-11
AI Technical Summary
In the high concurrency IoT scenario, traditional random access protocols are difficult to effectively solve the problem of packet collision, resulting in a decrease in communication efficiency and reliability.
A dynamic random access communication method based on deep reinforcement learning is adopted, and the receiver obtains the probability distribution of the number of transmitted data packets based on the transmission situation of the previous frame, and the user determines the number of packets and time slots to be sent in the transmission based on the probability distribution, so as to realize dynamic adjustments to reduce collisions.
It significantly reduces the retransmission demand caused by packet collision, improves system throughput, optimizes the allocation of channel resources, and reduces the waste of empty time slots.
Smart Images
Figure CN119603792A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of communication system access process control, and specifically relates to a dynamic random access communication method based on deep reinforcement learning. Background Art
[0002] With the rapid growth of the number of devices in large-scale machine-type communication scenarios, the Internet of Things technology has entered a stage of rapid development. Hundreds of millions of machine-type communication devices need to frequently and massively access wirelessly, resulting in high-concurrency access requests. Traditional authorization-based access protocols require complex multi-step interaction processes, and their signaling overhead and access delay increase dramatically when facing massive devices. Obviously, they cannot meet the needs of rapid access of a large number of devices in the Internet of Things scenario. For this reason, random access technology has become the focus of research. Random access greatly reduces scheduling overhead and access delay by simplifying or omitting the cumbersome multi-step interaction in the traditional authorization process, providing a more flexible and efficient access method for massive devices. However, this access mode is also accompanied by significant challenges. Especially in high-concurrency scenarios, the randomness of random access may lead to intensified resource competition among users, thereby causing collisions and packet loss problems, reducing overall communication efficiency and reliability.
[0003] ALOHA is the earliest proposed random access protocol to solve the problem of multi-user channel sharing in wireless communications. It is simple and flexible, but has certain limitations in spectrum utilization and conflict avoidance. SlottedALOHA is an improved version of the original ALOHA, which significantly reduces the probability of conflict by introducing a time slot mechanism. Theoretically, the maximum throughput of the SlottedALOHA system can reach 36.8%, which is about twice that of the original ALOHA. The Slotted ALOHA algorithm reduces some collisions and improves recognition efficiency by fixing the number of time slots, but it cannot dynamically adjust the number of time slots. When there are many tags, insufficient time slots lead to an increase in collision rate, a decrease in recognition efficiency and channel utilization; when there are few tags, the number of empty time slots increases, resulting in a waste of resources. Therefore, in order to meet the higher requirements of high-quality mobile Internet and IoT services for reliability, latency and efficiency, it is particularly important to design a more intelligent and optimized random access mechanism for massive terminals. Summary of the invention
[0004] The purpose of the present invention is to propose a dynamic random access communication method based on deep reinforcement learning, which effectively solves the problem of data packet collision between multiple users and can still decode normally even in the event of a collision, significantly improving the system throughput.
[0005] The present invention is achieved through the following technical solutions: A dynamic random access communication method based on deep reinforcement learning, comprising: The random access process of the user in the current frame includes: Step S11: At the beginning of the current frame, the user waits and receives the probability distribution of the number of sent data packets sent by the receiving end, wherein the probability distribution of the number of sent data packets is obtained by the receiving end through deep reinforcement learning according to the transmission situation of the previous frame; Step S12: The user determines the number of data packets to be sent in this transmission according to the received probability distribution of the number of sent data packets; Step S13: The user evenly selects a corresponding number of time slots in the current frame according to the number of data packets to be sent obtained in step S12, and sends data packets one by one in the selected time slots, thus realizing random access; The receiving process of the receiver at the receiving end for the user's random access signal includes: Step S21, the receiver decodes the decodable data packets one by one according to the time slot in the received random access signal of the current frame; Step S22: using the decoded data packet to eliminate interference in the random access signal of the current frame to obtain a signal after interference elimination; Step S23: for the signal after interference elimination, decode the decodable data packets one by one according to the time slot, and continue to eliminate interference according to step S22, and repeat this step continuously until all data packets in the random access signal of the current frame are successfully decoded or cannot be further decoded; Step S24, when all data packets are successfully decoded, the receiver obtains the probability distribution of the number of data packets again and sends it to all users to transmit new data; when there are still undecoded data packets and no further decoding is possible, the receiver sends a new probability distribution of the number of sent data packets to the users corresponding to the undecoded data packets, and these users retransmit the data packets according to the random access process until all data packets are successfully decoded or the upper limit of the number of retransmissions is reached.
[0006] Furthermore, in step S11, the receiving end includes a learning module and a receiver, and the receiving end obtains the probability distribution of the number of sent data packets, specifically including the following steps: Step S111: The learning module obtains the transmission matrix of the previous frame As the current state of the environment of the communication system , according to the policy network of the reinforcement learning model Make a decision, that is, get the probability distribution of the number of packets sent ,in, , , Indicates N User in k The activation status of the time slots, N Indicates the number of users,K Indicates the number of time slots in each frame, 1≤ k ≤ K , π represents the strategy of the decision network, Represents the parameters of the policy network; Step S112: The learning module simulates the probability distribution of the number of packets sent by the user. p t Uniformly select time slots to send data packets and decode the received data packets. At this time, the environment state changes to , then based on the transformation of the environmental state, according to the formula Get feedback rewards ,in, m Indicates the number of data packets sent by all users in one frame. A is the number of time slots contained in a subframe for a single user, n The number of users who successfully sent packets for this time slot, t now The current transmission time. t total is the total length of a frame; Step S113: The value network using the reinforcement learning model is Carry out as formula The scores of is the parameter of the current value network, represents the action state value estimated by the value network, and q() represents the action state value function simulated by the neural network; Step S114: Target network using reinforcement learning model To proceed as formula The scores of express t +1 moment of action, represents the parameters of the target network, Indicates the target network for t +1 The estimated action state value at the moment; Step S115: According to the formula and Calculate the time-division target and time-division error, where Represents the discount factor for future expected rewards; Step S116: Update the value network of the reinforcement learning model according to the formula and the policy network of the reinforcement learning model ,in, represents the learning rate of the value network, represents the gradient of the function with respect to the value network parameter w, represents the learning rate of the policy network, Represents the function for the policy network parameters The gradient of represents the updated value network parameters, represents the updated decision network parameters; Step S117: The learning module continuously repeats steps S111 to S116 for training until the reinforcement learning model converges, and transmits the final probability distribution of the number of sent data packets to the receiver.
[0007] Furthermore, the step S12 is specifically as follows: the probability distribution of the number of data packets sent includes a quantity level and multiple probabilities corresponding to each quantity level. After the user receives the probability distribution of the number of data packets sent, multiple sub-intervals corresponding to each quantity level are determined according to each probability and a set integer interval, and a random integer is generated within the set integer interval. The corresponding quantity level is determined according to the sub-interval to which the random integer belongs. The quantity level is the number of data packets that the user needs to send in this transmission.
[0008] Furthermore, in step S13, the user selects a corresponding number of time slots based on uniform distribution in the current frame according to the number of data packets that need to be sent.
[0009] Furthermore, in step S21, the receiver composes a virtual frame with the current frame and the V frames before the current frame, and the current frame and the V frames before the current frame are used as subframes of the virtual frame. The receiver decodes at the end of each subframe, and the data packets that are not successfully decoded are retained in the corresponding subframe. The data packets in the current subframe and other subframes are used as random access signals of the current frame, and the decodable data packets are decoded one by one according to time slots.
[0010] Furthermore, the step S22 is specifically as follows: the receiver removes the decoded data packets in the random access signal of the current frame to obtain a signal after interference elimination.
[0011] Furthermore, in the receiving process, the receiver uses a joint decoding mechanism to process data packets in the random access signal of the current frame.
[0012] The present invention has the following beneficial effects: 1. Before each frame starts, the receiving end obtains the probability distribution of the number of sent data packets through deep reinforcement learning based on the transmission status of the previous frame, and sends the probability distribution to all users. The user determines the number of data packets that need to be sent in this transmission based on the received probability distribution, and evenly selects the corresponding number of time slots in the current frame for transmission. The probability distribution of sending data packets is obtained through deep reinforcement learning, which enables the user's sending behavior to seamlessly adapt to the real-time communication environment, and realizes the optimal allocation of channel resources to effectively reduce the waste of empty time slots, and significantly reduces the retransmission demand caused by data packet collision. For the receiving end, the colliding data packets are not discarded, but the decoded data packets are used to eliminate the interference in the random access signal of the current frame to infer and decode the colliding data packets, thereby improving the signal decoding performance of the receiving end and significantly improving the system throughput. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] The present invention is further described in detail below with reference to the accompanying drawings.
[0014] Figure 1 It is a schematic diagram of the random access process of the present invention.
[0015] Figure 2 It is a schematic diagram of the receiving process of the present invention.
[0016] Figure 3 A schematic diagram of a user selecting a time slot to send a data packet according to the present invention.
[0017] Figure 4 Schematic diagram of the frame structure of the virtual frame of the present invention.
[0018] Figure 5 This is a simulation comparison chart of the average number of successfully transmitted users per frame between the present invention and the comparative solution. DETAILED DESCRIPTION
[0019] The dynamic random access communication method based on deep reinforcement learning includes: The random access process of the user in the current frame is as follows: Figure 1 As shown, including: Step S11: At the beginning of the current frame, the user waits and receives the probability distribution of the number of sent data packets sent by the receiving end, wherein the probability distribution of the number of sent data packets is obtained by the receiving end through deep reinforcement learning according to the transmission situation of the previous frame; The communication unit between users and receivers is frame, which is divided into multiple time slots. All users randomly access all time slots in a frame. At the beginning of each frame, all users receive instructions from the receiver and remain silent, waiting for the probability distribution of the number of packets sent by the receiver.
[0020] The receiving end includes a learning module and a receiver. The learning module trains the selected deep reinforcement learning model to obtain the probability distribution of the number of packets sent in the current frame. The core elements of deep reinforcement learning include an agent, which is responsible for performing actions and learning strategies in the environment; state s, which represents the instantaneous description of the environment, and the agent makes decisions based on it; the behavior selected by the agent in a specific state is called action a; the feedback obtained from the environment after the agent performs an action is called reward r, which is used to evaluate the quality of the action; the rule that determines how the agent chooses an action is called strategy π; and the value function Q, which is used to evaluate the long-term reward of a certain state. Deep reinforcement learning is a trial-and-error learning process in which the agent continuously interacts with the environment, learns from feedback, and gradually adjusts the action strategy. The deep reinforcement learning model selected in this embodiment is the A2C algorithm model.
[0021] In this embodiment, consider a communication system: there is N =40 users want to communicate with the receiver in a random access mode. The receiver and the users broadcast synchronization signals periodically to ensure the synchronization of all frames and time slots. Each frame is 1ms long and is divided into K =100 time slots, each time slot length is 0.01ms.
[0022] The receiving end obtains the probability distribution of the number of sent data packets, which specifically includes the following steps: In the learning module, the training parameters and environment of the reinforcement learning model are first initialized. The policy network, target network, and value network structures all use a 5-layer linear fully connected network. The initialization environment uses a scenario where 40 users randomly transmit in 100 time slots. Then, training is performed according to the following steps to obtain the probability distribution of the number of packets sent: Step S111: The learning module obtains the transmission matrix of the previous frame As the current (time t) environmental state of the communication system , according to the policy network of the reinforcement learning model Make a decision, that is, get the probability distribution of the number of packets sent ,in, , , Indicates N User in k The activation status of the time slots, N Indicates the number of users, K Indicates the number of time slots in each frame, 1≤ k ≤ K , π represents the strategy of the decision network, Represents the parameters of the policy network; Step S112: The learning module simulates the probability distribution of the number of packets sent by the user.p t Uniformly select time slots to send data packets and decode the received data packets. At this time, the environment state changes to , then based on the transformation of the environmental state, according to the formula Get feedback rewards ,in, m It represents the number of data packets sent by all users in one frame. During the training process, the sending and receiving process is continuously simulated, and the reward is calculated once for each simulation. m refers to the total number of data packets sent by all users in one frame of each simulation. A is the number of time slots contained in a subframe for a single user. There are multiple time slots in a frame. Each user can send a data packet in a time slot, so the maximum number of data packets sent is the number of time slots contained in the frame. n The number of users who successfully sent packets for this time slot, t now The current transmission time. t total is the total length of a frame; Step S113: The value network using the reinforcement learning model is Carry out as formula The scores of is the parameter of the current value network, represents the action state value estimated by the value network, and q() represents the action state value function simulated by the reinforcement learning model; Step S114: Target network using reinforcement learning model To proceed as formula The scores of represents the action at time t+1, represents the parameters of the target network, Represents the action state value estimated by the target network at time t+1; Step S115: According to the formula and Calculate the time-division target and time-division error, where Represents the discount factor for future expected rewards; Step S116: Update the value network of the reinforcement learning model according to the formula and the policy network of the reinforcement learning model ,in, represents the learning rate of the value network, represents the gradient of the function (i.e., q()) with respect to the value network parameter w, represents the learning rate of the policy network, Represents the function (i.e. ln()) for the policy network parameters The gradient of represents the updated value network parameters, represents the updated decision network parameters; Step S117: The learning module continuously repeats steps S111 to S116 for training until the reinforcement learning model converges, and transmits the final probability distribution of the number of sent data packets to the receiver.
[0023] Among them, the learning module can be implemented by the computer of the base station.
[0024] Step S12: The user determines the number of data packets to be sent in this transmission according to the received probability distribution of the number of sent data packets; The probability distribution of the number of data packets sent includes a quantity level and multiple probabilities corresponding to each quantity level. After the user receives the probability distribution of the number of data packets sent, multiple sub-intervals corresponding to each quantity level are determined according to each probability and a set integer interval, and a random integer is generated within the set integer interval. The corresponding quantity level is determined according to the sub-interval to which the random integer belongs. The quantity level is the number of data packets that the user needs to send in this transmission.
[0025] As shown in Table 1, the probability distribution of the number of data packets sent contains the number levels 1, 2, 3, and 4, and the probabilities corresponding to the number levels are 0.1, 0.5, 0.3, and 0.1, respectively. The set integer interval is [1,100]. The data volume of each sub-interval is obtained by multiplying the data volume of the integer interval by the probability of each number level, that is, the four sub-intervals are [1,10], [11,60], [61,90], and [91,100]. If the user generates a random number of 23 through its own random number generator, and the random number is in the sub-interval [11,60], the number level corresponding to the sub-interval is 2, and the number of data packets that the user needs to send in this transmission is 2. Among them, the number of data packets sent is 2 specifically means that in the current frame, the user needs to select two time slots to send two data packets respectively, and the two data packets are exactly the same data packets.
[0026] Table 1 Step S13: The user evenly selects a corresponding number of time slots in the current frame according to the number of data packets to be sent obtained in step S12, and sends data packets one by one in the selected time slots, thus realizing random access; Specifically, the user selects a corresponding number of time slots in the current frame based on a uniform distribution according to the number of data packets that need to be sent. An example is Figure 3 shown.
[0027] The receiving process of the receiver at the receiving end for the user's random access signal is as follows: Figure 2 As shown, including: Step S21, the receiver decodes the decodable data packets one by one according to the time slot in the received random access signal of the current frame; In this embodiment, in order to enable the data packets sent by the user to be decoded faster, the receiver combines the current frame and the V frames before the current frame into a virtual frame, and the current frame and the V frames before the current frame are used as subframes of the virtual frame, such as Figure 4 As shown, the current frame random access signal at this time is the data packet contained in the virtual frame, and the receiver uses a joint decoding mechanism to process the data packet in the current frame random access signal. Specifically, the receiver decodes at the end of each subframe, and the data packets that are not successfully decoded are retained in the corresponding subframe, and the decoded subframe state is recorded as a residual matrix.
[0028] Step S22: using the decoded data packet to eliminate interference in the random access signal of the current frame to obtain a signal after interference elimination; Interference cancellation specifically refers to the receiver removing the decoded data packets in the random access signal of the current frame to obtain the signal after interference cancellation. When performing interference cancellation, the receiver also considers the data of V subframes before the current frame.
[0029] Step S23: for the signal after interference elimination, decode the decodable data packets one by one according to the time slot, and continue to eliminate interference according to step S22, and repeat this step continuously until all data packets in the random access signal of the current frame are successfully decoded or cannot be further decoded; Step S24: When all data packets are successfully decoded, the receiver obtains the probability distribution of the number of data packets again and sends it to all users for new data transmission; when there are still undecoded data packets and they cannot be further decoded, the receiver sends a new probability distribution of the number of sent data packets to the users corresponding to the undecoded data packets, and these users retransmit data packets according to the random access process until all data packets are successfully decoded or the upper limit of the number of retransmissions is reached. Once a user's data packet is successfully recovered, the user completes access.
[0030] The better the sending strategy, the fewer user collisions there are after interference is eliminated, and the more users each frame successfully sends, that is, the same number of users can complete access with fewer frames. Figure 5What is measured is the average number of users who successfully transmit per frame when the number of access users is 40 and the users use different probability distributions of the number of packets to be sent. The number of training times in the figure refers to the number of training rounds in reinforcement learning. The more rounds, the more convergent the model and the better the performance. Since there is no training process for other comparison schemes, the performance does not change with the number of training times. It can be seen that after the number of training times is greater than 300 times, the performance of the present invention is much better than that of the comparison scheme. Among them, in the comparison scheme, the fixed scheme of sending one data packet means that each user always sends only one data packet in one frame and does not send a copy; the fixed probability distribution scheme means that the user always uses the same probability distribution to extract the number of packets to be sent. The specific probabilities used in the simulation are: the probability of sending two data packets is 0.5102, and the probability of sending 4 data packets is 0.4898.
[0031] The above description is only a preferred embodiment of the present invention, and therefore cannot be used to limit the scope of implementation of the present invention. That is, equivalent changes and modifications made according to the scope of the patent application of the present invention and the contents of the specification should still fall within the scope covered by the patent of the present invention.
Claims
1. A dynamic random access communication method based on deep reinforcement learning, characterized in that: include: The random access process of the user in the current frame includes: Step S11: At the beginning of the current frame, the user waits and receives the probability distribution of the number of sent data packets sent by the receiving end, wherein the probability distribution of the number of sent data packets is obtained by the receiving end through deep reinforcement learning according to the transmission situation of the previous frame; Step S12: The user determines the number of data packets to be sent in this transmission according to the received probability distribution of the number of sent data packets; Step S13: The user evenly selects a corresponding number of time slots in the current frame according to the number of data packets to be sent obtained in step S12, and sends data packets one by one in the selected time slots, thus realizing random access; The receiving process of the receiver at the receiving end for the user's random access signal includes: Step S21, the receiver decodes the decodable data packets one by one according to the time slot in the received random access signal of the current frame; Step S22: using the decoded data packet to eliminate interference in the random access signal of the current frame to obtain a signal after interference elimination; Step S23: for the signal after interference elimination, decode the decodable data packets one by one according to the time slot, and continue to eliminate interference according to step S22, and repeat this step continuously until all data packets in the random access signal of the current frame are successfully decoded or cannot be further decoded; Step S24, when all data packets are successfully decoded, the receiver obtains the probability distribution of the number of data packets again and sends it to all users to transmit new data; when there are still undecoded data packets and no further decoding is possible, the receiver sends a new probability distribution of the number of sent data packets to the users corresponding to the undecoded data packets, and these users retransmit the data packets according to the random access process until all data packets are successfully decoded or the upper limit of the number of retransmissions is reached.
2. According to claim 1, a dynamic random access communication method based on deep reinforcement learning is characterized in that: In step S11, the receiving end includes a learning module and a receiver, and the receiving end obtains the probability distribution of the number of sent data packets, specifically including the following steps: Step S111: The learning module obtains the transmission matrix of the previous frame As the current state of the environment of the communication system , according to the policy network of the reinforcement learning model Make a decision, that is, get the probability distribution of the number of packets sent ,in, , , Indicates N User in k The activation status of the time slots, N Indicates the number of users, K Indicates the number of time slots in each frame, 1≤ k ≤ K , π represents the strategy of the decision network, Represents the parameters of the policy network; Step S112: The learning module simulates the probability distribution of the number of packets sent by the user. p t Uniformly select time slots to send data packets and decode the received data packets. At this time, the environment state changes to , then based on the transformation of the environmental state, according to the formula Get feedback rewards ,in, m Indicates the number of data packets sent by all users in one frame. A is the number of time slots contained in a subframe for a single user, n The number of users who successfully sent packets for this time slot, t now The current transmission time. t total is the total length of a frame; Step S113: The value network using the reinforcement learning model is Carry out as formula The scores of is the parameter of the current value network, represents the action state value estimated by the value network, and q() represents the action state value function simulated by the reinforcement learning model; Step S114: Target network using reinforcement learning model To proceed as formula The scores of express t +1 moment of action, represents the parameters of the target network, Indicates the target network for t +1 The estimated action state value at the moment; Step S115: According to the formula and Calculate the time-division target and time-division error, where Represents the discount factor for future expected rewards; Step S116: Update the value network of the reinforcement learning model according to the formula and the policy network of the reinforcement learning model ,in, represents the learning rate of the value network, represents the gradient of the function with respect to the value network parameter w, represents the learning rate of the policy network, Represents the function for the policy network parameters The gradient of represents the updated value network parameters, represents the updated decision network parameters; Step S117: The learning module continuously repeats steps S111 to S116 for training until the reinforcement learning model converges, and transmits the final probability distribution of the number of sent data packets to the receiver.
3. The dynamic random access communication method based on deep reinforcement learning according to claim 1, characterized in that: The step S12 is specifically as follows: the probability distribution of the number of data packets sent includes a quantity level and multiple probabilities corresponding to each quantity level. After the user receives the probability distribution of the number of data packets sent, multiple sub-intervals corresponding to each quantity level are determined according to each probability and a set integer interval, and a random integer is generated within the set integer interval. The corresponding quantity level is determined according to the sub-interval to which the random integer belongs. The quantity level is the number of data packets that the user needs to send in this transmission.
4. A dynamic random access communication method based on deep reinforcement learning according to claim 1, 2 or 3, characterized in that: In step S13, the user selects a corresponding number of time slots based on uniform distribution in the current frame according to the number of data packets that need to be sent.
5. A dynamic random access communication method based on deep reinforcement learning according to claim 1, 2 or 3, characterized in that: In step S21, the receiver composes a virtual frame with the current frame and the V frames before the current frame, and the current frame and the V frames before the current frame are used as subframes of the virtual frame. The receiver decodes at the end of each subframe, and the data packets that are not successfully decoded are retained in the corresponding subframe. The data packets in the current subframe and other subframes are used as random access signals for the current frame, and the decodable data packets are decoded one by one according to the time slot.
6. A dynamic random access communication method based on deep reinforcement learning according to claim 1, 2 or 3, characterized in that: The step S22 is specifically as follows: the receiver removes the decoded data packets in the random access signal of the current frame to obtain a signal after interference elimination.
7. The dynamic random access communication method based on deep reinforcement learning according to claim 6, characterized in that: In the receiving process, the receiver uses a joint decoding mechanism to process data packets in the random access signal of the current frame.