A Distributed TDMA Time Slot Allocation Method Based on Multi-Agent Reinforcement Learning

Abstract

This invention proposes a distributed TDMA time slot allocation method based on multi-agent reinforcement learning, primarily addressing the problem of poor performance in mobile ad hoc networks (MAVs) due to nodes having no available time slots under high load conditions. The solution includes: 1) constructing a MAV scenario; 2) determining the TDMA frame structure based on the network scenario; 3) establishing a MAV simulation platform using the network scenario and frame structure to obtain performance metrics; 4) assigning each node to an agent and constructing a reinforcement learning module; 5) training a MADDPG model using the simulation platform and reinforcement learning module to obtain parameter files; and 6) loading the trained parameters and using the model to perform time slot prediction. This invention, by introducing multi-agent reinforcement learning and combining priority experience replay and exploration mechanisms, effectively improves time slot allocation efficiency and increases system throughput, thereby enhancing the overall performance of MAVs.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and further relates to a time division multiple access (TDMA) time slot allocation method in wireless communication technology. Specifically, it is a distributed TDMA time slot allocation method based on multi-agent reinforcement learning, which can be used in the MAC layer of mobile ad hoc networks. Technical Background

[0002] With the rapid development of wireless communication technology, mobile ad hoc networks (MANETs) are widely used in military, disaster relief, and temporary network scenarios due to their flexibility and adaptability. However, the dynamic and uncertain nature of these networks poses challenges to the management and scheduling of network resources. Time-division multiplexing (TDMA), as an effective wireless resource allocation method, achieves ordered communication between nodes by dividing time into multiple time slots, thereby avoiding signal collisions and improving spectrum utilization. In MANETs, ​​TDMA time slot allocation not only affects the overall network performance but also directly relates to the communication efficiency and throughput of nodes. Traditional time slot allocation methods are often based on static models and fail to effectively consider factors such as dynamic changes in nodes, network load fluctuations, and link quality, leading to uneven resource allocation and ultimately affecting communication quality.

[0003] C. David Young et al. proposed the USAP (Unifying Dynamic Distributed Multichannel TDMA Slot Assignment Protocol) in their paper "USAP: A Unifying Dynamic Distributed Multichannel TDMA Slot Assignment Protocol." This protocol adds a slot control phase before slot allocation. During this phase, all nodes broadcast their contention for slots to their neighbors. Each node receiving this message then rebroadcasts its neighbor's message to its neighbors. This mechanism allows each node in the network to obtain information about nodes within two hops, thus avoiding conflicts during node slot allocation and resolving conflicts caused by node movement, achieving dynamic slot allocation. However, the protocol's long data frames result in significant access latency. Chenxi Zhu et al. proposed the Five-Phase Reservation Protocol (FPRP) in their paper "A Five-Phase Reservation Protocol (FPRP) for Mobile Ad Hoc Networks." This protocol divides slots into a contention phase and a data phase. During contention, nodes can acquire the right to use data slots through a five-step handshake, ensuring the acquisition of conflict-free broadcast slots. However, this protocol has relatively high overhead and requires strict clock synchronization. Building on this, the original authors proposed E-TDMA (Evolutionary-TDMA) in their paper "An evolutionary-TDMA scheduling protocol (E-TDMA) for mobile ad hoc networks." This protocol enables hybrid unicast and broadcast scheduling, improving network flexibility, but its performance degrades significantly under heavy network load. Addressing the shortcomings of the USAP protocol, Peng Gexin et al. proposed the P-TDMA (priority-TDMA) protocol in their paper "A conflict-free dynamic time slot allocation algorithm based on fixed TDMA." Compared to USAP, it adds a priority-based time slot contention mechanism, enabling dynamic allocation of time slots based on service load and ensuring a degree of fairness in time slot allocation due to priority. However, this protocol still cannot reduce node latency and does not solve the fundamental problem of poor scalability with fixed latency.

[0004] In MANETs, ​​the traditional Time-Divided Distribut ... Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and propose a distributed TDMA time slot allocation method based on multi-agent reinforcement learning. By introducing multi-agent reinforcement learning and combining it with priority experience replay and sample generation optimization techniques, this invention aims to effectively improve the allocation efficiency of TDMA time slots, increase the system throughput, and solve the problem of no time slots available for allocation to nodes under high load conditions, thereby enhancing the overall performance and stability of mobile ad hoc networks.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] (1) Construct a mobile ad hoc network scenario, including nodes, topology, and traffic model;

[0008] (2) Determine the frame structure of TDMA based on the constructed mobile ad hoc network scenario;

[0009] (3) Using the mobile ad hoc network scenario and TDMA frame structure, a mobile ad hoc network simulation platform is constructed to obtain network performance indicators;

[0010] (4) Assign each node to an agent, design the state space, action space, reward function and network structure, and build a reinforcement learning module; and use Socket to establish a communication module for data transmission between the reinforcement learning module and the mobile ad hoc network simulation platform.

[0011] (5) Use the constructed simulation platform and reinforcement learning module to train the multi-agent deep deterministic policy gradient MADDPG model and obtain the trained MADDPG model parameter file;

[0012] (6) Load the trained MADDPG model parameter file and use the model to realize TDMA time slot prediction for mobile ad hoc networks.

[0013] Compared with the prior art, the present invention has the following beneficial effects:

[0014] First, by introducing a multi-agent reinforcement learning framework and using a simulation platform for joint training, the agents can work together more effectively to achieve fast and accurate time slot allocation, thereby significantly improving network throughput and resource utilization. At the same time, it can respond to dynamic changes in mobile ad hoc networks in real time, automatically adjust the time slot allocation strategy to adapt to node load fluctuations, solve the problem of nodes being unable to allocate time slots under high load conditions, and enhance the system's flexibility, stability, and fairness.

[0015] Secondly, because this invention uses a priority experience replay mechanism, the agent prioritizes the use of important experience samples, thereby accelerating the learning process, improving efficiency, reducing dependence on sample generation, and avoiding resource waste. At the same time, during the training process, by labeling and fine-tuning "good samples" quickly generated using existing samples, the quality of samples is effectively improved and the need for regenerating samples is reduced, making the overall sample generation process more efficient and reliable. This invention, through the joint design of the exploration mechanism and the experience replay algorithm, effectively reduces the reinforcement learning training time and enables the system to converge faster. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the implementation of the method of the present invention;

[0018] Figure 2 This is a schematic diagram of the TDMA frame structure provided in an embodiment of the present invention;

[0019] Figure 3 This is a framework diagram of the reinforcement learning simulation platform introduced in this invention;

[0020] Figure 4 This is a flowchart illustrating the training process of the multi-agent reinforcement learning model constructed in this invention.

[0021] Figure 5 This is a comparison chart of throughput simulation results under different load conditions when using the method of the present invention and existing methods for time slot allocation;

[0022] Figure 6 This is a comparison chart of the fairness simulation results under different load conditions when using the method of this invention and existing methods for time slot allocation. Detailed Implementation

[0023] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0024] Example 1: Refer to Figure 1 This example provides a distributed TDMA time slot allocation method based on multi-agent reinforcement learning. The specific implementation steps are as follows:

[0025] Step 1. Construct a mobile ad hoc network scenario, including nodes, topology, and traffic model. Implementation is as follows: Assume the mobile ad hoc network consists of n nodes, and each node has bidirectional connections with its neighbors; model the entire mobile ad hoc network as an undirected graph G = <V, E>, where V represents the set of nodes, |V| = n, V = {v1, v2, ... v...} n Let E be the set of edges. Each edge represents a TDMA communication link between two nodes in the network. Each edge has a bidirectional link, and all links satisfy the following conditions. That is, node v i and v j Located within each other's communication range; let L(v) i ,t) represents the i-th node v i The load at current time t, S(v) i ,t) represents the i-th node v i The number of time slots allocated at the current time t, T(v) i ,t) represents the i-th node v i The throughput at the current time t.

[0026] Step 2. Determine the TDMA frame structure based on the constructed mobile ad hoc network scenario. The TDMA frame structure in this embodiment includes dividing a superframe into N frames, where the first frame is a preemptive frame, and the following N-1 frames are information frames; further dividing a frame into Q time slots, with each time slot including P time slices, and each time slice lasting 1ms, meaning the duration of a superframe is N*Q*P milliseconds; setting the base time slice for each node to N-1, meaning each node is allocated at least N-1 time slices.

[0027] Step 3. Utilize the mobile ad hoc network scenario and TDMA frame structure to construct a mobile ad hoc network simulation platform and obtain network performance indicators. In this embodiment, the mobile ad hoc network simulation platform is used as the client of the reinforcement learning framework. The Exata simulation software is used to model the mobile ad hoc network and obtain network performance indicators; these indicators include at least system throughput, load, and fairness parameters.

[0028] Step 4. Assign each node to an agent, design the state space, action space, reward function, and network structure, and construct the reinforcement learning module; and use Socket to establish a communication module for data transmission between the reinforcement learning module and the mobile ad hoc network simulation platform.

[0029] The design of the state space, action space, reward function, and network structure is described above. The reinforcement learning module is constructed using the following steps:

[0030] (4.1) Design a state space containing four-dimensional data, wherein the four-dimensional data is the throughput of the node at the previous time step, the number of time slots at the previous time step, the current load of the node, and the sum of the throughput of each node within two hops of the current node;

[0031] (4.2) According to the frame structure in step 2, the maximum number of time slots that can be allocated to each node is (N-1)×Q×P, which is a discrete value. The action space is designed as a discrete value with the dimension of (N-1)×Q×P.

[0032] (4.3) The overall system throughput and fairness parameters are weighted and used as the reward function of the reinforcement learning system; assuming T i W represents the throughput of the i-th node. i Let be the weight of the i-th node, and the fairness parameter is expressed as follows:

[0033]

[0034] The larger the F value, the better the fairness.

[0035] The global reward function can be expressed as:

[0036]

[0037] Where w is the weight value of the fairness parameter, representing the ratio of the overall system throughput to the fairness parameter;

[0038] (4.4) The structure of the MADDPG model is used as the main framework for reinforcement learning to construct the network structure of the reinforcement learning module, train it, and use the priority experience replay method for sample generation and extraction.

[0039] Step 5. Train the multi-agent deep deterministic policy gradient (MADDPG) model using the constructed simulation platform and reinforcement learning module, and obtain the trained MADDPG model parameter file. Specifically, input the current load and current number of time slots of each node into the reinforcement learning framework, output the time slot allocation scheme for the next time step through the reinforcement learning framework, and transmit the scheme to the simulation platform for simulation to obtain the throughput and load for the next time step. The specific implementation steps include the following:

[0040] (5.1) Let the maximum total number of iterations be E1 and the maximum number of exploration iterations be E2, and E1>E2; initialize the model parameter saving directory and initialize the current iteration number to 0;

[0041] (5.2) The simulation platform generates an initial load L(v) for the i-th node. i ,0) and the initial number of time slots S(v i ,0), where the initial number of time slots for each node is N-1, and the initial throughput of the i-th node is T(v i ,0) is 0;

[0042] (5.3) Use the communication module to transfer L(v) i ,t)S(v i ,t) and T(v i The L(v,t) values ​​of all nodes are transmitted to the reinforcement learning module, which then transmits the L(v,t) values ​​of all nodes to the reinforcement learning module. i ,t)S(v i ,t) and T(v i The state space s at time t is constructed as follows: t ;

[0043] (5.4) Input the state space into the reinforcement learning module, and determine whether the current iteration count has reached the set maximum exploration iteration count. If not, enter the exploration module, and obtain the action A(v) of the i-th node from the exploration module. i ,t), conversely, the action A(v) of the i-th node is obtained from the neural network of the MADDPG model. i ,t); then construct the action space a at time t based on the actions of all nodes. t In other words, the time slot allocation scheme.

[0044] The exploration module in this embodiment is a novel exploration mechanism designed by the present invention. Specifically, in an exploration, if the throughput of the current exploration reaches a set threshold, the results are marked as "good results," and the next exploration will not use a random exploration mechanism but will use this exploration mechanism. In this exploration mechanism, based on the "good results," the time slot allocation is fine-tuned before being put into the experience pool to obtain a new set of samples. If the throughput of the new samples is improved, fine-tuning continues on this basis; otherwise, fine-tuning is performed again based on the previous "good results." At the same time, a maximum number of fine-tuning times is set in a fine-tuning stage to prevent entering an infinite loop.

[0045] The fine-tuning mentioned is in A(v) i If A(v) is higher than the average of all node time slot values, adjust A(v) i ,t+1)=A(v i ,t)-x, otherwise adjust A(v) i ,t+1)=A(v i ,t)+x, thus achieving; where x is the fine-tuning coefficient.

[0046] The experience pool uses a priority experience replay mechanism to select samples. Here, the priority experience replay mechanism uses the time series error (TD-error) as a metric to evaluate the priority of the sampled data.

[0047] (5.5) A(v) i The data (t) is input into the simulation platform for simulation to obtain the throughput, load, and number of node time slots T(v) at the next time step. i ,t+1)L(v i ,t+1) and S(v i (t+1), and the reward r at time t. t And construct the state space s for the next time step. t+1 ;

[0048] (5.6) s t ,a t ,r t ,s t+1 As a sample e t =(s t ,a t ,r t ,s t+1 Add it to the experience pool;

[0049] (5.7) Determine whether the current iteration number is greater than the maximum exploration iteration number. If so, select multiple samples from the experience pool to train the neural network of the MADDPG model and continue to execute step (5.8); otherwise, execute step (5.8) directly.

[0050] (5.8) After incrementing the current iteration number E3 by one, determine whether it has reached the maximum total iteration number E1. If it has not reached the maximum total iteration number E1, return to step (5.3); otherwise, continue to step (5.9).

[0051] (5.9) End the training process and generate the trained MADDPG model parameter file.

[0052] Step Six. Load the trained MADDPG model parameter file and use the model to implement TDMA slot prediction for mobile ad hoc networks. Specifically, in this embodiment, the trained model parameter file is imported into the MADDPG model, and the neural network in the model is used to generate the action A(v) of the i-th node. i ,t), and derive the time slot allocation scheme based on the actions of all nodes.

[0053] Example 2: The overall implementation steps of the time slot allocation method proposed in this example are the same as in Example 1. Please refer to the appendix for details. Figure 2-4 The following provides a detailed explanation of the implementation process of this invention, including specific parameter settings:

[0054] Step 1. Construct a mobile self-organizing network scenario:

[0055] Assume a mobile ad hoc network consists of 8 nodes, each capable of bidirectional connections with its neighbors. The entire mobile ad hoc network can be modeled as an undirected graph G = <V, E>, where V is the set of nodes, |V| = 8, V = {v1, v2, ..., v8}, and E is the set of edges. Each edge represents a TDMA communication link between two nodes in the network. Each edge has a bidirectional link, and these links all satisfy the following condition: That is, nodes u and v (both belonging to set E) must be within each other's transmission range. Each node has a set of attributes to describe its state, including: the node's current load L(v). i ,t), the number of time slots currently allocated to the node S(v i The throughput of a node at the current time can be expressed as T(v). i ,t).

[0056] Step 2. Determine the frame structure of TDMA:

[0057] Compared with the baseline scheme, a frame structure from the actual scheme is selected as the frame structure for TDMA based on the constructed mobile ad hoc network scenario. In this embodiment, a superframe is preferably divided into 8 time slots, where the first frame is a preemptive frame and the following 7 are information frames; one frame includes 32 time slots, each time slot includes 10 time slot slices, and each time slot slice lasts for 1ms, meaning the duration of one superframe is 2.56s. The default base time slot slice for each node is 7, meaning at least 7 time slot slices are allocated to determine the frame structure; for example... Figure 2 As shown.

[0058] Step 3. Obtain network performance metrics:

[0059] Using a mobile ad hoc network model as the client of a reinforcement learning framework, the Exata simulation software is used to model the mobile ad hoc network, thereby obtaining network metrics such as system throughput and latency.

[0060] Step 4. Construct the reinforcement learning module:

[0061] Each node is associated with an agent. A state space, action space, reward function, and network structure are designed to construct a reinforcement learning module. Here, the state S represents the state of the environment, the reward R represents the feedback value given by the environment, and the action A represents the action decided by the agent. A communication framework using sockets is used to enable data transmission between the reinforcement learning module and the simulation platform.

[0062] The state space, action space, reward function, and network structure designed in this embodiment are as follows:

[0063] (4-1) State space design: The state space is four-dimensional data, including the throughput of the node at the previous time step, the number of time slots at the previous time step, the current load of the node, and the total throughput of each node within two hops of the current node.

[0064] (4-2) Action space design: According to the frame structure in step 2, the maximum number of time slots that can be allocated to each node is 2240, which is a discrete number. Therefore, the action space is also designed as a discrete value with 2240 dimensions, ranging from 0 to 2239.

[0065] (4-3) Reward Function Design: To maximize system throughput while maintaining a high level of fairness, the reward function of the reinforcement learning system is a weighted average of the overall system throughput and fairness parameters. The fairness parameter uses Jain's Fairness index to represent the fairness of time slot allocation. Assume there are N nodes in total, T... i W represents the throughput of the i-th node. i For each node's weight, which is related to the load, Jain's Faireness index can be expressed as:

[0066]

[0067] The larger the value of F, the better the fairness; n represents the number of nodes.

[0068] The global reward function can be expressed as:

[0069]

[0070] Where w is the weight, and in this embodiment, it is preferably set to 200.

[0071] (4-4) Network structure design: Since the state of a node's time slot depends not only on the state of the node itself, but also on the state of other nodes, this invention uses a multi-agent reinforcement learning framework for training. The MADDPG structure is used as the main framework for reinforcement learning, and the priority experience replay method is used for sample generation and extraction.

[0072] The architecture of the load platform and reinforcement learning module in steps 3 and 4 above is as follows: Figure 3 As shown.

[0073] Step 5. Train the MADDPG model:

[0074] The current load and current number of time slots of each node are input into the reinforcement learning framework. The framework outputs the time slot allocation scheme for the next time step and transmits the scheme to the simulation platform for simulation to obtain the throughput and load for the next time step.

[0075] (5-1) The simulation platform generates an initial load L(v) for each node. i ,0) and the initial number of time slots S(v i ,0), where the initial number of time slots for each node is 7, and the initial throughput T(v) of the i-th node is 0. i ,0) is 0;

[0076] (5-2) Use the communication module to connect L(v) i ,t)S(v i ,t) and T(v i The L(v,t) values ​​of all nodes are transmitted to the reinforcement learning module, which then transmits the L(v,t) values ​​of all nodes to the reinforcement learning module. i ,t)S(v i ,t) and T(v i The state space s at time t is constructed as follows: t ;

[0077] (5-3) Input the state space into the reinforcement learning module to obtain the action of each node, and then construct the action space a at time t based on the actions of all nodes. t This yields the time slot allocation scheme. In this embodiment, the number of nodes is randomly generated in the first 1000 iterations. In subsequent iterations, the neural network output of the MADDPG model is used, and a portion of samples are selected from the experience pool for training to obtain the corresponding model file.

[0078] (5-4) Input the time slot allocation scheme parameters into the simulation platform for simulation to obtain the current throughput T(v) i ,t), current load L(v) i ,t), Current node time slot number S(v i ,t).

[0079] (5-5) Repeat steps (5-2) to (5-4), and set e t =(s t ,a t ,r t ,s t+1 ) is treated as a sample and placed into the experience pool, where r t For the reward at time t, s t+1 This is the state space to be constructed for the next time step.

[0080] Step 6. Import the model file trained in Step 5 and follow the same procedure as in Step 5. The difference is that in Step (5-3), training is not required; only the allocation scheme needs to be derived. The training process in Step 5 is improved as follows:

[0081] In step (5-3), the number of time slots for each node in the first 1000 cycles is randomly generated. However, for the MANET environment, since the number of selectable time slots for each node in TDMA is large and the number of nodes is large, the reinforcement learning action space is large. At this time, using the random exploration mechanism will lead to a long exploration period without finding a good time slot allocation result, so that the data in the experience pool cannot contain a lot of effective data, which will eventually lead to getting stuck in a local optimum during the training phase.

[0082] This invention proposes a novel exploration mechanism: in a single exploration, if the throughput of the current exploration reaches a set threshold, then this set of results is marked as a "good result." The next exploration will then bypass the random exploration mechanism and switch to this mechanism. In this mechanism, based on the "good results," a fine-tuning strategy is applied to the time slot allocation, and the samples are then placed back into the environment to obtain a new set of samples. If the throughput of the new samples improves, further fine-tuning will continue; if the throughput decreases, fine-tuning will be performed again on the previous "good samples." To prevent entering an infinite loop, a maximum of a finite number of fine-tuning operations are allowed per fine-tuning stage.

[0083] In addition, the experience pool selection method in step (5-3) adopts a priority experience replay method. This mechanism uses TD-error (temporal error) as a metric to evaluate the priority of the sampled data. TD-error refers to the difference between the current Q-value and its target Q-value; a larger error indicates that the data is more helpful for updating the network parameters. Training is performed by greedily sampling data with large TD-error values ​​(selecting the maximum value), thereby accelerating training convergence. The optimized training process is as follows: Figure 4 As shown.

[0084] The effects of the present invention will be further explained below with reference to simulation experiments.

[0085] 1. Simulation conditions:

[0086] The simulation experiments of this invention were conducted in a hardware environment with 32GB of memory, an AMD 5800H CPU, and a 3060 GPU, and a software environment with Exata 7.2.0, C++14, Python 3.6, and PyTorch 1.10.

[0087] 2. Simulation content:

[0088] Using the Exata network simulation platform, the throughput and fairness of the traditional TDMA method, FPRP method, and the method of this invention were compared under different load conditions by changing the packet transmission interval. Load-throughput curves and load-fairness curves were obtained, as detailed below. Figure 5 and Figure 6 As shown.

[0089] 3. Simulation results:

[0090] Depend on Figure 5 It is evident that the time slot allocation method proposed in this invention has higher throughput than traditional TDMA under different load conditions, and its throughput curve is closer to that of the FPRP protocol.

[0091] pass Figure 6 It can be seen that traditional TDMA has the strongest fairness, and the fairness of this invention can also achieve a good effect, while the fairness of FPRP is relatively poor.

[0092] Therefore, it can be seen that the present invention has stronger competitiveness in practical applications. While having high throughput, it also has good fairness, effectively addressing the shortcomings of traditional technologies and improving the overall performance of network communication.

[0093] The above simulation analysis proves the correctness and effectiveness of the method proposed in this invention.

[0094] The parts of this invention not described in detail are common knowledge to those skilled in the art. The above descriptions are merely preferred embodiments of this invention and are not intended to limit the invention. Obviously, those skilled in the art, after understanding the content and principles of this invention, may make various modifications and changes in form and detail without departing from the principles and structure of this invention. However, these schemes based on the ideas of this invention and using a multi-agent reinforcement learning algorithm and a time slot allocation algorithm for joint design are all within the scope of protection of this invention.

Claims

1. A distributed TDMA time slot allocation method based on multi-agent reinforcement learning, characterized in that, The implementation steps include the following: (1) Construct a mobile ad hoc network scenario, including nodes, topology, and traffic model; (2) Determine the frame structure of TDMA according to the constructed mobile ad hoc network scenario, and determine it in the following way: Divide a superframe into N frames, where the first frame is a preemptive frame and the following N-1 frames are information frames; then divide a frame into Q time slots, and each time slot includes P time slot slices, and the duration of each time slot slice is 1ms, that is, the duration of a superframe is N*Q*P milliseconds; set the basic time slot slice of each node to N-1, that is, each node is allocated at least N-1 time slot slices; (3) Using the mobile ad hoc network scenario and TDMA frame structure, a mobile ad hoc network simulation platform is constructed to obtain network performance indicators; (4) Assign each node to an agent, design the state space, action space, reward function and network structure, and construct a reinforcement learning module; A communication module for data transmission between the reinforcement learning module and the mobile ad hoc network simulation platform is established using Socket. The design of the state space, action space, reward function, and network structure, and the construction of the reinforcement learning module, are implemented through the following steps: (4.1) Design a state space containing four-dimensional data, wherein the four-dimensional data is the throughput of the node at the previous time step, the number of time slots at the previous time step, the current load of the node, and the sum of the throughput of each node within two hops of the current node; (4.2) Based on the frame structure in step (2), the maximum number of time slots that can be allocated to each node is: It is a discrete value, and the action space is designed as Discrete values ​​of a dimension; (4.3) The overall system throughput and fairness parameters are weighted and used as the reward function of the reinforcement learning system; assuming For the first The throughput of each node For the first The weights of each node, and the fairness parameter is expressed as follows: ， The larger the F value, the better the fairness; The global reward function can be expressed as: ， in The weight value of the fairness parameter represents the ratio of the overall system throughput to the fairness parameter. (4.4) The structure of the MADDPG model is used as the main framework for reinforcement learning to construct the network structure of the reinforcement learning module, train it, and use the priority experience replay method for sample generation and extraction. (5) Use the constructed simulation platform and reinforcement learning module to train the multi-agent deep deterministic policy gradient (MADDPG) model and obtain the trained MADDPG model parameter file; (6) Load the trained MADDPG model parameter file and use the model to realize TDMA time slot prediction for mobile ad hoc networks.

2. The method according to claim 1, characterized in that: The construction of the mobile ad hoc network scenario in step (1) is implemented as follows: Assume that the mobile ad hoc network consists of n nodes, and each node has bidirectional connections with its neighboring nodes; model the entire mobile ad hoc network as an undirected graph. Where V represents the set of nodes, , Let E be the set of edges. Each edge represents a TDMA communication link between two nodes in the network. Each edge has a bidirectional link, and all links satisfy the following conditions. , i.e., node and Located within each other's communication range; Indicates the first Nodes Current moment The load, Indicates the first Nodes Current moment The number of time slots allocated, Indicates the first Nodes Current moment throughput.

3. The method according to claim 1, characterized in that: The network performance metrics mentioned in step (3) are obtained by modeling mobile ad hoc networks using Exata simulation software, with the mobile ad hoc network simulation platform as the client of the reinforcement learning framework. These metrics include at least system throughput, load, and fairness parameters.

4. The method according to claim 1, characterized in that: Step (5) involves training the MADDPG model using the constructed simulation platform and reinforcement learning module. This involves inputting the current load and current number of time slots of each node into the reinforcement learning framework, outputting the time slot allocation scheme for the next time step through the reinforcement learning framework, and transmitting the scheme to the simulation platform for simulation to obtain the throughput and load for the next time step. The specific implementation steps include the following: (5.1) Let the maximum total number of iterations be E1 and the maximum number of exploration iterations be E2, and E1>E2; initialize the model parameter saving directory and initialize the current iteration number to 0; (5.2) The simulation platform is the first Each node generates the initial load. and initial number of time slots The initial number of time slots for each node is N-1, and the number of time slots for each node is... Initial throughput of each node =0; (5.3) Use a communication module to , and The data is transmitted to the reinforcement learning module, which then transmits it to all nodes. , and Construct the state space at time t ; (5.4) Input the state space into the reinforcement learning module, determine whether the current iteration number has reached the set maximum number of exploration iterations. If not, enter the exploration module, and obtain the nth iteration from the exploration module. Actions of each node Conversely, the first value is obtained from the neural network of the MADDPG model. Actions of each node Then, the action space at time t is constructed based on the actions of all nodes. In other words, the time slot allocation scheme; (5.5) will The data is input into a simulation platform to obtain the throughput, load, and number of node time slots at the next time step. , and and the reward at time t. And construct the state space for the next time step. ; (5.6) will As a sample Add to the experience pool; (5.7) Determine whether the current iteration number is greater than the maximum number of exploration iterations. If so, select multiple samples from the experience pool to train the neural network of the MADDPG model and continue to execute step (5.8); otherwise, execute step (5.8) directly. (5.8) After incrementing the current iteration number E3 by one, determine whether it has reached the maximum total iteration number E1. If it has not reached the maximum total iteration number E1, return to step (5.3); otherwise, continue to step (5.9). (5.9) End the training process and generate the trained MADDPG model parameter file.

5. The method according to claim 1, characterized in that: Step (6) describes the implementation of TDMA time slot prediction in mobile ad hoc networks. Specifically, it involves importing the trained model parameter file into the MADDPG model and using the neural network in the model to generate the first... Actions of each node And based on the actions of all nodes, a time slot allocation scheme is derived.

6. The method according to claim 4, characterized in that: The exploration module employs a novel exploration mechanism. Specifically, in an exploration, if the throughput reaches a set threshold, the results are marked as "good results," and the next exploration will use this mechanism instead of a random exploration mechanism. Within this mechanism, based on the "good results," the time slot allocation is fine-tuned before being added to the experience pool to obtain a new set of samples. If the throughput of the new samples is improved, further fine-tuning is performed. Conversely, fine-tune based on the previous "good result"; at the same time, set a maximum number of fine-tuning times in a fine-tuning phase to prevent entering an infinite loop.

7. The method according to claim 6, characterized in that: The fine-tuning is implemented as follows: like If it is higher than the average value of all node time slots, then adjust. Otherwise adjust ; where x is the fine-tuning coefficient.

8. The method according to claim 6, characterized in that: The experience pool uses a priority experience replay mechanism to select samples, and the priority experience replay mechanism uses the time series error (TD-error) as a metric to evaluate the priority of the sampled data.

Images