WiFi intelligent multi-link access method and system based on multi-agent reinforcement learning

Abstract

The present invention belongs to the field of network operation and service technology, and more particularly relates to a WiFi multi-link access method and system based on multi-agent reinforcement learning. The method comprises: modeling the WiFi multi-link access problem as a unified Markov decision process model, defining the actions, states, and reward functions therein; centrally training the access point (AP) with local observations of a single link from each multi-link device, and solving the optimal access strategy using the MADDPG multi-agent reinforcement learning algorithm; the AP distributes the trained multi-agents to the corresponding multi-link devices; and the multi-link devices, guided by the trained agents, perform distributed multi-link access. This invention can meet the high throughput requirements of future emerging applications, achieving higher network throughput on each link than the performance limit of traditional DCF-based access, with an optimization range of 11.2% to 23.9%.

Description

Technical Field

[0001] The present invention belongs to the technical field of network operation services, and in particular relates to a WiFi intelligent multi-link access method and system based on multi-agent reinforcement learning. Background Art

[0002] With the rapid development of communications, computing, and artificial intelligence technologies, emerging applications such as virtual reality, augmented reality, and telemedicine are rapidly emerging, with video resolution gradually shifting towards 4K and even 8K. These applications demand ultra-high throughput, ultra-high reliability, and ultra-low latency. Furthermore, these applications are often concentrated indoors, and communication is primarily based on WiFi, posing significant challenges to existing WiFi technology. Consequently, the IEEE 802.11be Working Group has released a new revised standard, IEEE 802.11be EHT (Ethernet HT), or WiFi 7, to enhance WiFi transmission capabilities. One of the key enhancements at the MAC layer is multi-link operation, which significantly increases bandwidth. WiFi 7 multi-link operation technology allows multi-link devices (MLDs) to operate simultaneously in the 2.4 GHz, 5 GHz, and 6 GHz bands. Multi-link-capable devices can concurrently transmit data frames over multiple links, aiming to achieve higher throughput, lower latency, and greater reliability. While these multi-link operations are potentially described, the specific channel access mechanisms for multi-link operation remain undefined. Existing Wi-Fi multi-link access methods are primarily based on mathematical modeling and analysis. These methods are limited by the backoff window in traditional access protocols, making it difficult to maximize link speeds. Fixed protocols also struggle to adapt to complex and changing network environments. On the other hand, the Wi-Fi multi-access problem can be modeled as a Markov decision process, making reinforcement learning a promising tool for solving this problem. However, existing intelligent access algorithms are limited to single-link networks and are difficult to extend to complex multi-link Wi-Fi networks with varying link states. Therefore, a new Wi-Fi intelligent multi-link access method is urgently needed to address the shortcomings and drawbacks of existing technologies.

[0003] Through the above analysis, the existing technology has the following problems and defects:

[0004] 1. Introducing multi-agent reinforcement learning methods: WiFi multiple access can be viewed as a decision-making problem, which can be optimized using reinforcement learning methods. However, the centralized execution of traditional reinforcement learning algorithms is not suitable for distributed terminal access scenarios. Therefore, a multi-agent reinforcement learning method is needed to enable distributed intelligent decision-making among smart terminals. Multi-agent reinforcement learning algorithms based on centralized training and distributed execution can improve training effectiveness and generalization capabilities, and are suitable for real-world distributed network scenarios.

[0005] 2. Optimizing throughput while ensuring fairness: In Wi-Fi multi-link scenarios, the quality of each link varies, and the transmission rate also varies. In this novel and complex scenario, it is necessary to optimize the total throughput of the entire system and ensure that all intelligent terminal nodes can access the network fairly, providing efficient and fair service quality for emerging services.

[0006] 3. Universal Markov Decision Process Modeling: Existing Wi-Fi smart access algorithms primarily use Markov decision process modeling for single-link network scenarios. This modeling process is strongly correlated with the transmission rate of a single link. However, in multi-link network scenarios, each link has different parameters and transmission rates. This traditional modeling process cannot be generalized to links of different rates. Therefore, a universal Markov decision process modeling method is needed.

[0007] 4. Adaptability to dynamic network environments: Network environments are complex and ever-changing. Varying numbers of nodes and changes in transmission rates can lead to changes in optimal access strategies. Therefore, the new access method can adapt to network links of varying quality and dynamically adjust access strategies to ensure optimal performance when the number of nodes changes. Summary of the Invention

[0008] In response to the problems existing in the prior art, the present invention provides a WiFi intelligent multi-link access method and system based on multi-agent reinforcement learning.

[0009] The present invention is implemented as follows: a WiFi intelligent multi-link access method based on multi-agent reinforcement learning, the method comprising:

[0010] Step 1: Model the WiFi multi-link access problem as a unified Markov decision process model, defining the actions, states, and reward functions.

[0011] In step 2, the AP obtains local observations of a single link from each multi-link device and conducts centralized training. The MADDPG multi-agent reinforcement learning algorithm is used to solve the optimal access strategy.

[0012] Step 3: AP distributes the trained multi-agents to the corresponding multi-link devices;

[0013] Step 4: Multi-link devices perform distributed multi-link access under the guidance of the trained intelligent agent.

[0014] Furthermore, the WiFi multi-link access method based on multi-agent reinforcement learning can adjust the multi-link access strategy according to the real-time link utilization status and respective access strategies, thereby maximizing link utilization.

[0015] Furthermore, a network consists of one AP and n multi-link devices, and the traffic of these n multi-link devices is all high-throughput saturated traffic type, that is, the network system needs to maximize its total throughput to meet the high throughput requirements and ensure fairness among all devices.

[0016] Furthermore, the step 1 specifically includes: introducing new parameter access opportunities as time points for Markov state transitions, and designing states independent of link transmission parameters, thereby establishing a unified distributed partially observable Markov decision process model for different links.

[0017] Furthermore, the unified distributed locally observable Markov decision process model includes action, state and reward functions, specifically:

[0018] Action, represented by a, where Its meaning is that the action of agent j∈{1, 2, ..., n} on the tth access opportunity on link j∈{1, 2, 3} is to transmit a data packet or to wait;

[0019] State, the state of a single agent is represented by s, which contains the action-observation pairs of the past M historical time steps, namely:

[0020]

[0021] in That is, the action-observation pair of agent i's t-th access opportunity on link j, That is, the local observation of agent i, which consists of two parts. The first part is the link observation value Specifically:

[0022]

[0023] That is, the local observation of the link by the agent includes four types, and in order to explain the link observation value The second part introduces new parameters represents the number of access opportunities of agent i on link j since the last successful transmission, and It represents the number of access opportunities of other agents except agent i on link j since the last successful transmission, and then the two are normalized to form The second part, in as well as

[0024] As for the joint state required for centralized training,

[0025] The reward function is represented by r, and the total reward of agent i for the tth access opportunity on link h is It is defined as:

[0026]

[0027] where r j，t，global represents the global reward, represents the individual reward of agent i, and α represents the balance coefficient. Specifically:

[0028]

[0029] in That is, the ratio of the number of waiting access opportunities of agent i to the total number of waiting access opportunities on link j;

[0030]

[0031] Furthermore, in step three, the intelligent multi-link device agent (Agent) adopts a multi-agent reinforcement learning algorithm under a centralized training distributed execution framework. The agent includes an Actor network and a Critic network. The Critic network is a traditional three-layer fully connected network Critic network with a hidden layer of 128 neurons, and the Actor network is a four-layer fully connected network with a hidden layer of 64 neurons and a gated recurrent unit (GRU). During centralized training, the Critic network is responsible for processing global observation information to better evaluate the execution strategy of the Actor network, and the Actor network is responsible for outputting access strategies based on local observation information. The experience replay pool is used to record all samples.<x,a,r,x′> , where x represents the joint state of the current access opportunity, a represents the joint action of each agent under the current access opportunity, x′ represents the next state to be migrated after executing action a in state x, and r represents the reward obtained after executing action s in state x.

[0032] Furthermore, in step 2, the MADDPG multi-agent reinforcement learning algorithm is used to solve the optimal access strategy, specifically:

[0033] Initialize the Actor network and Critic network parameters π θ (a|s);

[0034] Initialize the target network parameter value function parameter θ′=θ;

[0035] Initialize the experience replay pool EM;

[0036] Initialize other parameters such as n, γ, α, τ, M, I;

[0037] Centralized training phase: Each agent only needs to monitor the link on a single link j to obtain access opportunities. Each agent uses its own local observation state Get actions through the Actor network Perform joint actions Get rewards j，t and the new joint state x′ j，t ; The experience gained from each access opportunity t<x,a,r,x′> Deposit into the experience replay pool EM;

[0038] Perform a network update every time you access the network:

[0039] Randomly sample bs experience from EM as E, and define the loss function of the Critic network based on each experience:

[0040]

[0041] Update the Actor network according to the policy gradient:

[0042]

[0043] Then update the target network θ′ of each agent i ←τθ i +(1-τ)θ′ i ;

[0044] After a certain training period and training convergence, the Actor network parameters are the optimal access strategy and are applicable to any link with different parameters. The AP distributes the Actor network to each multi-link device, and the multi-link devices perform distributed access.

[0045] Another object of the present invention is to provide a WiFi multi-link access system based on multi-agent reinforcement learning that applies the WiFi multi-link access method based on multi-agent reinforcement learning, the system comprising:

[0046] The information acquisition module is used to use the AP to obtain the local link observation value corresponding to each multi-link device and aggregate it into a joint state;

[0047] The modeling module is used to model the link access problem with different WiFi parameters as a unified distributed partially observable Markov decision process model, defining the actions, states, and reward functions;

[0048] The solution module is used to solve the optimal access strategy during the centralized training phase using the MADDPG multi-agent reinforcement learning algorithm;

[0049] The execution module is used to control the multi-link device to access by using the trained optimal access strategy obtained from the AP side.

[0050] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the WiFi multi-link access method based on multi-agent reinforcement learning.

[0051] Another object of the present invention is to provide an information data processing terminal, which is used to implement the WiFi multi-link access system based on multi-agent reinforcement learning.

[0052] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0053] First, (1) the method of the present invention is applicable to multi-link scenarios with different link parameters, and can achieve a network throughput close to the optimal state while ensuring fairness among the multi-link devices. This means that the invention can be used in various practical application scenarios and has universal applicability.

[0054] (2) The present invention can achieve a higher network throughput than the performance limit of the traditional DCF-based access method on all links, with the optimization range ranging from 11.2% to 23.9%, which reflects the advantages of the intelligent access method.

[0055] (3) The WiFi intelligent multi-link access algorithm adopted by the present invention adopts a centralized training and distributed execution architecture. The centralized training phase can optimize the training effect and explore the optimal strategy, while the distributed execution phase is simple and efficient. It only requires each multi-link device to observe the local information of the network to obtain the optimal access method, thus realizing the simple operation of intelligent access and meeting the deployment conditions of actual scenarios.

[0056] (4) The WiFi intelligent multi-link access algorithm of the present invention uses a unified Markov model for different links, so that the optimal strategy trained in a single-link environment can be applied to different links. Currently, mainstream intelligent access algorithms are limited to single-link scenarios. This invention fills the research gap in the field of intelligent multi-link access.

[0057] (5) The WiFi intelligent multi-link access algorithm of the present invention is also robust to dynamic environments. That is, if the number of devices in the network changes, a pre-trained network model can be loaded to dynamically adapt to the changing environment and quickly adapt to the changed environment. Therefore, it can maintain access results close to the optimal throughput in a dynamically changing network environment, improving network performance and maximizing the new features of WiFi 7 multi-link.

[0058] (6) Each step in the present invention represents a significant technical advancement for WiFi 7 multi-link access, as follows:

[0059] Step 1: By introducing a new parameter: access opportunity, this paper models the WiFi multi-link access problem as a unified Markov decision process model, defining the actions, states, and reward functions. This is one of the key innovations of this paper. By doing so, the optimal policy trained in a single-link environment can be adapted to multi-link scenarios, filling a gap in this research area.

[0060] In step 2, the AP obtains local observations of a single link from each multi-link device and performs centralized training. The optimal access strategy is then solved using the MADDPG multi-agent reinforcement learning algorithm. This approach is another key innovation of the present invention. The MADDPG algorithm is a multi-agent reinforcement learning algorithm within a centralized training and distributed execution framework. Centralized training significantly improves training effectiveness, while distributed execution greatly simplifies the algorithm's deployment and execution.

[0061] Step 3: The AP distributes the centrally trained multi-agents, i.e., the corresponding Actor network parameters, to the corresponding multi-link devices.

[0062] In step 4, multi-link devices can load the trained intelligent agent and perform distributed multi-link access under its guidance. During distributed execution, each device only needs its own local observation information of the link to achieve optimal access on multiple links.

[0063] Second, the present invention can adaptively adjust the optimal multi-link access strategy based on the real-time link status and network device status. On the one hand, it makes the access strategy more precise, and on the other hand, it improves the adaptability and robustness of the network, allowing the network to always maintain a near-optimal state in various dynamic or static environments and states. This is also a significant technological advancement.

[0064] The technical solution of this invention fills a technological gap in the field of WiFi multi-access both domestically and internationally. Applying multi-agent reinforcement learning methods to the field of multi-access has been a hot topic in the industry in recent years. However, existing intelligent access algorithms and Markov modeling approaches are limited to single-link network environments and are difficult to extend to complex multi-link WiFi network scenarios with varying link states. No intelligent access algorithm suitable for multi-link scenarios has been proposed. The technical solution of this invention fills a technological gap in the field of WiFi multi-access both domestically and internationally. It models the WiFi multi-link access problem as a unified Markov decision process model. The proposed access algorithm can also perform distributed intelligent access across different links.

[0065] Third, the present invention aims to solve the following technical problems existing in the prior art:

[0066] Low network resource utilization: Traditional WiFi access methods easily lead to low network resource utilization when multiple devices access concurrently, and cannot fully utilize existing bandwidth and link resources.

[0067] Network congestion and interference: When multiple devices access the network simultaneously, traditional methods are unable to effectively avoid network congestion and interference, resulting in degraded network performance and poor user experience.

[0068] Lack of intelligent scheduling mechanism: Traditional methods lack an intelligent scheduling mechanism and cannot dynamically adjust access strategies based on real-time network status, resulting in unstable network performance.

[0069] Fairness issue: When multiple devices are connected, traditional methods cannot ensure that each device receives fair network resource allocation, and some devices may be at a disadvantage for a long time.

[0070] This paper introduces a multi-agent reinforcement learning algorithm to intelligently manage WiFi multi-link access, achieving the following significant technological advancements:

[0071] Improve network resource utilization: By modeling the WiFi multi-link access problem as a Markov decision process and optimizing it using a multi-agent reinforcement learning algorithm, each device can intelligently select the optimal access link based on the real-time network status, thereby improving overall network resource utilization.

[0072] Reduce network congestion and interference: Using the MADDPG multi-agent reinforcement learning algorithm, through centralized training and distributed execution, each device can efficiently access the network while avoiding congestion and interference, significantly improving network performance.

[0073] Realize intelligent scheduling: The present invention introduces a real-time dynamic adjustment mechanism. Each device can dynamically adjust its multi-link access mode according to the current link utilization status and access strategy to ensure optimal access under different network conditions.

[0074] Ensure fairness: By designing a reasonable reward function and multi-agent reinforcement learning algorithm, we ensure that each device can obtain fair resource allocation when accessing the network, preventing some devices from being at a disadvantage for a long time and improving the overall user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 This is an algorithm diagram illustrating a WiFi multi-link access method based on multi-agent reinforcement learning provided by an embodiment of the present invention;

[0076] Figure 2 This is a schematic diagram of a network scenario provided by an embodiment of the present invention;

[0077] Figure 3Schematic diagram of access opportunities and agent-guided multi-link access provided by an embodiment of the present invention;

[0078] Figure 4 1 is a schematic diagram of the architecture of a WiFi multi-link access algorithm based on multi-agent reinforcement learning provided by an embodiment of the present invention;

[0079] Figure 5 This is a diagram showing the results of centralized training of an agent in a single-link environment provided by an embodiment of the present invention;

[0080] Figure 6 This is a diagram showing simulation results of an agent under links with different transmission rates provided by an embodiment of the present invention;

[0081] Figure 7 This is a graph showing multi-link throughput results for different numbers of multi-link devices provided by an embodiment of the present invention;

[0082] Figure 8 This is a diagram showing the multi-link throughput results when the number of multi-link devices changes dynamically, as provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0084] like Figure 1 As shown, the WiFi multi-link access method based on multi-agent reinforcement learning provided by the embodiment of the present invention includes:

[0085] 1. Model the WiFi multi-link access problem as a unified Markov decision process model, defining the actions, states, and reward functions.

[0086] 2. The AP obtains local observations of a single link from each multi-link device and performs centralized training. The MADDPG multi-agent reinforcement learning algorithm is then used to solve the optimal access strategy.

[0087] 3. The AP distributes the trained multi-agents to the corresponding multi-link devices;

[0088] 4. Multi-link devices perform distributed multi-link access under the guidance of trained intelligent agents.

[0089] The WiFi multi-link access method based on multi-agent reinforcement learning can adjust the multi-link access strategy according to the real-time link utilization status and respective access strategies, thereby maximizing link utilization.

[0090] The network scenario of the present invention is as follows Figure 2As shown in Figure 1, n multi-link devices compete for links in three different frequency bands, 2.4GHz, 5GHz, and 6GHz, in a WiFi 7 multi-link scenario to obtain transmission opportunities to the access point. The multiple wireless interfaces in each multi-link device can access each link independently and can transmit frames asynchronously on different links. The transmission rate of each link is expressed as R j Mbps, j = 1, 2, 3, depending on the link bandwidth and modulation / coding scheme used by the three links. Furthermore, the traffic on these n multi-link devices is all high-throughput saturation traffic. This means that the network system needs to maximize its total throughput to meet high throughput requirements while ensuring fairness among devices.

[0091] The present invention introduces new parameter access opportunities as the time points of Markov state transitions, such as Figure 3 As shown in the figure, once a multi-link device senses a distributed interframe space (DIFS) idle time on a certain link, it obtains an access opportunity on the link, which is independent of the state of the link transmission parameters, thereby establishing a unified distributed partially observable Markov decision process model for different links.

[0092] A unified distributed locally observable Markov decision process model, including action, state, and reward functions, is:

[0093] Action, represented by a, where Its meaning is that the action of agent i∈{1, 2, ..., n} on the tth access opportunity on link j∈{1, 2, 3} is to transmit a data packet or to wait;

[0094] State, the state of a single agent is represented by s, which contains the action-observation pairs of the past M historical time steps, namely:

[0095]

[0096] in That is, the action-observation pair of agent i's t-th access opportunity on link j, That is, the local observation of agent i, which consists of two parts. The first part is the link observation value Specifically:

[0097]

[0098] That is, the local observation of the link by the agent includes four types. In order to explain the link observation value The second part introduces new parameters represents the number of access opportunities of agent i on link j since the last successful transmission, and It represents the number of access opportunities of other agents except agent i on link j since the last successful transmission, and then the two are normalized to form The second part, in as well as

[0099] As for the joint state required for centralized training,

[0100] The reward function is represented by r, and the total reward of agent i for the tth access opportunity on link j is represented by It is defined as:

[0101]

[0102] where r j，t，global represents the global reward, represents the individual reward of agent i, and α represents the balance coefficient. Specifically:

[0103]

[0104] in That is, the ratio of the number of waiting access opportunities of agent i to the total number of waiting access opportunities on link j.

[0105]

[0106] Figure 4 The schematic diagram of the WiFi multi-link access algorithm architecture based on multi-agent reinforcement learning provided by an embodiment of the present invention is shown. The multi-link device agent (Agent) adopts a multi-agent reinforcement learning algorithm under a centralized training distributed execution framework. The agent includes an Actor network and a Critic network. The Critic network is a traditional three-layer fully connected network with 128 neurons in the hidden layer, and the Actor network is a four-layer fully connected network with 64 neurons in the hidden layer and a gated recurrent unit (GRU). During centralized training, the Critic network is responsible for processing global observation information to better evaluate the execution strategy of the Actor network, and the Actor network is responsible for outputting the access strategy based on local observation information. The experience replay pool is used to record all samples.<x,a,r,x′> , where x represents the joint state of the current access opportunity, a represents the joint action of each agent under the current access opportunity, x′ represents the next state to be migrated after executing action a in state x, and x′ represents the reward obtained after executing action a in state x.

[0107] The MADDPG multi-agent reinforcement learning algorithm is used to solve the optimal access strategy, specifically:

[0108] Initialize the Actor network and Critic network parameters π θ (a|s);

[0109] Initialize the target network parameter value function parameter θ′=θ;

[0110] Initialize the experience replay pool EM;

[0111] Initialize other parameters such as n, γ, α, τ, M, I;

[0112] Centralized training phase: Each agent only needs to monitor the link on a single link j to obtain access opportunities. Each agent uses its own local observation state Get actions through the Actor network Perform joint actions Get rewards j，t and the new joint state x′ j，t . The experience gained from each access opportunity t<x,a,r,x′> Deposit into the experience replay pool EM;

[0113] Perform a network update every time you access the network:

[0114] Randomly sample bs experience from EM as E, and define the loss function of the Critic network based on each experience:

[0115]

[0116] Update the Actor network according to the policy gradient:

[0117]

[0118] Then update the target network θ′ of each agent i ←τθ i +(1-τ)θ′ i .

[0119] After a certain training period and training convergence, the Actor network parameters are the optimal access strategy and are applicable to any link with different parameters. The AP distributes the Actor network to each multi-link device, and the multi-link devices perform distributed access.

[0120] Preferably, the WiFi multi-link access system based on multi-agent reinforcement learning provided in an embodiment of the present invention includes:

[0121] The information acquisition module is used to use the AP to obtain the local link observation value corresponding to each multi-link device and aggregate it into a joint state;

[0122] The modeling module is used to model the link access problem with different WiFi parameters as a unified distributed partially observable Markov decision process model, defining the actions, states, and reward functions;

[0123] The solution module is used to solve the optimal access strategy during the centralized training phase using the MADDPG multi-agent reinforcement learning algorithm;

[0124] The execution module is used to control the multi-link device to access by using the trained optimal access strategy obtained from the AP side.

[0125] Figure 5 The results of centralized training of the WiFi multi-link access algorithm based on multi-agent reinforcement learning in a single-link environment are shown in the figure. The training scenario uses a 40MHz bandwidth link with 256-QAM modulation, and its transmission rate is 229.4Mbps. For comparison, the throughput of centralized multiple access is considered to be the upper limit of the throughput. Since there is still overhead caused by the header packet in each successful transmission, It is equal to the ratio of the packet payload length to the time it takes for each successful transmission. This training process is used for four MLDs in a multi-link network environment and can be applied to multi-link scenarios. Figure 5 It shows that the network eventually converged with a total throughput of 198.85Mbps, and the throughput of each device was almost the same, which indicates that the training achieved near-optimal network throughput performance and maintained good fairness between nodes.

[0126] Figure 6 The following are simulation results of the embodiment of the present invention under different transmission rate links. In the actual WiFi 7 network, 2.4GHz, 5GHz and 6GHz links often have different transmission rates due to different bandwidths and link conditions. It can be clearly seen that the present invention can achieve a higher network throughput than the performance limit of traditional DCF-based access on all links, with the optimization range ranging from 11.2% to 23.9%. In addition, regardless of how the transmission rate changes, the throughput of each multi-link device on each link is almost the same, and the JFI index measuring fairness is close to 1. This shows that the intelligent agent trained by the embodiment of the present invention in a single-link environment can adapt well to links with different transmission rates, reflecting the universality of the unified distributed partially observable Markov decision process model proposed by the present invention.

[0127] Figure 7The average throughput of the embodiment of the present invention in multi-link network scenarios with varying numbers of multi-link devices is shown in Figure 2. It can be clearly seen that regardless of network size, the total throughput on each link is always close to optimal, and the JFI fairness index is 0.9999. This demonstrates that the present invention does not experience a significant performance degradation with an increase in the number of devices, and that the agents trained by the embodiment of the present invention can always use local observation information to fairly share link resources in a distributed environment.

[0128] Figure 8 The following table shows the multi-link throughput results for a scenario where the number of multi-link devices changes dynamically, simulating a dynamically changing network size by having multi-link devices join or leave at given times. Specifically, the device number changes at [3, 6, 9, 12, 15], and the number of devices after the change is [2, 3, 4, 3, 5, 3]. It is clear that when the network size changes, the multi-link devices are able to quickly maintain fair throughput, and the total throughput on each link is insensitive to changes in the number of links and always approaches the ideal optimal throughput. This demonstrates the excellent adaptability of the present invention to time-varying environments. During implementation, each multi-link device can store multiple sets of training actor network parameters to accommodate networks of varying sizes. Multi-link devices can obtain information about network size changes through broadcast messages with access points and update the actor network accordingly.

[0129] At the same time, the present invention does not require a complicated execution process. The centrally trained intelligent agent only needs the local observation information of each multi-link device as the input of the neural network to obtain the optimal access strategy at each moment, which is simple to implement and fast to execute.

[0130] The following is an implementation scheme for each step in the embodiment of the present invention:

[0131] Step 1: Model the WiFi multi-link access problem as the aforementioned unified Markov decision process model, defining its actions, states, and reward functions. In specific implementations, each intelligent multi-link device's action represents the access strategy it chooses for the current access opportunity (access or non-access). State information includes each device's record of its previous actions and link transmission status, as well as the number of access opportunities since the last successful transmission. The reward function is derived by the access point (AP) through a simple calculation based on the transmission results after each multi-link device executes an action.

[0132] Step 2: Centralized training is performed on the AP side. During this centralized training phase, the AP is trained using the Actor-Critic approach and the MADDPG multi-agent reinforcement learning algorithm. Specifically, the AP is configured with the corresponding MADDPG code. The AP obtains local observations of each multi-link device on a single link and converts them into corresponding actions, states, and rewards, which are then passed into the AP's code program for centralized training. Through continuous iterative interaction, the optimal access strategy is obtained.

[0133] In step 3, the AP centrally trains each intelligent agent, obtaining the optimal access strategy and corresponding actor network parameters. The AP then sends a data packet containing these parameters to the target address of each multi-link device. Since only the parameters between neural network layers are sent, the data packet size does not impose additional network load. Each multi-link device parses the received data packet to obtain the corresponding actor network parameters.

[0134] In step 4, the multi-link device loads the trained actor network parameters and performs distributed multi-link access under its guidance. During the distributed execution process, each device obtains its own transmission records or ACK information from the network and converts them into digital status information as input to the actor network. Each time an access opportunity is observed, the actor network executes the action and outputs the optimal access action at that time.

[0135] The following are five specific embodiments of the present invention, which explain in detail how to implement the WiFi multi-link access method based on multi-agent reinforcement learning in different scenarios:

[0136] Example 1:

[0137] In a future home scenario, each family member has several multi-link devices, such as computers and mobile phones for watching HD videos and using VR devices. These n multi-link devices are connected to the same WiFi access point (AP). These intelligent multi-link devices are equipped with the multi-agent reinforcement learning-based WiFi multi-link access method and protocol described in this invention. Based on acquired information about themselves and their links, they dynamically adjust their multi-link access strategies in real time to ensure optimal network resource utilization, achieve optimal throughput, maintain optimal network speed, and maintain fairness across devices, significantly improving the user experience.

[0138] Example 2:

[0139] In a smart enterprise office scenario, enterprise collaboration relies on cloud computing services and online collaboration tools, requiring a fast and stable network to ensure the efficiency and accuracy of data transmission. Each employee has a corresponding intelligent multi-link device, and n multi-link devices are connected to the same WiFi AP. These intelligent multi-link devices are all equipped with the WiFi multi-link access method protocol based on multi-agent reinforcement learning described in the present invention. By dynamically adjusting the multi-link access strategy, it is ensured that network resources can always be utilized as much as possible to achieve optimal throughput, ensuring an optimal network speed and maintaining fairness between devices. This makes collaboration between employees more stable and efficient, ensuring that each employee has a good office experience and improving overall enterprise efficiency.

[0140] Example 3:

[0141] In smart education scenarios, such as universities and higher education institutions, students and faculty frequently upload and download learning materials and participate in online courses and video conferencing, all of which place extremely high throughput requirements. Teachers and students are equipped with corresponding intelligent multi-link devices, all equipped with the multi-agent reinforcement learning-based WiFi multi-link access method and protocol described in this paper. This ensures that network resources are always utilized to achieve optimal throughput and guarantee optimal network speeds, eliminating lags in video conferencing and online courses, and ensuring a positive teaching experience.

[0142] Example 4:

[0143] In smart manufacturing scenarios, automated and intelligent equipment on production lines require real-time transmission of large amounts of data. To improve transmission efficiency, each smart device is equipped with a multi-link network card and the multi-link WiFi access method and protocol based on multi-agent reinforcement learning described in this paper. Connecting n multi-link devices to the same WiFi access point, with access strategies guided by the smart device, ensures that each production device has a high network speed, maximizes network resources, and enables smoother interaction between devices, thereby improving production efficiency.

[0144] Example 5:

[0145] In large-scale intelligent conferences and exhibitions, numerous participants typically access the network simultaneously for real-time information exchange, requiring the WiFi network to handle high concurrent connections and high traffic. Consider n multi-link devices connected to the venue's WiFi. These intelligent multi-link devices are equipped with the multi-agent reinforcement learning-based WiFi multi-link access method and protocol described in this paper. Under the guidance of the intelligent agent, each device can execute the optimal access strategy, achieving near-zero-collision access, maximizing network resource utilization, and providing attendees with an excellent network experience.

[0146] Through the above embodiments, it can be seen that the WiFi multi-link access method based on multi-agent reinforcement learning has wide application value in different scenarios and can effectively solve the extremely high demands of future application scenarios on WiFi throughput and network speed.

[0147] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware portion can be implemented using dedicated logic; the software portion can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. Those skilled in the art will appreciate that the above-mentioned devices and methods can be implemented using computer-executable instructions and / or contained in processor control code, for example, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, or can be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0148] The multi-agent reinforcement learning-based WiFi multi-link access method provided by the embodiments of the present invention can achieve higher network throughput than traditional DCF-based WiFi access methods. Due to the inherent conflicts and backoff mechanisms of the traditional DCF protocol based on carrier sense multiple access (CSMA), this traditional protocol-based WiFi network has long suffered from limited access efficiency, and throughput has been difficult to reach the theoretical limit. However, the intelligent access algorithm proposed in the embodiments of the present invention can effectively avoid conflicts through intelligent perception and eliminates the need for backoff windows, saving resources and maintaining near-maximum throughput in a variety of scenarios.

[0149] The experimental scenario is as follows: Consider a multi-link WiFi network scenario, where the three link frequency bands are 2.4GHz, 5GHz and 6GHz respectively. Due to different bandwidths and link conditions, the three links have different transmission rates, which are 114.7Mbps, 229.4Mbps and 480.4Mbps respectively. There are four devices on the multi-link network for distributed access. Traditional multi-link devices independently use the DCF protocol based on CSMA / CA for access on each link; while the WiFi multi-link access method based on multi-agent reinforcement learning provided by the embodiment of the present invention performs distributed intelligent access through the steps described above. The experimental results are as follows: Figure 6As shown in the figure, it is clear that the present invention achieves significantly higher network throughput than traditional DCF-based access on all links, with optimization ranging from 11.2% to 23.9%. Furthermore, regardless of the transmission rate, the throughput of each multi-link device on each link remains nearly consistent, with the JFI index (JFI), a measure of fairness, approaching 1. This demonstrates that the agent trained in a single-link environment can adapt well to links with varying transmission rates, demonstrating the significant advantages of the present invention over existing technologies.

[0150] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with this technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered by the scope of protection of the present invention.

Claims

1. A WiFi intelligent multi-link access method based on multi-agent reinforcement learning, characterized in that: The method includes: Step 1: Model the WiFi multi-link access problem as a unified Markov decision process model, defining the actions, states, and reward functions. In step 2, the AP obtains local observations of a single link from each multi-link device and conducts centralized training. The MADDPG multi-agent reinforcement learning algorithm is used to solve the optimal access strategy. Step 3: AP distributes the trained multi-agents to the corresponding multi-link devices; Step 4: Multi-link devices perform distributed multi-link access under the guidance of the trained agent; The step 1 specifically includes: introducing new parameter access opportunities as time points for Markov state transitions, and designing states independent of link transmission parameters, thereby establishing a unified distributed partially observable Markov decision process model for different links; The unified distributed locally observable Markov decision process model includes action, state and reward functions, specifically: Action, represented by a, where Its meaning is that the action of agent i∈{1, 2, ..., n} on the tth access opportunity on link j∈{1, 2, 3} is to transmit a data packet or to wait; State, the state of a single agent is represented by s, which contains the action-observation pairs of the past M historical time steps, namely: in That is, the action-observation pair of agent i's t-th access opportunity on link j, That is, the local observation of agent i, which consists of two parts. The first part is the link observation value Specifically: That is, the local observation of the link by the agent includes four types, and in order to explain the link observation value The second part introduces new parameters represents the number of access opportunities of agent i on link j since the last successful transmission, and It represents the number of access opportunities of other agents except agent i on link j since the last successful transmission, and then the two are normalized to form The second part, in as well as As for the joint state required for centralized training, The reward function is represented by r, and the total reward of agent i for the tth access opportunity on link j is represented by It is defined as: where r j，t，global represents the global reward, represents the individual reward of agent i, and α represents the balance coefficient. Specifically: in That is, the ratio of the number of waiting access opportunities of agent i to the total number of waiting access opportunities on link j; 2. The WiFi multi-link access method based on multi-agent reinforcement learning according to claim 1, characterized in that: The WiFi multi-link access method based on multi-agent reinforcement learning can adjust the multi-link access strategy according to the real-time link utilization status and respective access strategies, thereby maximizing link utilization.

3. The WiFi multi-link access method based on multi-agent reinforcement learning according to claim 1, characterized in that: A network consists of one AP and n multi-link devices, all of which carry high-throughput saturated traffic. This means the network system must maximize its total throughput to meet high-throughput requirements while ensuring fairness among devices.

4. The WiFi multi-link access method based on multi-agent reinforcement learning according to claim 1, characterized in that: In step 3, the intelligent multi-link device agent (Agent) adopts a multi-agent reinforcement learning algorithm under a centralized training distributed execution framework. The agent includes an Actor network and a Critic network. The Critic network is a traditional three-layer fully connected network with 128 neurons in the hidden layer, and the Actor network is a four-layer fully connected network with 64 neurons in the hidden layer and a gated recurrent unit (GRU). During centralized training, the Critic network is responsible for processing global observation information to better evaluate the execution strategy of the Actor network, and the Actor network is responsible for outputting access strategies based on local observation information. The experience replay pool is used to record all samples.<x,a,r,x′> , where x represents the joint state of the current access opportunity, a represents the joint action of each agent under the current access opportunity, x′ represents the next state to be migrated after executing action a in state x, and r represents the reward obtained after executing action a in state x.

5. The WiFi multi-link access method based on multi-agent reinforcement learning according to claim 1, characterized in that: In step 2, the MADDPG multi-agent reinforcement learning algorithm is used to solve the optimal access strategy, specifically: Initialize the Actor network and Critic network parameters π θ (a|s); Initialize the target network parameter value function parameter θ′=θ; Initialize the experience replay pool EM; Initialize other parameters, including n, γ, α, τ, M, and I; Centralized training phase: Each agent only needs to monitor the link on a single link j to obtain access opportunities. Each agent uses its own local observation state Get actions through the Actor network Perform joint actions Get rewards j，t and the new joint state x′ j，t ; The experience gained from each access opportunity t<x,a,r,x′> Deposit into the experience replay pool EM; Perform a network update every time you access the network: Randomly sample bs experience from EM as E, and define the loss function of the Critic network based on each experience: Update the Actor network according to the policy gradient: Then update the target network θ′ of each agent i ←τθ i +(1-τ)θ′ i ; After a certain training period and training convergence, the Actor network parameters are the optimal access strategy and are applicable to any link with different parameters. The AP distributes the Actor network parameters to each multi-link device, and the multi-link devices perform distributed access.

6. A WiFi multi-link access system based on multi-agent reinforcement learning, using the WiFi multi-link access method based on multi-agent reinforcement learning according to any one of claims 1 to 5, characterized in that: The system includes: The information acquisition module is used to use the AP to obtain the local link observation value corresponding to each multi-link device and aggregate it into a joint state; The modeling module is used to model the link access problem with different WiFi parameters as a unified distributed partially observable Markov decision process model, defining the actions, states, and reward functions; The solution module is used to solve the optimal access strategy during the centralized training phase using the MADDPG multi-agent reinforcement learning algorithm; The execution module is used to control the multi-link device to access by using the trained optimal access strategy obtained from the AP side.

7. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the WiFi multi-link access method based on multi-agent reinforcement learning as described in any one of claims 1 to 5.

8. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the WiFi multi-link access system based on multi-agent reinforcement learning as described in claim 6.

Images