Multi-agent reinforcement learning-based WiFi (Wireless Fidelity) 7 network access point configuration strategy model training method, application method, training system, terminal equipment and medium

Abstract

The invention discloses a training method, an application method and a training system of a WiFi (Wireless Fidelity) 7 network access point configuration strategy model based on multi-agent reinforcement learning, terminal equipment and a medium, and relates to the technical field of wireless network configuration. The training method comprises the following steps: initializing an evaluation network and each access point execution network by using random weight; obtaining the network state of each access point, and executing a network generation action; obtaining a post-execution state and calculating a reward; the evaluation network calculates a Q value based on the action, the post-execution state and the reward; and updating the evaluation network parameters based on the Q value minimization loss function, and iteratively updating the execution network parameters. Through multi-agent collaborative optimization, dynamic adaptive configuration of the WiFi access point is realized, the network throughput and the packet delivery rate are improved, the transmission delay and the energy consumption are reduced, and the network operation stability and the configuration efficiency are guaranteed.

Description

Technical Field

[0001] This invention relates to the field of wireless network configuration technology, and in particular to a training method, application method, training system, terminal device, and medium for a WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning. Background Technology

[0002] With the explosion of data-intensive applications such as virtual reality and cloud gaming, WiFi networks need to meet the requirements of ultra-high peak throughput and ultra-low latency. WiFi 7 was born to meet this need. It integrates core technologies such as 4096-QAM modulation, multi-link operation (MLO), and preamble punching, providing the hardware foundation for the next generation of wireless communication.

[0003] However, existing WiFi network control strategies still have key limitations. First, their load balancing is inefficient, with some access points overloaded while neighboring devices remain idle during peak hours. Second, their energy utilization is low, with idle access points continuously operating at high power consumption. Furthermore, their static cell configuration cannot adapt to dynamic fluctuations in user density. Simultaneously, their resource sharing capabilities between neighboring access points are limited, resulting in insufficient spectrum utilization. Additionally, they lack autonomous adaptive capabilities, relying on manual or centralized scheduling, leading to lag in response. Even with WiFi 7's advanced hardware features, traditional static control logic still struggles to unleash its potential and cannot meet dynamic adaptation requirements.

[0004] Therefore, there is an urgent need for a dynamic WiFi 7 network access point cell configuration method to fill the gap in existing technology. Summary of the Invention

[0005] The technical problem this invention aims to solve is that, in the field of wireless network configuration technology, existing static access point configuration and inefficient scheduling lead to problems such as load imbalance, resource waste, high transmission latency, and low energy efficiency, which cannot adapt to dynamic network environments and the potential of WiFi 7 features. Therefore, an effective solution is urgently needed to address these technical problems.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides a training method for a WiFi 7 network access point configuration policy model based on multi-agent reinforcement learning, wherein the access point configuration policy model includes an execution network and an evaluation network, and the training method includes: Initialize the evaluation network and the execution network for each access point using random weights; Obtain the network status of each access point, and generate and execute actions for each access point through the execution network; Obtain the network status after each access point performs an action and calculate the reward; The Q-value is calculated using the evaluation network based on the action, the network state after the action, and the reward. Based on the Q value, the parameters of the evaluation network are iteratively updated by minimizing the loss function, and the parameters of the execution network are iteratively updated based on the optimized evaluation network.

[0007] In one implementation, the access point configuration policy model is applied to a WiFi 7 network comprising a controller and several access points, wherein the evaluation network is deployed on the controller and the execution network is deployed on each access point.

[0008] In one implementation, the evaluation network includes a main evaluation network and a target evaluation network, wherein the target evaluation network is a copy network with the same structure as the main evaluation network; the main evaluation network is a centralized double-Q evaluation network, which includes two evaluation sub-networks with the same structure and independent parameters, and the initial parameters of the two evaluation sub-networks are obtained by initializing them with random weights; before training begins, the initial parameters of the main evaluation network are completely assigned to the target evaluation network.

[0009] In one implementation, the execution network includes a main execution network and a target execution network, wherein the target execution network is a replica network with the same structure as the main execution network; the main execution network is a decentralized execution network, and the main execution network of each access point outputs the corresponding action through the network state; before training begins, the initial parameters of the main execution network are completely assigned to the target execution network.

[0010] In one implementation, obtaining the network status of each access point includes: For each access point, its own operational data is collected, and the operational data of adjacent access points are obtained through the communication link. By integrating its own operational data with the operational data of adjacent access points, the initial network state is obtained; The initial network states are filtered and normalized, and then combined to form the final network states; The network status includes the number of users connected to the access point, current traffic load, remaining energy, average load of adjacent access points, normalized effective bandwidth, multi-link operation enabled status, current cell size, frequency band used, and time normalization factor.

[0011] In one implementation, generating and executing the actions of each access point through the execution network includes: For each access point, a hybrid action vector is generated and output based on the network state through the execution network; Based on the hybrid action vector, the action is calculated and executed; The hybrid action vector includes at least a logarithmic probability of several discrete actions, wherein the discrete actions include at least one of merging, splitting, load sharing, resizing, and enabling multi-link operation.

[0012] In one implementation, calculating and executing the action based on the hybrid action vector includes: The logarithmic probabilities of discrete actions in the mixed action vector are subjected to a softmax operation to obtain the selection probability of each discrete action. The argmax function selects and executes the discrete action with the highest probability.

[0013] In one implementation, obtaining the network status after each access point performs an action and calculating the reward includes: Obtain the network status after each access point performs an action, as well as the global performance metrics of the WiFi 7 network; The transmission delay in the global performance metrics is normalized to obtain the normalized delay; The reward is calculated based on the global performance metrics and normalized latency. The global performance metrics include transmission latency, network throughput, packet delivery rate, and access point energy consumption.

[0014] In one implementation, calculating the Q-value through the evaluation network based on the action, the network state after the action, and the reward includes: Obtain the network status after all access points have performed actions, and integrate them to form a global network status; The global network state, the actions of all access points, and the calculated rewards are input into the main evaluation network to calculate the Q value output by the main evaluation network. The global network state, the target action generated by the target execution network, and the reward are input into the target evaluation network to obtain the target Q value output by the target evaluation network.

[0015] In one implementation, the access point configuration strategy model further includes an experience replay buffer, which stores experience samples. These experience samples include the current network state, actions, rewards, and the network state after the actions are executed. The steps for constructing and updating the experience replay buffer include: Initialize the experience replay buffer using historical sample data; After obtaining the network state after each action is performed by the access point and calculating the reward, the current network state, action, reward, and network state after the action are integrated to construct an experience sample, and the constructed experience sample is stored in the experience replay buffer.

[0016] In one implementation, the step of iteratively updating the parameters of the evaluation network by minimizing the loss function based on the Q-value, and iteratively updating the parameters of the execution network based on the optimized evaluation network, includes: Based on the Q value, the parameters of the main evaluation network are iteratively updated by minimizing the loss function; Based on the optimized main evaluation network, the parameters of the main execution network are iteratively updated; Based on the updated parameters of the main evaluation network and the main execution network, the parameters of the target evaluation network in the evaluation network and the parameters of the target execution network in the execution network are softly updated.

[0017] In one implementation, the step of iteratively updating the parameters of the main evaluation network based on the Q-value by minimizing the loss function includes: The temporal difference error is calculated based on the Q value, and the temporal difference error is the difference between the Q value output by the master evaluation network and the target Q value output by the target evaluation network. A loss function is constructed with the objective of minimizing the mean square value of the time-series difference error; A batch of samples is sampled from the experience replay buffer, and the parameters of the main evaluation network are iteratively updated using the gradient descent algorithm until the loss function value converges to a preset range.

[0018] In one implementation, iteratively updating the parameters of the main execution network based on the optimized main evaluation network includes: Set a delayed update period and count the number of parameter updates for the main evaluation network; When the number of parameter updates reaches the set delay update period, the deterministic policy gradient is calculated based on the Q value output by the optimized main evaluation network. The parameters of the main execution network are updated along the ascending direction of the deterministic policy gradient; The deterministic policy gradient is calculated by the expected value of the product of the gradient of the Q-value output by the main evaluation network with respect to the gradient of the main execution network's output action and the gradient of the main execution network's output action with respect to its own parameters.

[0019] In one implementation, the step of softly updating the parameters of the target evaluation network in the evaluation network and the target execution network in the execution network based on the updated parameters of the main evaluation network and the main execution network includes: Set a soft update coefficient, and based on the soft update coefficient, calculate the weighted sum of the updated parameters of the main evaluation network and the original parameters of the target evaluation network, as well as the weighted sum of the updated parameters of the main execution network and the original parameters of the target execution network. The two weighted sums obtained are used as the new parameters for the target evaluation network and the target execution network, respectively, and the target evaluation network and the target execution network are updated accordingly.

[0020] In one implementation, the convergence determination steps for iterative updates include: After each round of network evaluation and parameter updates, collect global performance metrics of the WiFi7 network. Calculate the fluctuation range of global performance indicators within consecutive preset rounds; If the fluctuation amplitude is less than a preset threshold, the access point configuration strategy model is determined to have converged.

[0021] Secondly, embodiments of the present invention also provide an application method for a WiFi 7 network access point configuration policy model based on multi-agent reinforcement learning, applying the access point configuration policy model obtained by the training method described in any one of the above schemes, the application method comprising: Obtain the network status of each access point; Based on the network status, the actions of each access point are generated and executed through the execution network in the access point configuration policy model to complete the access point configuration optimization. Obtain the network status after each access point performs an action and the global network status of the WiFi 7 network.

[0022] Thirdly, embodiments of the present invention also provide a training system for a WiFi 7 network access point configuration policy model based on multi-agent reinforcement learning, wherein the access point configuration policy model includes an execution network and an evaluation network, and the training system includes: The initialization module is used to initialize the evaluation network and the execution network for each access point using random weights. The action generation and execution module is used to obtain the network status of each access point, and generate and execute the actions of each access point through the execution network. The reward calculation module is used to obtain the network status after each access point performs an action and calculate the reward. The Q-value calculation module is used to calculate the Q-value based on the action, the network state after the action is performed, and the reward through the evaluation network. The iterative update module is used to iteratively update the parameters of the evaluation network by minimizing the loss function based on the Q value, and to iteratively update the parameters of the execution network based on the optimized evaluation network.

[0023] Fourthly, embodiments of the present invention also provide a terminal device, the terminal device including a memory, a processor, and a training program for a WiFi 7 network access point configuration policy model based on multi-agent reinforcement learning stored in the memory and executable on the processor. When the processor executes the training program for the WiFi 7 network access point configuration policy model based on multi-agent reinforcement learning, it implements the steps of the training method for the WiFi 7 network access point configuration policy model based on multi-agent reinforcement learning as described in any of the above schemes.

[0024] Fifthly, embodiments of the present invention also provide a computer-readable storage medium storing a training program for a WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning. When the training program for the WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning is executed by a processor, it implements the steps of the training method for the WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning as described in any of the above schemes.

[0025] Beneficial Effects: This invention discloses a training method, application method, training system, terminal device, and medium for a WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning, relating to the field of wireless network configuration technology. The access point configuration strategy model includes an execution network and an evaluation network. In the training method, firstly, random weights are used to initialize the evaluation network and the execution network for each access point. Then, the network state of each access point is acquired, and actions for each access point are generated and executed through the execution network. Subsequently, the network state after each access point executes its actions is acquired, and the reward is calculated. Next, the Q-value is calculated through the evaluation network based on the actions, the network state after the actions, and the reward. Finally, based on the Q-value, the parameters of the evaluation network are iteratively updated by minimizing the loss function, and the parameters of the execution network are iteratively updated based on the optimized evaluation network. This invention enables adaptive configuration of each access point through multi-agent collaboration, accurately matches network states with mixed action vectors, improves WiFi 7 network throughput and packet delivery rate, reduces transmission latency and energy consumption, adapts to dynamic network environments, and ensures efficient configuration and reliable operation. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating a specific implementation method for training a WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning, as provided in an embodiment of the present invention.

[0027] Figure 2 This is a schematic diagram of the network architecture for training a WiFi 7 network access point configuration policy model based on multi-agent reinforcement learning, as provided in an embodiment of the present invention.

[0028] Figure 3 This is a cell merging diagram illustrating the training method for a WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning provided in an embodiment of the present invention.

[0029] Figure 4 This is a cell splitting diagram illustrating the training method for the WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning provided in an embodiment of the present invention.

[0030] Figure 5 This is a resource sharing diagram illustrating the training method for a WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning, as provided in an embodiment of the present invention.

[0031] Figure 6 The graph shows the relationship between throughput and training rounds under different traffic loads for the training method of the WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning provided in the embodiments of the present invention.

[0032] Figure 7 The diagram shows the relationship between packet delivery rate and training rounds under different traffic loads for the training method of the WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning provided in the embodiments of the present invention.

[0033] Figure 8 The graph shows the relationship between average latency and training rounds under different traffic loads for the training method of the WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning provided in the embodiments of the present invention.

[0034] Figure 9 Training metrics diagrams for reward, loss, learning rate, and stability of the training method for the WiFi 7 network access point configuration policy model based on multi-agent reinforcement learning provided in the embodiments of the present invention.

[0035] Figure 10 The DC-MARL scheme feature attention map is provided for the training method of the WiFi 7 network access point configuration policy model based on multi-agent reinforcement learning in the embodiments of the present invention.

[0036] Figure 11 This is a WiFi 7 scheme feature attention map for the training method of the WiFi 7 network access point configuration policy model based on multi-agent reinforcement learning provided in the embodiments of the present invention.

[0037] Figure 12 The average traffic load heatmap is provided for the training method of the WiFi 7 network access point configuration policy model based on multi-agent reinforcement learning in the embodiments of the present invention.

[0038] Figure 13This is a schematic diagram illustrating the training and application process of a WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning, provided in an embodiment of the present invention.

[0039] Figure 14 This is a schematic diagram of a WiFi 7 network access point cell configuration device based on dynamic cell multi-agent reinforcement learning, provided in an embodiment of the present invention.

[0040] Figure 15 This is a block diagram illustrating the internal structure of the terminal device provided in an embodiment of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0042] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content, operations, or steps, nor does it require execution in the described order. For example, some operations or steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0043] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0044] It should be understood that, in order to clearly describe the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. For example, "first control information" and "second control information" are only used to distinguish different control information and do not limit their order.

[0045] Those skilled in the art will understand that the words "first" and "second" do not limit the quantity or the order of execution, and that the words "first" and "second" do not necessarily imply that they are different.

[0046] It should also be understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0047] As a core infrastructure for global connectivity, Wireless Local Area Networks (WLANs) not only provide seamless internet access for mobile devices in high-density urban scenarios but also supplement cellular network coverage blind spots, ensuring uninterrupted access to core services, online resources, and digital platforms. With the development of the digital economy, the explosive growth of data-intensive applications has placed new demands on WiFi network performance. Applications such as high-definition video, virtual reality, augmented reality, cloud gaming, remote work, and wireless screen projection not only require ultra-high-speed transmission capabilities but also have extremely low latency requirements. Furthermore, professional scenarios such as industrial automation and telemedicine also have high requirements for data transmission reliability to replace wired connections. To address these challenges, the next-generation WiFi standard, WiFi 7, has been proposed. Its goal is to support higher peak throughput and optimize latency and jitter under worst-case conditions, meeting the performance requirements of future applications through new Physical Layer (PHY) and Medium Access Control (MAC) technologies.

[0048] As a new wireless communication standard, WiFi 7 integrates several key innovative technologies, significantly improving network performance. First, 4096-QAM technology, compared to WiFi 6's 1024-QAM, encodes 12 bits of data per symbol, improving spectral efficiency by approximately 20%. Second, Multi-Link Operation (MLO) allows devices to simultaneously establish transmission links in multiple frequency bands (2.4GHz, 5GHz, and 6GHz), achieving higher throughput, interference-resistant handover, and concurrent transmission and reception through aggregation, handover, and duplex modes. Third, the Enhanced Multi-Resource Units (MRU) mechanism dynamically combines discontinuous resource units, avoiding bandwidth waste caused by spectrum fragmentation and improving throughput in congested environments. Fourth, Preamble Puncturing technology selectively disables occupied subcarriers, enabling high-throughput transmission using remaining spectrum in complex interference environments and reducing bandwidth waste. Fifth, Advanced Hybrid ARQ (HARQ) improves Packet Delivery Rate (PDR) by combining retransmission with soft combining. In addition, WiFi 7 incorporates features such as Time-Sensitive Networking (TSN) and Restricted Target Wake Time (R-TWT) to further optimize deterministic latency and energy efficiency.

[0049] Despite the advanced technological potential of WiFi 7, existing networks still have many key limitations that make it difficult to fully realize its performance advantages. Traditional WiFi systems lack a sophisticated dynamic load balancing mechanism. During peak hours, some access points (APs) become overloaded and congested, while others remain idle, resulting in poor overall network performance. Energy efficiency is low, with underutilized APs continuing to consume significant amounts of power, leading to unnecessary energy waste. Static cell size configurations cannot adapt to dynamically changing user densities, causing severe congestion in high-density areas and insufficient resource utilization in sparse areas. Resource sharing between adjacent APs is limited, resulting in low bandwidth utilization and significantly increased latency under peak load. Most existing systems rely on centralized control or manual intervention for configuration adjustments, lacking autonomous adaptability. This fails to meet the scalability requirements of large-scale networks and cannot quickly respond to dynamic network conditions such as user movement, traffic fluctuations, and interference changes. Furthermore, existing solutions lack sufficient integration of WiFi 7's core features, failing to leverage intelligent decision-making mechanisms to coordinate technologies such as 4096-QAM, MLO, and MRU. This makes it difficult to translate the theoretical performance of WiFi 7 into stable gains in practical applications, failing to fully meet the comprehensive requirements of high throughput, low latency, high reliability, and low energy consumption.

[0050] It is understandable that existing WiFi 7 networks suffer from problems such as low load balancing efficiency, unreasonable energy consumption control, lack of dynamic adaptability in cell configuration, insufficient resource sharing capabilities, lack of self-adaptation and scalability, insufficient QoS (Quality of Service) guarantee capabilities, and insufficient collaborative utilization of WiFi 7 features.

[0051] Therefore, this embodiment models WiFi 7 dynamic cell control as a multi-agent Markov Decision Process (MDP) integrating real-time user load, interference, and energy indicators. To achieve optimal decision-making in this MDP, this embodiment proposes a Dynamic Cell Multi-Agent Reinforcement Learning (DC-MARL) framework. This framework is based on the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm and extends it to form a core policy optimization algorithm (MATD3) adapted to multi-agent collaborative scenarios. As a policy optimization algorithm under the DC-MARL architecture, MATD3 has a dual Q-evaluation network and an adaptive target update mechanism, which can effectively eliminate the Q-value overestimation problem and ensure stable policy convergence in a dynamic network environment with multiple APs.

[0052] The network architecture of this embodiment is as follows: Figure 2 As shown, it includes a controller and several access points, with each access point responsible for the network connection of several user devices.

[0053] Within this DC-MARL framework, four strategies can be used: cell merging, cell splitting, cell adjustment, and access point coordination, as follows: Figure 3 , Figure 4 , Figure 5 , Figure 6 As shown, intelligent decisions can be made based on network traffic. Furthermore, WiFi 7's unique physical layer and media access control layer mechanisms, such as 4096 quadrature amplitude modulation, multi-link operation, and preamble punching, are integrated into the learning environment to achieve realistic performance evaluation. Finally, extensive simulation experiments demonstrate that the proposed framework significantly outperforms WiFi 7 benchmark solutions in terms of cumulative reward, throughput, latency, and energy efficiency.

[0054] This embodiment provides a training method for a WiFi 7 network access point configuration policy model based on multi-agent reinforcement learning, such as... Figure 1 As shown. The training method is the dual-delay deep deterministic policy gradient TD3 algorithm in the aforementioned dynamic cell multi-agent reinforcement learning DC-MARL framework.

[0055] The access point configuration strategy model includes an execution network and an evaluation network, and the training method specifically includes the following steps: Step S100: Initialize the evaluation network and the execution network for each access point using random weights.

[0056] In this embodiment, the Access Point (AP) is the core device in a WiFi 7 network that provides wireless access services to user terminals. It possesses signal transmission and reception, data forwarding, and status awareness functions, and can be deployed in high-density scenarios such as shopping malls and office buildings to directly establish wireless connections with user terminals and transmit data. The controller is the global coordination unit for the entire network. In the HarmonyOS scenario, it can act as a super device, responsible for integrating the status information of all access points, formulating global optimization strategies, issuing action commands, and iteratively updating model parameters. It is the core hub for achieving multi-access point collaborative optimization.

[0057] The execution network is a decentralized neural network deployed at each access point. Its core function is to generate adapted configuration actions based on the network state collected locally at each access point. Its input is a network state vector, and its output is a mixed action vector containing discrete action log probabilities and continuous adjustments. The evaluation network is a centralized neural network deployed on the controller. It is used to evaluate the long-term value of the actions generated by the execution network, providing a quantitative basis for parameter updates. Its input includes the global network state and the actions of all access points, and its output is the Q-value corresponding to the action, i.e., the action value estimate.

[0058] Random weights are initial values ​​that conform to a normal distribution used when initializing network parameters to avoid gradient vanishing or exploding problems during training due to excessively large or small initial parameters.

[0059] This step implemented the initialization architecture design of the decentralized execution network and the centralized evaluation network, as well as the reasonable selection of random weights. By independently initializing the execution network for each access point, it ensures that each access point has diverse action exploration capabilities in the early stages of training, avoiding the local optimum trap caused by homogeneous initial parameters. The initialization of the centralized evaluation network lays the foundation for subsequent global value evaluation. The normal distribution characteristics of the random weights can cover a wider parameter space, allowing the network to explore more potential action combinations in the early stages of training, gradually learning configuration strategies adapted to the dynamic environment of the WiFi 7 network, and providing rich empirical data support for subsequent iterative optimization.

[0060] In one implementation, the access point configuration policy model is applied to a WiFi 7 network comprising a controller and several access points, wherein the evaluation network is deployed on the controller and the execution network is deployed on each access point.

[0061] In this embodiment, the access point configuration strategy model is deployed on a WiFi 7 network that includes a controller and several access points, forming a centralized evaluation and decentralized execution architecture. The evaluation network is deployed on the controller, and the execution network is deployed on each access point.

[0062] From an implementation perspective, the controller, acting as the global coordination hub, possesses powerful computing and data integration capabilities. It can collect network status and action data from all access points to construct a global network view. The evaluation network, deployed on the controller, can assess the collaborative value of all access point actions from a global perspective, avoiding the local optimum trap caused by decentralized evaluation. For example, when multiple access points simultaneously execute split actions, the centralized evaluation network can identify potential load overlap and interference issues, guiding action coordination through Q-value adjustments.

[0063] The execution network is deployed at each access point, enabling each access point to generate actions in real time based on locally collected network status data, without waiting for remote commands from the controller. This design significantly reduces communication latency and adapts to the high-dynamic, low-latency configuration requirements of WiFi 7 networks. For example, when the number of users at an access point suddenly surges, the local execution network can quickly generate split actions to alleviate load pressure in a timely manner, without relying on global scheduling by the controller.

[0064] In this architecture, centralized evaluation ensures consistency in global optimization and avoids conflicts in access point actions, while decentralized execution guarantees the real-time and flexible nature of action generation, improving the network's response speed to dynamic environments. The combination of these two approaches enables the model to possess both global optimization capabilities and the real-time response capabilities of individual access points, fully leveraging the hardware performance potential of WiFi 7 networks and solving the core problems of high latency in traditional centralized scheduling and poor coordination in decentralized configuration.

[0065] In one implementation, the evaluation network includes a main evaluation network and a target evaluation network, wherein the target evaluation network is a copy network with the same structure as the main evaluation network; the main evaluation network is a centralized double-Q evaluation network, which includes two evaluation sub-networks with the same structure and independent parameters, and the initial parameters of the two evaluation sub-networks are obtained by initializing them with random weights; before training begins, the initial parameters of the main evaluation network are completely assigned to the target evaluation network.

[0066] In this embodiment, the evaluation network includes a main evaluation network and a target evaluation network. The target evaluation network is a copy network with the same structure as the main evaluation network. The main evaluation network adopts a centralized double-Q evaluation network architecture, which includes two evaluation sub-networks with the same structure and independent parameters. Before training begins, the initial parameters of the main evaluation network are completely assigned to the target evaluation network.

[0067] This embodiment proposes a dual-Q structure main evaluation network. The two evaluation sub-networks have identical structures, both being neural networks containing three fully connected layers. Specifically, the input layer dimension is the sum of the global network state vector dimension and the action vector dimensions of all access points, the hidden layers use the ReLU activation function, and the output layer is a one-dimensional Q-value. The initial parameters of both sub-networks follow a normal distribution, ensuring the randomness and consistency of the initial parameters while avoiding training problems caused by excessively large or small parameters.

[0068] The target evaluation network has the same structure as the main evaluation network, including the number of network layers, the number of neurons per layer, and the type of activation function; the only difference is the parameter update mechanism. The parameter assignment operation before training begins involves completely copying the random initial weights and biases of the main evaluation network to the target evaluation network. This ensures that the target network maintains parameter synchronization with the main network from the initial training phase, laying the foundation for stable calculation of the target Q-value later.

[0069] The technical advantage of the dual-Q evaluation network lies in its effective suppression of Q-value overestimation. Traditional single-Q networks are prone to Q-value overestimation due to noise or sample bias, leading the model to select suboptimal actions. In this embodiment, two independent evaluation sub-networks calculate Q-values ​​separately, and the minimum of the two is taken as the output Q-value of the main evaluation network. A double-validation mechanism filters out overestimated Q-values, ensuring the accuracy of action value assessment. The target evaluation network provides a stable target Q-value reference. Specifically, the parameters of the main evaluation network are updated rapidly during training, while the parameters of the target evaluation network are updated later, using a soft update mechanism to adjust slowly, avoiding target Q-value oscillations caused by fluctuations in the main network parameters, thus improving training stability. The combination of the two makes the Q-value calculation of the evaluation network both accurate and stable, providing a reliable basis for subsequent parameter updates, accelerating model convergence, and improving final configuration performance.

[0070] In one implementation, the execution network includes a main execution network and a target execution network, wherein the target execution network is a replica network with the same structure as the main execution network; the main execution network is a decentralized execution network, and the main execution network of each access point outputs the corresponding action through the network state; before training begins, the initial parameters of the main execution network are completely assigned to the target execution network.

[0071] In this embodiment, the execution network includes a main execution network and a target execution network. The target execution network is a replica network with the same structure as the main execution network. The main execution network is a decentralized execution network. The main execution network of each access point outputs the corresponding action through the network state. Before training begins, the initial parameters of the main execution network are completely assigned to the target execution network.

[0072] The decentralization of the main execution network is reflected in the independent deployment of a set at each access point, enabling action generation without relying on state data from other access points or controllers. Its network structure can be a three-layer fully connected layer. Specifically, the input layer dimension is the dimension of the access point's local network state vector, the hidden layer uses the ReLU activation function, and the output layer is a hybrid action vector. This structure maps local states to the action space, ensuring the targeted and efficient generation of actions.

[0073] The target execution network has the same structure as the main execution network, including the number of neurons in each layer, activation functions, and output layer design; the only difference is the parameter update mechanism. The parameter assignment operation before training begins involves completely copying the random initial parameters of the main execution network at each access point to the corresponding target execution network, ensuring that the initial state of the target execution network is consistent with that of the main execution network.

[0074] The technical advantages of the decentralized master execution network are reflected in its real-time performance and flexibility. Specifically, each access point quickly generates actions based on its local state, reducing dependence on communication links and avoiding scheduling delays caused by centralized execution. For example, when an access point detects that the load of a neighboring access point is low and its own remaining energy is insufficient, the local master execution network can quickly generate a merge action to reduce energy consumption in a timely manner, without waiting for the controller's global decision. The technical advantage of the target execution network is that it provides stable target actions: its parameter updates lag behind the master network, and it is slowly adjusted through a soft update mechanism. The generated target actions are not affected by fluctuations in the master network parameters, providing reliable input for the calculation of the target Q-value, avoiding distortion of the target Q-value caused by rapid changes in the master execution network parameters, and ensuring the stability of the training process.

[0075] Step S200: Obtain the network status of each access point, and generate and execute the actions of each access point through the execution network.

[0076] In this embodiment, network status is a comprehensive data set reflecting the access point's own operating status and the characteristics of its surrounding environment. It covers key indicators such as the number of connected users, traffic load, and remaining energy, comprehensively depicting the access point's operating conditions and dynamic changes in the network environment. Actions are configuration optimization operations that the access point can perform, specifically including merging, splitting, load sharing, resizing, and enabling Multi-Link Operation (MLO). These actions directly affect the access point's cell configuration and resource allocation, and are means to optimize network performance.

[0077] This embodiment proposes a decentralized action generation mechanism involving multiple agents. Each access point independently generates actions based on locally collected network conditions, without waiting for global scheduling instructions from the controller. This mechanism is well-suited to the widely distributed and highly dynamic nature of WiFi 7 network access points. Specifically, decentralized execution reduces communication overhead between the controller and access points, minimizes action generation latency, and ensures that access points can respond in real-time to changes in local network conditions, such as sudden user growth. Independent exploration by each access point enriches the diversity of experience data, avoids the limitations of local exploration by a single access point, and makes the overall network configuration strategy more adaptable. Furthermore, the diversified design of action types covers core scenarios of cell topology adjustment and resource scheduling, and can specifically address key issues in traditional networks such as load imbalance and resource waste.

[0078] In one implementation, obtaining the network status of each access point specifically includes the following steps: Step S210: For each access point, collect its own operating data and obtain the operating data of adjacent access points through the communication link; Step S220: Integrate its own operating data with the operating data of adjacent access points to obtain the initial network state; Step S230: Filter and normalize the initial network state, and combine them to form the final network state; The network status includes the number of users connected to the access point, current traffic load, remaining energy, average load of adjacent access points, normalized effective bandwidth, multi-link operation enabled status, current cell size, frequency band used, and time normalization factor.

[0079] In this embodiment, obtaining the network status of each access point includes three core steps: collection, integration, filtering, and normalization, ultimately forming a standardized network status vector that can support action generation.

[0080] The first step is data acquisition. Specifically, each access point collects its own operational data through built-in sensors and communication modules, including the number of connected users, current traffic load, and remaining energy. Simultaneously, it acquires operational data from neighboring access points via communication links, such as the MLO link in WiFi 7, including the average load of neighboring access points and the status of multi-link operation. This ensures that access points can not only perceive their own status but also obtain information about the surrounding network environment, providing data support for generating collaborative actions.

[0081] Secondly, there is data integration. Specifically, the access point integrates its own operational data with the operational data of neighboring access points to form the initial network state. The dimensions of the initial network state cover the core indicators of access point operation and are represented using a network state vector: Equation (1)

[0082] in, Indicates the first Each access point at time The local state vector, Given a 9-dimensional real number space, the definitions and physical meanings of the parameters in each dimension are as follows: This indicates the number of users currently associated with this access point, reflecting the service pressure on the access point. This represents the average user load of adjacent access points, obtained through real-time data interaction with surrounding APs, and is used to estimate the congestion of the surrounding network. This represents the normalized energy consumption of the access point, with a value range of [0,1]. It is calculated from the ratio of the current remaining energy of the access point to the maximum rated energy, reflecting the energy consumption status of the equipment. This represents the normalized coverage radius of the access point, with a value range of [0,1]. It is adjusted in real time by dynamic cell operations and corresponds to the standardized result of the actual physical coverage area. This represents a binary frequency band indicator, with a value of 0 or 1, used to specify the frequency band currently operating at the access point; Indicates the multi-link operation (MLO) status, 0 for disabled and 1 for enabled, which is associated with the multi-band concurrent transmission capability of WiFi 7; Represents traffic intensity, normalized to [0,1], calculated from the ratio of the data packet arrival rate to the service rate of the current access point, reflecting the level of transmission busyness; This represents the average packet delay, normalized to [0,1]. It is obtained by statistically analyzing the average time from the generation to the reception of data packets per unit time, reflecting the transmission timeliness. This represents the normalized time period characteristics, with a value range of [0,1]. It is calculated by the ratio of the current time to the total time period of the day and is used to capture the daily variation patterns of user demand and network behavior, such as the difference between peak and off-peak periods.

[0083] When the access point collects its own operational data, it obtains real-time data through hardware units such as the built-in user connection detection module, traffic statistics module, and energy consumption monitoring module. , , It acquires its own status parameters. When obtaining operational data from adjacent access points, it uses a dedicated communication link between the access points to achieve this. , Real-time sharing of status information of neighboring nodes.

[0084] Finally, there is the filtering and normalization process. Specifically, the initial network status of the access points is filtered to remove outliers, such as data exceeding reasonable ranges due to sensor malfunctions. Then, each indicator is normalized, mapping all data to the [0,1] interval to eliminate numerical magnitude differences caused by different units and avoid affecting the decision-making accuracy of subsequent controllers. For example, the number of users is normalized to a reasonable range of "0-50 people", and the latency is normalized to a range of "0-10ms".

[0085] Following the dimensional order of equation (1), the preprocessed parameters are combined to form the local state vector. This allows the vectors to comprehensively and accurately reflect the real-time status of the access point itself and the surrounding local network, providing high-quality basic data for the controller to construct the global network status.

[0086] The comprehensiveness of data acquisition ensures that the network state vector can fully characterize the access point itself and its surrounding environment, providing a foundation for accurate action generation. The standardized state vector design of Equation (1) unifies the data structure and adapts to the input requirements of the execution network. Filtering and normalization processes improve data quality, avoid interference from outliers and magnitude differences in training, and accelerate model convergence. At the same time, the design of multi-dimensional state vectors not only covers the core characteristics of WiFi7 networks but also takes into account network performance and device status, enabling decisions on diverse actions such as merging and splitting.

[0087] In one implementation, the step of generating and executing the actions of each access point through the execution network specifically includes the following steps: Step S240: For each access point, based on the network state, generate and output a hybrid action vector through the execution network; Step S250: Calculate and execute the action based on the hybrid action vector; The hybrid action vector includes at least a logarithmic probability of several discrete actions, wherein the discrete actions include at least one of merging, splitting, load sharing, resizing, and enabling multi-link operation.

[0088] In this embodiment, the actions of each access point generated by the network are executed. The core is to generate a hybrid action vector and transform it into a specific action, thereby achieving the synergy between discrete topology adjustment and continuous parameter optimization.

[0089] The first step is the generation of the hybrid action vector. Specifically, the main execution network of each access point calculates and outputs the hybrid action vector through forward propagation based on the network state vector in equation (1). The hybrid action vector has a dimension of 6, represented as: Equation (2)

[0090] Among them, the first 5 dimensions The logits represent the discrete actions, corresponding to five preset discrete topology operations: Merge, Split, Share Load, Resize, and Enable MLO; the sixth dimension... This is the adjustment amount for the continuous cell radius, and its value range is... , This is the preset maximum radius adjustment range.

[0091] Subsequently, actions are calculated and executed based on the hybrid action vector.

[0092] The hybrid action vector design combines discrete topology adjustment and continuous parameter optimization. It can solve the load imbalance problem at the network topology level through actions such as merging and splitting, and optimize the coverage range through continuous radius adjustment, adapting to the complex optimization needs of WiFi 7 networks.

[0093] In one implementation, the step of calculating and executing the action based on the hybrid action vector specifically includes the following steps: Step S251: Perform a softmax operation on the logarithmic probabilities of the discrete actions in the mixed action vector to obtain the selection probability of each discrete action; Step S252: Select the discrete action with the highest probability using the argmax function and execute it.

[0094] In this embodiment, actions are calculated and executed based on hybrid action vectors. Specifically, the precise selection of discrete actions is achieved through softmax operation and argmax function, ensuring the rationality and efficiency of action execution.

[0095] First, there's the softmax operation. Specifically, after the mixed action vector is generated, it's transformed into specific executable actions through subsequent processing. First, a softmax operation is performed on the log-probability of the first 5 discrete actions, converting it into a selection probability between 0 and 1, expressed as: Equation (3)

[0096] in, For the first The probability of choosing a discrete action is given, and the sum of all probabilities is 1. A softmax operation is performed on the 5-dimensional discrete action log odds in the mixed action vector to transform the unnormalized log odds into a probability distribution between 0 and 1, with the sum of the probabilities of all discrete actions being 1. The log odds themselves are unbounded real numbers; they are mapped to positive numbers using an exponential function, and then divided by the sum of all exponents to achieve probability normalization. For example, if the log odds of splitting the action in the mixed action vector are 2.1, the log odds of merging the action are 1.3, and the log odds of other actions are all less than 1, then after the softmax operation, the probability of splitting the action will be significantly higher than that of other actions, reflecting its optimality in the current state.

[0097] Then, the discrete action with the highest probability is selected using the argmax function, represented as: Equation (4)

[0098] If the selected discrete action is to adjust the size, that is... Then combine with continuous adjustment amount Perform cell radius adjustment; for other discrete actions, directly execute the corresponding topology operation, such as merging or splitting. The argmax function iterates through the five action probabilities obtained from the softmax operation and selects the discrete action with the highest probability as the final action to be executed. This ensures that the access point can choose the action that is most likely to optimize network performance in the current state, avoiding the execution of invalid actions. For example, when the probability of the load sharing action is 0.6, and the probabilities of other actions are all less than 0.2, the argmax function will directly select the load sharing action, guiding the access point to transfer some of the load to adjacent access points, alleviating its own congestion.

[0099] The softmax operation provides a probabilistic basis for action selection, making it flexible and avoiding overexploration of single actions. It also provides probabilistic interpretability, preventing insufficient exploration caused by deterministic choices. The argmax function ensures the selection of the highest-priority action, guaranteeing targeted configuration and improving efficiency. This process is closely integrated with the execution network's output, achieving a smooth transformation from action vectors to specific actions, ensuring the precise implementation of the network's optimization goals. For example, when the execution network learns during training that splitting actions is better under high load, it will output a higher log-probability of splitting actions. After softmax and argmax processing, the access point will prioritize splitting actions, optimizing network performance. This design allows access points to flexibly select action types based on network conditions, such as prioritizing splitting actions under high load and merging actions under low load and high energy consumption, achieving dynamic adaptive configuration.

[0100] Step S300: Obtain the network status after each access point performs the action and calculate the reward.

[0101] In this embodiment, the network state after the action is executed is the comprehensive state data updated after the access point completes the configuration operation. It can directly reflect the impact of the action execution on network operation and is the basis for evaluating the effect of the action. The reward is a quantitative evaluation index designed based on global network performance indicators. It is used to guide the iterative optimization of model parameters. Its calculation integrates core performance dimensions such as network throughput, packet delivery rate, transmission delay, and access point energy consumption, and can achieve multi-objective collaborative optimization.

[0102] This embodiment proposes a closed-loop design for real-time state feedback and multi-dimensional reward calculation. Real-time acquisition of the network state after an action is executed ensures the accuracy and timeliness of reward calculation, enabling the model to quickly perceive the impact of actions on the network. The multi-dimensional reward design overcomes the limitations of traditional single-index optimization. By balancing objectives such as throughput improvement, latency reduction, and energy consumption optimization through weight allocation, it avoids network performance imbalances caused by optimizing a single index. For example, the normalization of transmission latency in the reward function ensures consistent evaluation of latency indicators under different traffic loads, guiding the model to generate configuration actions that balance real-time performance and energy efficiency, providing reliable value guidance for subsequent Q-value calculation and parameter updates.

[0103] In one implementation, obtaining the network status after each access point performs an action and calculating the reward specifically includes the following steps: Step S310: Obtain the network status after each access point performs the action, as well as the global performance indicators of the WiFi 7 network; Step S320: Normalize the transmission delay in the global performance metrics to obtain the normalized delay; Step S330: Based on the global performance indicators and normalized latency, the reward is calculated by fusion. The global performance metrics include transmission latency, network throughput, packet delivery rate, and access point energy consumption.

[0104] In this embodiment, the network status after the access point performs an action is obtained and the reward is calculated. Specifically, by fusing multi-dimensional performance indicators, a clear optimization guide is provided for the model, achieving a synergistic improvement in throughput, latency, and energy consumption.

[0105] First, data collection is performed. Specifically, after the access point executes its action, the controller collects two types of data through the communication link: one is the network status of each access point after execution, including the updated number of users, load, coverage radius, etc., corresponding to the state vector update in equation (1); the other is the global performance indicators of the WiFi 7 network, specifically including transmission delay, network throughput, packet delivery ratio (PDR), and access point energy consumption. These indicators comprehensively cover network service quality and equipment operating efficiency, and can comprehensively reflect the effect of action execution.

[0106] Next is the normalization of transmission delay. Specifically, since the reasonable range of transmission delay varies greatly under different traffic loads, directly using it for reward calculation will lead to evaluation bias. Therefore, it is necessary to normalize the transmission delay to obtain a normalized delay. Normalization uses a bounded function, expressed as: Equation (6)

[0107] in, Access point Normalization delay, For the original transmission delay, This is the preset maximum acceptable latency. When the original latency is less than... When the normalized delay is constant, the normalized delay has a linear relationship with the original delay; when the original delay exceeds a certain threshold... At this time, the normalized delay is fixed at 1 to avoid the excessive impact of extreme delay on the reward.

[0108] against In the case where the normalized delay is determined, the sensitivity of the reward to the original delay can be quantified by equation (7): Equation (7)

[0109] In this formula, Indicates reward For the original delay The partial derivative reflects the change in reward for every unit change in the original delay; The weight of the delay metric in the reward function. The maximum acceptable delay is given. Equation (7) reveals the sensitivity of rewards to changes in delay. The larger the value, the smaller the absolute value of the sensitivity, and the smoother the response of the reward to delay fluctuations. The smaller the value, the larger the absolute value of the sensitivity, and the more significant the penalty for latency changes. This sensitivity adjustment mechanism makes the penalty for latency during training more controllable, avoiding training oscillations caused by differences in the reasonable range of latency in different scenarios.

[0110] Finally, the reward is calculated using a weighted summation method. Specifically, the reward function combines global performance metrics and normalized latency, and is expressed as: Equation (5)

[0111] in, To normalize throughput, To normalize the group delivery rate, Normalized energy consumption; , , , These are non-negative weights used to balance the importance of different metrics. The design logic of the reward function is: increasing throughput and group delivery rate will increase the reward, while increasing latency and energy consumption will decrease the reward, guiding the model to generate actions that take into account multiple objectives.

[0112] Comprehensive data collection ensures that reward calculations reflect the true effects of actions, avoiding evaluation biases caused by single metrics. Normalization makes different metrics comparable, ensuring the stability of reward calculations. The weighted fusion reward function achieves multi-objective collaborative optimization, preventing the model from overemphasizing one metric while neglecting others. For example, when an action increases throughput but significantly increases energy consumption, the reward function reduces the value of that action through the penalty effect of energy consumption weight, guiding the model to select a more balanced action configuration.

[0113] Step S400: Calculate the Q-value using the evaluation network based on the action, the network state after the action, and the reward.

[0114] In this embodiment, the Q-value is a quantitative evaluation result of the action's value, representing the expected cumulative reward that can be obtained in the future after performing a certain action in the current network state. Its role is to provide a quantitative standard for the quality of actions and help the model distinguish the long-term value of actions with different configurations. The calculation of the Q-value is based on the evaluation network's modeling of the mapping relationship between network state, action, and reward, integrating the weighted sum of current reward and future reward, reflecting the long-term optimization effect of the action.

[0115] This embodiment proposes a multi-agent collaborative Q-value calculation logic. The network evaluation calculates the Q-value based on the global network state and the actions of all access points, rather than evaluating the value of a single access point's action in isolation. This logic considers the synergistic effect of multiple access point actions, avoiding the global performance degradation caused by local optima of a single access point. For example, when an access point performs a merging action, the Q-value calculation simultaneously considers its impact on the load distribution of adjacent access points and network interference, ensuring that the action selection aligns with the global optimization objective. The introduction of the Q-value allows the model to break free from the limitations of traditional rule-based configuration, learning optimal action selection strategies adapted to dynamic network environments through a data-driven approach, providing a core basis for subsequent parameter updates.

[0116] In one implementation, the step of calculating the Q-value through the evaluation network based on the action, the network state after the action, and the reward specifically includes the following steps: Step S410: Obtain the network status after all access points have performed actions, and integrate them to form a global network status; Step S420: Input the global network status, the actions of all access points and the calculated rewards into the main evaluation network, and calculate the Q value output by the main evaluation network; Step S430: Input the global network state, the target action generated by the target execution network, and the reward into the target evaluation network to obtain the target Q value output by the target evaluation network.

[0117] In this embodiment, the Q-value is calculated by evaluating the network, integrating the global network status with the actions and rewards of all access points, and the current Q-value and target Q-value are calculated by the main evaluation network and the target evaluation network respectively, providing a valuable basis for parameter updates.

[0118] The first step is the construction of the global network state. Specifically, the controller collects the network state after all access points have performed actions and integrates it to form the global network state. ,in This represents the number of access points. The global network status reflects the operational condition of the entire WiFi 7 network, providing a holistic perspective for centralized evaluation.

[0119] Then the main evaluation network calculates the current Q-value. Specifically, it considers the global network state and the actions of all access points. and the calculated reward Input the main evaluation network. The dual-Q structure of the main evaluation network will calculate the two Q values ​​separately. and 2. Finally, the minimum value between the two is selected as the current Q-value output by the main evaluation network. It effectively suppresses overestimation of the Q value, ensuring the accuracy of the current Q value.

[0120] Finally, the target evaluation network calculates the target Q-value. Specifically, this involves evaluating the global network state and the target actions generated by the target execution network. And reward input target evaluation network. Target action The target execution network generates actions based on the next state, and also includes discrete actions and continuous adjustments. The target evaluation network also uses a dual-Q structure to calculate two target Q values. and The minimum value is selected as the final target Q value. .

[0121] The construction of the global network state ensures that Q-value calculations take into account the synergistic effects of access point actions, avoiding misjudgments caused by local perspectives. The dual-Q structure of the main evaluation network and the target evaluation network respectively guarantees the accuracy of the current Q-value and the target Q-value, providing reliable data for subsequent time-series difference error calculations. For example, when multiple access points collaboratively execute load-sharing actions, the global network state can reflect the balance of load distribution. The main evaluation network will output a higher current Q-value, while the target evaluation network predicts subsequent rewards based on the next state, guiding the model to continuously select collaborative actions.

[0122] In one implementation, the access point configuration strategy model further includes an experience replay buffer, which stores experience samples. These experience samples include the current network state, actions, rewards, and the network state after the actions are executed. The steps for constructing and updating the experience replay buffer include: Step S401: Initialize the experience replay buffer using historical sample data; Step S402: After obtaining the network status after the access point performs an action and calculating the reward each time, integrate the current network status, action, reward, and network status after the action to construct an experience sample, and store the constructed experience sample in the experience replay buffer.

[0123] In this embodiment, the experience replay buffer of the access point configuration strategy model is used to store interaction samples during training, providing diverse and uncorrelated training data for evaluating network parameter optimization and improving training stability. The construction and updating of the experience replay buffer includes three core steps: initialization, sample storage, and sample sampling.

[0124] The first step is initialization. Specifically, the buffer is initialized using the interactive sample data from the initial training phase. This is used to represent the maximum storage capacity of the buffer, ensuring that the buffer initially has a certain amount of samples to support training. Initial samples can be generated through random actions at the access points, covering different network states and action combinations, avoiding training bias caused by a single initial sample.

[0125] Next comes sample storage. Specifically, after each step of acquiring the network state, generating an action, executing the action, acquiring the post-execution state, and calculating the reward, the current network state is stored. ,action ,award Network state after the action is executed Transferred samples are stored in a buffer. Sample storage follows a first-in, first-out (FIFO) principle; when the buffer reaches its maximum capacity, the oldest sample is overwritten by the newest sample, ensuring that the samples in the buffer always reflect recent network interaction characteristics. A sample is defined as... .

[0126] Finally, there is sample sampling. Specifically, when the number of samples in the buffer reaches a preset threshold, a batch of samples is randomly sampled from the buffer before iteratively updating the evaluation network parameters. These samples are then used to optimize the network parameters. Random sampling breaks the temporal correlation of samples, avoids gradient oscillations during training, and improves the stability of model convergence. For example, the sampled samples may cover various scenarios such as high load, low load, strong interference, and weak interference, enabling the evaluation network to learn general Q-value prediction patterns.

[0127] Sample storage enables the reuse of training data, avoiding the waste of data from single interactions and improving data utilization efficiency. Random sampling breaks the temporal correlation of samples, solving the gradient instability problem caused by the correlation of training data in reinforcement learning. The capacity limit of the buffer ensures the timeliness of samples, enabling the model to adapt to dynamic changes in network state. For example, when the network traffic load transitions from low to high periods, the samples in the buffer are quickly updated with the interaction data of the high period, allowing the optimization of the evaluation network to fit the current network scenario and improving the model's adaptability.

[0128] Step S500: Based on the Q value, iteratively update the parameters of the evaluation network by minimizing the loss function, and iteratively update the parameters of the execution network based on the optimized evaluation network.

[0129] In this embodiment, the loss function is a quantitative tool for measuring the difference between the Q-value predicted by the evaluation network and the actual action value. This embodiment uses the mean square function of temporal difference error, which can accurately characterize the degree of deviation in Q-value prediction. Iterative updates are a process of gradually adjusting network parameters through optimization algorithms such as gradient descent. The core is to adjust the parameters of the evaluation network and the execution network based on the gradient direction of the loss function, thereby continuously optimizing model performance. The gradient descent algorithm is the core method for parameter optimization. By calculating the gradient of the loss function with respect to the network parameters, the parameters are adjusted along the gradient descent direction to minimize prediction bias.

[0130] This embodiment proposes a collaborative iterative optimization mechanism between the evaluation network and the execution network. The evaluation network achieves accurate Q-value prediction by minimizing the loss function, providing the execution network with a reliable assessment of action value. The execution network adjusts its action generation strategy based on the optimized Q-value output of the evaluation network, making the generated actions more aligned with the global optimization objective. This collaborative optimization mechanism ensures continuous model learning. Specifically, the accurate predictions of the evaluation network provide a clear optimization direction for the execution network, while the action exploration of the execution network provides rich training data for the evaluation network, forming a closed-loop iteration. Simultaneously, designs such as delayed updates and soft updates further enhance training stability, avoiding model oscillations caused by parameter fluctuations, ultimately achieving collaborative optimization of multiple objectives such as network throughput, latency, and energy consumption.

[0131] In one implementation, the step of iteratively updating the parameters of the evaluation network by minimizing the loss function based on the Q-value, and iteratively updating the parameters of the execution network based on the optimized evaluation network, specifically includes the following steps: Step S510: Based on the Q value, iteratively update the parameters of the main evaluation network by minimizing the loss function; Step S520: Based on the optimized main evaluation network, iteratively update the parameters of the main execution network; Step S530: Based on the updated parameters of the main evaluation network and the main execution network, perform a soft update on the parameters of the target evaluation network in the evaluation network and the parameters of the target execution network in the execution network.

[0132] In this embodiment, the parameters of the evaluation network and the execution network are updated iteratively based on the Q-value. Specifically, the formula logic and order in the technical disclosure are followed. First, the preparatory work for updating the parameters of the main evaluation network is completed, namely, the generation of the target action. Then, the parameters of the main execution network and the soft update of the target network are implemented in sequence to achieve stable optimization of the model.

[0133] The first step is the generation of the target action. During training, when the target execution network generates the target action based on the next state, Gaussian noise is added to improve training stability. Specifically, this is expressed as follows: Equation (8)

[0134] in, Zero-mean Gaussian noise, cropped to the range [ δ, δ], The standard deviation of noise. This is the noise clipping threshold; clipping is used to prevent excessive noise from distorting the target's motion. Execute the network for the target. Its parameters, This represents the next network state after the action is performed. The generation of the target action is the basis for subsequent calculation of the target Q-value, providing stable action input for updating the parameters of the main evaluation network.

[0135] After generating the target action, the parameters of the main evaluation network are updated first, followed by the parameter update of the main execution network. Specifically, a delayed update period is set, and the parameter update of the main execution network is only triggered when the parameter update count of the main evaluation network reaches the period. This design provides a convergence time window for the Q-value prediction of the evaluation network, ensuring that the execution network is optimized based on a relatively stable value assessment.

[0136] The noise mechanism in Equation (8) injects appropriate perturbation into the target action, avoiding overfitting caused by the target action being too simple and improving the generalization ability of the model. The delayed update mechanism ensures the stability of the main execution network update, avoiding frequent changes in the execution network strategy due to fluctuations in the evaluation network parameters, and laying the foundation for smooth convergence of the subsequent overall parameter update.

[0137] In one implementation, the step of iteratively updating the parameters of the main evaluation network based on the Q-value by minimizing the loss function specifically includes the following steps: Step S511: Calculate the temporal difference error based on the Q value, wherein the temporal difference error is the difference between the Q value output by the master evaluation network and the target Q value output by the target evaluation network; Step S512: Construct a loss function with the objective of minimizing the mean square value of the time-series difference error; Step S513: Sample a batch of samples from the experience replay buffer and iteratively update the parameters of the main evaluation network using the gradient descent algorithm until the loss function value converges to a preset range.

[0138] In this embodiment, the parameters of the main execution network are iteratively updated based on the optimized main evaluation network. Specifically, the parameters of the main evaluation network are first optimized by calculating the target Q value and constructing the loss function. Then, the parameters of the main execution network are adjusted by guiding the gradient of the deterministic policy to ensure that the actions generated by the execution network can maximize the Q value.

[0139] First, the parameters of the main evaluation network are optimized. Specifically, based on the generated target action and current reward, the target Q-value is calculated using a dual-objective evaluation network, and the minimum of the two values ​​is taken to suppress overestimation bias, expressed as: Equation (9)

[0140] in, For the target Q value, For the current reward, As a discount factor, it balances the importance of current and future rewards; For the first A target evaluation subnetwork, Its parameters, The target action is defined. Calculating the target Q-value provides a reliable reference standard for constructing the loss function.

[0141] Subsequently, a loss function is constructed with the objective of minimizing the mean squared error between the predicted Q-value and the target Q-value of the main evaluation network. The expression is as follows: Equation (10)

[0142] in, For the first The loss value of the individual evaluation subnetwork. The parameters of the main evaluation subnetwork are used. The main evaluation network is used to predict the Q-value of the current state-action pair. The buffer serves as an experience replay buffer, and the expected computation is based on batch samples sampled from this buffer. Batch samples are sampled from the experience replay buffer, and the gradient of the loss function with respect to the parameters of the main evaluation network is calculated using the gradient descent algorithm. The parameters are then adjusted along the gradient descent direction until the loss function value converges to a preset range, thus completing the optimization of the main evaluation network parameters.

[0143] After the main evaluation network parameters are optimized, the main execution network parameter update phase will begin. This step uses a dual-Q network with a minimum design to suppress Q-value overestimation, the mean squared loss function to accurately characterize prediction bias, and the gradient descent algorithm to ensure smooth convergence of the main evaluation network parameters, ultimately achieving accurate Q-value prediction and providing a value assessment basis for the optimization of the main execution network.

[0144] In one implementation, the iterative update of the parameters of the main execution network based on the optimized main evaluation network specifically includes the following steps: Step S521: Set a delayed update period and count the number of parameter updates of the main evaluation network; Step S522: When the number of parameter updates reaches the set delay update period, calculate the deterministic policy gradient based on the Q value output by the optimized master evaluation network. Step S523: Update the parameters of the main execution network along the ascending direction of the deterministic policy gradient; The deterministic policy gradient is calculated by the expected value of the product of the gradient of the Q-value output by the main evaluation network with respect to the gradient of the main execution network's output action and the gradient of the main execution network's output action with respect to its own parameters.

[0145] In this embodiment, the parameters of the main execution network are iteratively updated based on the optimized main evaluation network. Specifically, the parameter adjustment is guided by a deterministic policy gradient to ensure that the actions generated by the execution network can maximize the Q value.

[0146] The first step is setting a delayed update period. Specifically, to avoid interference from fluctuations in the parameters of the main evaluation network on the updates of the execution network, a delayed update period is set. The parameter update of the main execution network is only triggered when the parameter update count of the main evaluation network reaches the set period. This design provides a convergence time window for the Q-value prediction of the evaluation network, ensuring that the execution network is optimized based on a relatively stable value assessment.

[0147] The second step is to calculate the gradient of the deterministic policy. Specifically, based on the Q-value of the optimized main evaluation network output, the parameter gradient of the main execution network is calculated, as follows: Equation (11)

[0148] in, To execute the network's policy gradient, The parameters for the main execution network; The gradient of the Q-value with respect to the action reflects the degree to which changes in the action affect the Q-value. The gradient of the action with respect to the network parameters reflects the impact of parameter changes on action generation; the expectation operation is based on state samples sampled from the empirical replay buffer. .

[0149] Finally, the parameters of the main execution network are updated. Specifically, the parameters are adjusted along the ascending direction of the deterministic policy gradient to ensure that the actions generated by the main execution network continuously improve the Q-value. The step size of the parameter update is controlled by the learning rate to avoid action oscillations caused by excessive parameter adjustments, ensuring smooth policy optimization of the execution network. After the main execution network parameter update is completed, the soft update phase of the target network begins.

[0150] The deterministic policy gradient provides a clear optimization direction for updating the execution network parameters, ensuring that the action generation policy remains consistent with the value assessment of the main evaluation network. The delayed update mechanism improves training stability, avoiding frequent changes in the execution network policy due to fluctuations in the evaluation network parameters, and laying the foundation for subsequent soft updates to the target network.

[0151] In one implementation, the step of softly updating the parameters of the target evaluation network in the evaluation network and the target execution network in the execution network based on the updated parameters of the main evaluation network and the main execution network specifically includes the following steps: Step S531: Set the soft update coefficient, and based on the soft update coefficient, calculate the weighted sum of the updated parameters of the main evaluation network and the original parameters of the target evaluation network, as well as the weighted sum of the updated parameters of the main execution network and the original parameters of the target execution network. Step S532: Use the two weighted sums obtained from the calculation as the new target evaluation network parameters and target execution network parameters, respectively, and update the target evaluation network and target execution network.

[0152] In this embodiment, the target network is softly updated based on the updated parameters of the main evaluation network and the main execution network. Specifically, the target network parameters are slowly adjusted through a weighted sum mechanism to ensure the stability of the training process.

[0153] The first step is setting the soft update coefficient. Specifically, the soft update coefficient... The value of this coefficient can be set to the range of 0.001 to 0.01. This coefficient directly determines the degree of influence of the main network parameters on the target network parameters. The smaller the coefficient, the smoother the update of the target network parameters and the higher the training stability; the larger the coefficient, the faster the target network parameters follow the main network, but it may introduce the risk of oscillation. This embodiment selects a smaller soft update coefficient to balance training stability and convergence efficiency.

[0154] The second step is the calculation of the weighted sum. Specifically, based on the soft update coefficient, the weighted sums of the parameters of the main evaluation network and the target evaluation network, and the main execution network and the target execution network are calculated separately, as follows: Equation (12)

[0155] Equation (13)

[0156] in, For the updated target evaluation subnetwork parameters, The updated main evaluation subnetwork parameters; Execute network parameters for the updated target. These are the updated parameters of the main execution network. The weighted sum formula shows that only a small portion of the new parameters of the target network come from the main network, while the vast majority retain the original parameters, ensuring a smooth parameter change.

[0157] Finally, the target network parameters are updated. Specifically, the two weighted sums are used as the new target evaluation network parameters and target execution network parameters, respectively, overwriting the original parameters and completing the soft update. This process ensures that the target network parameters can slowly keep up with the optimization results of the main network, avoiding sudden parameter changes that could cause oscillations in the target Q-value and target actions, and providing a stable reference standard for the next round of training.

[0158] The soft update mechanism avoids drastic fluctuations in the target network parameters, keeping the target Q-value and target action stable, and providing a reliable basis for the next round of parameter updates in the main evaluation network and the main execution network. The slow adjustment of parameter updates makes the training process converge smoothly, avoiding model oscillations caused by fluctuations in the main network parameters, improving the robustness of the entire training process, and ensuring that the model can continuously learn the optimal policy in a dynamic WiFi 7 network environment.

[0159] In one implementation, the convergence determination steps for iterative updates include: Step S541: After each round of network evaluation and parameter update, collect the global performance metrics of the WiFi7 network. Step S542: Calculate the fluctuation range of the global performance index within consecutive preset rounds; Step S543: If the fluctuation amplitude is less than a preset threshold, then it is determined that the access point configuration strategy model has converged.

[0160] In this embodiment, the convergence determination of the iterative update is made by monitoring the fluctuation range of the global performance index of the network to determine whether the model has reached the optimal configuration state, thereby avoiding resource waste caused by overtraining.

[0161] The first step is the collection of global performance metrics. Specifically, after each round of parameter updates for the evaluation network, execution network, and target network, the controller collects global performance metrics for the WiFi 7 network, including transmission latency, network throughput, packet delivery rate, and access point power consumption. These metrics are consistent with those used in reward calculations and directly reflect the model's configuration effectiveness.

[0162] The next step is to calculate the fluctuation range. Specifically, the fluctuation range of the global performance index within a consecutive preset number of rounds is calculated. The fluctuation range is calculated by taking the difference between the maximum and minimum values ​​within consecutive rounds for each performance index, and then dividing it by the average value to obtain the relative fluctuation range.

[0163] Finally, convergence is determined. Specifically, if the fluctuation range of all global performance indicators is less than a preset threshold, the access point configuration strategy model is considered to have converged. Convergence means that the model's configuration strategy has stabilized, and further training cannot significantly improve network performance. At this point, iterative updates can be stopped, and the optimal model can be output.

[0164] Convergence criteria provide a clear termination condition for the training process, avoiding the waste of computational resources caused by overtraining. Based on the fluctuation range of global performance metrics, it ensures that the network performance reaches a stable optimal state when the model converges. For example, when the model converges, if the fluctuation range of network throughput is less than a preset threshold and the transmission latency is stable below a preset threshold, it indicates that the model's configuration strategy has adapted to dynamic network changes and continuously outputs stable optimization results.

[0165] In summary, this invention discloses a training method for a WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning. Based on a dynamic cell multi-agent reinforcement learning framework and combined with the core characteristics of WiFi 7, it achieves intelligent optimization of access point cell configuration. Compared with existing technologies, it significantly improves network throughput and energy efficiency, reduces transmission latency, and adapts to user and load fluctuations through dynamic merging and splitting, solving the problems of load imbalance and resource waste. Simultaneously, relying on dual-band coordination and intelligent decision-making in multi-link operation, it ensures service continuity and spectrum utilization, adapts to high-density network scenarios, and provides highly reliable and high-performance wireless connectivity support for data-intensive applications.

[0166] This embodiment provides an application method for a WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning. The application method specifically includes the following steps: Step S10: Obtain the network status of each access point; Step S20: Based on the network status, generate and execute the actions of each access point through the execution network in the access point configuration strategy model to complete the access point configuration optimization; Step S30: Obtain the network status of each access point after performing the action and the global network status of the WiFi7 network.

[0167] In this embodiment, the application method specifically involves deploying the converged access point configuration strategy model on an actual WiFi 7 network to achieve dynamic optimization of access point configuration. Its execution logic is closely connected with the training method but does not involve training-related mechanisms.

[0168] The first step is acquiring the network status. Specifically, each access point collects its local network status at a preset period. The collected data is consistent with the network status data in the training method, including core indicators such as the number of connected users, traffic load, and remaining energy. After collection, the access point does not need to upload the status to the controller; it is directly used for local action generation, ensuring real-time performance.

[0169] Following this is action generation and execution. Specifically, the main execution network of each access point receives the real-time network state vector, generates a hybrid action vector through forward propagation, and determines the final configured action through softmax operation and argmax function before execution. For example, when an access point detects that its own load is too high, the main execution network generates a split action, which the access point immediately executes to split into two cells, relieving load pressure. If the access point detects that its own load and that of its neighboring access points are both low, a merge action is generated to reduce energy consumption.

[0170] Finally, the post-execution status is obtained. Specifically, after the access point executes an action, it collects its own post-execution network status and feeds back the global network status to the controller for subsequent configuration effect monitoring. The controller can determine whether the configuration meets the requirements based on the global network status. If the network performance in a certain area does not meet expectations, it can trigger the access point to re-execute the action generation process until the requirements are met.

[0171] Model loading and local execution ensure real-time configuration, adapting to the highly dynamic demands of WiFi 7 networks. The action generation and execution logic is consistent with the training method, ensuring that the optimization strategies learned during training are accurately implemented. A dynamic feedback mechanism ensures the reliability of the configuration effect, avoiding network performance degradation caused by invalid actions. For example, in high-density office scenarios, access points can dynamically generate merging and splitting actions based on real-time user counts, ensuring stable network throughput and low latency, fully leveraging the hardware performance of WiFi 7.

[0172] The specific training and application steps of the DC-MARL framework described above are as follows: Figure 13 As shown in the diagram. In this process, the network is first established, then the state of each access point is observed, followed by the training round. In the subsequent training round, it is first determined whether Distributed Multi-Agent Reinforcement Learning (DC-MARL) is enabled. If not enabled, the process sequentially initializes the Q-value metric, processes state-action pairs, evaluates rewards, and updates the Q-table to obtain the optimized value. If enabled, the DC-MARL framework is entered, and it is sequentially determined whether to perform actions such as merging, splitting, resizing, and load sharing. After generating rewards, these are processed by the decision engine, while simultaneously incorporating data from the experience replay buffer.

[0173] Based on the DC-MARL framework, this embodiment can also be combined with the characteristics of WiFi 7 networks. Each access point integrates core WiFi 7 features to optimize transmission performance; these core WiFi 7 features include 4096 quadrature amplitude modulation technology, preamble punching technology, hybrid automatic repeat request technology, and resource unit allocation based on the number of service users.

[0174] It consists of a set of N autonomous access points (APs), labeled as a set. Equipped with DC-MARL optimization and aggressive mode. Each AP can access one or more data sources from the collection. The frequency band.

[0175] For Multi-Link Operation (MLO), MLO supports concurrent transmission across multiple frequency bands, and its basic effective bandwidth formula is: Equation (19)

[0176] in, It is the MLO coordination overhead factor. It represents the percentage of bandwidth loss caused by preamble puncturing. Access point The number of active MLO links.

[0177] To understand how multi-link operations improve throughput, consider the incremental gain when enabling a new MLO link: Equation (20)

[0178] like and The changes are relatively small, and the first-order approximation is: Equation (21)

[0179] Furthermore, WiFi 7 introduces a more advanced modulation scheme, 4096-QAM, allowing each symbol to carry higher data density. This paper employs a high-level analytical model for WiFi 7 characteristics. 4096-QAM is represented as a fixed PHY rate improvement relative to the WiFi 6 benchmark; preamble puncturing reduces the effective bandwidth according to the puncturing ratio; and HARQ is modeled as a multiplicative improvement in packet delivery rate (PDR). These approximations allow for comparison of the relative performance of different schemes under consistent assumptions without needing to capture all the protocol details of IEEE 802.11be.

[0180] The modulation gain of 4096-QAM is calculated as follows: Equation (22)

[0181] Therefore, 4096-QAM offers a 20% throughput improvement compared to WiFi 6. Furthermore, preamble punching allows the AP to avoid congested sub-channels within the wide channel, thus the effective bandwidth is: Equation (23)

[0182] in, It is full bandwidth. This is due to the spectral puncture rate caused by interference. This ensures high throughput even in interference environments, where the effective throughput becomes: Equation (24)

[0183] To improve reliability, Hybrid Automatic Repeat Request (HARQ) is used to retransmit packets when transmission fails, which improves packet delivery rate. Equation (25)

[0184] in, This is the baseline packet delivery rate without HARQ. Ensure HARQ is enabled when a packet fails. It is an efficiency gain factor. HARQ can typically improve PDR by 10-15%. In addition to core PHY / MAC improvements, the framework also leverages the enhanced system management features of WiFi 7 to achieve fair resource allocation and energy saving. Furthermore, enhanced OFDMA and Resource Unit (RU) allocation in WiFi 7 improve OFDMA performance through finer RU allocation and multi-user scheduling. If the AP is... If each user is provided with an equal RU allocation, then: Equation (26)

[0185] Therefore, the throughput per user is: Equation (27)

[0186] The above formula ensures fair resource sharing and avoids starvation. The AP can set a running timer for each user, and users can enter sleep mode at other times, thus improving energy efficiency for each user as follows: Equation (28)

[0187] The above formula calculates the optimal sleep duration for devices within the DC-MARL framework, dynamically adjusting based on current throughput performance, access point load conditions, and cell size characteristics. By allowing longer sleep times when network conditions permit, while maintaining responsive operation during high-demand periods, it intelligently balances energy efficiency and service quality.

[0188] During resizing and load balancing, the AP can switch between frequency bands based on coverage area and traffic congestion: Equation (29)

[0189] The path loss is calculated using a frequency band-specific logarithmic propagation path loss model: Equation (30)

[0190] Where 32.4 is the fixed offset of the free space path loss model. It is the operating frequency. It is the exponential path loss factor. Cell size adjustment and optimal MLO band are selected based on path loss.

[0191] Each AP generates a Poisson distributed flow with a mean of λ. Assume that uplink and downlink flow are uniformly distributed: Equation (30)

[0192] Due to competition and protocol overhead, delivery throughput will decrease by a factor. Therefore, the delivery throughput in each direction is: Equation (31)

[0193] Packet delivery rate depends on interference, link quality, and traffic load. For the DC-MARL scheme, PDR is modeled as follows: Equation (32)

[0194] in, Indicates the baseline reliability based on interference. Indicates the MLO enhancement factor. Represents the spatial flow factor. Indicates traffic load. and It is the shape parameter of the Sigmoid.

[0195] The above formulas directly correspond to the performance quantification of WiFi 7's core features. Access points integrate these features and calculate and optimize transmission parameters according to the formulas to improve transmission performance, working in synergy with cell configuration optimization strategies.

[0196] Based on this, a specific simulation experiment was designed to verify the performance of this embodiment. The relevant parameters of the simulation experiment are shown in Table 1.

[0197] Table 1

[0198] The simulation experiment was conducted in a 100m×100m indoor WiFi 7 network environment, deploying 6 randomly distributed access points (APs) supporting the 2.4GHz, 5GHz, and 6GHz frequency bands. Poisson distribution was used to simulate user traffic (3-10 users per AP initially), and traffic load was covered at five levels: 200MHz, 400MHz, 600MHz, 800MHz, and 1000MHz. The experiment used the dual-delay deep deterministic policy gradient (TD3) algorithm to train a dynamic cell multi-agent reinforcement learning (DC-MARL) model, with the actuator learning rate set accordingly. evaluator learning rate The discount factor was 0.95, and the training run consisted of 500 rounds (20 seconds per round). The comparison schemes included the standard WiFi 7 scheme and the aggressive WiFi 7 MLO scheme. The core test metrics were network throughput, packet delivery rate (PDR), transmission latency, training stability, and feature attention distribution.

[0199] The experiment was conducted in three phases. The first phase involved environment deployment and parameter configuration, building a simulation network according to the aforementioned conditions. The second phase involved model training, where the DC-MARL agent learned strategies such as cell merging, splitting, and load sharing through interactive learning. The third phase involved performance testing, collecting performance data for each scheme under different traffic loads and generating corresponding performance curves and feature attention maps.

[0200] Experimental results passed Figures 6 to 11 exhibit.

[0201] Specifically, Figure 6 The relationship between throughput and training epochs under different traffic loads was demonstrated, showing the throughput trends of the three schemes under low, medium, and high loads. The DC-MARL enhanced scheme converged the fastest (stable within 100 epochs), achieving throughput exceeding 10Gbps under low load (200MHz), maintaining 33-35Gbps under medium load (400-600MHz), and breaking through 30Gbps under high load (800-1000MHz), with a peak of 38Gbps. The standard WiFi7 scheme achieved a throughput of only 21-25Gbps under high load, while the WiFi7 aggressive MLO scheme achieved approximately 25-30Gbps. The results indicate that DC-MARL significantly improves network throughput under different loads through dynamic cell configuration and MLO collaboration.

[0202] Figure 7The relationship between Packet Delivery Rate (PDR) and training rounds under different traffic loads is shown, reflecting changes in data transmission reliability. The PDR of the DC-MARL series solutions is consistently higher than that of traditional solutions. Under low load, the enhanced DC-MARL solution achieves a PDR of 99%, maintains 96% under medium load, and remains above 95% under high load; the standard WiFi 7 solution's PDR drops to 87% under high load, while the WiFi 7 aggressive MLO solution reaches approximately 90%. DC-MARL's interference avoidance and HARQ collaborative strategy effectively reduces packet loss caused by congestion.

[0203] Figure 8 The study demonstrates the relationship between average latency and training rounds under different traffic loads, showcasing the optimization effect on transmission timeliness. The DC-MARL solution maintains low latency throughout. It achieves approximately 2.5-3.0ms under low load, 3.5-4.0ms under medium load, and no more than 4.5ms under high load; in contrast, traditional solutions experience latency spikes to 6-15ms under high load. DC-MARL reduces queuing delays and interference through dynamic cell splitting and intelligent frequency band switching.

[0204] Figure 9 Training metrics such as reward, loss, learning rate, and stability are presented, demonstrating the dynamic process of model training. The cumulative reward of the DC-MARL enhancement scheme eventually reaches 800, significantly higher than that of traditional schemes (below 400). The loss of the double-Q evaluation network converges to [value missing] within 100 rounds. The magnitude of the algorithm avoids overestimation of the Q-value. A controllable decay strategy for the learning rate balances exploration and exploitation; the reward variance is less than 100, demonstrating better stability than traditional schemes (variance > 250), validating the model's training stability and convergence efficiency.

[0205] Figure 10 The presentation showcases the attention features of the DC-MARL scheme, revealing the attention weight distribution of DC-MARL agents. Collaborative features such as neighbor load, effective bandwidth, and MLO enable status account for over 60%, while energy metrics account for 8-12%. This demonstrates that the model can focus on the global network state and optimize resource allocation through multi-AP collaborative optimization, avoiding the trap of local optima.

[0206] Figure 11 The study demonstrates the attention characteristics of WiFi 7 solutions, showing that traditional WiFi 7 solutions focus on self-reference indicators such as their own load and frequency band (accounting for >70%), lacking attention to neighbor status and global coordination, resulting in rigid resource allocation and difficulty in adapting to dynamic load changes.

[0207] Therefore, it can be concluded that the DC-MARL framework, through multi-agent collaboration and dynamic cell configuration, significantly outperforms traditional WiFi7 solutions in throughput (28%-52% improvement), PDR (5%-8% improvement), transmission latency (25%-30% reduction), and training stability. It can adapt to high-density, high-load network scenarios and fully unleash the hardware performance potential of WiFi7.

[0208] In addition, performance verification (mean traffic) is set up under different average traffic loads, such as... Figure 12 As shown in the figure. This experiment verifies the model's adaptability under different traffic loads ranging from 5-40 Mbps, comparing it with traditional static strategies and single-agent strategies. The results show that under light load, the model reduces latency by 12%-15%; under medium load, the throughput is improved by 28%-33% compared to the traditional strategy, with PDR maintained above 99.2%; under heavy load, the traditional strategy suffers from severe congestion (PDR 82%), while the model still achieves a throughput of 38.5 Mbps, a PDR of 95.7%, and a latency of less than 22 ms. These results indicate that the model can dynamically adapt to traffic changes, and its congestion mitigation effect is significant under medium and heavy loads.

[0209] An experiment using feature attention was conducted, quantifying the contribution of state features through gradient and permutation methods. The results showed that the access point queue length (32%), user signal-to-noise ratio (28%), and current throughput (21%) were the core features, contributing a total of 81%, while marginal features accounted for only 4%. The feature attention mechanism can focus on core states, reduce redundant interference, improve decision accuracy, and provide a basis for lightweight model optimization.

[0210] The ablation study, using the complete model as a baseline, removed core components to verify the performance impact: without centralized commentators or experience replay, throughput decreased by 21%-27% and training oscillations occurred; without feature attention or soft updates to the target network, the number of convergence epochs increased by 15%-22%, and latency increased by 18%-35%. The experiments confirmed that the synergy of each component is indispensable, validating the rationality of the DC-MARL framework and the MATD3 algorithm architecture. The ablation study results are shown in Table 2.

[0211] Table 2

[0212] A queuing-theoretic interpretation experiment was conducted, mapping access points (APs) to service desks and data packets to customers. The model dynamically configured the corresponding service desk resource reallocation. The experiment showed that the model reduced the average AP queue length by 42%-51%, decreased user waiting time by 38%-45%, and improved service desk utilization balance to 85%. From a queuing-theoretic perspective, this validated the core logic of the solution in addressing the mismatch between service rate and arrival rate, demonstrating the theoretical rationale behind the technical solution.

[0213] like Figure 14 As shown in the figure, this embodiment of the invention provides a training system for a WiFi 7 network access point configuration policy model based on multi-agent reinforcement learning. The system is applied to a controller and several access points, and the controller and the access points are connected to the network. The training system includes: an initialization module 10, an action generation and execution module 20, a reward calculation module 30, a Q-value calculation module 40, and an iterative update module 50.

[0214] Specifically, the initialization module 10 is used to initialize the evaluation network and the execution network of each access point using random weights; the action generation and execution module 20 is used to obtain the network state of each access point, and generate and execute the actions of each access point through the execution network; the reward calculation module 30 is used to obtain the network state of each access point after executing the action and calculate the reward; the Q-value calculation module 40 is used to calculate the Q-value based on the action, the network state after executing the action, and the reward through the evaluation network; the iterative update module 50 is used to iteratively update the parameters of the evaluation network by minimizing the loss function based on the Q-value, and iteratively update the parameters of the execution network based on the optimized evaluation network.

[0215] Based on the above embodiments, the present invention also provides a terminal device, the principle block diagram of which can be as follows: Figure 15 As shown, the terminal device includes a processor, memory, network interface, display screen, and temperature sensor connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a training method for a WiFi 7 network access point configuration policy model based on multi-agent reinforcement learning. The display screen can be an LCD screen or an e-ink screen. The temperature sensor is pre-installed inside the terminal device to detect the operating temperature of the internal components.

[0216] Those skilled in the art will understand that Figure 15 The schematic diagram shown is only a partial structural diagram related to the present invention and does not constitute a limitation on the terminal device to which the present invention is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0217] In one embodiment, a terminal device is provided, including a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors, the one or more programs including instructions for performing operations as described in the embodiments of the methods above.

[0218] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0219] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0220] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A training method for a WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning, characterized in that, The access point configuration strategy model includes an execution network and an evaluation network, and the training method includes: Initialize the evaluation network and the execution network for each access point using random weights; Obtain the network status of each access point, and generate and execute actions for each access point through the execution network; Obtain the network status after each access point performs an action and calculate the reward; The Q-value is calculated using the evaluation network based on the action, the network state after the action, and the reward. Based on the Q value, the parameters of the evaluation network are iteratively updated by minimizing the loss function, and the parameters of the execution network are iteratively updated based on the optimized evaluation network.

2. The training method for the WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning according to claim 1, characterized in that, The access point configuration strategy model is applied to a WiFi 7 network that includes a controller and several access points, wherein the evaluation network is deployed on the controller and the execution network is deployed on each access point.

3. The training method for the WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning according to claim 2, characterized in that, The evaluation network includes a main evaluation network and a target evaluation network. The target evaluation network is a copy network with the same structure as the main evaluation network. The main evaluation network is a centralized double-Q evaluation network, which contains two evaluation sub-networks with the same structure and independent parameters. The initial parameters of the two evaluation sub-networks are obtained by initializing them with random weights. Before training begins, the initial parameters of the main evaluation network are completely assigned to the target evaluation network.

4. The training method for the WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning according to claim 3, characterized in that, The execution network includes a main execution network and a target execution network. The target execution network is a replica network with the same structure as the main execution network. The main execution network is a decentralized execution network, and each access point's main execution network outputs corresponding actions based on the network status. Before training begins, the initial parameters of the main execution network are fully assigned to the target execution network.

5. The training method for the WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning according to claim 1, characterized in that, The process of obtaining the network status of each access point includes: For each access point, its own operational data is collected, and the operational data of adjacent access points are obtained through the communication link. By integrating its own operational data with the operational data of adjacent access points, the initial network state is obtained; The initial network states are filtered and normalized, and then combined to form the final network states; The network status includes the number of users connected to the access point, current traffic load, remaining energy, average load of adjacent access points, normalized effective bandwidth, multi-link operation enabled status, current cell size, frequency band used, and time normalization factor.

6. The training method for the WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning according to claim 5, characterized in that, The process of generating and executing actions for each access point through the execution network includes: For each access point, a hybrid action vector is generated and output based on the network state through the execution network; Based on the hybrid action vector, the action is calculated and executed; The hybrid action vector includes at least a logarithmic probability of several discrete actions, wherein the discrete actions include at least one of merging, splitting, load sharing, resizing, and enabling multi-link operation.

7. The training method for the WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning according to claim 6, characterized in that, The process of calculating and executing actions based on the hybrid action vector includes: The logarithmic probabilities of discrete actions in the mixed action vector are subjected to a softmax operation to obtain the selection probability of each discrete action. The argmax function selects and executes the discrete action with the highest probability.

8. The training method for the WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning according to claim 1, characterized in that, The process of obtaining the network status after each access point performs an action and calculating the reward includes: Obtain the network status after each access point performs an action, as well as the global performance metrics of the WiFi 7 network; The transmission delay in the global performance metrics is normalized to obtain the normalized delay; The reward is calculated based on the global performance metrics and normalized latency. The global performance metrics include transmission latency, network throughput, packet delivery rate, and access point energy consumption.

9. The training method for the WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning according to claim 4, characterized in that, The step of calculating the Q-value through the evaluation network, based on the action, the network state after the action, and the reward, includes: Obtain the network status after all access points have performed actions, and integrate them to form a global network status; The global network state, the actions of all access points, and the calculated rewards are input into the main evaluation network to calculate the Q value output by the main evaluation network. The global network state, the target action generated by the target execution network, and the reward are input into the target evaluation network to obtain the target Q value output by the target evaluation network.

10. The training method for the WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning according to claim 9, characterized in that, The access point configuration strategy model also includes an experience replay buffer, which stores experience samples. These experience samples include the current network state, actions, rewards, and the network state after the actions are executed. The steps for constructing and updating the experience replay buffer include: Initialize the experience replay buffer using historical sample data; After obtaining the network state after each action is performed by the access point and calculating the reward, the current network state, action, reward, and network state after the action are integrated to construct an experience sample, and the constructed experience sample is stored in the experience replay buffer.

11. The training method for the WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning according to claim 10, characterized in that, The process of iteratively updating the parameters of the evaluation network by minimizing the loss function based on the Q-value, and iteratively updating the parameters of the execution network based on the optimized evaluation network, includes: Based on the Q value, the parameters of the main evaluation network are iteratively updated by minimizing the loss function; Based on the optimized main evaluation network, the parameters of the main execution network are iteratively updated; Based on the updated parameters of the main evaluation network and the main execution network, the parameters of the target evaluation network in the evaluation network and the parameters of the target execution network in the execution network are softly updated.

12. The training method for the WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning according to claim 11, characterized in that, The step of iteratively updating the parameters of the main evaluation network based on the Q-value by minimizing the loss function includes: The temporal difference error is calculated based on the Q value, and the temporal difference error is the difference between the Q value output by the master evaluation network and the target Q value output by the target evaluation network. A loss function is constructed with the objective of minimizing the mean square value of the time-series difference error; A batch of samples is sampled from the experience replay buffer, and the parameters of the main evaluation network are iteratively updated using the gradient descent algorithm until the loss function value converges to a preset range.

13. The training method for the WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning according to claim 12, characterized in that, The iterative update of the parameters of the main execution network based on the optimized main evaluation network includes: Set a delayed update period and count the number of parameter updates for the main evaluation network; When the number of parameter updates reaches the set delay update period, the deterministic policy gradient is calculated based on the Q value output by the optimized main evaluation network. The parameters of the main execution network are updated along the ascending direction of the deterministic policy gradient; The deterministic policy gradient is calculated by the expected value of the product of the gradient of the Q-value output by the main evaluation network with respect to the gradient of the main execution network's output action and the gradient of the main execution network's output action with respect to its own parameters.

14. The training method for the WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning according to claim 11, characterized in that, The step of softly updating the parameters of the target evaluation network in the evaluation network and the target execution network in the execution network based on the updated parameters of the main evaluation network and the main execution network includes: Set a soft update coefficient, and based on the soft update coefficient, calculate the weighted sum of the updated parameters of the main evaluation network and the original parameters of the target evaluation network, as well as the weighted sum of the updated parameters of the main execution network and the original parameters of the target execution network. The two weighted sums obtained are used as the new parameters for the target evaluation network and the target execution network, respectively, and the target evaluation network and the target execution network are updated accordingly.

15. The training method for the WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning according to claim 14, characterized in that, The convergence determination steps for iterative updates include: After each round of network evaluation and parameter updates, collect global performance metrics of the WiFi7 network. Calculate the fluctuation range of global performance indicators within consecutive preset rounds; If the fluctuation amplitude is less than a preset threshold, the access point configuration strategy model is determined to have converged.

16. An application method for a WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning, characterized in that, The application method of the access point configuration strategy model obtained by the training method described in claims 1-15 includes: Obtain the network status of each access point; Based on the network status, the actions of each access point are generated and executed through the execution network in the access point configuration policy model to complete the access point configuration optimization. Obtain the network status after each access point performs an action and the global network status of the WiFi 7 network.

17. A training system for a WiFi 7 network access point configuration strategy model based on multi-agent reinforcement learning, characterized in that, The access point configuration strategy model includes an execution network and an evaluation network, and the training system includes: The initialization module is used to initialize the evaluation network and the execution network for each access point using random weights. The action generation and execution module is used to obtain the network status of each access point, and generate and execute the actions of each access point through the execution network. The reward calculation module is used to obtain the network status after each access point performs an action and calculate the reward. The Q-value calculation module is used to calculate the Q-value based on the action, the network state after the action is performed, and the reward through the evaluation network. The iterative update module is used to iteratively update the parameters of the evaluation network by minimizing the loss function based on the Q value, and to iteratively update the parameters of the execution network based on the optimized evaluation network.

18. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a training program for a WiFi 7 network access point configuration policy model based on multi-agent reinforcement learning, stored in the memory and executable on the processor. When the processor executes the training program for the WiFi 7 network access point configuration policy model based on multi-agent reinforcement learning, it implements the steps of the training method for the WiFi 7 network access point configuration policy model based on multi-agent reinforcement learning as described in any one of claims 1-15.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a training program for a WiFi 7 network access point configuration policy model based on multi-agent reinforcement learning. When the training program for the WiFi 7 network access point configuration policy model based on multi-agent reinforcement learning is executed by a processor, it implements the steps of the training method for the WiFi 7 network access point configuration policy model based on multi-agent reinforcement learning as described in any one of claims 1-15.

Images