A seamless WIFI network distribution method based on Q-learning algorithm

Abstract

The present invention discloses a seamless WIFI network allocation method based on Q-learning algorithm, which specifically relates to the field of networking of POE switch and AP panel, including selecting AP through two-stage decision-making. n～1 or A n～2 ; First stage decision: select AP panel based on static indicators; Signal strength: evaluate signal strength and select AP that can cover the required area n Bandwidth utilization: Consider the current network traffic situation and select A with the lowest bandwidth utilization. n ; Packet loss rate: Select A with the lowest packet loss rate n ; Combining the above three indicators, determine the preferred A under static conditions n Decisions in this phase depend on the current network status and environmental characteristics. The second phase involves dynamic optimization and adjustment. This two-stage decision-making process, combined with the Q-learning algorithm, optimizes the networking and Wi-Fi allocation process for POE switches and AP panels, resolving issues such as inappropriate network resource allocation and poor performance, and improving network efficiency and performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the networking technology field of POE switch cooperating with AP panel, more particularly, the present application relates to a seamless WIFI network distribution method based on Q-learning algorithm. BACKGROUND

[0002] POE switch cooperating with AP panel networking refers to the process of using POE switch to provide power and network connection for AP panel, POE switch supplies power and transmits data signals to AP panel through network cable, without additional power line, simplifying wiring and installation process, AP panel as the access point of wireless local area network, is connected to local area network through POE switch, so that users can wirelessly connect and access network resources, this networking mode is commonly used in offices, schools, shops and other places to provide convenient wireless network access.

[0003] In the process of POE switch cooperating with AP panel networking and distribution, the traditional distribution mode only relies on signal source strength distribution, and does not consider other factors, such as the movement of equipment or the burst change of network load, which cannot adjust resource distribution in time to respond to these changes, resulting in instability of network performance and decline of user experience, in view of the above, the traditional distribution mode is too simple and rough, and cannot meet the complex network environment and user demand, therefore, a more intelligent and flexible method is needed to optimize the networking and resource distribution process of network equipment. SUMMARY

[0004] In order to overcome the above-mentioned defects of the prior art, the embodiments of the present application provide a seamless WIFI network distribution method based on Q-learning algorithm, which can optimize the networking and WiFi distribution process of POE switch and AP panel through two-stage decision and Q-learning algorithm, solve the problems of unreasonable network resource configuration and low performance, and improve network efficiency and performance, so as to solve the problems proposed in the above background.

[0005] In order to achieve the above-mentioned purpose, the present application provides the following technical scheme: a seamless WIFI network distribution method based on Q-learning algorithm, comprising:

[0006] S1: planning network topology, drawing a network topology diagram according to the wiring demand of network, marking the position and connection mode of each AP, the wiring demand of network includes position, coverage range and device quantity;

[0007] S2: network equipment selection, selecting POE switch and AP panel suitable for network scale and demand, for meeting the bandwidth and network coverage demand of network;

[0008] S3: Installation of POE switch and AP panel;

[0009] S4: Connect POE switch and AP panel with required Ethernet cables, each port of POE switch connects one AP panel, and plan each AP panel as A1 A2 A3...A n , POE switch as P 总 ;

[0010] S5: Configure network, configure VLAN, port security and other network settings in P 总 , to meet network security and performance optimization;

[0011] S6: Test and optimize, after completing network configuration, conduct network test, check coverage and signal strength of A1 to A n , and optimize network according to test results, to meet network performance and coverage;

[0012] S7: Network discovery: discover other devices around and establish connection;

[0013] Deploy sensors or monitoring devices in the network, collect network performance data, including signal strength D n , bandwidth utilization L n , packet loss rate S n of the corresponding area A n after accessing the device, these data are input to intelligent algorithm for network optimization decision;

[0014] Determine A n according to D n , L n and S n of the area after device access;

[0015] Any one A n includes at least 2.4GHz frequency band WIFI and 5GHz frequency band WIFI, plan 2.4GHz frequency band WIFI in this state as A n～1 , and plan 5GHz frequency band WIFI in this state as A n～2 ;

[0016] Select A n～1 or A n～2 through two-stage decision;

[0017] First stage decision: select AP panel based on static indicators;

[0018] Signal strength: evaluate signal strength and select A n that can cover the required area;

[0019] Bandwidth utilization: Consider the current network traffic situation and select A with the lowest bandwidth utilization. n ;

[0020] Packet loss rate: Select the A with the lowest packet loss rate n ;

[0021] Combining the above three indicators, we can determine the preferred A under static conditions. n ,The decisions made at this stage depend on the current network ,status and environmental characteristics;

[0022] Second stage decision-making: dynamic optimization and adjustment;

[0023] Real-time monitoring: Continuously monitor network performance indicators, including signal strength, bandwidth utilization, and packet loss rate;

[0024] Adaptive adjustment: Dynamically adjust A based on real-time monitored data n～1 and A n～2 choice;

[0025] Feedback mechanism: Feedback real-time monitoring data to the decision-making system to continuously optimize the selection strategy.

[0026] In a preferred embodiment, the two-stage decision making of reinforcement learning is modeled by a Markov decision process, in which the process learns how to choose actions in different states to maximize the cumulative reward through interaction with the environment;

[0027] First, define the state space and action space:

[0028] State space: includes different combinations of signal strength, bandwidth utilization, and packet loss rate;

[0029] Action space: different A selected by the agent n ;

[0030] Then, define the reward function:

[0031] Reward function: According to the current state and the action of the agent, the corresponding reward is given. The reward is a weighted combination of signal strength, bandwidth utilization and packet loss rate, as well as A n Defined by the choice of

[0032] Next, establish the state transition probability, which defines the probability distribution of the next state given the current state and action;

[0033] Finally, define the value function or policy function in the reinforcement learning algorithm to implement the two-stage decision-making:

[0034] First stage decision: select action based on current state, using value function or policy function to determine the preferred A n ;

[0035] Second stage decision: dynamically adjust action selection A based on feedback from the environment and learned knowledge to maximize long-term cumulative reward. n～1 or A n～2 .

[0036] In a preferred embodiment, the formula is established by a Q-learning algorithm, in which we define a Q function that represents the long-term cumulative reward that can be obtained by taking a certain action in a given state, denoted as Q(m, a):

[0037] Q(m, a) = (1 - p) * Q(m, a) + p * [r + g * max(a') Q(m', a')];

[0038] Q(m, a) is the estimated long-term cumulative reward for taking action a in state m, where state m is defined as a combination of signal strength D n , bandwidth utilization L n , and packet loss rate S n ; p is the learning rate, controlling the degree of influence of new rewards; r is the immediate reward obtained after taking action a in state m; g is the discount factor, used to weigh the importance of current rewards and future rewards; m' is the next state after taking action a, and a' is the action in the next state m'; max(a') Q(m', a') represents the action in the next state m' that maximizes the Q value;

[0039] Q(m, a) is used for the first stage decision or the second stage decision.

[0040] In a preferred embodiment, when it comes to optimizing the Q-learning algorithm or similar algorithms, the following includes testing and optimization steps:

[0041] 1) Define the problem:

[0042] Determine the type of problem;

[0043] Determine the state space and action space;

[0044] Determine hyperparameters such as reward function and discount factor;

[0045] 2) Initialize Q-table:

[0046] Create a Q-table to store the Q value of each state-action pair;

[0047] 3) Select action:

[0048] Select an action by policy;

[0049] 4) Perform the action:

[0050] Observe the next state and immediate reward according to the selected action execution environment;

[0051] 5) Update Q values by Q-learning algorithm;

[0052] 6) Test and evaluate:

[0053] Test using the trained Q-table;

[0054] Calculate the frequency or expected return of selecting the optimal action in each state;

[0055] Analyze performance and make evaluation;

[0056] 7) Optimization:

[0057] Adjust hyperparameters to optimize algorithm performance;

[0058] Adjust the policy to achieve better convergence and performance;

[0059] 8) Repeat iteration:

[0060] According to the test results and optimization strategy, repeat the above steps until a satisfactory performance level is reached or the optimal strategy is converged.

[0061] In a preferred embodiment, a Q-table is created to store the initial Q values of each state-action pair, and the Q-table is initialized to zero or random values.

[0062] In a preferred embodiment, random initialization: randomly assign a small initial value range to the Q value of each state-action pair, ranging from [-0.5, 0.5] or [-1, 1].

[0063] In a preferred embodiment, initialization according to prior knowledge: if prior knowledge is available, initialize the Q-table according to the knowledge.

[0064] In a preferred embodiment, initialization based on function approximation: in some cases, the Q-table is too large to be directly stored, and a function approximation method is used to estimate the Q value, and the weights of the neural network are randomly initialized.

[0065] Technical effects and advantages of the present application:

[0066] 1. By two-stage decision and Q-learning algorithm, the networking and WiFi distribution process of POE switch and AP panel can be optimized, the problem of unreasonable network resource configuration and low performance is solved, and the network efficiency and performance are improved;

[0067] 2. Through intelligent learning and optimization, the system can dynamically adjust network resource allocation according to real-time environment and demand, improve network stability, reliability and throughput, and provide better user experience and service quality. DETAILED DESCRIPTION

[0068] The technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0069] An embodiment of the present application is a seamless WIFI network distribution method based on Q-learning algorithm, which comprises:

[0070] S1: planning network topology, drawing a network topology map according to the wiring requirements of the network, marking the position and connection mode of each AP, and the wiring requirements of the network including position, coverage range and device quantity;

[0071] S2: network device selection, selecting POE switch and AP panel suitable for network size and demand to meet the bandwidth and network coverage requirements of the network;

[0072] S3: installation of POE switch and AP panel, installing POE switch in network room or appropriate position to ensure that it can be connected to power supply and network center, installing AP panel in each AP coverage area according to network topology map to ensure that AP panel is installed in appropriate position to maximize the optimization of wireless signal coverage range and signal quality;

[0073] S4: connecting POE switch and AP panel with required Ethernet cable, connecting one AP panel with each port of POE switch, and planning each AP panel as A1A2A3...A n , POE switch as P 总 ; when actually connecting, it is necessary to ensure stable and reliable connection to avoid signal attenuation caused by too long or curved cable;

[0074] S5: configuration of network, configuring network in P 总The VLAN, port security and other network settings are configured to meet the network security and performance optimization requirements. In addition, the wireless network settings of the AP panel, including SSID, encryption method, channel, etc., are configured to meet the network requirements and security requirements.

[0075] S6: Testing and optimization. After completing the network configuration, network testing is performed to check the coverage and signal strength of A1 to A n . Based on the test results, the network is optimized and adjusted to meet the network performance and coverage requirements and to achieve the expected target.

[0076] S7: Network discovery. The device discovers other devices around it and establishes a connection.

[0077] The device is characterized in that,

[0078] Sensors or monitoring devices are deployed in the network to collect network performance data, including the signal strength D n , bandwidth utilization L n , and packet loss rate S n of the area corresponding to A n after accessing the device. These data are used as input for the intelligent algorithm to make network optimization decisions.

[0079] Based on the D n , L n , and S n of the area obtained after the device accesses, the selection of A n is determined.

[0080] Any A n includes at least 2.4GHz frequency band WIFI and 5GHz frequency band WIFI. The 2.4GHz frequency band WIFI in this state is designated as A n～1 , and the 5GHz frequency band WIFI in this state is designated as A n～2 .

[0081] A n～1 or A n～2 is selected through two-stage decision-making.

[0082] First-stage decision: select the AP panel based on static indicators.

[0083] Signal strength: evaluate the signal strength and select A n that can cover the required area.

[0084] Bandwidth utilization: consider the current network traffic situation and select A n with the lowest bandwidth utilization.

[0085] Packet loss rate: select An ;

[0086] Based on the above three indicators, determine the preferred A n under static conditions ;

[0087] Second stage decision: dynamic optimization adjustment ;

[0088] Real-time monitoring: continuously monitor network performance indicators, including signal strength, bandwidth utilization and packet loss rate ;

[0089] Adaptive adjustment: dynamically adjust the selection of A n～1 and A n～2 according to real-time monitoring data ;

[0090] Feedback mechanism: feedback real-time monitoring data to decision system, continuously optimize selection strategy .

[0091] In addition, the two-stage decision of reinforcement learning is modeled through Markov decision process, in which the agent learns how to choose actions in different states to maximize cumulative rewards through interaction with the environment ;

[0092] First, define the state space and action space :

[0093] State space: includes different combinations of signal strength, bandwidth utilization and packet loss rate .

[0094] Action space: different A n selected by the agent ;

[0095] Then, define the reward function :

[0096] Reward function: according to the current state and the action of the agent, give the corresponding reward, the reward is defined according to the weighted combination of signal strength, bandwidth utilization and packet loss rate, and the selection of A n ;

[0097] Next, establish the state transition probability, define the probability distribution of the next state given the current state and action ;

[0098] Finally, define the value function or policy function in reinforcement learning algorithm to realize two-stage decision :

[0099] First stage decision: select action based on current state, use value function or policy function to determine the preferred A n .

[0100] Second stage decision: dynamically adjust the action selection A n～1 or A n～2to maximize long-term cumulative reward.

[0101] The formula is established by Q-learning algorithm, in which a Q function is defined for the long-term cumulative reward that can be obtained by taking a certain action in a given state, and the function Q(m, a) is expressed as:

[0102] Q(m, a) = (1-p) * Q(m, a) + p * [r + g * max(a') Q(m', a')];

[0103] Q(m, a) is the estimated long-term cumulative reward for taking action a in state m, and state m is defined as a combination containing signal strength D n , bandwidth utilization L n , and packet loss rate S n ; p is the learning rate, which controls the degree of influence of new rewards; r is the immediate reward obtained after taking action a in state m; g is the discount factor, which is used to weigh the importance of current rewards and future rewards; m' is the next state after taking action a, and a' is the possible action in the next state m'; max(a') Q(m', a') indicates selecting the action with the maximum Q value in the next state m', where Q usually refers to action, that is, the value function, in the reinforcement learning of the present scheme, Q(m, a) represents the expected cumulative return that can be obtained by taking action a in state m and following the logical strategy of the present scheme. In other words, Q(m, a) represents the expected reward that the agent can obtain after selecting action a in state m, which can help the agent make optimal decisions between different states and actions to maximize long-term cumulative return;

[0104] Q(m, a) is used for first-stage decision or second-stage decision, the application of m in first-stage decision is as described above, and when Q(m, a) is applied in second-stage decision, m is used to determine the selected A n ; and action a is the selected A n～1 or A n～2 ; of course, the present scheme can also be used only for second-stage decision;

[0105] It should be noted that if the learning rate is small, the updating amplitude will be small each time, the algorithm will be more conservative, and more iteration times may be needed to converge to the optimal solution, but it helps to maintain stability and avoid converging to a local optimal solution. On the contrary, if the learning rate is large, the updating amplitude will be large, the algorithm may converge faster, but it may also lead to instability or miss the optimal solution;

[0106] Therefore, selecting a suitable learning rate is crucial for the performance of the Q-learning algorithm, and the optimal learning rate value usually needs to be determined through experiments and parameter tuning.

[0107] When it comes to optimizing Q-learning algorithms or similar algorithms, the following steps include testing and optimization:

[0108] 1) Define the problem:

[0109] Determine the type of problem; (e.g., Markov Decision Process)

[0110] Determine the state space and action space;

[0111] Determine hyperparameters such as reward function and discount factor;

[0112] 2) Initialize Q-table:

[0113] Create a Q-table to store Q-values for each state-action pair;

[0114] 3) Select an action:

[0115] Select an action through a certain strategy (e.g., ε-greedy strategy); ε-greedy strategy allows balancing between exploration (randomly selecting actions) and exploitation (selecting the current best action)

[0116] 4) Execute the action:

[0117] Execute the environment according to the selected action, observe the next state and immediate reward;

[0118] 5) Update Q-values through the Q-learning algorithm;

[0119] 6) Test and evaluate:

[0120] Test using the trained Q-table;

[0121] Calculate the frequency of selecting the optimal action or expected return in each state;

[0122] Analyze performance and make evaluations;

[0123] 7) Optimization:

[0124] Adjust hyperparameters (learning rate, discount factor, etc.) to optimize algorithm performance;

[0125] Adjust the strategy (such as exploration rate) to achieve better convergence and performance;

[0126] Use more advanced techniques such as Deep Q-Networks (Deep Q-Networks) to handle complex problems and improve performance;

[0127] 8) Repeat iterations:

[0128] According to the test results and optimization strategies, repeat the above steps until a satisfactory performance level is reached or the optimal strategy is converged.

[0129] In practical application of this scheme, the performance of Q-learning algorithm can be verified and optimized through experiments and simulations, and adjusted and improved according to actual needs.

[0130] In the initialization of Q-table scheme, a Q-table is created to store the initial Q values of each state-action pair. The Q-table can be initialized to zero or random values. The Q-table, also known as Q-value table, is a data structure used in Q-learning algorithm to store the Q values of each state-action pair. In Q-learning, the agent (such as a robot, an intelligent agent, etc.) tries to learn to choose the optimal action to achieve the maximum cumulative reward. The Q-table is a two-dimensional array, where the rows represent possible states and the columns represent possible actions. Each table entry (state-action pair) contains a Q value, which represents the expected return of performing a specific action in a particular state. By continuously updating the Q-table, the agent can learn the strategy of choosing the best action in each state.

[0131] The first initialization scheme is random initialization: a small initial value range is randomly assigned to the Q value of each state-action pair, which is proposed to be between [-0.5, 0.5] or [-1, 1]. This method can help the algorithm to explore in the initial stage without falling into local optimal solution.

[0132] The second initialization scheme is to initialize according to prior knowledge: if prior knowledge is available, initialize the Q-table according to the knowledge. For example, if it is known that some actions are very important to achieve the target state, the Q values of these actions can be initialized to larger values.

[0133] The third initialization scheme is based on function approximation initialization: in some cases, the Q-table will be very large and difficult to store directly. Function approximation methods (such as neural network-based methods) can be used to estimate Q values, and the weights of the neural network can be randomly initialized.

[0134] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A seamless WIFI network allocation method based on Q-learning algorithm, characterized in that: include: S1: Plan the network topology. Draw a network topology diagram based on the network cabling requirements, marking the location and connection method of each AP. The network cabling requirements include location, coverage, and number of devices. S2: Network equipment selection: Choose POE switches and AP panels that are suitable for the network scale and needs to meet the network bandwidth and network coverage requirements; S3: Installation of POE switch and AP panel; S4: Use the required Ethernet cables to connect the POE switch and the AP panel. Each port of the POE switch is connected to an AP panel, and each AP panel is designated as A1A2A3...A n , POE switch is P 总 ; S5: Configure the network, in P 总 Configure VLAN, port security, and other network settings to ensure network security and performance optimization; S6: Test and optimize, after completing the network configuration, perform network testing to check the A1 to A n The coverage and signal strength of the network are measured, and the network is optimized and adjusted according to the test results to meet the network performance and coverage conditions; S7: Network discovery: The device discovers other devices around it and establishes connections; It is characterized in that Deploy sensors or monitoring devices in the network to collect network performance data. The network performance data includes the corresponding A n The signal strength D in this area n , bandwidth utilization L n , packet loss rate S n ,These data serve as input to intelligent algorithms for network optimization ,decisions; After the device is connected, the D n 、L n and S n Decision A n choice; Any A n At least including 2.4GHz band WIFI and 5GHz band WIFI, it is planned that the 2.4GHz band WIFI in this state is A n～1 , the 5GHz band WIFI in this state is planned to be A n～2 ; Choose A through a two-stage decision-making process n～1 or A n～2 ; First stage decision-making: select AP panels based on static indicators; Signal strength: Evaluate the signal strength and select the signal that can cover the required area. n ; Bandwidth utilization: Consider the current network traffic situation and select A with the lowest bandwidth utilization. n ; Packet loss rate: Select the A with the lowest packet loss rate n ; Combining the above three indicators, we can determine the preferred A under static conditions. n ,The decisions made at this stage depend on the current network ,status and environmental characteristics; Second stage decision-making: dynamic optimization and adjustment; Real-time monitoring: Continuously monitor network performance indicators, including signal strength, bandwidth utilization, and packet loss rate; Adaptive adjustment: Dynamically adjust A based on real-time monitored data n～1 and A n～2 choice; Feedback mechanism: Feedback real-time monitoring data to the decision-making system to continuously optimize the selection strategy.

2. The seamless WIFI network allocation method based on Q-learning algorithm according to claim 1, characterized in that: The two-stage decision-making of reinforcement learning is modeled by a Markov decision process, in which the process learns how to choose actions in different states to maximize the cumulative reward through interaction with the environment; First, define the state space and action space: State space: includes different combinations of signal strength, bandwidth utilization, and packet loss rate; Action space: different A selected by the agent n ; Then, define the reward function: Reward function: According to the current state and the action of the agent, the corresponding reward is given. The reward is a weighted combination of signal strength, bandwidth utilization and packet loss rate, as well as A n Defined by the choice of Next, establish the state transition probability, which defines the probability distribution of the next state given the current state and action; Finally, define the value function or policy function in the reinforcement learning algorithm to implement the two-stage decision-making: The first stage of decision-making: select an action based on the current state, using a value function or a policy function to determine the preferred A n ; Second stage decision-making: Dynamically adjust action selection A based on environmental feedback and learned knowledge n～1 or A n～2 , in order to maximize the long-term cumulative rewards.

3. The seamless WIFI network allocation method based on Q-learning algorithm according to claim 2, characterized in that: The formula is established through the Q-learning algorithm. In Q-learning, we define a Q function for the long-term cumulative reward that can be obtained by taking a certain action in a given state. The function Q(m,a) is expressed as: Q(m,a)=(1-p)*Q(m,a)+p*[r+g*max(a')Q(m',a')]; Q(m,a) is the estimated long-term cumulative reward for taking action a in state m, where state m is defined as the state containing the signal strength D n , bandwidth utilization L n , packet loss rate S n p is the learning rate, which controls the impact of the new reward; r is the immediate reward obtained after taking action a in state m; g is the discount factor, which is used to weigh the importance of current rewards and future rewards; m' is the next state after taking action a, and a' is the action in the next state m'; max(a')Q(m',a') means choosing the action that maximizes the Q value in the next state m'; Q(m,a) is used for first-stage decision making or second-stage decision making.

4. The seamless WIFI network allocation method based on Q-learning algorithm according to claim 3, characterized in that: When it comes to optimizing Q-learning or similar algorithms, the following steps are involved in testing and optimization: 1) Define the problem: Identify the type of problem; Determine the state space and action space; Determine the reward function and discount factor; 2) Initialize Q-table: Create a Q-table to store the Q value of each state-action pair; 3) Select an action: Select actions through strategies; 4) Execute the action: Execute the environment based on the selected action, observing the next state and immediate reward; 5) Update the Q value through the Q-learning algorithm; 6) Testing and Evaluation: Use the trained Q-table for testing; Calculate the frequency or expected reward of choosing the optimal action in each state; Analyze performance and conduct evaluations; 7) Optimization: Adjust hyperparameters to optimize algorithm performance; Adjust strategies for better convergence and performance; 8) Repeat iteration: Based on the test results and optimization strategy, the above steps are repeated until a satisfactory performance level is achieved or convergence to the optimal strategy is achieved.

5. The seamless WIFI network allocation method based on Q-learning algorithm according to claim 4, characterized in that: Create a Q-table to store the initial Q value of each state-action pair. The Q-table is initialized to zero or random values.

6. The seamless WIFI network allocation method based on Q-learning algorithm according to claim 5, characterized in that: Random initialization: Randomly assign a small initial value range to the Q value of each state-action pair, preferably between [-0.5, 0.5] or [-1, 1].

7. The seamless WIFI network allocation method based on Q-learning algorithm according to claim 5, characterized in that: Initialization based on prior knowledge: If prior knowledge is available, initialize the Q-table based on that knowledge.

8. The seamless WIFI network allocation method based on Q-learning algorithm according to claim 5, characterized in that: Initialization based on function approximation: In some cases, the Q-table will be too large to store directly. Function approximation methods are used to estimate the Q value and the weights of the neural network are randomly initialized.