A wireless access point version upgrade method and system based on reinforcement learning

Abstract

The present invention belongs to the field of wireless communication technology, and specifically relates to a wireless access point version upgrade method based on reinforcement learning, which includes the following steps: S1: start initializing parameters, assuming that N APs are connected to the same AC; S2: AC receives AP upgrade request, sets set variable K, count variable Count and cumulative count variable Total; S3: judge that the number of APs to be upgraded = M or the number of remaining APs < M, jump to step S4; otherwise jump to the step S2; S4: AC obtains a strategy according to a reinforcement learning algorithm, sends a version upgrade file to AP according to the strategy, and AP restarts after completing the version upgrade to complete the upgrade; S5: AP upgrade selection and bandwidth allocation strategy; S6: judge that all APs have completed the upgrade, and the steps end; otherwise jump to the step S2. The present invention can adaptively select a suitable AP to be upgraded according to dynamic conditions, and allocate a suitable bandwidth, reduce the probability of AP upgrade failure, and shorten the overall upgrade time.

Description

Technical Field

[0001] The present invention belongs to the technical field of wireless communications, and in particular relates to a method and system for upgrading a wireless access point version based on reinforcement learning. Background Art

[0002] Wireless Local Area Network (WLAN) network consists of wireless access points (AP) and wireless controllers (AC). Wireless access points are responsible for wireless connection and data transmission with client devices, while wireless controllers are responsible for centralized management and control of wireless access points. With the rapid development of mobile Internet, wireless networks need to support a large number of users to access at the same time. However, traditional distributed wireless networks are difficult to manage and maintain in high-density user scenarios. Therefore, the deployment mode combining centralized AC with lightweight access points (FitAP) has gradually become the mainstream.

[0003] During the deployment and operation of the wireless LAN, in order to support new functions and improve user experience, it is necessary to upgrade the AP version. The specific process of AP version upgrade is as follows: First, the AC pre-stores the AP version file. Then, after the AP to be upgraded establishes a CAPWAP tunnel with the AC, it sends a file request message to the AC to obtain the AP version file pre-stored in the AC. After receiving the file request message, the AC sends the AP version file to the AP to be upgraded. Finally, the AP to be upgraded replaces the local AP version file with the obtained version file, thereby completing the AP upgrade process.

[0004] During the AP upgrade process, the AP needs to download the version upgrade file from the AC. The following two factors affect the success of the version upgrade and the overall upgrade time:

[0005] First, bandwidth allocation is performed. When the AC sends version upgrade files to multiple APs at the same time, bandwidth allocation is required. If some APs obtain too little bandwidth, the upgrade file transmission rate will be slow, increasing the overall upgrade time or causing the upgrade to fail due to upgrade timeout.

[0006] Secondly, AP memory affects the time and success of version upgrades. If the AP currently has insufficient memory, that is, the current memory is smaller than the size of the version upgrade file, it is necessary to wait for the AP memory to be released before upgrading, resulting in failure of the AP upgrade. On the other hand, business continuity should be considered during the AP upgrade process. If the AC upgrades immediately after receiving each upgrade request, the AP cannot provide wireless services during the upgrade process, which will cause frequent business interruptions.

[0007] Therefore, in this technology, after receiving upgrade requests from multiple APs at the same time, the AC adopts a batch upgrade method. When the number of APs to be upgraded reaches the set value, the group of APs is upgraded by comprehensively considering the bandwidth allocation and AP memory conditions. After the upgrade of the current group is completed, the next group is upgraded. In summary, the present invention proposes a wireless access point version upgrade method and system that adopts a reinforcement learning-based solution, can adaptively select appropriate APs to be upgraded according to dynamic conditions, allocate appropriate bandwidth, reduce the probability of AP upgrade failure, and shorten the overall upgrade time. Summary of the invention

[0008] The purpose of the present invention is to provide a wireless access point version upgrade method and system based on reinforcement learning, which can adaptively select a suitable AP to be upgraded according to dynamic conditions and allocate appropriate bandwidth, thereby reducing the probability of AP upgrade failure and shortening the overall upgrade time.

[0009] The technical solution adopted by the present invention is as follows:

[0010] A wireless access point version upgrade method based on reinforcement learning includes the following steps:

[0011] S1: Start initializing parameters. Assume that N APs are connected to the same AC.

[0012] S2: AC receives AP upgrade request and sets collection variables Count variable Count and cumulative count variable Total;

[0013] S3: Determine whether the number of APs to be upgraded = M or the number of remaining APs < M, and jump to step S4; otherwise, jump to step S2;

[0014] S4: The AC obtains a strategy based on the reinforcement learning algorithm and sends a version upgrade file to the AP based on the strategy. After the AP completes the version upgrade, it restarts to complete the upgrade.

[0015] S5: AP upgrade selection and bandwidth allocation strategy;

[0016] S6: If it is determined that all APs have completed upgrading, the step ends; otherwise, the process goes to step S2.

[0017] The step S1 is Indicates the AP set. The AP sends a version upgrade request to the AC.

[0018] In step S2, the initial The initial value of the count variable Count and the cumulative count variable Total are both 0; when the AC receives an upgrade request from the AP, the Count value and the Total value are both incremented by 1 and the Count value at that time is added to the collection middle.

[0019] After the step S4 is completed, The variables are reset to empty, Count is reset to 0, and the Total value is not reset.

[0020] A wireless access point version upgrade system based on reinforcement learning includes a parameter initialization module, a version upgrade file, a central processing module, a channel bandwidth allocation module and an AP upgrade completion time management module.

[0021] The version upgrade file of the APi is represented by S i ={D i ,F i}, where D i Indicates the file size, F i Indicates the file version number.

[0022] The channel bandwidth allocation module is for channel bandwidth allocation between AC and AP. Assuming that the total communication bandwidth between AC and N APs is B, the bandwidth allocated between AC and APi during the upgrade process is expressed as B i , we can get:

[0023]

[0024] That is, each time the AC sends a version upgrade file to a group of APs, the total communication bandwidth between the AC and the APs to be upgraded cannot exceed the total bandwidth B.

[0025] The AP upgrade completion time management module considers a group of APs to be upgraded for the AP upgrade completion time. The transmission rate between AC and APi(i∈K) can be expressed as:

[0026] r i =B i log2(1+SNR i ). (2)

[0027] Among them, B i is the allocated channel bandwidth, SNR i is the signal-to-noise ratio of the signal transmitted between AC and APi;

[0028] Then, the version upgrade time for APi can be expressed as:

[0029]

[0030] For the AP group to be upgraded The upgrade completion time of this group depends on the slowest AP. The upgrade time of this group can be obtained as:

[0031] T K=max{T i tr},i∈K. (4)

[0032] During the AP upgrade process, in order to minimize the overall upgrade time, the upgrade time of each group of APs to be upgraded must be minimized. Therefore, the group of APs to be upgraded is selected and bandwidth is allocated reasonably. Therefore, for each group of APs to be upgraded, the relationship to minimize the upgrade completion time is as follows:

[0033]

[0034] Where C={C1,C2,...,C M} is the upgrade selection strategy, the elements are "0-1" variables, "1" means to upgrade, "0" means not to upgrade, B={B1,B2,...,B M} is the bandwidth allocation strategy, which allocates bandwidth to the AP being upgraded.

[0035] The AP upgrade selection and bandwidth allocation strategy is first transformed into an MDP problem and then solved using the reinforcement learning method; and the MDP problem is defined by a tuple {S, A, T, R}, where S is the system state set, A is the action set, T = {p(s'|s, a)} is the transition probability set, and R: S × A → R is a real-valued reward function;

[0036] The MDP problem is defined as follows:

[0037] M1, state: When count = M, the current state is defined as the set of AP version upgrade file size and channel state information, as follows:

[0038] s={D,H,L}. (6)

[0039] Where D = {D1, D2, ..., D M} is the version upgrade file size information, H = {SNR1, SNR2, ..., SNR M} is the signal-to-noise ratio of the signal transmission between AC and AP, L={l1,l2,...,l M} is the current available memory status information of AP;

[0040] M2, Action: When count = M, the action is defined as the selection of each AP to be upgraded and the allocated bandwidth, as follows:

[0041] a={B1,B2,...,B M ,C1,C2,...,C M}. (7)

[0042] M3, Reward: To minimize the overall upgrade time, the reward function is defined using the upgrade completion time of each group of APs, as follows:

[0043] r=f(T K ). (8)

[0044] Among them, f(·) is a negative exponential function that plays a normalization role. The longer the upgrade completion time is, the smaller the reward function is.

[0045] The DDPG algorithm in the reinforcement learning algorithm is used to process the MDP problem. AC is a reinforcement learning agent. The DDPG algorithm includes an Actor network, a Critic network, an Actor network output strategy, and a Critic network evaluation strategy, which are as follows:

[0046] The Actor network consists of two networks, one of which is an online actor network, whose parameters are θ μ Indicates that the other is the targetactor network, whose parameters are θ μ 'express;

[0047] The deterministic policy output by the Actor network is denoted by μ(s|θ μ ) indicates that the strategy is determined by the parameter θ μ Sure;

[0048] The critic network consists of an onlineQ network and a targetQ network, and its network parameters are θ Q and θ Q' express;

[0049] The critic network evaluation strategy sets the onlineQ network as the Q function used for evaluation, which is defined as Q(s,a|θ Q ) is used to indicate.

[0050] In the training process of the DDPG algorithm, B represents a batch of sample data randomly drawn from the experience pool, where the kth sample data can be represented by {s k ,a k ,r k ,s k+1}, the policy gradient is expressed as:

[0051]

[0052] Among them, Q(s,a) represents the Q function value output by the Critic network;

[0053] The onlineQ network obtains the Q estimate based on the sample data randomly drawn from the experience pool B. Given the sample data {s k,a k ,r k ,s k+1}, we can get the Q estimate Q(s k ,a k |θ Q ); and the Q target value can be expressed as:

[0054] y k =r k +γQ'[s k+1 μ'(s k+1 |θ μ' )|θ Q' ]. (10)

[0055] Among them, μ'(s k+1 |θ μ' ) is the output of the targetactor network, so the loss function for updating the onlineQ network can be obtained as follows:

[0056]

[0057] During the network update process, for the online actor network, the network parameters are updated according to the policy gradient algorithm, which can be expressed as:

[0058]

[0059] Among them, ρ μ Represents the learning rate of actor network updates;

[0060] For the parameter update of the online Q network, the method of minimizing the loss function is considered, and the gradient descent method is used, which can be expressed as:

[0061]

[0062] Among them, ρ Q is the learning rate of the critic network;

[0063] For the corresponding target network in the Actor network and the Critic network, the update process depends on the parameters of the online network. The target network is assigned a value using the soft update method, which ensures that the target network is updated in each iteration, which can be expressed as:

[0064]

[0065] Among them, τ∈(0,1) is the rate of soft update;

[0066] After the DDPG algorithm training is completed, the optimal strategy can be obtained by inputting the current state, as follows:

[0067] μ * (s) = arg max a Q * (s,a) (16).

[0068] In summary, through steps S1-S6, bandwidth and AP memory factors are comprehensively considered, and real-time decisions can be made for AP upgrades. As a reinforcement learning agent, AC continuously interacts with the dynamic environment to learn and update network model parameters to achieve optimal upgrade selection and bandwidth allocation strategy learning. After the reinforcement learning model training is completed, AC can adaptively make upgrade decisions in real time based on current status information, improve the success rate of AP upgrades, reduce the overall upgrade time, and achieve fast AP upgrades.

[0069] The technical effects achieved by the present invention are:

[0070] The present invention provides a wireless access point version upgrade method based on reinforcement learning. Through steps S1-S6, it comprehensively considers bandwidth and AP memory factors, and can provide real-time decision-making for AP upgrades. AC, as a reinforcement learning agent, continuously interacts with the dynamic environment for learning, updates network model parameters to achieve optimal upgrade selection and bandwidth allocation strategy learning. After the reinforcement learning model training is completed, AC can adaptively make upgrade decisions based on current state information in real time, improve the success rate of AP upgrades, reduce the overall upgrade time, and achieve rapid AP upgrades. It can adaptively select appropriate APs to be upgraded according to dynamic conditions, and allocate appropriate bandwidth, reduce the probability of AP upgrade failure, and shorten the overall upgrade time.

[0071] The invention discloses a wireless access point version upgrade system based on reinforcement learning. The MDP problem is defined. Through the three steps of state, action and reward, it is concluded that f(·) is a negative exponential function that plays a normalization role. The longer the upgrade completion time is, the smaller the reward function is. This serves as a theoretical guide for shortening the overall upgrade time.

[0072] The invention discloses a wireless access point version upgrade system based on reinforcement learning. The DDPG algorithm adopted by the invention includes Actor and Critic networks, Actor network output strategy, and Critic network evaluation strategy. The invention can perform soft update and provide learning theory guidance for optimal upgrade selection and bandwidth allocation strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 It is a flow chart of a wireless access point version upgrade method based on reinforcement learning of the present invention.

[0074] Figure 2 It is a system block diagram of a wireless access point version upgrade system based on reinforcement learning of the present invention. Detailed Implementation Modes

[0075] To make the objectives and advantages of the present invention more clear and understandable, the present invention will be specifically described below in conjunction with embodiments. It should be understood that the following text is only used to describe one or several specific implementation modes of the present invention, and does not strictly limit the scope of protection specifically claimed for the present invention.

[0076] Embodiment 1:

[0077] As Figure 1 shown, a method for upgrading the version of a wireless access point based on reinforcement learning includes the following steps:

[0078] Consider a wireless local area network scenario where N APs are connected to the same AC. Let represent the set of APs. The AP sends a version upgrade request to the AC. When the AC receives a certain number of requests, it sorts the upgrade order of the APs to be upgraded; there is a set variable initialized to be empty at the AC. At the same time, a counting variable Count and an accumulated counting variable Total are set, and their initial values are both 0; whenever the AC receives an upgrade request from an AP, both the Count value and the Total value are incremented by 1, and the current Count value is added to the set . When Count = M (M < N) or N - Total < M, the AC obtains a policy according to the reinforcement learning algorithm and sends the version upgrade file to the AP according to this policy. After the AP completes the version upgrade, it restarts to complete this upgrade, the variable is reset to be empty, Count is reset to 0, and the Total value is not reset.

[0079] Embodiment 2:

[0080] As Figure 2 shown, a system for upgrading the version of a wireless access point based on reinforcement learning includes:

[0081] A parameter initialization module for initializing parameters with one key;

[0082] A version upgrade file stored in the memory and can be called during the upgrade process;

[0083] A channel bandwidth allocation module for allocating bandwidth according to the actual situation between each AC and AP;

[0084] An AP upgrade completion time management module for allocating different resources according to the AP upgrade completion time;

[0085] A central processing module that can use the system resources of the server and is used to overall control the parameter initialization module, the version upgrade file, the channel bandwidth allocation module, and the AP upgrade completion time management module.

[0086] For AP version upgrade files, the version upgrade file of AP i is represented as S i ={D i ,F i}, where D i Indicates the file size, F i Indicates the file version number;

[0087] The channel bandwidth allocation module allocates channel bandwidth between AC and AP. Assuming that the total communication bandwidth between AC and N APs is B, the bandwidth allocated between AC and AP i during the upgrade process is expressed as B i , we can get:

[0088]

[0089] That is, each time the AC sends a version upgrade file to a group of APs, the total communication bandwidth between the AC and the APs to be upgraded cannot exceed the total bandwidth B;

[0090] The AP upgrade completion time management module considers a group of APs to be upgraded. The transmission rate between AC and AP i (i∈K) can be expressed as:

[0091] r i =B i log2(1+SNR i ). (2)

[0092] Among them, B i is the allocated channel bandwidth, SNR i is the signal-to-noise ratio of the signal transmitted between AC and API; and the version upgrade time for API can be expressed as:

[0093]

[0094] For the AP group to be upgraded The upgrade completion time of this group depends on the slowest AP. The upgrade time of this group can be obtained as:

[0095] T K =max{T i tr},i∈K. (4)

[0096] During the AP upgrade process, in order to minimize the overall upgrade time, the upgrade time of each group of APs to be upgraded must be minimized. Therefore, it is necessary to select the group of APs to be upgraded and allocate bandwidth reasonably. This leads to the following optimization problem: for each group of APs to be upgraded, minimize the upgrade completion time:

[0097]

[0098] Where C={C1,C2,...,C M} is the upgrade selection strategy, the elements are "0-1" variables, "1" means to upgrade, "0" means not to upgrade, B={B1,B2,...,B M} is the bandwidth allocation strategy, which allocates bandwidth to the APs being upgraded; since the complexity of the traditional solution to the above optimization problem increases sharply with the increase in the number of APs, it cannot meet the requirements of real-time decision-making; therefore, based on the reinforcement learning method, this system can continuously learn to obtain the optimal bandwidth allocation strategy in the dynamic interaction with the environment, and can allocate bandwidth in real time to minimize the overall upgrade time;

[0099] For the above optimization problem, AP upgrade selection and bandwidth allocation strategy are first transformed into MDP problem, Markov decision process (MDP), and then solved by reinforcement learning method; and the MDP problem is defined by tuple {S, A, T, R}, where S is the system state set, A is the action set, T = {p(s'|s, a)} is the transition probability set, and R: S × A → R is the real-valued reward function;

[0100] The MDP problem is defined as follows:

[0101] (1) Status: When count = M, the current status is defined as the set of AP version upgrade file size and channel status information.

[0102] s={D,H,L}. (6)

[0103] Where D = {D1, D2, ..., D M} is the version upgrade file size information, H = {SNR1, SNR2, ..., SNR M} is the signal-to-noise ratio of the signal transmission between AC and AP, L={l1,l2,...,l M} is the current available memory status information of AP;

[0104] (2) Action: When count = M, the action is defined as the selection of each AP to be upgraded in the group and the allocated bandwidth, as follows:

[0105] a={B1,B2,...,B M ,C1,C2,...,C M}. (7)

[0106] (3) Reward: To minimize the overall upgrade time, the reward function is defined using the upgrade completion time of each group of APs as follows:

[0107] r=f(TK ). (8)

[0108] Among them, f(·) is a negative exponential function that plays a normalization role. The longer the upgrade completion time is, the smaller the reward function is;

[0109] This technology uses the DDPG algorithm in the reinforcement learning algorithm to deal with the above MDP problem. AC is a reinforcement learning agent. It updates the network model parameters through DDPG algorithm training and learns the optimal strategy.

[0110] The DDPG algorithm has Actor and Critic networks, Actor network output strategy, and Critic network evaluation strategy, as shown below:

[0111] The Actor network consists of two networks, one of which is an online actor network, whose parameters are θ μ Indicates that the other is the targetactor network, whose parameters are θ μ' express;

[0112] The deterministic policy output by the Actor network is represented by μ(s|θ μ ) indicates that the strategy is determined by the parameter θ μ Sure;

[0113] The Critic network is similar to the Actor network in structure and is also composed of an onlineQ network and a targetQ network. Its network parameters are θ Q and θ Q' express;

[0114] Critic network evaluation strategy, set the onlineQ network as the Q function used for evaluation, defined as Q(s,a|θ Q ) is used to indicate;

[0115] For the training process of the DDPG algorithm, B is used to represent a batch of sample data randomly selected from the experience pool, where the kth sample data can be represented by {s k ,a k ,r k ,s k+1}, the policy gradient is expressed as:

[0116]

[0117] Among them, Q(s,a) represents the Q function value output by the Critic network;

[0118] The onlineQ network obtains the Q estimate based on the sample data randomly drawn from the experience pool B. Given the sample data {s k ,a k,r k ,s k+1}, we can get the Q estimate Q(s k ,a k |θ Q ); and the Q target value can be expressed as:

[0119] y k =r k +γQ'[s k+1 μ'(s k+1 |θ μ' )|θ Q' ]. (10)

[0120] Among them, μ'(s k+1 |θ μ' ) is the output of the targetactor network, so the loss function for updating the onlineQ network can be obtained:

[0121]

[0122] During the network update process, for the online actor network, the network parameters are updated according to the policy gradient algorithm, which can be expressed as:

[0123]

[0124] Among them, ρ μ Represents the learning rate of the actor network update;

[0125] For the parameter update of the online Q network, we consider the method of minimizing the loss function and adopt the gradient descent method, which can be expressed as:

[0126]

[0127] Among them, ρ Q is the learning rate of the critic network;

[0128] For the corresponding target networks in the Actor network and the Critic network, the update process depends on the parameters of the online network. The target network is assigned values ​​using the soft update method, which ensures that the target network is updated in each iteration. It can be expressed as:

[0129]

[0130] Among them, τ∈(0,1) is the rate of soft update;

[0131] After the DDPG algorithm training is completed, the optimal strategy can be obtained by inputting the current state, as follows:

[0132] μ* (s) = arg max a Q * (s,a) (16)

[0133] Algorithm 1-DDPG algorithm

[0134]

[0135] In summary, during the AP version upgrade process, the bandwidth allocation between AC and AP and AP memory affect the upgrade time and whether the upgrade is successful, thus affecting the normal operation of the service. Therefore, the upgraded AP should be selected and bandwidth allocated reasonably to ensure the success rate of AP version upgrade, reduce the probability of AP upgrade failure, and minimize the overall upgrade time. This technology first models the AP upgrade process and forms an optimization problem with the minimum upgrade time to shorten the overall upgrade time. Due to the large number of access APs, the complexity of traditional algorithms is huge, and it is difficult to solve the optimization problem and quickly obtain the optimal strategy.

[0136] This technology takes into account bandwidth and AP memory factors, and proposes an AP version upgrade method based on reinforcement learning, which can provide real-time decision-making for AP upgrades. As a reinforcement learning agent, AC continuously interacts with the dynamic environment to learn and update network model parameters to achieve optimal upgrade selection and bandwidth allocation strategy learning. After the reinforcement learning model training is completed, AC can adaptively make upgrade decisions in real time based on current status information, improve the success rate of AP upgrades, reduce the overall upgrade time, and achieve fast AP upgrades.

[0137] The above is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention. The structures, devices and operating methods not specifically described and explained in the present invention shall be implemented according to the conventional means in the art unless otherwise specified and limited.

Claims

1. A wireless access point version upgrade method based on reinforcement learning, characterized in that: The following steps are involved: S1: Start initializing parameters. Assume that N APs are connected to the same AC. S2: AC receives the AP upgrade request and sets the collection variable K, the count variable Count and the cumulative count variable Total; S3: Determine whether the number of APs to be upgraded = M or the number of remaining APs < M, and jump to step S4; otherwise, jump to step S2; S4: The AC obtains a strategy based on the reinforcement learning algorithm and sends the version upgrade file to the AP based on the strategy. S5: AP upgrade selection and bandwidth allocation strategy; S6: It is determined that all APs have completed the upgrade, and the step ends; Otherwise, jump to step S2; The strategy in step S4 specifically includes: Consider a group of APs K to be upgraded, where the transmission rate between AC and APi(i∈K) is expressed as: r i =B i log2(1+SNR i ). (2) Among them, B i is the allocated channel bandwidth, that is, the bandwidth allocated between AC and AP i during the upgrade process is expressed as B i , SNR i is the signal-to-noise ratio of the signal transmitted between AC and APi; Then, the version upgrade time of APi is expressed as: The AP version upgrade file is represented by S i ={D i ,F i }, where D i Indicates the file size, F i Indicates the file version number; For AP group K to be upgraded, the upgrade completion time of this group depends on the slowest AP. The upgrade time of this group can be obtained as: T K =max{T i tr },i∈K. (4) During the AP upgrade process, in order to minimize the overall upgrade time, the upgrade time of each group of APs to be upgraded must be minimized. Therefore, the group of APs to be upgraded is selected and bandwidth is allocated reasonably. Therefore, for each group of APs to be upgraded, the relationship to minimize the upgrade completion time is as follows: Where B={B1,B2,...,B M } is the bandwidth allocation strategy, which allocates bandwidth to the AP being upgraded; The AP upgrade selection and bandwidth allocation strategy is first transformed into an MDP problem and then solved using the reinforcement learning method. The MDP problem is defined as a tuple {S, A, T, R}, where S is the system state set, A is the action set, T = {p(s'|s, a)} is the transition probability set, and R: S × A → R is a real-valued reward function. The MDP problem is defined as follows: M1, state: When count = M, the current state is defined as the set of AP version upgrade file size and channel state information, as follows: s={D,H,L}. (6) Where D = {D1, D2, ..., D M } is the version upgrade file size information, H = {SNR1, SNR2, ..., SNR M } is the signal-to-noise ratio of the signal transmission between AC and AP, L={l1,l2,...,l M } is the current available memory status information of AP; M2, Action: When count = M, the action is defined as the selection of each AP to be upgraded and the allocated bandwidth, as follows: a={B1,B2,...,B M ,C1,C2,...,C M }. (7) Where C={C1,C2,...,C M } is the upgrade selection strategy, the element is a "0-1" variable, "1" means to upgrade, "0" means not to upgrade; M3, Reward: To minimize the overall upgrade time, the reward function is defined using the upgrade completion time of each group of APs, as follows: r=f(T K ). (8) Among them, f(·) is a negative exponential function that plays a normalization role. The longer the upgrade completion time is, the smaller the reward function is.

2. The method for upgrading a wireless access point version based on reinforcement learning according to claim 1, characterized in that: In step S1, N={1, 2, ..., N} represents an AP set, and the AP sends a version upgrade request to the AC.

3. The method for upgrading a wireless access point version based on reinforcement learning according to claim 1, characterized in that: In step S2, K is initially empty, and the initial values ​​of the counting variable Count and the cumulative counting variable Total are both 0; when the AC receives an upgrade request from the AP, the Count value and the Total value are both incremented by 1 and the Count value at this time is added to the set K.

4. The method for upgrading a wireless access point version based on reinforcement learning according to claim 1, characterized in that: After the restart in step S4 is completed, the K variable is reset to empty, Count is reset to 0, and the Total value is not reset.

5. A wireless access point version upgrade system based on reinforcement learning, executing the wireless access point version upgrade method based on reinforcement learning as claimed in any one of claims 1 to 4, characterized in that: It includes parameter initialization module, version upgrade file, central processing module, channel bandwidth allocation module and AP upgrade completion time management module.

6. The wireless access point version upgrade system based on reinforcement learning according to claim 5, characterized in that: The channel bandwidth allocation module is for channel bandwidth allocation between AC and AP. Assuming that the total communication bandwidth between AC and N APs is B, it can be obtained: That is, each time the AC sends a version upgrade file to a group of APs, the total communication bandwidth between the AC and the APs to be upgraded cannot exceed the total bandwidth B.

7. The wireless access point version upgrade system based on reinforcement learning according to claim 6, characterized in that: The MDP problem is processed by the DDPG algorithm in the reinforcement learning algorithm. AC is a reinforcement learning agent. The DDPG algorithm includes an Actor network, a Critic network, an Actor network output strategy, and a Critic network evaluation strategy, which are as follows: The Actor network consists of two networks, one of which is an online actor network, whose parameters are θ μ The other is the targetactor network, whose parameters are θ μ' express; The deterministic policy output by the Actor network is denoted by μ(sθ μ ) indicates that the strategy is determined by the parameter θ μ Sure; The critic network consists of an onlineQ network and a targetQ network, and its network parameters are θ Q and θ Q 'express; The critic network evaluation strategy sets the onlineQ network as the Q function used for evaluation, which is defined as Q(s, aθ Q ) is used to indicate.

8. The wireless access point version upgrade system based on reinforcement learning according to claim 7, characterized in that: In the training process of the DDPG algorithm, B represents a batch of sample data randomly drawn from the experience pool, where the kth sample data can be represented by {s k ,a k ,r k ,s k+1 }express.

Images