A collaborative charging method based on multi-agent reinforcement learning

Abstract

The present invention provides a collaborative charging method based on multi-agent reinforcement learning. The method supports a trajectory collection mechanism for asynchronous decision-making, so that ASM-PPO can collect more concise and useful information in asynchronous decision-making environments such as wireless rechargeable sensor networks, thereby accelerating the training speed of the algorithm. The present invention processes global information through a feature extraction module composed of a special-shaped convolution kernel and a GRU unit, so that ASM-PPO can keep the number of parameters that need to be learned unchanged when the number of charging vehicles increases, thereby increasing the stability of the algorithm.

Description

Technical Field

[0001] The present invention relates to the technical field of wireless rechargeable sensor networks and multi-agent deep reinforcement learning, and more specifically, to a collaborative charging method based on multi-agent reinforcement learning. Background Art

[0002] Wireless sensor networks refer to the deployment of a large number of sensor nodes in the environment to perform data collection and processing tasks. In recent years, wireless sensor networks have developed rapidly and have been widely used in many fields such as environmental surveys and building status monitoring. However, due to the limitations of sensor node battery capacity, wireless sensor networks are difficult to meet large-scale long-term monitoring tasks. With the increasing maturity of wireless charging technology, wireless rechargeable sensor networks have greatly extended the life cycle of sensor nodes.

[0003] The wireless rechargeable sensor network adds multiple mobile wireless charging devices to the wireless sensor network. Through the scheduling of the charging algorithm, the charging vehicle can move to the location of the sensor node with low power to replenish power to extend the life cycle of the sensor node. However, since the power consumption rate of different nodes will change dynamically, how to make the charging vehicle shorten its own moving distance and improve charging efficiency while ensuring the survival of the node, and transmit more power to the network has become an important issue in the wireless rechargeable sensor network.

[0004] Multi-agent deep reinforcement learning has great potential in solving sequential decision-making problems involving multiple agents. However, most multi-agent deep reinforcement learning algorithms can only be applied to scenarios where multiple agents make synchronous decisions. In wireless rechargeable sensor networks, due to factors such as different charging times and moving distances of nodes corresponding to different charging vehicles, each charging vehicle cannot complete its own charging task at the same time and decide on the next charging node. That is, there is asynchronous decision-making between charging vehicles, and multi-agent deep reinforcement learning algorithms cannot be used directly.

[0005] In order to solve the problem of coordinated charging in wireless rechargeable sensor networks and the problem of asynchronous decision-making of charging vehicles, we proposed a multi-agent deep reinforcement learning algorithm ASM-PPO that supports asynchronous decision-making. Unlike other multi-agent deep reinforcement learning algorithms that need to synchronously collect information from all agents, each agent in ASM-PPO only needs to asynchronously collect the required information according to its own time step. At the same time, in order to make the algorithm scalable and keep the parameters that need to be learned unchanged when the number of agents increases, ASM-PPO uses a special feature extraction module to process global information. This feature extraction module contains special-shaped convolution kernels and GRU units. Summary of the invention

[0006] The present invention provides a collaborative charging method based on multi-agent reinforcement learning, which accelerates the training speed and stability of the algorithm.

[0007] In order to achieve the above technical effects, the technical solution of the present invention is as follows:

[0008] A collaborative charging method based on multi-agent reinforcement learning comprises the following steps:

[0009] S1: Establish a wireless rechargeable sensor network simulation environment;

[0010] S2: Build a scalable multi-agent deep reinforcement learning model ASM-PPO;

[0011] S3: Collect data using a trajectory collection mechanism that supports asynchronous decision making in a simulation environment and train ASM-PPO;

[0012] S4: Use the trained ASM-PPO model for collaborative charging in a real environment.

[0013] Furthermore, in step S1, the wireless rechargeable sensor network simulation environment includes:

[0014] A fixed base station located in the center of the target area Ω is used to centrally process the data transmitted by the sensor nodes and to replenish the power for the charging vehicle;

[0015] N fixed-position sensor nodes are randomly deployed in the target area Ω, and the battery capacity of the sensor node is E s ; The power of sensor nodes is mainly used for environmental perception, data transmission and reception, so the energy consumption of sensor nodes will change with the occurrence of random events around the nodes and the change of the amount of transmitted data; when the power of the sensor node is lower than the preset threshold, it will send a charging request to the base station, and the base station will broadcast it to all charging vehicles after receiving the charging request;

[0016] There are M mobile charging vehicles waiting to be dispatched at the base station, and the battery capacity of the charging vehicle is E. c The charging vehicle can move between any node and base station. When the charging vehicle is in an idle state, it will select a request in its own request queue, move to the location of the request, and replenish the power of the node. After the node is fully charged, the charging vehicle will re-enter the idle state; when the power of the charging vehicle is lower than the preset threshold, the charging vehicle will return to the base station to replenish the power; the content of the charging request sent by the sensor node to the base station includes: node ID, location, remaining power, current energy consumption rate, and current timestamp.

[0017] Furthermore, in step S2, the scalable multi-agent deep reinforcement learning model ASM-PPO includes two neural networks:

[0018] Actor network with parameter φ, whose input is the observation of the charging vehicle itself and output is the probability distribution of the charging requests available in the request queue;

[0019] With parameters The input of the Critic network is the observation value of all charging vehicles, that is, the global information. The input needs to first pass through a scalable feature extraction module, which is composed of a special-shaped convolution kernel and a GRU unit. The output of the feature extraction module is input into the Critic network. The Critic network outputs an evaluation of the strategy, which is used to update the Actor network.

[0020] Furthermore, in step S3, the trajectory collection mechanism supporting asynchronous decision-making specifically refers to each charging vehicle asynchronously collecting its own trajectory. When the charging vehicle is in an idle state, it collects its own local observation information, global state information, execution action, reward information and other information needed for the training model, wherein the global state information refers to the local observation values ​​of all charging vehicles, including charging vehicles that are in an idle state and a busy state at the current moment.

[0021] Furthermore, the specific steps of the trajectory collection mechanism supporting asynchronous decision-making are:

[0022] For a specific charging vehicle, the charging vehicle only needs to asynchronously collect its own trajectory according to its own time step; when the charging vehicle is in an idle state, it collects its own local observation information, global state information, execution action, reward information, etc. required for the training model, where the global state information refers to the local observation values ​​of all charging vehicles, including those in the idle and busy states at the current moment;

[0023] After collecting training information for a time step, the charging vehicle stores the information in a cache pool shared by all charging vehicles, and inputs its own observation value into the Actor network to obtain the corresponding decision;

[0024] Repeat the above steps until the charging cycle ends, that is, the charging vehicle is too low to continue the charging task and returns to the base station to replenish power; after all charging vehicles have completed the charging cycle, samples are extracted from the shared cache pool to train the Actor network and Critic network shared by the charging vehicles.

[0025] Furthermore, the training process of the ASM-PPO model in step S3 is:

[0026] S31: randomly redeploy the sensor nodes in the target area;

[0027] S32: using a trajectory collection mechanism that supports asynchronous decision-making to asynchronously collect trajectory information of each charging vehicle from the time it departs from the base station to the time it returns to the base station when the battery level is too low;

[0028] S33: Update the Actor network by obtaining evaluation information of the trajectory from the Critic network according to the trajectory information, and use the evaluation information and the actual evaluation including the current reward information as the supervision information of the Critic network to update the Critic network.

[0029] Furthermore, the Actor network with parameter φ first inputs the observation value of the charging vehicle itself into the feature extraction module. The observation value includes: the location and remaining power of the charging vehicle, the N nearest to the charging vehicle, and the near The feature extraction module is composed of a special-shaped convolution kernel and a GRU, in which the special-shaped convolution kernel is a two-layer fully connected layer. After the feature extraction module proposes the corresponding features, it is input into the Actor network. The Actor network is composed of two layers of fully connected layers, and the output is the probability distribution of the optional charging requests in the request queue.

[0030] Furthermore, with parameters The Critic network first inputs the observation values ​​of all charging vehicles, that is, the global information, into the feature extraction module. The feature extraction module has the same structure as the feature extraction module in the Actor network. The observation values ​​of different charging vehicles are sequentially input into the same feature extraction module. After the feature extraction module proposes the features of the global information, it is input into the Critic network. The Critic network consists of two fully connected layers; the Critic network outputs an evaluation of the strategy to update the Actor network.

[0031] Furthermore, the specific training steps of the multi-agent deep reinforcement learning model ASM-PPO are:

[0032] After all the trajectories of all charging vehicles in a charging cycle are collected using the trajectory collection mechanism, the Actor network and the Critic network are updated;

[0033] The loss function of the Actor network during the update process is:

[0034]

[0035] Among them, r t (φ) represents the ratio of the updated strategy to the pre-updated strategy, that is:

[0036]

[0037] represents the advantage function, which can be obtained by subtracting the V value obtained by the Critic network from the reward. clip(·) is the truncation function, which means r t (φ) is intercepted to the range of 1-∈ to 1+∈;

[0038] The loss function of the Critic network during the update process is:

[0039]

[0040] in, Indicates receiving a reward.

[0041] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0042] The present invention can support the trajectory collection mechanism of asynchronous decision-making, so that ASM-PPO can collect more concise and useful information in asynchronous decision-making environments such as wireless rechargeable sensor networks, thereby accelerating the training speed of the algorithm; the present invention processes global information through a feature extraction module composed of a special-shaped convolution kernel and a GRU unit, so that ASM-PPO can keep the number of parameters that need to be learned unchanged when the number of charging vehicles increases, thereby increasing the stability of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of the trajectory collection mechanism of the present invention;

[0044] Figure 2 This is a training flow chart of the ASM-PPO model of the present invention. DETAILED DESCRIPTION

[0045] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0046] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0047] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0048] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0049] Example 1

[0050] A collaborative charging method based on multi-agent reinforcement learning includes the following steps:

[0051] S1: Establish a wireless rechargeable sensor network simulation environment;

[0052] S2: Build a scalable multi-agent deep reinforcement learning model (ASM-PPO);

[0053] S3: Collect data using a trajectory collection mechanism that supports asynchronous decision making in a simulation environment and train ASM-PPO;

[0054] S4: Use the trained ASM-PPO model for collaborative charging in a real environment.

[0055] Step S1: the wireless rechargeable sensor network simulation environment includes:

[0056] A fixed base station located in the center of the target area Ω is used to centrally process the data transmitted by the sensor nodes and to replenish the power for the charging vehicle;

[0057] N fixed-position sensor nodes are randomly deployed in the target area Ω, and the battery capacity of the sensor node is E s ; The power of sensor nodes is mainly used for environmental perception, data transmission and reception, so the energy consumption of sensor nodes will change with the occurrence of random events around the nodes and the change of the amount of transmitted data; when the power of the sensor node is lower than the preset threshold, it will send a charging request to the base station. The content of the request includes: node ID, location, remaining power, current energy consumption rate, and current timestamp. After receiving the charging request, the base station will broadcast it to all charging vehicles;

[0058] There are M mobile charging vehicles waiting to be dispatched at the base station, and the battery capacity of the charging vehicle is E. c The charging vehicle can move between any node and base station. When the charging vehicle is in an idle state, it will select a request in its own request queue, move to the location of the request, and replenish the power of the node. When the node is fully charged, the charging vehicle will re-enter the idle state; when the charging vehicle's power is lower than the preset threshold, the charging vehicle will return to the base station to replenish power;

[0059] The scalable multi-agent deep reinforcement learning model ASM-PPO in step S2 consists of two neural networks:

[0060] The first is an Actor network with parameter φ, whose input is the observation value of the charging vehicle itself and whose output is the probability distribution of the optional charging requests in the request queue;

[0061] The second is with parameters The input of the Critic network is the observation value of all charging vehicles, that is, the global information. The input needs to first pass through a scalable feature extraction module, which is composed of a special-shaped convolution kernel and a GRU unit. The output of the feature extraction module is input into the Critic network. The Critic network outputs an evaluation of the strategy, which is used to update the Actor network.

[0062] The trajectory collection mechanism supporting asynchronous decision-making in step S3 specifically refers to each charging vehicle asynchronously collecting its own trajectory. When the charging vehicle is in an idle state, it collects its own local observation information, global state information, execution action, reward information, etc. required for the training model, where the global state information refers to the local observation values ​​of all charging vehicles, including charging vehicles that are in an idle state and a busy state at the current moment; the training process of the ASM-PPO model in step S3 is:

[0063] S31: randomly redeploy the sensor nodes in the target area;

[0064] S32: using a trajectory collection mechanism that supports asynchronous decision-making to asynchronously collect trajectory information of each charging vehicle from the time it departs from the base station to the time it returns to the base station when the battery level is too low;

[0065] S33: Update the Actor network by obtaining evaluation information of the trajectory from the Critic network according to the trajectory information, and use the evaluation information and the actual evaluation including the current reward information as the supervision information of the Critic network to update the Critic network.

[0066] Example 2

[0067] A collaborative charging method based on multi-agent reinforcement learning comprises the following steps:

[0068] S1: Establish a wireless rechargeable sensor network simulation environment;

[0069] S2: Build a scalable multi-agent deep reinforcement learning model ASM-PPO;

[0070] S3: Collect data using a trajectory collection mechanism that supports asynchronous decision making in a simulation environment and train ASM-PPO;

[0071] S4: Use the trained ASM-PPO model for collaborative charging in a real environment.

[0072] In step S1, the wireless rechargeable sensor network simulation environment includes:

[0073] A fixed base station located in the center of the target area Ω is used to centrally process the data transmitted by the sensor nodes and to replenish the power for the charging vehicle;

[0074] N fixed-position sensor nodes are randomly deployed in the target area Ω, and the battery capacity of the sensor node is E s; The power of sensor nodes is mainly used for environmental perception, data transmission and reception, so the energy consumption of sensor nodes will change with the occurrence of random events around the nodes and the change of the amount of transmitted data; when the power of the sensor node is lower than the preset threshold, it will send a charging request to the base station, and the base station will broadcast it to all charging vehicles after receiving the charging request;

[0075] There are M mobile charging vehicles waiting to be dispatched at the base station, and the battery capacity of the charging vehicle is E. c The charging vehicle can move between any node and base station. When the charging vehicle is in an idle state, it will select a request in its own request queue, move to the location of the request, and replenish the power of the node. After the node is fully charged, the charging vehicle will re-enter the idle state; when the power of the charging vehicle is lower than the preset threshold, the charging vehicle will return to the base station to replenish the power; the content of the charging request sent by the sensor node to the base station includes: node ID, location, remaining power, current energy consumption rate, and current timestamp.

[0076] In step S2, the scalable multi-agent deep reinforcement learning model ASM-PPO contains two neural networks:

[0077] Actor network with parameter φ, whose input is the observation of the charging vehicle itself and output is the probability distribution of the charging requests available in the request queue;

[0078] With parameters The input of the Critic network is the observation value of all charging vehicles, that is, the global information. The input needs to first pass through a scalable feature extraction module, which is composed of a special-shaped convolution kernel and a GRU unit. The output of the feature extraction module is input into the Critic network. The Critic network outputs an evaluation of the strategy, which is used to update the Actor network.

[0079] In step S3, the trajectory collection mechanism supporting asynchronous decision-making specifically refers to each charging vehicle asynchronously collecting its own trajectory. When the charging vehicle is in an idle state, it collects its own local observation information, global state information, execution action, reward information, etc. required for the training model. The global state information refers to the local observation values ​​of all charging vehicles, including charging vehicles that are idle and busy at the current moment.

[0080] like Figure 1 As shown in Figure 2, the specific steps of the trajectory collection mechanism supporting asynchronous decision-making are:

[0081] For a specific charging vehicle, the charging vehicle only needs to asynchronously collect its own trajectory according to its own time step; when the charging vehicle is in an idle state, it collects its own local observation information, global state information, execution action, reward information, etc. required for the training model, where the global state information refers to the local observation values ​​of all charging vehicles, including those in the idle and busy states at the current moment;

[0082] After collecting training information for a time step, the charging vehicle stores the information in a cache pool shared by all charging vehicles, and inputs its own observation value into the Actor network to obtain the corresponding decision;

[0083] Repeat the above steps until the charging cycle ends, that is, the charging vehicle is too low to continue the charging task and returns to the base station to replenish power; after all charging vehicles have completed the charging cycle, samples are extracted from the shared cache pool to train the Actor network and Critic network shared by the charging vehicles.

[0084] like Figure 2 As shown, the training process of the ASM-PPO model in step S3 is:

[0085] S31: randomly redeploy the sensor nodes in the target area;

[0086] S32: using a trajectory collection mechanism that supports asynchronous decision-making to asynchronously collect trajectory information of each charging vehicle from the time it departs from the base station to the time it returns to the base station when the battery level is too low;

[0087] S33: Update the Actor network by obtaining evaluation information of the trajectory from the Critic network according to the trajectory information, and use the evaluation information and the actual evaluation including the current reward information as the supervision information of the Critic network to update the Critic network.

[0088] The Actor network with parameter φ first inputs the observation value of the charging vehicle into the feature extraction module. The observation value includes: the location and remaining power of the charging vehicle, the N nearest charging vehicles, and the near The feature extraction module is composed of a special-shaped convolution kernel and a GRU, in which the special-shaped convolution kernel is a two-layer fully connected layer. After the feature extraction module proposes the corresponding features, it is input into the Actor network. The Actor network is composed of two layers of fully connected layers, and the output is the probability distribution of the optional charging requests in the request queue.

[0089] With parameters The Critic network first inputs the observation values ​​of all charging vehicles, that is, the global information, into the feature extraction module. The feature extraction module has the same structure as the feature extraction module in the Actor network. The observation values ​​of different charging vehicles are sequentially input into the same feature extraction module. After the feature extraction module proposes the features of the global information, it is input into the Critic network. The Critic network consists of two fully connected layers; the Critic network outputs an evaluation of the strategy to update the Actor network.

[0090] Example 3

[0091] A collaborative charging method based on multi-agent reinforcement learning comprises the following steps:

[0092] S1: Establish a wireless rechargeable sensor network simulation environment;

[0093] S2: Build a scalable multi-agent deep reinforcement learning model ASM-PPO;

[0094] S3: Collect data using a trajectory collection mechanism that supports asynchronous decision making in a simulation environment and train ASM-PPO;

[0095] S4: Use the trained ASM-PPO model for collaborative charging in a real environment.

[0096] In step S1, the wireless rechargeable sensor network simulation environment includes:

[0097] A fixed base station located in the center of the target area Ω is used to centrally process the data transmitted by the sensor nodes and to replenish the power for the charging vehicle;

[0098] N fixed-position sensor nodes are randomly deployed in the target area Ω, and the battery capacity of the sensor node is E s ; The power of sensor nodes is mainly used for environmental perception, data transmission and reception, so the energy consumption of sensor nodes will change with the occurrence of random events around the nodes and the change of the amount of transmitted data; when the power of the sensor node is lower than the preset threshold, it will send a charging request to the base station, and the base station will broadcast it to all charging vehicles after receiving the charging request;

[0099] There are M mobile charging vehicles waiting to be dispatched at the base station, and the battery capacity of the charging vehicle is E. cThe charging vehicle can move between any node and base station. When the charging vehicle is in an idle state, it will select a request in its own request queue, move to the location of the request, and replenish the power of the node. After the node is fully charged, the charging vehicle will re-enter the idle state; when the power of the charging vehicle is lower than the preset threshold, the charging vehicle will return to the base station to replenish the power; the content of the charging request sent by the sensor node to the base station includes: node ID, location, remaining power, current energy consumption rate, and current timestamp.

[0100] In step S2, the scalable multi-agent deep reinforcement learning model ASM-PPO contains two neural networks:

[0101] Actor network with parameter φ, whose input is the observation of the charging vehicle itself and output is the probability distribution of the charging requests available in the request queue;

[0102] With parameters The input of the Critic network is the observation value of all charging vehicles, that is, the global information. The input needs to first pass through a scalable feature extraction module, which is composed of a special-shaped convolution kernel and a GRU unit. The output of the feature extraction module is input into the Critic network. The Critic network outputs an evaluation of the strategy, which is used to update the Actor network.

[0103] In step S3, the trajectory collection mechanism supporting asynchronous decision-making specifically refers to each charging vehicle asynchronously collecting its own trajectory. When the charging vehicle is in an idle state, it collects its own local observation information, global state information, execution action, reward information, etc. required for the training model. The global state information refers to the local observation values ​​of all charging vehicles, including charging vehicles that are idle and busy at the current moment.

[0104] like Figure 1 As shown in Figure 2, the specific steps of the trajectory collection mechanism supporting asynchronous decision-making are:

[0105] For a specific charging vehicle, the charging vehicle only needs to asynchronously collect its own trajectory according to its own time step; when the charging vehicle is in an idle state, it collects its own local observation information, global state information, execution action, reward information, etc. required for the training model, where the global state information refers to the local observation values ​​of all charging vehicles, including those in the idle and busy states at the current moment;

[0106] After collecting training information for a time step, the charging vehicle stores the information in a cache pool shared by all charging vehicles, and inputs its own observation value into the Actor network to obtain the corresponding decision;

[0107] Repeat the above steps until the charging cycle ends, that is, the charging vehicle is too low to continue the charging task and returns to the base station to replenish power; after all charging vehicles have completed the charging cycle, samples are extracted from the shared cache pool to train the Actor network and Critic network shared by the charging vehicles.

[0108] like Figure 2 As shown, the training process of the ASM-PPO model in step S3 is:

[0109] S31: randomly redeploy the sensor nodes in the target area;

[0110] S32: using a trajectory collection mechanism that supports asynchronous decision-making to asynchronously collect trajectory information of each charging vehicle from the time it departs from the base station to the time it returns to the base station when the battery level is too low;

[0111] S33: Update the Actor network by obtaining evaluation information of the trajectory from the Critic network according to the trajectory information, and use the evaluation information and the actual evaluation including the current reward information as the supervision information of the Critic network to update the Critic network.

[0112] The Actor network with parameter φ first inputs the observation value of the charging vehicle into the feature extraction module. The observation value includes: the location and remaining power of the charging vehicle, the N nearest charging vehicles, and the near The feature extraction module is composed of a special-shaped convolution kernel and a GRU, in which the special-shaped convolution kernel is a two-layer fully connected layer. After the feature extraction module proposes the corresponding features, it is input into the Actor network. The Actor network is composed of two layers of fully connected layers, and the output is the probability distribution of the optional charging requests in the request queue.

[0113] With parameters The Critic network first inputs the observation values ​​of all charging vehicles, that is, the global information, into the feature extraction module. The feature extraction module has the same structure as the feature extraction module in the Actor network. The observation values ​​of different charging vehicles are sequentially input into the same feature extraction module. After the feature extraction module proposes the features of the global information, it is input into the Critic network. The Critic network consists of two fully connected layers; the Critic network outputs an evaluation of the strategy to update the Actor network.

[0114] The specific training steps of the multi-agent deep reinforcement learning model ASM-PPO are:

[0115] After all the trajectories of all charging vehicles in a charging cycle are collected using the trajectory collection mechanism, the Actor network and the Critic network are updated;

[0116] The loss function of the Actor network during the update process is:

[0117]

[0118] Among them, r t (φ) represents the ratio of the updated strategy to the pre-updated strategy, that is:

[0119]

[0120] represents the advantage function, which can be obtained by subtracting the V value obtained by the Critic network from the reward. clip(·) is the truncation function, which means r t (φ) is intercepted to the range of 1-∈ to 1+∈;

[0121] The loss function of the Critic network during the update process is:

[0122]

[0123] in, Indicates receiving a reward.

[0124] The present invention enables multiple charging vehicles in a wireless rechargeable sensor network to work together efficiently when the battery capacity of the charging vehicle and the node is limited, replenishing the power of sensor nodes with low power in a timely manner and extending the life cycle of the sensor network, while reducing the moving distance of the charging vehicle and improving the charging efficiency of the charging vehicle.

[0125] The same or similar reference numerals correspond to the same or similar components;

[0126] The positional relationships described in the drawings are only for illustrative purposes and should not be construed as limiting the present patent.

[0127] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A collaborative charging method based on multi-agent reinforcement learning, characterized in that: The following steps are involved: S1: Establish a wireless rechargeable sensor network simulation environment; In step S1, the wireless rechargeable sensor network simulation environment includes: A fixed base station located in the center of the target area Ω is used to centrally process the data transmitted by the sensor nodes and to replenish the power for the charging vehicle; N fixed-position sensor nodes are randomly deployed in the target area Ω, and the battery capacity of the sensor node is E s ; The power of sensor nodes is mainly used for environmental perception, data transmission and reception, so the energy consumption of sensor nodes will change with the occurrence of random events around the nodes and the change of the amount of transmitted data; when the power of the sensor node is lower than the preset threshold, it will send a charging request to the base station, and the base station will broadcast it to all charging vehicles after receiving the charging request; There are M mobile charging vehicles waiting to be dispatched at the base station, and the battery capacity of the charging vehicle is E. c The charging vehicle can move between any node and base station. When the charging vehicle is in an idle state, it will select a request in its own request queue, move to the location of the request, and replenish the power of the node. After the node is fully charged, the charging vehicle will re-enter the idle state; when the charging vehicle's power is lower than the preset threshold, the charging vehicle will return to the base station to replenish the power; S2: Build a scalable multi-agent deep reinforcement learning model ASM-PPO; In step S2, the scalable multi-agent deep reinforcement learning model ASM-PPO includes two neural networks: Actor network with parameter φ, whose input is the observation of the charging vehicle itself and output is the probability distribution of the charging requests available in the request queue; With parameters The input of the Critic network is the observation value of all charging vehicles, that is, global information. The input needs to first pass through a scalable feature extraction module, which is composed of a special-shaped convolution kernel and a GRU unit. The output of the feature extraction module is input into the Critic network. The Critic network outputs an evaluation of the strategy, which is used to update the Actor network. S3: Collect data using a trajectory collection mechanism that supports asynchronous decision making in a simulation environment and train ASM-PPO; In step S3, the trajectory collection mechanism supporting asynchronous decision-making specifically refers to each charging vehicle asynchronously collecting its own trajectory. When the charging vehicle is in an idle state, it collects its own local observation information, global state information, execution action and reward information required for the training model, where the global state information refers to the local observation values ​​of all charging vehicles, including charging vehicles that are in an idle state and a busy state at the current moment; The specific steps of the trajectory collection mechanism that supports asynchronous decision-making are: For a specific charging vehicle, the charging vehicle only needs to asynchronously collect its own trajectory according to its own time step; when the charging vehicle is in an idle state, it collects its own local observation information, global state information, execution action and reward information required for the training model, where the global state information refers to the local observation values ​​of all charging vehicles, including those in the idle and busy states at the current moment; After collecting training information for a time step, the charging vehicle stores the information in a cache pool shared by all charging vehicles, and inputs its own observation value into the Actor network to obtain the corresponding decision; Repeat the above steps until the charging cycle ends, that is, the charging vehicle is too low to continue the charging task and returns to the base station to recharge. After all charging vehicles have completed the charging cycle, samples are extracted from the shared cache pool to train the Actor network and Critic network shared by the charging vehicles. The training process of the ASM-PPO model in step S3 is: S31: randomly redeploy the sensor nodes in the target area; S32: using a trajectory collection mechanism that supports asynchronous decision-making to asynchronously collect trajectory information of each charging vehicle from the time it departs from the base station to the time it returns to the base station when the battery level is too low; S33: Update the Actor network by obtaining evaluation information of the trajectory from the Critic network according to the trajectory information, and use the evaluation information and the actual evaluation including the current reward information as the supervision information of the Critic network to update the Critic network; S4: Use the trained ASM-PPO model for collaborative charging in a real environment.

2. The collaborative charging method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The content of the charging request sent by the sensor node to the base station includes: node ID, location, remaining power, current energy consumption rate, and current timestamp.

3. The collaborative charging method based on multi-agent reinforcement learning according to claim 2 is characterized in that: The Actor network with parameter φ first inputs the observation value of the charging vehicle into the feature extraction module. The observation value includes: the location and remaining power of the charging vehicle, the N nearest charging vehicles, and the near The feature extraction module is composed of a special-shaped convolution kernel and a GRU, in which the special-shaped convolution kernel is a two-layer fully connected layer. After the feature extraction module proposes the corresponding features, it is input into the Actor network. The Actor network is composed of two layers of fully connected layers, and the output is the probability distribution of the optional charging requests in the request queue.

4. The collaborative charging method based on multi-agent reinforcement learning according to claim 3 is characterized in that: With parameters The Critic network first inputs the observation values ​​of all charging vehicles, that is, the global information, into the feature extraction module. The feature extraction module has the same structure as the feature extraction module in the Actor network. The observation values ​​of different charging vehicles are sequentially input into the same feature extraction module. After the feature extraction module proposes the features of the global information, it is input into the Critic network. The Critic network consists of two fully connected layers; the Critic network outputs an evaluation of the strategy to update the Actor network.

5. The collaborative charging method based on multi-agent reinforcement learning according to claim 4 is characterized in that: The specific training steps of the multi-agent deep reinforcement learning model ASM-PPO are: After all the trajectories of all charging vehicles in a charging cycle are collected using the trajectory collection mechanism, the Actor network and the Critic network are updated; The loss function of the Actor network during the update process is: Among them, r t (φ) represents the ratio of the updated strategy to the pre-updated strategy, that is: represents the advantage function, which can be obtained by subtracting the V value obtained by the Critic network from the reward. clip(·) is the truncation function, which means r t (φ) is intercepted to the range of 1-∈ to 1+∈; The loss function of the Critic network during the update process is: in, Indicates receiving a reward.

Images