Mobile wireless charger mobile trajectory and charging combined scheduling method and trolley

Abstract

The present application aims at the problem that the rechargeable battery of the sensor node configuration of the wireless rechargeable sensor network in the prior art generally needs to replace the battery to maintain the energy supply, and provides a joint scheduling method of the mobile wireless charger mobile track and charging, and belongs to the technical field of control. The mobile wireless charger is moved in the wireless rechargeable sensor network, searches the sensor node in the network in the moving process, detects the residual power of the sensor node when the sensor node is in the chargeable range of the mobile wireless charger, and performs wireless charging on the sensor node when the residual power is not full. The present application can intelligently perceive the initial power of the battery of the sensor node, the distribution density and the energy consumption rate in the WRSN, and intelligently gives a collision-free safe charging scheduling strategy according to the node parameter change in the network under the condition that the information position of the sensor node and the obstacle is unknown.

Description

Technical Field

[0001] This invention belongs to the field of control technology, specifically relating to a method for jointly scheduling the movement trajectory and charging of a mobile wireless charger, and a trolley thereof. Background Technology

[0002] Wireless Rechargeable Sensor Networks (WSNs) for environmental monitoring and data acquisition consist of several rechargeable sensor nodes and one or more mobile wireless chargers powered by wireless charging technology. Rechargeable sensor nodes are typically equipped with rechargeable batteries with limited capacity. Traditional methods often assume precise location information of the sensor nodes in advance, requiring frequent battery replacements. Battery supply solutions need to be designed in advance and adapt to changes in the WSN network. Since the pioneering work on resonant coupling-based wireless power transfer (WPT), research on how WPT technology can help address the energy bottleneck in wireless sensor networks (WSNs) has increased. Sensor networks typically consist of sensors with limited battery capacity, and the cost of manually replacing batteries from time to time is high, especially when the sensor network is deployed in hazardous areas. While much literature focuses on applying energy harvesting (EH) technology to WSNs, enabling sensors to collect energy from the environment, the problems of low and fluctuating energy harvesting rates remain unresolved. To recharge sensor nodes in WSNs using WPT technology, static chargers can be deployed within the network, or mobile wireless chargers (MCs) can be scheduled to periodically access nodes with rechargeable batteries. This can extend network lifespan and even maintain sustainable network operation. Compared to static chargers, mobile wireless chargers offer advantages such as flexibility, low cost, and ease of maintenance. MCs provide an efficient way to extend the lifespan of WSN systems or other IoT devices. However, because MCs carry limited battery energy and can only successfully charge sensor nodes within a limited charging range, inappropriate MC movement trajectories can lead to low charging efficiency, long charging delays, and high energy costs. Furthermore, considering the complex environments typically used for deploying wireless sensor networks, MCs often face hazardous environmental factors during charging tasks, such as enemy or friendly fire points in battlefield surveillance and ground obstacles in dangerous terrain. These factors can be highly destructive to MCs; therefore, their movement trajectories must be carefully designed to avoid collisions with obstacles during charging tasks. Summary of the Invention

[0003] This invention addresses the problem that rechargeable batteries configured in sensor nodes of existing wireless rechargeable sensor networks often need to be replaced to maintain energy supply. It proposes a joint scheduling method for the movement trajectory and charging of a mobile wireless charger, as well as a vehicle.

[0004] The objective of this invention is achieved through the following technical solution:

[0005] A method for jointly scheduling the movement trajectory and charging of a mobile wireless charger includes an autonomously moving mobile wireless charger and a wireless rechargeable sensor network. The wireless rechargeable sensor network includes several rechargeable sensor nodes, and includes the following steps:

[0006] The first step is for a fully charged portable wireless charger to depart from the charging station;

[0007] The second step involves the mobile wireless charger moving within a wireless rechargeable sensor network. During this movement, it searches for sensor nodes in the network. When a sensor node is within the rechargeable range of the mobile wireless charger, it detects the remaining power of that sensor node. If the remaining power is not full, it wirelessly charges the sensor node.

[0008] The third step is to return the mobile wireless charger to the charging station to recharge after the rechargeable power is exhausted.

[0009] The fourth step involves the fully charged wireless charger restarting the first step, and this process is repeated sequentially.

[0010] The mobile wireless charger's movement trajectory in the network is set as a two-dimensional plane movement trajectory. The movement trajectory design and charging scheduling problem of the mobile wireless charger are established by a Markov decision model (MDP). Based on the established MDP, reinforcement learning (RL) technology is introduced to form a mobile safety policy intervention algorithm based on soft actor criticism. This decouples maximizing charging utility from minimizing collision risk and finds a safe movement trajectory with the maximum achievable charging utility.

[0011] By using reinforcement learning technology, mobile wireless chargers can learn through trial and error to jointly optimize their movement trajectory and charging plan by interacting with the environment and tracking feedback signals from sensor nodes and obstacles in real time during the charging process.

[0012] The obstacles include static obstacles and dynamic obstacles.

[0013] Preferably, the sensor node is equipped with a wirelessly rechargeable battery, and a mobile wireless charger wirelessly charges the battery of the sensor node.

[0014] Preferably, the sensor nodes are randomly distributed.

[0015] Preferably, the mobile wireless charger includes a GPS module and a battery management system.

[0016] Preferably, the mobile wireless charger includes a main battery and a backup battery. The main battery is used to charge sensor nodes in the network, and the backup battery is used for the mobile wireless charger to return to the charging station from any location in the network. The backup battery is activated after the main battery is depleted.

[0017] Preferably, the mobile wireless charger's movement trajectory starts at the charging station and ends when the main battery is depleted.

[0018] Preferably, the system status of the mobile wireless charger, including its real-time location and remaining battery power, is as follows: Where t is the equal-length time slot, Let be the coordinates of time slot t in the two-dimensional plane. The remaining power of the mobile wireless charger on its movement trajectory at time slot t;

[0019] The status information is The mobile wireless charger uses this information to decide on its next move and power allocation.

[0020] As a preferred approach, the Markov decision model (MDP) is defined as a quintuple. Where S and A represent the state space and action space of MC, P represents the state transition probability space, R represents the reward space, and γ represents the discount factor used to weigh short-term and long-term cumulative rewards; based on MDP, the mobile wireless charger improves its joint trajectory design and charging scheduling to maximize the positive feedback of the environment through continuous interaction with the environment; the mobile wireless charger is defined as a real RL agent, and the task agent, safety agent, and intervention agent are defined as virtual RL agents, each of the three virtual agents representing an independent training architecture based on the soft actor criticism algorithm;

[0021] The task agent, security agent, and intervention agent learn the following policies: task policy, security policy, and intervention policy, respectively. The task agent or security agent can directly control the mobile wireless charger through their respective learned policies. The intervention agent only determines when the security agent controls the mobile wireless charger, rather than directly controlling the mobile wireless charger.

[0022] As a preferred approach, the optimization objective of the task agent is to maximize charging utility, while the optimization objective of the security agent is to minimize the cost of safety violations. The task agent and the security agent are independent of each other. The mobile security policy intervention algorithm controls the adaptive game between the task agent and the security agent. Whenever the mobile wireless charger reaches a certain state, the two agents coordinate with each other to learn and predict future joint actions, minimizing the occurrence of unsafe states in future trajectories while maximizing the cumulative charging amount of the system.

[0023] A mobile wireless charging vehicle, the operation of which is controlled using any of the methods described above.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] The algorithm proposed in this invention can intelligently perceive the initial battery charge, distribution density, and energy consumption rate of sensor nodes in WRSN. Even when the locations of sensor nodes and obstacles are unknown, it intelligently provides a collision-free, safe charging scheduling strategy based on changes in node parameters within the network. Specifically, it can achieve the following objectives:

[0026] 1) Intelligent charging scheduling based on the initial power distribution in the sensor network. When all nodes have the same energy consumption, MC performs charging tasks in areas with lower initial power, replenishing the power of nodes with lower remaining energy in a timely manner, reducing the number of node failures in the entire WRSN;

[0027] 2) Intelligent charging scheduling based on the distribution density of nodes in the sensor network. Results show that the MC trajectory can move along a dense trajectory until the main battery it carries is depleted, at which point the charging task ends.

[0028] 3) Intelligent charging scheduling based on the energy consumption of sensor nodes in the sensor network. The unit energy consumption of sensor nodes is crucial to the duration of the entire WRSN operation, and the charging trajectory of MC always moves along the areas where sensor nodes have higher energy consumption. Attached Figure Description

[0029] Figure 1 This is a model of the random distribution of sensor nodes in the wireless rechargeable sensor network of the present invention on a two-dimensional plane.

[0030] Figure 2 The algorithm framework for SAC-MSPI;

[0031] Figure 3 The algorithm flow of this invention is as follows. Detailed Implementation

[0032] The present invention will be further described below with reference to the embodiments illustrated in the accompanying drawings:

[0033] like Figure 1As shown, sensor nodes are randomly distributed in the network. Due to node operations (such as sensing and communication), each node may randomly consume a certain amount of energy in each time slot. Therefore, the real-time remaining power of each node may change over time. To replenish the energy of the nodes to maintain their normal operation, MCs need to be dispatched to visit these nodes periodically. For a specific charging process, the MC starts from the charging station, moves through the network, and finally returns to the charging station. During the charging process, the MC keeps moving and uses omnidirectional WPT technology to charge the sensor nodes within its charging range for a certain period of time, that is, all nodes within the MC's charging range can be charged simultaneously. In addition, we assume that there are both static and moving obstacles in the network. The MC needs to avoid colliding with obstacles when moving in the network; otherwise, the MC may be damaged by obstacles, causing the charging task to fail. Therefore, this invention proposes a joint scheduling method for the movement trajectory and charging of a mobile wireless charger.

[0034] We define the WRSN as a two-dimensional rectangular region with multiple sensor nodes (nodes) S = {s1, s2, ..., s3}. N The nodes are randomly distributed, where N is the total number of nodes. We also assume that some static obstacles are randomly distributed within this area, and some moving obstacles may move randomly within this area. Each node s i From a block with a maximum capacity B s Powered by the same rechargeable battery. We assume that time is divided into equal-length time slots, with nodes s within each time slot. i It will consume a certain amount of energy. It is derived from a certain probability distribution The nodes are randomly, equally, and independently sampled, where i = 1, 2, ..., N. Mobile wireless chargers (MCs) are deployed to periodically access these nodes to charge them and maintain the normal operation of the WRSN. For example, in... Figure 1 In the system, the MC starts from a charging station, moves within the network, and provides wireless charging services to nodes, eventually returning to the charging station to recharge itself. The MC is equipped with a main battery and a backup battery. When the main battery runs out of power during charging, the MC uses the backup battery to return directly to the charging station. The energy stored in the backup battery is sufficient for the MC to return to the charging station from anywhere in the network.

[0035] The MC can move freely within a two-dimensional rectangular area. Since there are both static and moving obstacles in the 2D area, the MC needs to carefully avoid collisions. At the beginning of each time slot t, the MC determines its direction of movement within time slot t according to a specific control strategy π. movement speed ,in ∈[0,360◦). It can be divided into two components, namely the x-axis velocity. and y-axis velocity It satisfies the following constraint.

[0036]

[0037] Assuming the length of each time slot is one unit of time, the distance traveled per unit of time is... We denote δ as the movement cost per unit distance of MC (energy used for movement), and the movement cost of MC in time slot t is... However, the cost of moving through time slot t cannot exceed the remaining power of MC at the start of the current time slot. ,Right now Represent the position of MC at the start of time slot t as two-dimensional coordinates. The real-time position of MC is updated according to equation (2), that is...

[0038]

[0039] We define This represents the movement trajectory of the MC, starting from the charging station and ending at the location where the MC's main battery has just been depleted. (Definition) Let P be the total time (the total number of time slots t) consumed by the MC on its movement trajectory P. In this work, our goal is to find the optimal movement trajectory assuming that the MC cannot access the exact location information of each sensor node before charging. And an optimal charging plan to maximize the average effective charging rate. Furthermore, the MC cannot know the location of every static obstacle or the movement pattern of moving obstacles within the network before charging. These assumptions are to accommodate the realities of nodes deployed in complex, unstructured, and dynamic environments such as battlefields and disaster zones. Under these assumptions, the MC needs to learn to optimize its movement trajectory P through interaction with the environment during the charging process.

[0040] Because of the presence of both static and moving obstacles in the 2D area, the MC should carefully avoid collisions while moving. Otherwise, the MC is easily damaged, causing the charging task to fail. We assume that each obstacle is much larger than the MC, because even if the MC hits a small obstacle (such as a small stone or small animal), it may not cause serious damage. Based on this assumption, the size of the MC can be ignored, and we only consider the size of the obstacles. Specifically, we will... Represented as an obstacle in a two-dimensional region at time slot t The location of the center point This represents the maximum distance between the center point and any point on the surface of each obstacle. We assume that MC can detect a certain range using a ranging radar. Any obstacle within the area. In each time slot t, if the distance between MC and the center point of the obstacle is less than or equal to... ,Right now - If MC collides with the obstacle, then MC will collide with the obstacle. We define the cost of a safety violation collision of MC within time slot t as Equation (3), where α represents the scaling factor, which is a constant. - This indicates the collision intensity when a collision occurs. The cost of violating safety rules is zero when the MC does not collide with an obstacle. The safety violation costs defined in this work can benefit the charging task while preventing the MC from colliding with obstacles.

[0041]

[0042] We divide the task of MC in each time slot t into two phases. The first phase is for the movement of MC itself. After moving in the first phase of time slot t, MC reaches its new position. Then MC enters the second stage, adopting the wireless charging model (4) developed in the literature "L. Fu, P. Cheng, Y. Gu, J. Chen, and T. He, “Minimizing Charging Delay in Wireless Rechargeable Sensor Networks,” in Proc. of IEEE INFOCOM, Turin, Italy, Apr. 2013", to transfer energy to surrounding nodes for charging through omnidirectional WPT technology.

[0043]

[0044] in and d represents the charging power of the MC and the receiving power of the sensor node, respectively; d represents the distance between the mobile charger and the sensor node during charging; R represents the maximum charging coverage radius (i.e., radiation range) of the mobile wireless charger. This indicates that the sensor node is within the charging coverage area of ​​the MC. To adjust the Friis equation to accommodate short-distance charging, the remaining parameters defined in (4) are constants determined by the nodes and MC. It is the transmission channel gain of the charger; It is the sensor node's received channel gain; These are the hardware physical parameters. We will Let represent the i-th node of MC charging within time slot t, where i = 1, 2, ..., , This represents the total number of sensor nodes charged by MC within time slot t. This applies when MC decides not to charge any nodes or when no nodes are within MC's charging range in time slot t. In the second phase of time slot t, multiple nodes can charge simultaneously as long as they are within the charging range of the MC. At the end of the second phase, the MC collects a feedback signal containing information on the effective charge amount given to all sensor nodes during the second phase. This indicates the i-th sensor node within time slot t. The actual effective amount of electricity received when being charged by MC. Represents a node The current remaining power when not being charged, then the sensor node Effective charging amount within time slot t Represented as equation (5), that is, where This represents the energy consumed by the sensor node itself during sensing operations within the time slot t. The remaining battery power of the sensor node changes over time as shown in equation (6). That is, at the beginning of time slot t, the remaining battery power of the sensor is subtracted from the energy consumed for sensing to obtain the current remaining battery power of the sensor node within a unit time slot (minimum is 0, all consumed). This is then added to the energy received from MC. The total cannot exceed the maximum battery capacity of the sensor. The current battery level of the sensor is obtained before the start of the next time slot. When calculating the effective charge of a sensor node, it is only necessary to subtract the remaining energy at the beginning of the time slot. That's all.

[0045]

[0046]

[0047] Based on the energy change formula of the above sensor nodes, it can be seen that the effective charging amount of MC in each time slot t is expressed as: The change in the remaining battery power of MC in this time slot is expressed by equation (7), which is the current remaining power minus the energy consumed during movement. And the energy emitted to charge the sensor. .

[0048]

[0049] In summary, the dynamic changes of each element in WRSN can be seen as follows:

[0050] The sensor undergoes two main phases of change: energy is consumed while performing the sensing task, and it may be recharged if it is within the charging range of the MC.

[0051] The energy variation of a mobile wireless charger is mainly affected by two factors: its own energy consumption during mobility and the energy consumption for charging sensor nodes within its coverage area.

[0052] The objective of this invention is to maximize the charging efficiency of the MC, that is, in The total effective charging energy of MC within a given time period. Due to... It is variable, depending on the main battery capacity of the MC and how the MC consumes onboard energy for mobility and charging. Therefore, it is advisable to maximize the average effective charging rate of the MC, as defined below:

[0053]

[0054] Let represent the effective charging amount that MC provides to the i-th sensor node in time slot t. Therefore, the overall goal of this invention is to maximize the effective charging amount that MC provides to all sensor nodes throughout the entire charging journey, as expressed by equation (9). Equation (10) indicates that the total energy consumed by MC on trajectory P cannot exceed the maximum capacity of the battery. Equation (11) specifies that the moving distance of MC in each time slot cannot exceed the longest distance that can be supported by the unit maximum speed and remaining power. Equation (12) indicates that the charging transmission power in each time slot cannot exceed the current remaining power of the battery. In addition, MC should try to avoid collisions with static and moving obstacles in the network. Generally, we need to design the optimal movement trajectory and charging schedule based on energy consumption constraints, motion constraints, power allocation constraints, and safety constraints to maximize the average effective charging rate.

[0055]

[0056] Therefore, the specific implementation scheme of the present invention is as follows:

[0057] First, the joint trajectory design and charging scheduling problem is formulated as an MDP, where the MC makes decisions (i.e., movement and charging) based on its state at the beginning of each time slot t. Then, we detail how to develop a reinforcement learning-based algorithm to find the optimal policy, which maps the MC's current state to the best decision for each time slot. Specifically, at the beginning of each time slot t, the MC determines its future movement and charging power allocation based on its current location and the remaining charge of the main battery. Within the RL framework, the MC... The system sequentially makes decisions regarding the maximum average effective charging rate. It also considers the charging process. Safety issues during the process (i.e., collision avoidance). A Soft Actor-Critic based Mobile SecurityPolicy Intervened Algorithm (SAC-MSPI) was developed. This algorithm combines a distributed exploration-based safety training method with a SAC-based stochastic policy algorithm to jointly optimize the safe movement and charging scheduling trajectory design of mobile MCs.

[0058] MDP definition

[0059] We define an MDP as a quintuple. Here, S and A represent the state space and action space of MC, P represents the state transition probability space, R represents the reward space, and γ represents the discount factor used to weigh short-term and long-term cumulative rewards. Based on MDP, MC improves its joint trajectory design and charge scheduling to maximize the positive feedback (reward) of the environment through continuous interaction with it. Specifically, MC is defined as a real RL agent. Three virtual RL agents are also defined: a task agent, a safety agent, and an intervention agent. Each of these three virtual agents represents an independent SAC-based training architecture.

[0060] State: We define the system state at the start of time slot t as the real-time location of the MC and the remaining power, represented as follows: Because the MC has an embedded Global Positioning System (GPS) module and a battery management system that can continuously monitor the MC's real-time location and remaining battery power, the MC can fully access its own status information. It inherits all historical state information prior to t, that is The MC uses this information to determine its next movement and power allocation. The MC is fully charged before the charging task begins; that is, when t = 0, .

[0061] Action: We define action as the MC's decision regarding its own movement and charging power allocation within each time slot t, expressed as... Specifically, velocity components From respectively The charging power is selected independently within the range and remains constant within the time slot. It should be noted that the action of MC selection within each time slot must satisfy the constraints of equations (11) and (12) above.

[0062] Reward: Because we focus on optimizing a safe movement trajectory by maximizing the MC charging rate under finite energy constraints, i.e., the total Energy is used for MC's movement and to charge sensor nodes in the network until it is depleted. We define the reward function as follows: , representing the sum of the effective charging amounts provided by MC to the sensor nodes within time slot t, and the number of nodes. This represents the parameter for adjusting the weights. MC performs actions in each time slot t. Later received a reward Furthermore, we assume that the MC will incur a penalty when it collides with an obstacle during training, namely the safety violation cost defined in (3). Therefore, the goal of the developed training algorithm is to maximize the average cumulative reward while minimizing the average cumulative cost of the safety violation. It can be seen that the cost of violating safety is separate and not incorporated into the reward function. Because combining reward and cost as a single feedback signal may prevent the MC from exploring better charging locations near obstacles in order to reduce the risk of colliding with them. State transition probability: The state transition probability defines the system dynamics that depend only on the previous state and action, representing the probability that the random variable S and these values ​​will occur in time slot t given specific values ​​of the current state and action. Since the state space and action space are both continuous and infinite, the space of the state transition probability P is also continuous and infinite and cannot be explicitly modeled. Therefore, in the absence of an explicit state transition probability model, we adopt a model-free RL technique that enables the MC to learn to optimize joint trajectory design and charging scheduling through interaction with the environment.

[0063] SAC-MSPI framework

[0064] Because agents need to explore and learn unknown environments through trial and error to improve their movement and charging strategies, attempting dangerous actions can harm them. In this embodiment, the MC (Moving Agent) needs to consider not only maximizing charging utility but also the safety of movement within the network due to static and mobile obstacles. Therefore, maintaining the safety of the MC during charging tasks is extremely important, especially when deployed in unknown and complex environments. This invention proposes the SAC-MSPI algorithm, which combines a distributed exploration-based safe training method with a SAC-based stochastic policy algorithm to optimize joint trajectory design and charging scheduling. In fact, the proposed SAC-MSPI improves upon traditional safe RL frameworks without sacrificing charging utility for movement safety. Specifically, to ensure the agent's safety during environmental interactions, traditional safe reinforcement learning techniques either use prior knowledge of certain dangerous states to restrict action choices or incorporate the cost of violating safety into the reward function. This typically sacrifices task utility for safe exploration in unknown environments.

[0065] The core architecture of SAC-MSPI comprises three virtual RL agents: the task agent, the security agent, and the intervention agent. At any given time, either the task agent or the security agent can directly control the MC (Machine Capacitor) using their respective learned policies. The intervention agent only determines when the security agent controls the MC, rather than directly controlling it. Specifically, we define the task policy, security policy, and intervention policy as the policies learned by the task agent, security agent, and intervention agent, respectively. Each agent independently and sequentially makes action decisions based on the system state defined by the MDP (Machine Dependency Program) and predictions of future actions of other agents. The control logic for how the task agent and the security agent alternately control the MC based on the intervention agent's decisions is as follows:

[0066] Safety Intervention Strategy Switching Mechanism: To address the trade-off between maximizing charging utility and minimizing the cost of safety violations, we introduce an intervention agent to determine the appropriate time for the safety agent to control the MC. This is because during a charging task, the task agent controls the MC primarily to maximize charging utility. Once a safety agent is selected to control the MC and execute the corresponding safety policy, it means the MC has already encountered an obstacle or is likely to do so in the near future. Therefore, the intervention agent needs to learn and improve its intervention strategy, i.e., based on the MC's state and the actions performed by the safety agent, to understand the optimal time for the MC to execute the safety policy instead of the task policy. This allows the MC to avoid encountering obstacles during the charging task. (Based on each system state...) Both the task agent and the security agent will make their own action decisions, which will be recorded as tasks. Safety actions Then, the control unit G decides whether to make a safety decision. For control unit G, MC only performs safety actions under certain conditions where the intervention agent assumes a violation (or potential violation) of safety. For all other states, MC focuses on executing and optimizing task actions. This approach maximizes the average effective charging rate. By decoupling the shared goals of maximizing charging utility and minimizing the cost of safety violations into two independent objectives, assigned to the task agent and the safety agent respectively, we note that this decoupling method reduces the reward loss caused by the trade-off between charging utility and motion safety in the task agent.

[0067] To address the trade-off between the MC agent's objectives of security violations and maximizing charging tasks during the trial-and-error process, we need a third-party agent to control the optimal state and timing of actions performed by both agents through state observation, especially regarding the execution of security policies. This is because the execution of a security policy indicates a system state violation or an impending violation at some future time. The third-party control agent learns to predict future joint actions by reacting to the security agent's behavior, thus maintaining the system's current and future security. Figure 2 As shown, our algorithm framework comprises three intelligent agents and one intelligent control unit. The security agent, determined by the intervention agent, decides whether to exercise control over the system state. Our model is a classic sequential decision problem, where in each system state, the task agent and the security agent simultaneously make action decisions. and The control unit G handles the action switching, and the safety intervention framework ultimately outputs only one action. Therefore, actions addressing safety issues are only executed within the set of states it assumes control over (where violations may occur), while the task agent maximizes environmental rewards in other states. The goals of minimizing safety violations and maximizing task rewards are now delegated to two independent agents with different objectives, reducing the loss of reward trade-offs caused by coupling.

[0068] Task Agent: The goal of the task agent is to maximize the expected cumulative reward, i.e., the total amount of energy effectively charged by MC during each training session. The task agent can freely choose and execute actions according to a random policy in any state without intervention from the security agent. The reward function of the task agent is defined as:

[0069]

[0070] in This refers to the system reward defined above. This indicates that the agent is interfering with the agent's action decision in time slot t. 0，1} ,when When =1, MC executes the security action given by the security agent. And receive rewards from the task agent. Otherwise, MC executes the action given by the task agent. And receive a reward Therefore, at any given time, the reward earned by the task agent depends solely on the charging utility, excluding the safety costs of a collision, regardless of the task agent's state. Initially, we seek action decisions that maximize the expected cumulative reward, expressed as equation (14), where This represents the policy of the task-based agent at time slot t. The given action decision, Indicates a security agent-based policy The given action decision.

[0071]

[0072] Security Agent: Typically, the task agent controls MC movement and charging nodes unless the intervention agent decides to allow the security agent to control the MC, which incurs security violation and intervention costs. The goal of the security agent is to minimize the expected cumulative costs of security violations and interventions, thereby reducing the number of security violation states during training and operations. Without loss of generality, we define the reward function of the security agent (15), where As expressed by equation (3), It is a constant representing the intervention cost of implementing the intervention strategy when the intervening agent decides to let the safety agent control the MC.

[0073]

[0074] Since the intervention agent ensures the safety of the MC by intermittently intervening in the operation of the task agent, it is crucial that the intervention agent intervene at the appropriate time during the charging task when the MC truly needs to be alerted for safety violations. Therefore, under the control of the intervention agent, the safety agent seeks to maintain stability in any state. The initial expected cumulative reward is maximized, as expressed in equation (16).

[0075]

[0076] 3) Design and implementation of the SAC-MSPI algorithm

[0077] Architecture and Main Functions: The SAC-based mobile safety policy intervention algorithm is designed based on the SAC algorithm. Its core is the adaptive game process between the safety agent controlled by the control unit and the task agent. Each time the control unit reaches a state, the two agents coordinate and learn to predict future joint actions, minimizing unsafe states in future trajectories while maximizing the system's cumulative charge over rounds. The algorithm mainly consists of three parts, as detailed below. Figure 2 As shown, this includes data acquisition, policy switching control, and policy updates. The MC obtains quadruple data by interacting with the action decisions and environment provided by the intelligent policy switching framework and stores it in the data buffer. When the number of steps accumulates to the number of iterations required for network training, the three SAC instances sample data from the experience buffer and train synchronously.

[0078] Algorithm execution flow: The entire system in Figure 3 It runs under the control of the algorithm. Specifically, we first initialize three data buffers. And the hyperparameters of each SAC module, including The agent MC starts from the initial state. Upon starting, the strategy switching module outputs specific actions. Then get a new state. and receive corresponding rewards. After several iterations, the system generates a sufficient number of data samples, which are stored in data buffers. As training progresses, each data buffer accumulates data samples, and each agent collects data samples from its corresponding data buffer. Each training round ends when the main battery is depleted. To achieve precise policy switching between the task agent and the security agent, we convert the continuous actions of the intervention agent into two binary decision variables: When the action value output by the SAC module of the intervention agent is greater than zero, the MC executes a safety action. Otherwise, the MC executes the task operation. Each agent is trained using the Policy Gradient Algorithm (SAC) and a non-policy training method is used to reduce sampling complexity. Each SAC module updates its hyperparameters at intervals of a certain number of training steps according to the needs of policy learning.

[0079] Algorithm Implementation Module: SAC (Soft Actor-Critic) combines off-policy updates with a stochastic actor-critic, and is an algorithm for optimizing stochastic policies offline. The actor's goal is to maximize the expected reward while maximizing the entropy of the policy action in each access state. By adding an entropy term to the goal, the agent is encouraged to explore more extensively while abandoning actions that are clearly unpromising. Therefore, the goal of MC is defined throughout the task execution as shown in Equation (17), where α is an automatically learnable entropy coefficient. We will... Represented by strategy The resulting trajectory distribution of state-action pairs, where H represents the entropy of each visited state. The goal of the SAC module is to make the objective function... maximize.

[0080]

[0081] In order to Maximizing requires a strategy We perform iterative improvement, meaning we use a soft Q-function to iteratively evaluate the newly updated policy and use it to improve the old policy. Since both the state and action spaces of Monte Carlo Trees (MC) are continuous, we construct three function approximators (neural networks), including two... -network( ) and a strategy -network( ), respectively parameterize the state-action value function and policy function Two training methods were used alternately to train the system using stochastic gradient descent. -Network and -Network. To alleviate The problem of overestimation, each network They are all accompanied by a goal network Each -network Training and updating are performed by minimizing the soft Bellman residuals defined in (18), where the V function is defined in (19). Each objective... -network By using Exponential moving average replacement Perform asynchronous updates.

[0082]

[0083] In policy network In the improvement steps, since the output of the policy network is the mean and covariance, actions are directly sampled from the policy. Since the gradient cannot be calculated, a resampling technique is used to sample from the approximate distribution. Then, perform backpropagation of the gradient to update the policy network parameters. The loss expression for the policy network is as follows:

[0084]

[0085] Effects of the invention:

[0086] Because sensor node-side information (energy consumption, location, etc.) and obstacle-side information are inaccessible to MC, traditional optimization techniques cannot solve (or even explicitly articulate) our problem. In real-world scenarios, the workload of each node can vary over time, causing its energy consumption rate to fluctuate. For example, a camera sensor can increase the frame rate when aiming at a moving object. In most cases, MC cannot know how the workload of each node changes because workload typically depends on random events in the environment. Therefore, MC cannot know how the energy consumption rate of each node changes over time, and without node-side information, MC cannot decide when and where to charge nodes in the network. Furthermore, obstacle location information is also unknown to MC. Although MC can detect any object within a limited range using radar, without prior path planning, even if distance can be detected, a safe movement trajectory may not be maintained.

[0087] To address the challenges posed by the lack of accurate system information, we introduce reinforcement learning (RL) techniques. This allows the MC (Machine Mapper) to learn and optimize its movement trajectory and charging schedule through trial and error by interacting with the environment and tracking feedback signals from nodes and obstacles in real time. In fact, the learning process is complex for several reasons. First, the energy consumption rate of each node is time-varying, and nodes are randomly distributed within the network. Therefore, the real-time remaining battery power of each node exhibits spatiotemporal dynamics within the network. When making movement and charging decisions in each time slot, the MC must consider the number of nodes to be charged and their remaining battery power. For example, even if the number of nodes charging simultaneously in a certain area is large, it is not cost-effective for the MC to travel long distances to charge all nodes with relatively high remaining battery power or low energy consumption rates. Second, sometimes the most "profitable" nodes—those with low remaining battery power or high energy consumption rates—are located near obstacles. A trade-off must be struck between maximizing effective charging energy and minimizing the risk of colliding with obstacles, and this should be handled with caution during the charging process.

[0088] The algorithm proposed in this invention can intelligently perceive the initial battery charge, distribution density, and energy consumption rate of sensor nodes in WRSN. Even when the locations of sensor nodes and obstacles are unknown, it intelligently provides a collision-free, safe charging scheduling strategy based on changes in node parameters within the network. Specifically, it can achieve the following objectives:

[0089] 1) Intelligent charging scheduling based on the initial power distribution in the sensor network. When all nodes have the same energy consumption, MC performs charging tasks in areas with lower initial power, replenishing the power of nodes with lower remaining energy in a timely manner, reducing the number of node failures in the entire WRSN.

[0090] 2) Intelligent charging scheduling based on node distribution density in the sensor network. We verified the impact of node distribution density in the WRSN on the intelligent scheduling decision of the MC. While maintaining the same initial average charge value for all nodes in the network and the same average energy consumption per node per unit time, we verified the MC's movement trajectory by setting different node densities in different areas. The results show that the MC can move along a dense trajectory until the main battery it carries is depleted, at which point the charging task ends.

[0091] 3) Intelligent charging scheduling based on the energy consumption of sensor nodes in the sensor network. The unit energy consumption of sensor nodes is crucial to the duration of the entire WRSN operation. MC needs to optimize its movement trajectory to replenish the power of nodes with high energy consumption in a timely manner, thereby increasing the continuous operation time of the entire network. We verified that when all nodes have the same initial average power and the node distribution density is consistent, the charging trajectory of MC always moves along the area where sensor nodes have high energy consumption.

[0092] The method proposed in this invention maximizes the charging utility of the MC (Mobile Concentrator) by jointly optimizing its movement trajectory and charging schedule without obtaining precise system information. We design an efficient model-free RL algorithm, SAC-MSPI, which decouples maximizing charging utility from minimizing collision risk. Extensive evaluation results demonstrate that our algorithm outperforms existing leading RL solutions and traditional baseline algorithms in terms of charging utility and collision avoidance capabilities.

[0093] The present invention also provides a mobile wireless charging vehicle, which is controlled by the above method.

[0094] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to replace them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A method for joint scheduling of mobile wireless charger moving trajectory and charging, comprising an automatically walking mobile wireless charger and a wireless rechargeable sensor network, the wireless rechargeable sensor network comprising a plurality of rechargeable sensor nodes, characterized in that, Includes the following steps: The first step is for a fully charged portable wireless charger to depart from the charging station; The second step involves the mobile wireless charger moving within a wireless rechargeable sensor network. During this movement, it searches for sensor nodes in the network. When a sensor node is within the rechargeable range of the mobile wireless charger, it detects the remaining power of that sensor node. If the remaining power is not full, it wirelessly charges the sensor node. The third step is to return the mobile wireless charger to the charging station to recharge after the rechargeable power is exhausted. The fourth step involves the fully charged wireless charger restarting the first step, and this process is repeated sequentially. The mobile wireless charger's movement trajectory in the network is set as a two-dimensional plane movement trajectory. The movement trajectory design and charging scheduling problem of the mobile wireless charger are established by a Markov decision model (MDP). Based on the established MDP, reinforcement learning (RL) technology is introduced to form a mobile safety policy intervention algorithm based on soft actor criticism. This decouples maximizing charging utility from minimizing collision risk and finds a safe movement trajectory with the maximum achievable charging utility. By using reinforcement learning technology, mobile wireless chargers can learn through trial and error to jointly optimize their movement trajectory and charging plan by interacting with the environment and tracking feedback signals from sensor nodes and obstacles in real time during the charging process. The obstacles include static obstacles and dynamic obstacles; The Markov decision model (MDP) is defined as a quintuple. Where S and A represent the state space and action space of MC, P represents the state transition probability space, R represents the reward space, and γ represents the discount factor used to weigh short-term and long-term cumulative rewards; based on MDP, the mobile wireless charger improves its joint trajectory design and charging scheduling to maximize the positive feedback of the environment through continuous interaction with the environment; the mobile wireless charger is defined as a real RL agent, and the task agent, safety agent, and intervention agent are defined as virtual RL agents, each of the three virtual agents representing an independent training architecture based on the soft actor criticism algorithm; The task agent, security agent, and intervention agent learn the following policies: task policy, security policy, and intervention policy, respectively. The task agent or security agent can directly control the mobile wireless charger through their respective learned policies. The intervention agent only determines when the security agent controls the mobile wireless charger, rather than directly controlling the mobile wireless charger.

2. The method for jointly scheduling the movement trajectory and charging of a mobile wireless charger according to claim 1, characterized in that, The sensor node is equipped with a wirelessly rechargeable battery, and a mobile wireless charger wirelessly charges the battery on the sensor node.

3. The method for jointly scheduling the movement trajectory and charging of a mobile wireless charger according to claim 2, characterized in that, The sensor nodes are randomly distributed.

4. The method for jointly scheduling the movement trajectory and charging of a mobile wireless charger according to claim 2, characterized in that, The mobile wireless charger is equipped with a GPS module and a battery management system.

5. The method for jointly scheduling the movement trajectory and charging of a mobile wireless charger according to claim 2, characterized in that, The mobile wireless charger includes a main battery and a backup battery. The main battery is used to charge sensor nodes in the network, and the backup battery is used for the mobile wireless charger to return to the charging station from any location in the network. The backup battery is activated after the main battery is depleted.

6. The method for jointly scheduling the movement trajectory and charging of a mobile wireless charger according to claim 2, characterized in that, The mobile wireless charger's movement trajectory begins at the charging station and ends when the main battery is completely depleted.

7. The method for jointly scheduling the movement trajectory and charging of a mobile wireless charger according to claim 6, characterized in that, The system status of the mobile wireless charger, including its real-time location and remaining battery power, is as follows: Where t is the equal-length time slot, Let be the coordinates of time slot t in the two-dimensional plane. The remaining power of the mobile wireless charger on its movement trajectory at time slot t; The status information is The mobile wireless charger uses this information to decide on its next move and power allocation.

8. The method for jointly scheduling the movement trajectory and charging of a mobile wireless charger according to claim 7, characterized in that, The optimization objective of the task agent is to maximize charging utility, while the optimization objective of the security agent is to minimize the cost of safety violations. The task agent and the security agent are independent of each other. The mobile security policy intervention algorithm controls the adaptive game between the task agent and the security agent. When the mobile wireless charger reaches a certain state, the two agents coordinate with each other to learn and predict future joint actions, so as to minimize the occurrence of unsafe states in future trajectories and maximize the cumulative charging amount of the system.

9. A mobile wireless charging vehicle, characterized in that, The operation of the mobile wireless charging vehicle can be controlled using any of the methods described in claims 1 to 8.

Images