Mobile charging station scheduling method and system for charging unmanned aerial vehicle

Abstract

The invention relates to the technical field of mobile charging station scheduling, and discloses a mobile charging station scheduling method and system for charging unmanned aerial vehicles, and the method comprises the steps: obtaining the real-time position and residual electric quantity of each unmanned aerial vehicle, the real-time position of a mobile charging station, and the residual electric quantity of each unmanned aerial vehicle; based on the real-time position of the unmanned aerial vehicle and the real-time position of the mobile charging station, the distance between the two is obtained; constructing an observation state of the mobile charging station based on the real-time position and the residual electric quantity of each unmanned aerial vehicle, the real-time position of the mobile charging station and the distance between the unmanned aerial vehicle and the mobile charging station; and inputting the observation state of the mobile charging station into a reinforcement learning model trained based on strategy gradient to obtain an action strategy for scheduling the mobile charging station to charge the unmanned aerial vehicle. According to the embodiment of the invention, the mobile charging station is dynamically scheduled to move to the position where the unmanned aerial vehicle may arrive or pass, and energy supplement is provided for the unmanned aerial vehicle in time so as to support the unmanned aerial vehicle cluster to continuously monitor the environment.

Description

Technical Field

[0001] This invention relates to the field of mobile charging station scheduling technology, and specifically to a mobile charging station scheduling method and system for charging drones. Background Technology

[0002] Real-time urban sensing data collection is a crucial application of big data technology and forms the foundation for effective traffic management, safety assurance, and emergency response. Fixed sensing devices typically cover only a limited number of key sensors, and deployment and maintenance costs are high in many cities. Mobile crowdsourced data often faces critical issues such as low sampling rates, missing data, and inaccuracies. Compared to traditional methods, drones offer significant advantages: their powerful sensing capabilities allow them to capture richly informative and expansive "bird's-eye view" images or videos using cameras, and their flexible maneuverability enables them to reach desired locations promptly, unaffected by terrain or traffic congestion. However, their main disadvantage lies in their susceptibility to onboard power limitations, hindering their ability to perform long-duration sensing tasks.

[0003] Existing work has focused on one-time scheduling of UAVs without considering the energy constraints in their continuous monitoring. Other work has considered the energy limitations of UAVs during deployment, but only as part of the optimization objective, not as a hard constraint in actual deployment, which does not meet the application constraints in real-world application scenarios. Summary of the Invention

[0004] In view of this, the present invention provides a mobile charging station scheduling method and system for charging drones, in order to solve the problem of how to schedule mobile charging stations to replenish the energy of drones so as to support their continuous environmental perception and data collection.

[0005] In a first aspect, the present invention provides a mobile charging station scheduling method for charging drones, the method comprising:

[0006] The system obtains the real-time location and remaining battery power of each drone, as well as the real-time location of the mobile charging station, and calculates the distance between the two based on their real-time locations.

[0007] Based on the real-time location and remaining battery power of each drone, the real-time location of the mobile charging station and the distance between them, the observation status of the mobile charging station is constructed.

[0008] The observed state of the mobile charging station is input into a reinforcement learning model trained based on policy gradients to obtain an action policy for scheduling the mobile charging station to charge the drone.

[0009] The mobile charging station scheduling method for charging drones provided in this invention constructs the observation status of mobile charging stations based on the drone's real-time location, remaining battery power, and distance to the mobile charging station. This helps the model accurately determine which mobile charging station should charge which drone, avoiding resource waste and unreasonable allocation. Combining real-time location and distance information with the drone's remaining battery power, the reinforcement learning model can plan the optimal charging path for the mobile charging station. This path planning based on real-time data considers the dynamic position changes of the drone and the mobile charging station, enabling the mobile charging station to reach the target drone via the shortest path, reducing time consumption and energy loss during the charging process, and further improving the efficiency of the entire charging scheduling system to support continuous environmental monitoring by the drone swarm.

[0010] In one alternative implementation, the observed status of the mobile charging station Defined as:

[0011]

[0012] Among them, g t (v n ) and g t (u col ) represent the grid positions of the mobile charging station and the drone in time slot t, respectively, where n represents the number of the mobile charging station and m represents the number of the drone. d represents the remaining battery power of the drone. M (u n ,v col ) is the Manhattan distance from the drone equipment to the mobile charging station.

[0013] This invention, through a grid layout including mobile charging stations and drones, provides a clear understanding of the spatial distribution of all elements in the system. This helps to intuitively grasp the spatial relationship between mobile charging stations and drones in need. Considering the remaining battery power of drones, it can predict in advance which drones will require charging services. By introducing the Manhattan distance from the drone to the mobile charging station, it provides an intuitive and effective way to measure distance. The Manhattan distance accurately reflects the actual travel distance to the drone's location, which helps to optimize charging scheduling strategies and reduce the travel time and energy consumption of mobile charging stations.

[0014] In one optional implementation, the process of training the reinforcement learning model based on policy gradients includes:

[0015] Initialize experience pool D v and the state-value function parameter θ of the reinforcement learning model v ;

[0016] Based on historical data from drones and mobile charging stations, including historical drone battery data, historical location data, and historical mobile charging station location data, an observation status of the mobile charging station is generated. And based on strategy Select Action

[0017] Perform the action Collect and execute actions to obtain instant rewards from various mobile charging stations after the interaction environment changes. t And transfer its stored interactive experiences to the experience pool D v ;

[0018] From experience pool D v Extract the transfer experience and calculate the corresponding target value using the target value function;

[0019] The parameters of the reinforcement learning model are updated using a preset loss function based on the target value, and the parameters are optimized based on the preset policy gradient until the reinforcement learning model converges, thus obtaining a trained reinforcement learning model.

[0020] This invention utilizes historical data from drones and mobile charging stations to generate observational states of mobile charging stations. This helps the model understand the system's operational patterns and characteristics. Combined with optimization through preset policy gradients, it effectively improves the model's performance, enabling the model to gradually converge to a better policy, continuously reducing the gap between predicted and target values, and ultimately obtaining a well-trained and high-performance reinforcement learning model.

[0021] In an alternative implementation, the value function is constructed using a fully connected network, as shown below:

[0022]

[0023] Among them, W p and b p These are the parameters that the fully connected network needs to train, used to estimate the state values ​​based on the observed state values.

[0024] This invention employs a fully connected network to effectively approximate various complex value functions. In the drone charging scheduling scenario, the observed state includes multiple complex variables such as the location of the mobile charging station and the drone, the drone's battery level, and distance, and the relationships between these variables may be non-linear. The fully connected network, through the connections of multiple neurons and activation functions, can learn the complex mapping relationships between these variables, thereby accurately estimating the value.

[0025] In one optional implementation, the preset loss function is:

[0026]

[0027] Where T is the number of time slots in the entire support cycle, V is the number of all mobile charging stations, and y t The objective value function is expressed by the following formula:

[0028]

[0029] Where γ is the discount factor, r t This is the reward value.

[0030] The loss function in this embodiment of the invention includes the number of time slots throughout the entire support cycle. This allows the model to consider not only immediate rewards but also long-term gains during training. By comprehensively evaluating the value across multiple time slots, the model can learn a more forward-looking strategy. As time slots change, the system's state (the drone's location and battery level, the location of mobile charging stations, etc.) also changes dynamically. The loss function, by considering the value differences across different time slots, enables the model to learn to make optimal decisions over time.

[0031] Secondly, the present invention provides a mobile charging station scheduling system for charging drones, comprising:

[0032] The real-time data acquisition module is used to acquire the real-time location and remaining battery power of each drone and the real-time location of the mobile charging station, and to obtain the distance between the two based on the real-time location of the drone and the real-time location of the mobile charging station.

[0033] The observation status construction module is used to construct the observation status of the mobile charging station based on the real-time location and remaining power of each drone, the real-time location of the mobile charging station and the distance between the two.

[0034] The scheduling policy generation module is used to input the observed states of the mobile charging station into a reinforcement learning model trained based on policy gradients, and obtain action policies to schedule the mobile charging station to charge the drone.

[0035] In one optional implementation, the scheduling policy generation module includes:

[0036] An initialization unit is used to initialize the experience pool D. v and the state-value function parameter θ of the reinforcement learning model v ;

[0037] The action selection unit is used to generate the mobile charging station observation status based on historical data of the drone and the mobile charging station, including historical power data and location data of the drone and historical location data of the mobile charging station. And based on strategy Select Action

[0038] The reward acquisition unit is used to perform the action. Collect and execute actions to obtain instant rewards from various mobile charging stations after the interaction environment changes. t And transfer its stored interactive experiences to the experience pool D v ;

[0039] The target value calculation unit is used to calculate the target value from the experience pool D. v Extract the transfer experience and calculate the corresponding target value using the target value function;

[0040] The optimization unit is used to update the parameters of the reinforcement learning model based on the target value using a preset loss function, and optimize the parameters based on the preset policy gradient until the reinforcement learning model converges, thus obtaining a trained reinforcement learning model.

[0041] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the mobile charging station scheduling method for charging drones described in the first aspect or any corresponding embodiment.

[0042] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the mobile charging station scheduling method for charging a drone as described in the first aspect or any corresponding embodiment.

[0043] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the mobile charging station scheduling method for charging a drone as described in the first aspect or any corresponding embodiment. Attached Figure Description

[0044] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating a mobile charging station scheduling method for charging drones according to an embodiment of the present invention.

[0046] Figure 2This is a flowchart illustrating a reinforcement learning model trained based on policy gradients according to an embodiment of the present invention.

[0047] Figure 3 A structural block diagram of a mobile charging station scheduling system for charging drones according to an embodiment of the present invention;

[0048] Figure 4 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] To schedule mobile charging stations to replenish the energy of drones and support their continuous environmental perception and data acquisition, this invention provides an embodiment of a mobile charging station scheduling method for charging drones. It should be noted that the steps shown in the flowcharts can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that presented here.

[0051] This embodiment provides a mobile charging station scheduling method for charging drones. Figure 1 This is a flowchart of a mobile charging station scheduling method for charging drones according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0052] S101: Obtain the real-time location and remaining battery power of each drone, and the real-time location of the mobile charging station, and calculate the distance between the two based on the real-time location of the drone and the real-time location of the mobile charging station.

[0053] Specifically, drones are unmanned aircraft controlled by radio remote control equipment and their own program control devices, or operated autonomously, either completely or intermittently, by an onboard computer. Mobile charging stations include charging delivery vehicles and charging robots. Delivery vehicles use minivans, pickup trucks, and other similar vehicles as mobile platforms to provide charging services for drones.

[0054] This invention, by acquiring the real-time location and remaining battery power of the drone, as well as the real-time location of the mobile charging station, can accurately calculate the distance. This allows for timely dispatching of the mobile charging station to recharge the drone when its battery is low, under conditions of strong energy constraints. This prevents the drone from losing control due to insufficient power and enables it to continuously perceive the environment and collect data.

[0055] S102, based on the real-time location and remaining power of each drone, the real-time location of the mobile charging station and the distance between the two, constructs the observation status of the mobile charging station.

[0056] Specifically, the observation status of the mobile charging station constructed in this embodiment of the invention. Defined as:

[0057]

[0058] Among them, g t (v n ) and g t (u col ) represent the grid positions of the mobile charging station and the drone in time slot t, respectively, where n represents the number of the mobile charging station and m represents the number of the drone. d represents the remaining battery power of the drone. M (u n ,v col ) is the Manhattan distance from the drone equipment to the mobile charging station.

[0059] By including the grid positions of mobile charging stations and drones in time slot t, the spatial distribution of all elements in the entire system can be clearly understood. This is similar to having a real-time map that precisely marks the positions of mobile charging stations and drones at a specific point in time (time slot t), helping to intuitively grasp the spatial relationship between resources (mobile charging stations) and demand objects (drones). Considering the remaining battery power of drones, it is possible to predict in advance which drones will need charging services. Introducing the Manhattan distance from drone equipment to mobile charging stations provides an intuitive and effective way to measure distance. In actual scheduling environments, especially in scenarios with regular layouts such as city streets and industrial parks, the Manhattan distance is more in line with actual movement costs. For example, when mobile charging stations move between blocks, due to road limitations, they usually move horizontally and vertically. In this case, the Manhattan distance can accurately reflect the actual movement distance to reach the drone's location, which helps to optimize charging scheduling strategies and reduce the movement time and energy consumption of mobile charging stations.

[0060] S103, input the observed state of the mobile charging station into the reinforcement learning model trained based on the policy gradient to obtain the action policy for scheduling the mobile charging station to charge the drone.

[0061] Specifically, the goal of reinforcement learning models is usually to maximize long-term rewards, while the goal of policy gradient reinforcement learning is to learn an optimal policy to help mobile charging stations obtain the maximum expected long-term discount reward, that is, to provide more replenishment within a given time.

[0062] This invention incorporates the observed state of a mobile charging station into a reinforcement learning model trained on policy gradients. The model can then perform comprehensive analysis based on the rich input information and output a reasonable action strategy. In actual drone charging scheduling, the environment is dynamically changing. The drone's position constantly shifts as the task is executed, its remaining battery power fluctuates, and the mobile charging station's position may also change for various reasons. The reinforcement learning model trained on policy gradients, with its continuously input real-time observed state, can adapt to these changes and adjust its action strategy in real time under strong energy constraints. This provides energy replenishment for the drone swarm and continuously monitors and collects environmental data. For example, when a drone that didn't require emergency charging experiences a rapid drop in battery power due to performing additional tasks, or when a mobile charging station encounters obstacles on its way to charging and has to change routes, the model can quickly re-plan the charging scheduling strategy based on the new observed state. This ensures the entire charging scheduling process continues smoothly, guarantees the drone's continuous operational capability, and allows the entire charging scheduling system to achieve optimal performance over long-term operation.

[0063] In reinforcement learning, three basic definitions are observed state, action, and reward. Each agent, based on the observed state and according to the reinforcement learning algorithm, executes appropriate actions to influence the environmental state and receives rewards from the environment. In this interaction process, the network is continuously trained based on transition experience, improving the action control strategy until convergence. The policy gradient reflects the rate of change of the policy function with varying parameters; it indicates how to adjust the model's parameters so that the adopted action policy improves towards obtaining higher long-term rewards. For example, if the current action policy leads to low charging efficiency, the policy gradient indicates in which direction the model parameter space should be adjusted to make subsequent action policies better. This invention describes the process of training a reinforcement learning model based on policy gradients, as follows: Figure 2 As shown, it includes:

[0064] S201, Initialize the storage experience pool D v and the state-value function parameter θ of the reinforcement learning model v ;

[0065] Specifically, the initialization of the experience pool provides a space to store historical interaction information, offering a data foundation for subsequent model learning. The model prepares a knowledge base containing various past experiences, ensuring the model has a basis for subsequent training and avoiding the blindness of learning from scratch. Initializing the state value function parameters provides a starting point for the model to evaluate the value of different states. Reasonable initialization can guide the model to converge to a better strategy more quickly. For example, by using initial parameters based on prior knowledge or simple heuristics, the model can make a relatively reasonable judgment on state value in the early stages, reducing the exploration cost in the early training phase.

[0066] Specifically, the value function is constructed using a fully connected network, as shown below:

[0067]

[0068] Among them, W p and b p These are the parameters that the fully connected network needs to train, so that it can be used to estimate the state of the observed environment.

[0069] S202, based on historical data from the drone and mobile charging station, including historical drone battery data, historical location data, and historical mobile charging station location data, generate the mobile charging station observation status. v and based on strategy Select Action

[0070] Specifically, historical data from drones and mobile charging stations is used to generate the observation status of mobile charging stations, fully leveraging the value of past information. This historical data contains key information such as battery level and location, helping the model understand the system's operational patterns and characteristics. For example, by analyzing historical location data, the model can understand the distribution patterns of drones and mobile charging stations in different areas, as well as their common relative positional relationships, thereby better constructing the observation status. This embodiment of the invention uses historical data from drones and mobile charging stations to generate the observation status of mobile charging stations in time slot t.

[0071] Strategy Used for action selection, it describes observation-based... Execute action Given a probability distribution, the softmax function is used to obtain the probability of choosing each possible action, and its formula is as follows:

[0072]

[0073] Policy-based action selection enables the model to choose actions strategically based on the current observation state and learned policy knowledge. This approach avoids the inefficiency that can result from random action selection, instead making relatively reasonable decisions based on past experience and current state assessment. For example, if historical data indicates that drones are more prone to running out of power in certain areas, the action of selecting mobile charging stations will prioritize moving closer to these areas.

[0074] S203, Execute action Collect and execute actions to obtain instant rewards from various mobile charging stations after the interaction environment changes. t and transfer its storage interaction experience to D v ;

[0075] Specifically, you receive an instant reward for accessing n mobile charging stations. And store interactive transfer experience Go to experience pool D v .

[0076] Collecting immediate rewards after actions provides the model with direct feedback signals, allowing it to quickly understand the merits of each action. If an action leads to a good outcome (such as successfully charging a drone with extremely low battery), the model receives a positive reward; conversely, it may receive a negative reward. In this way, the model can quickly adjust its strategy, moving towards obtaining more rewards. Storing interaction transfer experiences in an experience pool allows the model to accumulate rich experience. As the experience pool grows, the model can learn from a large number of different situations, improving its generalization ability. For example, when facing various complex drone mission scenarios and mobile charging station layouts, a rich experience pool can provide the model with more reference cases, helping it better cope with different situations.

[0077] S204, from experience pool D v Extract the transfer experience and calculate the corresponding target value using the target value function;

[0078] Specifically, y t The objective value function is expressed by the following formula:

[0079]

[0080] Where γ is the discount factor, r t This is the reward value.

[0081] The objective value function calculates the corresponding target value, providing a clear direction for model parameter updates. This target value reflects the value assessment standard that the model should achieve under ideal conditions. The discount factor helps balance the relationship between immediate and long-term rewards. During training, the model needs to find a balance between pursuing immediate rewards (such as quickly providing charging services to drones in urgent need) and obtaining long-term rewards (such as optimizing the efficiency of the entire charging scheduling system to obtain more rewards throughout the support cycle). By adjusting the value of the discount factor, the model's preference for immediate and long-term rewards can be controlled, enabling the model to learn a strategy that ensures the successful completion of the current task while also considering the long-term stable operation of the system.

[0082] S205, based on the target value, update the parameters of the reinforcement learning model using a preset loss function, and optimize the parameters based on the preset policy gradient until the reinforcement learning model converges, thus obtaining the trained reinforcement learning model.

[0083] Specifically, the preset loss function is:

[0084]

[0085] Where T is the number of time slots in the entire support cycle, V is the number of all mobile charging stations, and y t The target value function value.

[0086] Since the goal of policy gradient reinforcement learning is to learn an optimal policy to help mobile charging stations obtain the maximum expected long-term discount reward, i.e., to provide more refueling within a given time, the objective function can be expressed as:

[0087]

[0088] V(o v ) represents the value network for a mobile charging station state o v The rating is defined as follows:

[0089]

[0090] Its specific meaning refers to each possible action in action set A. The score is the product of its possible probability. Based on this, the policy gradient to be optimized can be expressed as:

[0091]

[0092] When the network converges during the iterative process, the trained reinforcement learning model is considered complete. Updating the model's parameters using the loss function based on the target value and optimizing it through a pre-set policy gradient effectively improves the model's performance. This update method allows the model to gradually converge to a better policy, continuously reducing the gap between the predicted and target values. For example, by continuously adjusting the parameters of the state value function and policy function, the model's evaluation of the observed state becomes increasingly accurate, and the selected actions become more aligned with the goal of efficiently scheduling mobile charging stations to charge drones, ultimately resulting in a well-trained and high-performing reinforcement learning model.

[0093] It's important to note that even after training is complete, the model continues to interact with the environment. After each action is performed, the environment provides corresponding reward feedback and state transition results. The model uses this new feedback to further confirm the effectiveness of the current strategy and continuously optimize the selection of subsequent actions. Although the degree of optimization is much smaller than during the training phase, this continuous interactive feedback mechanism is always in effect, ensuring that the model always adapts to changes in the environment and provides the optimal scheduling strategy to complete the task efficiently.

[0094] This embodiment also provides a mobile charging station scheduling system for charging drones. This system implements the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the systems described in the following embodiments are preferably implemented in software, hardware implementations, or combinations of software and hardware, are also possible and contemplated.

[0095] This embodiment provides a mobile charging station scheduling system for charging drones, such as... Figure 3 As shown, it includes:

[0096] The real-time data acquisition module 301 is used to acquire the real-time location and remaining power of each drone and the real-time location of the mobile charging station, and to obtain the distance between the two based on the real-time location of the drone and the real-time location of the mobile charging station.

[0097] The observation status construction module 302 is used to construct the observation status of the mobile charging station based on the real-time location and remaining power of each UAV, the real-time location of the mobile charging station and the distance between the two.

[0098] The scheduling strategy generation module 303 is used to input the observed state of the mobile charging station into the reinforcement learning model trained based on the policy gradient to obtain the action strategy for scheduling the mobile charging station to charge the drone.

[0099] In some optional implementations, the reinforcement learning model in the scheduling policy generation module includes the following during training:

[0100] An initialization unit is used to initialize the experience pool D. v and the state-value function parameter θ of the reinforcement learning model v ;

[0101] The action selection unit is used to generate the mobile charging station observation status based on historical data of the drone and the mobile charging station, including historical power data and location data of the drone and historical location data of the mobile charging station. And based on strategy Select Action

[0102] The reward acquisition unit is used to perform the action. Collect and execute actions to obtain instant rewards from various mobile charging stations after the interaction environment changes. t And transfer its stored interactive experiences to the experience pool D v ;

[0103] The target value calculation unit is used to calculate the target value from the experience pool D. v Extract the transfer experience and calculate the corresponding target value using the target value function;

[0104] The optimization unit is used to update the parameters of the reinforcement learning model based on the target value using a preset loss function, and optimize the parameters based on the preset policy gradient until the reinforcement learning model converges, thus obtaining a trained reinforcement learning model.

[0105] In some alternative implementations, the value function is constructed using a fully connected network, as shown below:

[0106]

[0107] Among them, W p and b p These are the parameters that the fully connected network needs to train, used to estimate the state values ​​based on the observed state values.

[0108] In some optional implementations, the preset loss function is:

[0109]

[0110] Where T is the number of time slots in the entire support cycle, V is the number of all mobile charging stations, and y t The objective value function is expressed by the following formula:

[0111]

[0112] Where γ is the discount factor, r t This is the reward value.

[0113] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0114] In this embodiment, the mobile charging station scheduling system for charging drones is presented in the form of a functional unit, including a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0115] This invention also provides a computer device having the above-described features. Figure 3 The mobile charging station dispatch system shown is used to charge drones.

[0116] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 4 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 4 Take a processor 10 as an example.

[0117] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0118] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0119] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0120] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0121] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0122] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor central control systems, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the methods shown in the above embodiments are implemented.

[0123] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0124] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for scheduling a mobile charging station for charging a drone, characterized in that: include: Obtain the real-time location and remaining power of each drone, the real-time location of the mobile charging station, and obtain the distance between the real-time location of the drone and the real-time location of the mobile charging station; Based on the real-time location and remaining power of each drone, the real-time location of the mobile charging station and the distance between them, the observation status of the mobile charging station is constructed; The observed state of the mobile charging station is input into the reinforcement learning model trained based on policy gradient, and the action strategy is obtained to schedule the mobile charging station to charge the drone.

2. The method according to claim 1, characterized in that Observation status of the mobile charging station Defined as: Among them, g t (v n ) and g t (u col ) represent the grid positions of the mobile charging station and the UAV in time slot t, n represents the number of the mobile charging station, m represents the number of the UAV, Indicates the remaining battery power of the drone, d M (u n ,v col ) is the Manhattan distance from the drone device to the mobile charging station.

3. The method according to claim 2, characterized in that The process of training the reinforcement learning model based on policy gradient includes: Initialize experience pool D v And the state value function parameter θ of the reinforcement learning model v ; Generate the observation status of the mobile charging station based on the historical data of the drone and the mobile charging station, including the historical power data and location data of the drone and the historical location data of the mobile charging station Based on strategy Select Action Execute the action Collect and execute actions after the interactive environment changes to obtain instant rewards for each mobile charging station t , and transfer its stored interactive experience to the experience pool D v ; From the experience pool D v Extract transfer experience and calculate the corresponding target value using the target value function; Based on the target value, the parameters of the reinforcement learning model are updated using the preset loss function, and the parameters are optimized based on the preset policy gradient until the reinforcement learning model converges to obtain a trained reinforcement learning model.

4. The method according to claim 3, characterized in that: The value function is constructed using a fully connected network and is expressed as follows: Among them, W p and b p It is the parameter that needs to be trained for the fully connected network to estimate the observed state value using the fully connected network.

5. The method according to claim 4, characterized in that The preset loss function is: Where, T is the number of time slots in the entire support period, V is the number of all mobile charging stations, and y t is the target value function, which is expressed by the following formula: Among them, γ is the discount factor, r t is the reward value.

6. A mobile charging station dispatching system for charging drones, characterized in that: include: A real-time data acquisition module is used to obtain the real-time location and remaining power of each drone and the real-time location of the mobile charging station, and obtain the distance between the real-time location of the drone and the real-time location of the mobile charging station based on the real-time location of the drone; An observation status construction module is used to construct the observation status of the mobile charging station based on the real-time location and remaining power of each drone, the real-time location of the mobile charging station and the distance between the two; The scheduling strategy generation module is used to input the observed state of the mobile charging station into the reinforcement learning model trained based on the policy gradient, and obtain the action strategy for scheduling the mobile charging station to charge the drone.

7. The system according to claim 6, characterized in that The reinforcement learning model in the scheduling strategy generation module includes: Initialization unit, used to initialize the experience pool D v And the state value function parameter θ of the reinforcement learning model v ; The action selection unit is used to generate the observation state of the mobile charging station based on the historical data of the drone and the mobile charging station, including: the historical power data and historical location data of the drone and the historical location data of the mobile charging station Based on strategy Select Action A reward acquisition unit, used to perform the above actions Collect and execute actions after the interactive environment changes to obtain instant rewards for each mobile charging station t , and transfer its stored interactive experience to the experience pool D v ; The target value calculation unit is used to obtain the target value from the experience pool D v Extract transfer experience and calculate the corresponding target value using the target value function; The optimization unit is used to update the parameters of the reinforcement learning model based on the target value using a preset loss function, and optimize the parameters based on a preset policy gradient until the reinforcement learning model converges to obtain a trained reinforcement learning model.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the mobile charging station scheduling method for charging a drone according to any one of claims 1 to 5 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the mobile charging station scheduling method for charging a drone according to any one of claims 1 to 5.

10. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to cause a computer to execute the mobile charging station scheduling method for charging a drone according to any one of claims 1 to 5.

Images