Multi-unmanned aerial vehicle scheduling method, system and device and storage medium

Abstract

The invention relates to the technical field of unmanned aerial vehicles, and particularly provides a multi-unmanned aerial vehicle scheduling method, system and device and a storage medium, and the method comprises the steps: obtaining the state information of unmanned aerial vehicles, and constructing a state space and an action space based on the state information; acquiring an image acquired by the unmanned aerial vehicle, identifying and tracking a fire point from the image, and extracting the size, trajectory and moving speed of the fire point; generating an unmanned aerial vehicle task based on the size, the track and the moving speed of the fire point; setting a reward function, wherein the reward function comprises a task coverage reward, a resource effective utilization reward, a task response speed reward and a task completion rate reward; and based on the state space, the action space and the reward function, solving an optimal scheduling scheme for executing the unmanned aerial vehicle task. The method can better adapt to the complex forest environment, and the response timeliness of the unmanned aerial vehicle to the forest fire is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicles, and in particular relates to a multi-unmanned aerial vehicle scheduling method, system, device and storage medium. Background Art

[0002] With the rapid development of drone technology, drones are increasingly used in forest environments, especially in forest monitoring and dispatching. Traditional forest monitoring methods usually rely on patrols and monitoring by ground personnel, but this method is inefficient and costly, and it is often difficult to obtain monitoring data in real time and comprehensively when faced with vast and complex forest terrain. The introduction of drones provides a new technical means for forest monitoring. Through the high-altitude perspective of drones, large areas of forests can be monitored quickly and effectively, while reducing the consumption of human resources.

[0003] However, as the complexity of monitoring tasks increases, the problem of coordinated scheduling of multiple UAVs has become increasingly prominent. In practical applications, multiple UAVs are required to perform monitoring tasks in forest environments to cover a wider area or perform different tasks at the same time. In complex forest environments, UAV scheduling faces many challenges, such as path planning, task allocation, obstacle avoidance, and energy consumption management. The efficient solution of these problems is directly related to the monitoring effect and resource utilization of the multi-UAV system. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a multi-UAV scheduling method, system, device and storage medium to solve the above-mentioned technical problems.

[0005] In a first aspect, the present invention provides a multi-UAV scheduling method, comprising: Acquire state information of the drone, and construct a state space and an action space based on the state information; Obtain images collected by drones, and identify and track fire points from the images, and extract the size, trajectory, and movement speed of the fire points; Generate a drone mission based on the size, trajectory, and movement speed of the fire point; Setting a reward function, wherein the reward function includes a task coverage reward, a resource effective utilization reward, a task response speed reward, and a task completion rate reward; Based on the state space and action space, and the reward function, an optimal scheduling solution for executing the drone mission is solved.

[0006] In an optional implementation, obtaining state information of the drone and constructing a state space and an action space based on the state information includes: Use sensors to obtain the status information of the drone; The formula for defining the state space is as follows:

[0007] in, is the global state at time t, which contains the states of all drones; For the i-th drone, its state Further defined as:

[0008] in, is the position of drone i at time t, expressed as a three-dimensional coordinate ; is the speed of drone i at time t, expressed as a velocity vector ; is the remaining power of drone i at time t; is the current mission status of UAV i; is the coverage area of ​​the monitoring area of ​​UAV i at the current moment; The action space describes the specific operations that each drone can perform in the current state, covering flight action control, throwing actions, and shouting actions. The action space is defined as follows:

[0009] in, is the global action at time t, which includes the actions of all drones; For the i-th drone, its action Defined as:

[0010] in, is the flight control action of UAV i, expressed as the position adjustment at time t, expressed as a three-dimensional vector ; It is the throwing action of drone i, indicating whether the throwing action is performed at the current moment. 0 means no throwing, and 1 means throwing.

[0011] In an optional embodiment, an image captured by a drone is obtained, and a fire point is identified and tracked from the image, and the size, trajectory, and moving speed of the fire point are extracted, including: Extracting image frame sequences from video streams captured by drones; Extracting potential fire points from image frames by setting color thresholds; Extracting a subsequence containing a fire point from the image frame sequence, and inputting the subsequence into a pre-trained YOLO model to identify the fire point, the location of the fire point, the size of the fire point, and the confidence level; Based on the position difference of the ignition point between adjacent image frames in the subsequence, the moving speed and moving direction of the ignition point are calculated.

[0012] In an optional embodiment, generating a drone mission based on the size, trajectory, and moving speed of the fire point includes: Predict where the drone mission will be based on the location of the fire, its movement speed, direction, and the drone’s flight time; Determine whether the confidence level reaches a set threshold: If so, setting the drone mission requires drones carrying fire extinguishing agents, and setting the number of drones required based on the size of the fire point; If not, the drone mission is set to require the drone to hover for observation, and the required number of drones is set based on the size of the fire point.

[0013] In an optional implementation, a reward function is set, and the reward function includes a task coverage reward, a resource effective utilization reward, a task response speed reward, and a task completion rate reward, including: The reward function is:

[0014] w1, w2, w3, and w4 are the weights of the four optimization objectives, with values ​​of 0.3, 0.2, 0.2, and 0.3 respectively; The tasks cover rewards It is used to reward the UAV system for effectively covering the monitoring area and avoiding area duplication. The formula is:

[0015] is the effective monitoring area covered, is the total area that needs to be monitored; Resource efficiency rewards Used to reward drone systems for their efficiency in using resources, including battery power and flight time:

[0016] is the amount of electricity used by the i-th drone, is the total power of the i-th drone, is the flight time of the i-th UAV, is the total flight time of the ith UAV, is the total number of drones; Task Response Speed ​​Reward Used to reward the UAV system for its quick response after receiving the mission command:

[0017] is the expected task response time, is the actual task response time; Mission Completion Rate Rewards Used to reward drone systems for completing monitoring tasks on time and completely:

[0018] is the number of tasks successfully completed, is the total number of tasks assigned.

[0019] In an optional embodiment, based on the state space and the action space, and the reward function, solving the optimal scheduling scheme for executing the drone mission includes: Define heuristic rules and generate candidate solutions based on the heuristic rules; defining a reinforcement learning environment in which the drone selects an action based on the current state and action space and calculates a corresponding reward based on a reward function; Use the candidate solution as the initial policy of the reinforcement learning model, collect data through simulation execution, and use this data to train the reinforcement learning model so that it can learn and optimize the scheduling policy; During the training process, the reinforcement learning model is used to evaluate the performance of each candidate solution, and the optimal scheduling solution is selected based on the evaluation results.

[0020] In a second aspect, the present invention provides a multi-UAV dispatching system, comprising: A state processing module, used to obtain the state information of the drone and construct a state space and an action space based on the state information; An image processing module is used to obtain images collected by the drone, identify and track the fire point from the image, and extract the size, trajectory, and movement speed of the fire point; A task generation module, used to generate a UAV task based on the size, trajectory, and moving speed of the fire point; A reward setting module, used to set a reward function, wherein the reward function includes a task coverage reward, a resource effective utilization reward, a task response speed reward, and a task completion rate reward; A solution solving module is used to solve the optimal scheduling solution for executing the drone mission based on the state space and the action space, and the reward function.

[0021] In a third aspect, a device is provided, including: A memory for storing a multi-UAV scheduling program; A processor is used to implement the steps of the multi-UAV scheduling method provided in the first aspect when executing the multi-UAV scheduling program.

[0022] In a fourth aspect, a computer-readable storage medium is provided, on which a multi-UAV scheduling program is stored. When the multi-UAV scheduling program is executed by a processor, the steps of the multi-UAV scheduling method provided in the first aspect are implemented.

[0023] The beneficial effect of the present invention is that the multi-UAV scheduling method, system, device and storage medium provided by the present invention obtain the state information of the UAV and the cycle image of the UAV, construct the state space and action space, generate the UAV task, and further iterate and optimize the best UAV task execution plan based on the reward function. The present invention better adapts to the complex forest environment and improves the timeliness of the UAV response to forest fires.

[0024] In addition, the invention has a reliable design principle, a simple structure and a very broad application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention.

[0027] Figure 2 The figure is a schematic flow chart of image processing of a method according to an embodiment of the present invention.

[0028] Figure 3 It is a schematic principle diagram of a scheduling scheme generation method according to an embodiment of the present invention.

[0029] Figure 4 It is a schematic flow chart of the generation and execution of a scheduling scheme of a method according to an embodiment of the present invention.

[0030] Figure 5 is a schematic block diagram of a system according to an embodiment of the present invention.

[0031] Figure 6 A schematic diagram of the structure of a device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0034] The key terms appearing in the present invention are explained below.

[0035] YOLO (You Only Look Once) is a real-time target detection algorithm. It is the first to convert the target detection task into a single forward pass regression problem. It is efficient and accurate and has been widely used in many fields including security monitoring. In the fire warning system, the YOLO neural network can identify the fire point. The following is a brief introduction to YOLO's fire point identification: Its working principle includes: YOLO neural network uses convolutional neural network (CNN) to extract features from input images. Regression analysis with loss function is used to classify and locate the extracted features to identify the fire point.

[0036] The advantages of YOLO include: Compared with traditional target detection methods, YOLO has a significant advantage in speed and can process a large amount of image data in a short time. YOLO can accurately identify the fire point and provide its location information. YOLO's real-time performance enables it to issue an alarm at the early stage of a fire, providing strong support for the timely detection and disposal of fires.

[0037] Reinforcement Learning (RL) is an important method in the field of machine learning. It is inspired by behavioral psychology and focuses on how agents take different actions in the environment to maximize cumulative rewards. The following is a detailed introduction to the reinforcement learning model: 1. Basic components The reinforcement learning model mainly consists of the following key elements: Agent: The ontology of reinforcement learning, as a learner or decision maker, analogous to the human brain. The agent is responsible for selecting actions based on the current state of the environment and receiving feedback (rewards or penalties) from the environment.

[0038] Environment: Everything outside the agent, mainly a collection of states, analogous to the human body and the surrounding natural environment. The environment receives the actions of the agent and returns a new state and corresponding rewards.

[0039] State: A data that represents an environment. The state set is all possible states in the environment. The agent chooses an action based on the current state.

[0040] Action: The action taken by the agent. The action set is the set of all possible actions that the agent can take. The agent changes the state of the environment by performing actions.

[0041] Reward: The feedback information obtained by the agent after performing an action is a positive reward (reward) or a negative reward (punishment). The reward is the goal of the agent's learning. The agent learns the best strategy by maximizing the cumulative reward.

[0042] 2. Learning Process The learning process of reinforcement learning is a process of continuous trial and error and optimization. The agent interacts with the environment through the following steps: Perceiving the state of the environment: The agent first perceives the current state of the environment.

[0043] Select action: Based on the current state and a certain policy, the agent selects an action to perform.

[0044] Taking actions: The agent performs the selected action and observes the changes in the state of the environment.

[0045] Receive rewards: The environment returns a new state and corresponding reward based on the agent's actions.

[0046] Update policy: The agent updates its policy based on the rewards it receives in order to better choose actions in the future.

[0047] Monte Carlo Tree Search (MCTS) is a heuristic search algorithm that is particularly suitable for problems with randomness and / or incomplete information. It combines the Monte Carlo method with tree search techniques to evaluate the value of different strategies by simulating possible future scenarios and make decisions based on this. The following is a detailed introduction to Monte Carlo Tree: 1. Basic principles Monte Carlo Tree Search is based on the following basic principles: Tree structure: MCTS uses a search tree to represent all possible decision sequences starting from the current state. Each node of the tree represents a state, and the path from the root node to the leaf node represents a series of decisions.

[0048] Simulation: During the search process, MCTS estimates the future value of each decision sequence through random sampling (i.e. simulation). This is usually done by simulating multiple complete instances of the game or problem.

[0049] Selection: When choosing actions, MCTS uses heuristics to balance exploration (i.e., trying new, not yet fully evaluated decisions) and exploitation (i.e., choosing decisions that are known to have high value).

[0050] Backtracking: After the simulation is completed, MCTS backtracks the results into the search tree and updates the statistics of the nodes (such as the number of visits and estimated value).

[0051] The multi-UAV scheduling method provided in the embodiment of the present invention is executed by a computer device, and accordingly, the multi-UAV scheduling system runs in the computer device.

[0052] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention. Figure 1 The execution subject is a multi-UAV dispatching system. According to different requirements, the order of the steps in the flowchart is changed and some are omitted.

[0053] like Figure 1 As shown, the method includes: S1. Obtain the state information of the drone, and construct a state space and an action space based on the state information.

[0054] Obtain various key status information from the drone, including but not limited to the drone's location, speed, altitude, remaining battery, camera direction, etc. This information is the basis for the drone to perform its mission and is also an important basis for subsequent decision-making.

[0055] Based on this state information, a state space is constructed, which contains all possible states of the drone. At the same time, an action space needs to be constructed, which defines all possible actions that the drone can take, such as forward, backward, ascending, descending, rotating the camera, etc. These two spaces together constitute the basic framework for drone decision-making.

[0056] S2. Obtain images collected by the drone, identify and track the fire point from the image, and extract the size, trajectory, and moving speed of the fire point.

[0057] Drones are usually equipped with high-definition cameras to collect environmental images in real time. In this step, image processing technologies such as target detection and target tracking are needed to accurately identify and track the fire point from the collected images.

[0058] Once the fire point is identified, key information is extracted, including the size of the fire point (usually obtained by calculating the pixel area of ​​the fire point in the image), the trajectory (i.e. the path of the fire point moving over time), and the moving speed (obtained by calculating the distance and time difference between adjacent points in the trajectory). This information is crucial for subsequent task generation and scheduling decisions.

[0059] S3. Generate a drone mission based on the size, trajectory, and moving speed of the fire point.

[0060] A series of specific missions are generated for the drone based on the size, trajectory and speed of the fire. For example, if the fire is small and moving slowly, the drone may need to perform a close-range surveillance mission; if the fire is large and moving quickly, the drone may need to perform tracking and recording missions to provide important information for subsequent fire control and rescue operations.

[0061] These missions should have clear objectives and priorities to ensure that the drone can perform its duties efficiently and accurately.

[0062] S4. Setting a reward function, wherein the reward function includes a task coverage reward, a resource effective utilization reward, a task response speed reward, and a task completion rate reward.

[0063] In order to evaluate the performance of drone missions, a reasonable reward function needs to be set. This function should be able to reflect the various performances of the drone during the mission execution process, including the breadth of mission coverage, effective use of resources, response speed to the mission, and mission completion rate.

[0064] Specifically, the mission coverage reward reflects the extent to which the drone covers the mission area; the resource efficiency utilization reward evaluates whether the drone wastes too many resources (such as power, time, etc.) when performing the mission; the mission response speed reward measures whether the drone can respond quickly after receiving the mission; and the mission completion rate reward reflects the extent to which the drone successfully completes the mission.

[0065] The settings of these reward functions should be adjusted and optimized according to specific application scenarios and task requirements.

[0066] S5. Based on the state space and action space, and the reward function, solve the optimal scheduling plan for executing the drone mission.

[0067] The last step is to find the optimal scheduling solution for executing drone missions. This usually involves complex optimization problems that require the use of algorithms such as reinforcement learning and dynamic programming.

[0068] In this process, the state space, action space and reward function of the drone are used as inputs, and the optimal scheduling solution is found through algorithm iteration and optimization. This solution should be able to maximize the key indicators of the drone, such as mission coverage, resource utilization, mission response speed and mission completion rate.

[0069] The final dispatching plan guides drones to perform tasks efficiently in practical applications, providing strong support for the timely detection and disposal of fires.

[0070] To facilitate understanding of the present invention, the multi-UAV scheduling method provided by the present invention is further described below based on the principle of the multi-UAV scheduling method of the present invention and in combination with the process of scheduling multiple UAVs in the embodiment.

[0071] In an embodiment of the present invention, based on step S1, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0072] Use sensors to obtain the status information of the drone; The formula for defining the state space is as follows:

[0073] in, is the global state at time t, which contains the states of all drones; For the i-th drone, its state Further defined as:

[0074] in, is the position of drone i at time t, expressed as a three-dimensional coordinate ; is the speed of drone i at time t, expressed as a velocity vector ; is the remaining power of drone i at time t; is the current mission status of UAV i; is the coverage area of ​​the monitoring area of ​​UAV i at the current moment; The action space describes the specific operations that each drone can perform in the current state, covering flight action control, throwing actions, and shouting actions. The action space is defined as follows:

[0075] in, is the global action at time t, which includes the actions of all drones; For the i-th drone, its action Defined as:

[0076] in, is the flight control action of UAV i, expressed as the position adjustment at time t, expressed as a three-dimensional vector ; It is the throwing action of drone i, indicating whether the throwing action is performed at the current moment. 0 means no throwing, and 1 means throwing.

[0077] In one embodiment of the present invention, based on step S2, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner, such as Figure 2 As shown in the figure, the system receives and processes signals from drones and other devices to ensure that the system can obtain and analyze monitoring data in real time. The image data collected by the drone is corrected and cropped to convert it into standardized image input for visual algorithms. It ensures that subsequent image recognition and target detection can be performed with optimized image quality. Using the corrected image data, abnormal targets in the forest, i.e., fire points, are detected and identified. This module can quickly identify anomalies and initiate corresponding response mechanisms.

[0078] In a specific example, the cruise video processing method is as follows: S201. Extract image frame sequence from the video stream collected by the drone When drones are performing fire monitoring or rescue missions, they are usually equipped with high-definition cameras to collect real-time video streams from the scene. For subsequent processing and analysis, it is first necessary to extract a series of image frames from this continuous video stream to form an image frame sequence. This step usually involves technologies such as video decoding and frame extraction to ensure that clear and complete image frames can be obtained.

[0079] S202. Extracting potential fire points from the image frame by setting a color threshold When a fire occurs, the flame usually presents specific color characteristics, such as red, orange, etc. Based on this characteristic, the potential fire points are initially screened out from the image frame by setting the color threshold. This step usually involves image processing techniques, such as color space conversion, threshold segmentation, etc., so that possible flame areas can be quickly located.

[0080] S203. Extract a subsequence containing the fire point from the image frame sequence, and input the subsequence into a pre-trained YOLO model to identify the fire point, the fire point location, the fire point size and the confidence level. After initially screening out potential fire points, it is necessary to further confirm whether these areas actually contain flames and obtain more detailed information, such as the specific location, size, and confidence level of the fire points. To this end, subsequences containing these potential fire points are extracted from the image frame sequence and input into the pre-trained YOLO (You Only LookOnce) model.

[0081] YOLO is an advanced target detection algorithm that can simultaneously predict the location, size, and category of multiple targets in a single forward propagation. Through training, the YOLO model learns the characteristics of flames, accurately identifies the fire point in the image, and gives information such as its location, size, and confidence. This information is crucial for subsequent fire monitoring and response.

[0082] S204. Calculate the moving speed and moving direction of the ignition point based on the position difference of the ignition point in adjacent image frames in the subsequence After identifying the fire point, we also need to pay attention to its moving speed and direction in order to better understand the development and spread of the fire. To this end, the moving speed and direction of the fire point are calculated by using the position difference of the fire point in adjacent image frames in the subsequence.

[0083] Specifically, first determine the position coordinates of the fire point in two adjacent frames, then calculate the Euclidean distance or Manhattan distance between them as the distance the fire point moves. Next, determine the moving direction of the fire point by comparing the positional relationship of the fire point in two adjacent frames. Finally, divide the moving distance by the time interval between two adjacent frames to get the moving speed of the fire point.

[0084] Through these steps, the fire point can be accurately identified from the video stream collected by the drone, and key information such as its location, size, confidence, moving speed and direction can be obtained, providing strong support for subsequent fire monitoring and response.

[0085] In an embodiment of the present invention, based on step S3, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0086] After obtaining key information such as the location, speed and direction of the fire, combined with the flight time of the drone, we can predict the location where the drone mission will be performed. This step usually involves algorithms such as spatial geometry and dynamic programming to ensure that we can accurately predict where the drone may arrive at a certain point in the future.

[0087] Specifically, first, based on the current location and movement speed of the fire point, the possible location of the fire point in the future is calculated. Then, combined with the flight speed and time of the drone, the area that the drone can reach at a certain point in the future is determined. Finally, by comparing these two areas, the execution location of the drone mission is predicted.

[0088] This step is crucial for the mission planning and scheduling of drones, because it helps us understand in advance where the drones may arrive, so as to reasonably arrange the flight routes and task allocation of the drones.

[0089] After identifying the fire point and obtaining its confidence, we determine whether the confidence reaches the set threshold. This step usually involves processes such as threshold comparison and decision making to ensure that we can accurately determine the authenticity of the fire point.

[0090] Specifically, the confidence of the identified fire point is compared with the preset threshold. If the confidence is higher than or equal to the threshold, the fire point is considered to be real and corresponding actions are taken; if the confidence is lower than the threshold, we may need to further observe or take other measures to confirm the authenticity of the fire point.

[0091] This step is important to reduce false alarms and missed alarms because it helps us improve the accuracy and reliability of fire monitoring.

[0092] If yes, then the drone mission requires a drone carrying a fire extinguisher, and the number of drones required is set based on the size of the fire point. If the confidence of the fire point reaches the set threshold, the fire point is considered to be real and corresponding fire extinguishing measures need to be taken. In this step, the drone mission is set as a drone carrying fire extinguishing agent, and the number of drones required is set according to the size of the fire point.

[0093] Specifically, the amount of fire extinguishing agent and the number of drones required are evaluated based on the area of ​​the fire and the severity of the fire. If the fire is large or fierce, we may need to dispatch more drones and carry more fire extinguishing agents to deal with it. At the same time, we also need to consider factors such as the flight capability and endurance of the drone to ensure that the drone can successfully complete the mission.

[0094] If not, set the drone mission to require drones to hover and observe, and set the number of drones required based on the size of the fire point. If the confidence of the fire point does not reach the set threshold, we may need to further observe the situation of the fire point. In this step, set the drone mission to hover observation and set the number of drones required according to the size of the fire point.

[0095] Specifically, a certain number of drones are dispatched to hover near the fire point to obtain more information and data. Through continuous observation and analysis, the authenticity of the fire point can be further confirmed, and corresponding measures can be taken to deal with it.

[0096] Through these steps, drone missions can be reasonably planned based on information such as the location, movement speed, direction, and confidence level of the fire point, which can not only improve the accuracy and efficiency of fire monitoring and response, but also provide strong support for subsequent firefighting and rescue operations.

[0097] In an embodiment of the present invention, based on step S4, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0098] The reward function includes task coverage reward, resource effective utilization reward, task response speed reward, and task completion rate reward, including: The reward function is:

[0099] w1, w2, w3, and w4 are the weights of the four optimization objectives, with values ​​of 0.3, 0.2, 0.2, and 0.3 respectively; The tasks cover rewards It is used to reward the UAV system for effectively covering the monitoring area and avoiding area duplication. The formula is:

[0100] is the effective monitoring area covered, is the total area that needs to be monitored; Resource efficiency rewards Used to reward drone systems for their efficiency in using resources, including battery power and flight time:

[0101] is the amount of electricity used by the i-th drone, is the total power of the i-th drone, is the flight time of the i-th UAV, is the total flight time of the ith UAV, is the total number of drones; Task Response Speed ​​Reward Used to reward the UAV system for its quick response after receiving the mission command:

[0102] is the expected task response time, is the actual task response time; Mission Completion Rate Rewards Used to reward drone systems for completing monitoring tasks on time and completely:

[0103] is the number of tasks successfully completed, is the total number of tasks assigned.

[0104] In an embodiment of the present invention, based on step S5, a possible embodiment is given below to illustrate its specific implementation in a non-limiting manner.

[0105] Specifically, the present invention uses MADDPG (Multi-Agent Deep Deterministic Policy Gradient) to solve the optimal scheduling solution. The MADDPG algorithm achieves optimized scheduling through the Actor-Critic architecture. Each drone (agent) has an independent Actor network, which is responsible for generating actions based on the current state, and jointly evaluates the actions of all drones through the Critic network to ensure the optimal scheduling strategy. Figure 3 As shown, the specific process is as follows: initialization: Initialize the Actor network for each agent (π i ) and Critic network (Q i ), and their target networks (π' i and Q' i ).

[0106] Initialize the experience replay buffer D.

[0107] Interaction and storage experience: The agents interact with the environment. At each time t, each agent acts according to its strategy π i and state S i Observation i , select action a i .

[0108] Execute all agents’ actions [a1, ...,a N ], observe the new state S' and reward R.

[0109] For each agent i, transform (t = (o i , a i , R i , o' i )) is stored in the experience replay buffer D.

[0110] Sampling and updating: Randomly sample a batch of transitions from the experience replay buffer D.

[0111] For each sampled transformation, a target Q value is computed using the target network.

[0112] Update the Critic network Qi to minimize the loss: L = (Q i (o i , a i ) - target Q)^2.

[0113] Update the Actor network πi using the policy gradient method.

[0114] Soft update target network parameters: π' i ←τπ i + (1 - τ)π' i , Q' i ←τQ i + (1 - τ)Q' i .

[0115] repeat: The interaction and updating process is repeated until the training is completed.

[0116] Please refer to Figure 4 ,After getting the scheduling plan, the basic flight operations of the UAV are controlled based on the scheduling plan, including take-off, hovering, landing and path planning, etc., to ensure that the UAV can complete the flight mission safely and accurately according to the scheduled mission requirements.,Throwing action control, according to the mission requirements, the UAV can perform specific throwing operations, such as dropping fire extinguishing bombs, to respond to sudden forest fires or other emergencies.

[0117] On the basis of the above embodiments, in order to further improve the reliability of solving the optimal scheduling solution provided by the above embodiments, an implementable method is to seek the optimal scheduling solution by combining the reinforcement learning environment and the Monte Carlo tree.

[0118] Heuristics: Prioritize drones based on the size and urgency of the fire (e.g., flame spread and fire intensity). Prioritize the drones closest to the fire. Consider the flight speed and endurance of the drones to ensure they can complete the mission and return to base.

[0119] Candidate solution generation: Based on the heuristic rules, a set of candidate UAV task scheduling solutions are generated. Each solution includes information such as the take-off order, flight path, and task allocation of the UAVs.

[0120] Environment setting: When the drone is performing a mission, its current status includes its location, remaining battery power, and the amount of fire extinguishing agent it carries. The action space includes taking off, flying to a designated location, hovering for observation, spraying fire extinguishing agent, and returning to base.

[0121] Reward function: The reward function combines the task coverage reward, resource effective utilization reward, task response speed reward and task completion rate reward. In specific implementation, the weight coefficients w1, w2, w3, w4 can be adjusted according to the actual situation. In this embodiment, the reward function of step S4 is adopted.

[0122] The generated candidate solutions are used as the initial strategy of the reinforcement learning model, and data is collected through simulation execution. During the simulation, the drone selects actions based on the current state and action space, and calculates the corresponding rewards based on the reward function.

[0123] The collected data is used to train a reinforcement learning model, which enables it to learn and optimize the scheduling strategy. During the training process, the model will continuously try different actions and adjust its strategy based on reward feedback.

[0124] During the training process, the reinforcement learning model is used to evaluate the performance of each candidate solution. Through simulation execution, the efficiency, resource utilization and other indicators of different candidate solutions in completing the task are compared.

[0125] In another embodiment of the present invention, the Monte Carlo tree is used to further optimize the results output by the reinforcement learning model.

[0126] Build a Monte Carlo tree where each node represents a state and each branch represents an action. Starting from the current state, simulate future states and evaluate the expected reward for each action.

[0127] In the Monte Carlo tree, the action with the highest expected reward is selected as the current optimal action. The Monte Carlo tree is continuously updated iteratively until the predetermined number of iterations is reached or other stopping conditions are met.

[0128] The optimal actions obtained from each Monte Carlo tree update are integrated into the optimal scheduling solution, which takes into account factors such as the take-off order, flight path, and task allocation of drones to ensure efficient execution of tasks.

[0129] Execute drone missions based on the optimal scheduling plan. During the execution of the mission, new status information and mission execution results are continuously collected. Based on this information, the parameters of the reinforcement learning model and Monte Carlo tree search are updated in real time to adjust the optimal scheduling plan.

[0130] In some embodiments, the multi-drone dispatching system includes a plurality of functional modules composed of computer program segments. The computer programs of each program segment in the multi-drone dispatching system are stored in the memory of the computer device and executed by at least one processor to perform (see Figure 1 Description) Function of multi-UAV scheduling.

[0131] In this embodiment, the multi-UAV dispatching system is divided into multiple functional modules according to the functions it performs, such as Figure 5 As shown. The functional modules of the system 500 include: a state processing module 510, an image processing module 520, a task generation module 530, a reward setting module 540 and a solution solving module 550. The module referred to in the present invention refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0132] A state processing module, used to obtain the state information of the drone and construct a state space and an action space based on the state information; An image processing module is used to obtain images collected by the drone, identify and track the fire point from the image, and extract the size, trajectory, and movement speed of the fire point; A task generation module, used to generate a UAV task based on the size, trajectory, and moving speed of the fire point; A reward setting module, used to set a reward function, wherein the reward function includes a task coverage reward, a resource effective utilization reward, a task response speed reward, and a task completion rate reward; A solution solving module is used to solve the optimal scheduling solution for executing the drone mission based on the state space and the action space, and the reward function.

[0133] Optionally, as an embodiment of the present invention, the state processing module includes: Use sensors to obtain the status information of the drone; The formula for defining the state space is as follows:

[0134] in, is the global state at time t, which contains the states of all drones; For the i-th drone, its state Further defined as:

[0135] in, is the position of drone i at time t, expressed as a three-dimensional coordinate ; is the speed of drone i at time t, expressed as a velocity vector ; is the remaining power of drone i at time t; is the current mission status of UAV i; is the coverage area of ​​the monitoring area of ​​UAV i at the current moment; The action space describes the specific operations that each drone can perform in the current state, covering flight action control, throwing actions, and shouting actions. The action space is defined as follows:

[0136] in, is the global action at time t, which includes the actions of all drones; For the i-th drone, its action Defined as:

[0137] in, is the flight control action of UAV i, expressed as the position adjustment at time t, expressed as a three-dimensional vector ; It is the throwing action of drone i, indicating whether the throwing action is performed at the current moment. 0 means no throwing, and 1 means throwing.

[0138] Optionally, as an embodiment of the present invention, the image processing module includes: Extracting image frame sequences from video streams captured by drones; Extracting potential fire points from image frames by setting color thresholds; Extracting a subsequence containing a fire point from the image frame sequence, and inputting the subsequence into a pre-trained YOLO model to identify the fire point, the location of the fire point, the size of the fire point, and the confidence level; Based on the position difference of the ignition point between adjacent image frames in the subsequence, the moving speed and moving direction of the ignition point are calculated.

[0139] Optionally, as an embodiment of the present invention, the task generation module includes: Predict where the drone mission will be based on the location of the fire, its movement speed, direction, and the drone’s flight time; Determine whether the confidence level reaches a set threshold: If so, setting the drone mission requires drones carrying fire extinguishing agents, and setting the number of drones required based on the size of the fire point; If not, the drone mission is set to require the drone to hover for observation, and the required number of drones is set based on the size of the fire point.

[0140] Optionally, as an embodiment of the present invention, the reward setting module includes: The reward function is:

[0141] w1, w2, w3, and w4 are the weights of the four optimization objectives, with values ​​of 0.3, 0.2, 0.2, and 0.3 respectively; The tasks cover rewards It is used to reward the UAV system for effectively covering the monitoring area and avoiding area duplication. The formula is:

[0142] is the effective monitoring area covered, is the total area that needs to be monitored; Resource efficiency rewards Used to reward drone systems for their efficiency in using resources, including battery power and flight time:

[0143] is the amount of electricity used by the i-th drone, is the total power of the i-th drone, is the flight time of the i-th UAV, is the total flight time of the ith UAV, is the total number of drones; Task Response Speed ​​Reward Used to reward the UAV system for its quick response after receiving the mission command:

[0144] is the expected task response time, is the actual task response time; Mission Completion Rate Rewards Used to reward drone systems for completing monitoring tasks on time and completely:

[0145] is the number of tasks successfully completed, is the total number of tasks assigned.

[0146] Optionally, as an embodiment of the present invention, the solution solving module includes: Define heuristic rules and generate candidate solutions based on the heuristic rules; defining a reinforcement learning environment in which the drone selects an action based on the current state and action space and calculates a corresponding reward based on a reward function; Use the candidate solution as the initial policy of the reinforcement learning model, collect data through simulation execution, and use this data to train the reinforcement learning model so that it can learn and optimize the scheduling policy; During the training process, the reinforcement learning model is used to evaluate the performance of each candidate solution, and the best candidate solution is selected based on the evaluation results; Build a Monte Carlo tree where each node represents a state and each branch represents an action; Starting from the current state, simulate future states and evaluate the expected reward of each action, and choose the action with the highest expected reward as the current optimal action; The Monte Carlo tree is continuously updated iteratively until the predetermined number of iterations is reached or other stopping conditions are met; Integrate the optimal actions obtained from each update of the Monte Carlo tree into the optimal scheduling solution; Execute the UAV mission based on the optimal scheduling solution, and continuously collect new status information and mission execution results during the execution of the mission; The parameters of the reinforcement learning model and the Monte Carlo tree search are updated in real time according to this information to adjust the optimal scheduling solution.

[0147] Figure 6 The multi-UAV scheduling method provided for the embodiment of the present application is applied to the device. Those skilled in the art understand that the device structure involved in the embodiment of the present invention does not constitute a limitation on the device, and the device includes more or fewer components than shown in the figure, or combines certain components, or arranges different components. In an embodiment of the present invention, the device includes but is not limited to a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The device also represents various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present application described and / or required herein.

[0148] The device 300 includes: a processor 310, a memory 320 and a communication unit 330. These components communicate through one or more buses. Those skilled in the art understand that the structure of the server shown in the figure does not constitute a limitation of the present invention. It is both a bus structure and a star structure, and also includes more or less components than shown in the figure, or combines certain components, or arranges components differently.

[0149] The memory 320 is used to store the execution instructions of the processor 310. The memory 320 is implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the device 300 is enabled to perform some or all of the steps in the following method embodiments.

[0150] The processor 310 is the control center of the storage device, and uses various interfaces and lines to connect various parts of the entire electronic device. It runs or executes software programs and / or modules stored in the memory 320, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor is composed of an integrated circuit (IC), such as a single packaged IC, or a plurality of packaged ICs with the same or different functions. For example, the processor 310 only includes a central processing unit (CPU). In the embodiment of the present invention, the CPU is a single computing core, and also includes multiple computing cores.

[0151] The communication unit 330 is used to establish a communication channel so that the storage device can communicate with other devices, receive user data sent by other devices or send user data to other devices.

[0152] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program may include some or all of the steps in each embodiment provided by the present invention. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0153] Those skilled in the art clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or partly contributed to the prior art in the form of a software product, which is stored in a storage medium such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc., which stores program codes, and includes several instructions for enabling a computer device (a personal computer, a server, or a second device, a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present invention.

[0154] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.

[0155] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods are implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there are other division methods in actual implementation, such as combining or integrating multiple modules or components into another system, or ignoring or not executing some features. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed is an indirect coupling or communication connection of some interfaces, systems or modules, which is electrical, mechanical or other forms.

[0156] The modules described as separate components are or are not physically separated, and the components shown as modules are or are not physical modules, that is, located in one place, or distributed on multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment.

[0157] In addition, each functional module in each embodiment of the present invention is integrated into one processing module, each module exists physically separately, or two or more modules are integrated into one module.

[0158] Although the present invention has been described in detail by referring to the accompanying drawings and in combination with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person skilled in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions shall be within the scope of the present invention. Any person skilled in the art who is familiar with the present invention may easily conceive of changes or substitutions within the technical scope disclosed by the present invention, and these shall be within the scope of protection of the present invention.

Claims

1. A multi-UAV scheduling method, characterized in that: include: Acquire state information of the drone, and construct a state space and an action space based on the state information; Obtain images collected by drones, and identify and track fire points from the images, and extract the size, trajectory, and movement speed of the fire points; Generate a drone mission based on the size, trajectory, and movement speed of the fire point; Setting a reward function, wherein the reward function includes a task coverage reward, a resource effective utilization reward, a task response speed reward, and a task completion rate reward; Based on the state space and action space, and the reward function, an optimal scheduling solution for executing the drone mission is solved.

2. The method according to claim 1, characterized in that Obtain the state information of the drone, and construct the state space and action space based on the state information, including: Use sensors to obtain the status information of the drone; The formula for defining the state space is as follows: in, is the global state at time t, which contains the states of all drones; For the i-th drone, its state Further defined as: in, is the position of drone i at time t, expressed as a three-dimensional coordinate ; is the speed of drone i at time t, expressed as a velocity vector ; is the remaining power of drone i at time t; is the current mission status of UAV i; is the coverage area of ​​the monitoring area of ​​UAV i at the current moment; The action space describes the specific operations that each drone can perform in the current state, covering flight action control, throwing actions, and shouting actions. The action space is defined as follows: in, is the global action at time t, which includes the actions of all drones; For the i-th drone, its action Defined as: in, is the flight control action of UAV i, expressed as the position adjustment at time t, expressed as a three-dimensional vector ; It is the throwing action of drone i, indicating whether the throwing action is performed at the current moment. 0 means no throwing, and 1 means throwing.

3. The method according to claim 1, characterized in that Obtain images collected by drones, identify and track the fire point from the images, and extract the size, trajectory, and movement speed of the fire point, including: Extracting image frame sequences from video streams captured by drones; Extracting potential fire points from image frames by setting color thresholds; Extracting a subsequence containing a fire point from the image frame sequence, and inputting the subsequence into a pre-trained YOLO model to identify the fire point, the location of the fire point, the size of the fire point, and the confidence level; Based on the position difference of the ignition point between adjacent image frames in the subsequence, the moving speed and moving direction of the ignition point are calculated.

4. The method according to claim 3, characterized in that Based on the size, trajectory, and moving speed of the fire point, a drone mission is generated, including: Predict where the drone mission will be based on the location of the fire, its movement speed, direction, and the drone’s flight time; Determine whether the confidence level reaches a set threshold: If so, setting the drone mission requires drones carrying fire extinguishing agents, and setting the number of drones required based on the size of the fire point; If not, the drone mission is set to require the drone to hover for observation, and the required number of drones is set based on the size of the fire point.

5. The method according to claim 1, characterized in that A reward function is set, which includes a task coverage reward, a resource effective utilization reward, a task response speed reward, and a task completion rate reward, including: The reward function is: w1, w2, w3, and w4 are the weights of the four optimization objectives, with values ​​of 0.3, 0.2, 0.2, and 0.3 respectively; The tasks cover rewards It is used to reward the UAV system for effectively covering the monitoring area and avoiding area duplication. The formula is: is the effective monitoring area covered, is the total area that needs to be monitored; Resource efficiency rewards Used to reward drone systems for their efficiency in using resources, including battery power and flight time: is the amount of electricity used by the i-th drone, is the total power of the i-th drone, is the flight time of the ith UAV, is the total flight time of the ith UAV, is the total number of drones; Task response speed reward Used to reward the UAV system for its quick response after receiving the mission command: is the expected task response time, is the actual task response time; Mission Completion Rate Rewards Used to reward drone systems for completing monitoring tasks on time and completely: is the number of tasks successfully completed, is the total number of tasks assigned.

6. The method according to claim 1, characterized in that Based on the state space and the action space, and the reward function, solving the optimal scheduling scheme for executing the UAV mission includes: Define heuristic rules and generate candidate solutions based on the heuristic rules; defining a reinforcement learning environment in which the drone selects an action based on the current state and action space and calculates a corresponding reward based on a reward function; Use the candidate solution as the initial policy of the reinforcement learning model, collect data through simulation execution, and use this data to train the reinforcement learning model so that it can learn and optimize the scheduling policy; During the training process, the reinforcement learning model is used to evaluate the performance of each candidate solution, and the optimal scheduling solution is selected based on the evaluation results.

7. A multi-UAV dispatching system, characterized in that: include: A state processing module, used to obtain the state information of the drone and construct a state space and an action space based on the state information; An image processing module is used to obtain images collected by the drone, identify and track the fire point from the image, and extract the size, trajectory, and movement speed of the fire point; A task generation module, used to generate a UAV task based on the size, trajectory, and moving speed of the fire point; A reward setting module, used to set a reward function, wherein the reward function includes a task coverage reward, a resource effective utilization reward, a task response speed reward, and a task completion rate reward; A solution solving module is used to solve the optimal scheduling solution for executing the drone mission based on the state space and the action space, and the reward function.

8. The system according to claim 7, characterized in that The state processing module comprises: Use sensors to obtain the status information of the drone; The formula for defining the state space is as follows: in, is the global state at time t, which contains the states of all drones; For the i-th drone, its state Further defined as: in, is the position of drone i at time t, expressed as a three-dimensional coordinate ; is the speed of drone i at time t, expressed as a velocity vector ; is the remaining power of drone i at time t; is the current mission status of UAV i; is the coverage area of ​​the monitoring area of ​​UAV i at the current moment; The action space describes the specific operations that each drone can perform in the current state, covering flight action control, throwing actions, and shouting actions. The action space is defined as follows: in, is the global action at time t, which includes the actions of all drones; For the i-th drone, its action Defined as: in, is the flight control action of UAV i, expressed as the position adjustment at time t, expressed as a three-dimensional vector ; It is the throwing action of drone i, indicating whether the throwing action is performed at the current moment. 0 means no throwing, and 1 means throwing.

9. A device, characterized in that: include: A memory for storing a multi-UAV scheduling program; A processor, configured to implement the steps of the multi-UAV scheduling method as described in any one of claims 1 to 6 when executing the multi-UAV scheduling program.

10. A computer-readable storage medium storing a computer program, characterized in that: The readable storage medium stores a multi-UAV scheduling program, which, when executed by a processor, implements the steps of the multi-UAV scheduling method according to any one of claims 1 to 6.

Images