Multi-unmanned aerial vehicle collaborative search scheduling method based on deep reinforcement learning

Abstract

The invention discloses a multi-unmanned aerial vehicle collaborative search scheduling method based on deep reinforcement learning. The method is oriented to a complex environment with dynamic tasks and space risk constraints. According to the method, a hierarchical collaborative decision framework is constructed, system-level task scheduling and individual-level continuous control are decoupled and modeled, a graph structure and an attention mechanism are introduced into a scheduling layer, and global optimization distribution of multiple unmanned aerial vehicles and multiple tasks is realized; in a control layer, a deep reinforcement learning strategy is adopted to generate a continuous control action based on local observation, and the coordination of scheduling and execution is improved through joint training. The method can significantly improve the task completion rate and the emergency response efficiency, optimizes the energy utilization, reduces the system oscillation, and is suitable for disaster monitoring, emergency rescue and other multi-unmanned aerial vehicle cooperative operation scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle cluster cooperative control and intelligent decision-making, and particularly relates to a multi-unmanned aerial vehicle cooperative search scheduling method based on deep reinforcement learning. BACKGROUND

[0002] With the deepening application of multi-unmanned aerial vehicle cooperative search technology in disaster monitoring, emergency rescue and complex scene situation awareness, the problem of group intelligence cooperation in large-scale and high-dynamic environment has gradually become an important direction of unmanned system research. Compared with single unmanned aerial vehicle system, multi-unmanned aerial vehicle cluster can significantly improve search coverage efficiency and task response ability through parallel search, information sharing and task division, especially in forest fire, earthquake rescue and other scenes with large spatial scale and strong environmental uncertainty. However, such application scenarios usually have the characteristics of random task generation, dynamic evolution of risk distribution and limited communication conditions, which puts forward higher requirements for the cooperative decision-making ability and system robustness of unmanned aerial vehicle cluster.

[0003] In view of the above requirements, existing researches have carried out a lot of explorations around multi-unmanned aerial vehicle task allocation, path planning and multi-agent reinforcement learning. At the task allocation level, centralized optimization method and heuristic algorithm can obtain better solution in static or quasi-static environment, but they often rely on complete global information and are difficult to adapt to the actual scene with frequent task changes and limited communication. At the control level, the method based on multi-agent reinforcement learning learns individual strategy through environment interaction, to some extent, it alleviates the deficiency of model dependence and artificial design rules, and promotes the development of distributed cooperative control framework. However, most of the existing methods usually assume that the task set is relatively stable, the risk factors can be ignored or the environment changes slowly, and it is difficult to effectively cope with the complex situations such as continuous task generation, priority change over time and rapid risk diffusion in real disaster scenes.

[0004] Further, in the rapidly evolving high-risk environment such as forest fires, the task often presents obvious time sensitivity and spatial correlation, and different types of tasks require different capabilities of unmanned aerial vehicles; at the same time, due to the differences in role division, residual energy, motion performance and communication range between unmanned aerial vehicles, significant heterogeneous collaborative characteristics are formed. This makes the multi-unmanned aerial vehicle cooperative search and task scheduling problem present high-dimensional coupling, nonlinear dynamics and hierarchical dependence: on the one hand, a single centralized scheduling strategy is difficult to guarantee real-time while taking into account system size and robustness; on the other hand, a completely distributed reinforcement learning control lacks a global coordination mechanism, which can easily lead to low resource matching efficiency, frequent task conflicts and unstable strategy convergence. Therefore, a multi-unmanned aerial vehicle cooperative search and task scheduling method is needed, which can simultaneously describe global task scheduling and individual behavior optimization, and take into account the adaptability of dynamic environment and the stability of cooperation, in order to improve the overall performance and reliability of the system in complex and high-risk scenarios. SUMMARY

[0005] The purpose of the present application is to solve the problems of insufficient global coordination ability, task allocation and individual control fragmentation, low heterogeneous unmanned aerial vehicle resource matching efficiency, and poor strategy stability in high-risk scenarios in the existing multi-unmanned aerial vehicle cooperative search system. A multi-unmanned aerial vehicle cooperative search scheduling method based on deep reinforcement learning is proposed, which builds a collaborative control framework combining hierarchical decision-making and centralized training and distributed execution, and realizes the joint optimization of dynamic task scheduling, path planning and risk avoidance in a unified model, thereby improving the search efficiency, cooperative stability and overall task completion ability of the multi-unmanned aerial vehicle system in complex disaster scenarios.

[0006] Technical scheme: In order to achieve the above-mentioned purpose of the application, the following technical scheme is adopted:

[0007] In the first aspect, the present application provides a multi-unmanned aerial vehicle cooperative search scheduling method based on deep reinforcement learning, comprising the following steps:

[0008] First, the environment state information, unmanned aerial vehicle cluster state information, dynamic task set information and spatial risk distribution information in the multi-unmanned aerial vehicle cooperative search scene are obtained, and a unified multi-unmanned aerial vehicle cooperative search environment model is constructed therefrom. The environment model includes multiple unmanned aerial vehicles, dynamically generated task nodes and risk distribution representing spatial safety, wherein the unmanned aerial vehicle state information at least includes position, speed, residual energy and role capability attributes, the task information at least includes task position, task type, priority and generation time, and the risk distribution is used to describe the potential danger degree of different spatial regions, thereby providing a complete state description for subsequent collaborative decision-making.

[0009] Further, the multi-UAV cooperative search and task scheduling problem is formalized as a multi-agent Markov decision process, and a hierarchical cooperative decision-making model is constructed under the centralized training and distributed execution framework. In the scheduling layer, a centralized task scheduling network based on deep reinforcement learning is introduced, which takes the global environment state as input and generates dynamic task allocation decisions by learning the matching relationship between UAVs and tasks. This allows the system to adaptively adjust task assignments based on changes in task priority, UAV resource state, and overall load, achieving global-level cooperative optimization.

[0010] Further, in the execution layer, an individual control policy network based on local observations is constructed for each UAV, which includes at least the UAV's own state, neighborhood task information, neighboring UAV state, and local risk perception results. By encoding the above local information, the individual control policy network outputs continuous control actions to guide the UAV in path planning and real-time risk avoidance during search and task execution, thus completing distributed cooperative execution without relying on global information.

[0011] Further, to enhance the ability of the multi-UAV system to describe group cooperation, the invention introduces a global feature aggregation mechanism during scheduling and execution, which aggregates the state and task distribution information of multiple UAVs to construct global cooperative features reflecting the overall load, spatial coverage, and task density of the system. These global features assist the scheduling layer in decision-making and complement the individual layer's local features, allowing the system to capture the coupling between local behavior and global situation.

[0012] Further, to improve the stability of deep reinforcement learning models in multi-layer decision-making and long-term training, the invention introduces joint reward modeling and additive residual structures during policy network updates. By superimposing new policy update results on the basis of preserving historical decision information, the invention alleviates the common problems of non-stationarity, gradient vanishing, and policy oscillation in multi-agent learning, thereby improving the convergence speed and stability of cooperative strategies.

[0013] Further, in the policy output stage, the scheduling layer generates discrete task allocation decisions to determine the execution UAVs for each dynamic task, and the execution layer outputs continuous control actions to constrain the flight direction and speed of the UAVs. These actions are then normalized and clipped based on physical constraints such as energy consumption, risk avoidance, and task progress to ensure that the policy output can be directly executed in the actual UAV system.

[0014] Further, to avoid relying on artificial rules or static planning models, the application unifies factors such as task completion efficiency, risk cost, energy consumption, and cooperative stability into the reward function design through the construction of an end-to-end reinforcement learning training mechanism. Through continuous interaction with the environment, the system gradually learns the optimal cooperative strategy under unsupervised conditions, thereby realizing the joint optimization of dynamic task scheduling and path control.

[0015] Further, to enhance the adaptability of the method under different environmental scales, task densities, and risk distribution conditions, the application introduces a strategy generalization mechanism based on environmental disturbance in the training phase. By randomly adjusting the task generation position, disturbing the risk distribution form, changing the initial deployment of unmanned aerial vehicles, or simulating communication restricted scenarios, the model experiences diverse cooperative search situations during training, thereby improving the generalization ability and robustness of the learned strategy in real complex scenarios.

[0016] In a second aspect, the application provides a device for a multi-unmanned aerial vehicle cooperative search scheduling method based on deep reinforcement learning, comprising:

[0017] A scene modeling and risk field generation module is configured to establish a multi-unmanned aerial vehicle cooperative search model containing multiple unmanned aerial vehicles, a dynamic task pool, and a risk distribution environment, divide the search area into grids, deploy risk sources, and configure task generation rules, construct a search scenario with spatial heterogeneity, task randomness, and environmental dynamics, and form an environmental state set based on this to describe the global situation.

[0018] A central scheduling and graph structure construction module is configured to receive global state information output by the scene modeling and risk field generation module, construct a bipartite graph structure composed of unmanned aerial vehicle nodes and task nodes, introduce a graph attention network to jointly model unmanned aerial vehicle capabilities, task priorities, spatial distances, accessibility constraints, and resource competition relationships, output matching relationships and assignment results between multiple unmanned aerial vehicles and multiple tasks, and realize globally optimized task scheduling.

[0019] An unmanned aerial vehicle observation processing and state coding module is configured to collect local observation information for each unmanned aerial vehicle, including position, velocity, remaining energy, neighbor unmanned aerial vehicle state, adjacent task attribute, and local risk patch, normalize, splice, and encode the above information, generate a fixed-dimensional local observation vector, and form a state input for the deep reinforcement learning control strategy call in combination with the central scheduling result.

[0020] The deep reinforcement learning training module is used for constructing a multi-agent control network based on an actor-critic framework, adopting a global or centralized critic network and a distributed actor network structure to train a continuous action policy of the unmanned aerial vehicle; and through defining a composite reward function including a task progress, a task completion, a priority response, an energy consumption penalty, a risk penalty and a redistribution penalty, and combining experience replay, target network updating and a federal parameter sharing mechanism, the unmanned aerial vehicle policy is iteratively optimized.

[0021] The cooperative control and online execution module is used for calling the converged strategy network obtained by the deep reinforcement learning training module in an execution phase, so that each unmanned aerial vehicle generates a dynamic trajectory and an action decision according to a current local observation and a central scheduling result, and realizes path planning, risk avoidance and task approaching; meanwhile, in response to environmental changes and new task generation requests, the central scheduling and the control strategy are linked to complete an efficient search, a reasonable allocation and a cooperative execution process of the multiple unmanned aerial vehicles in a dynamic task scene.

[0022] In a third aspect, the present application provides a computer system, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the multi-unmanned aerial vehicle cooperative search scheduling method based on deep reinforcement learning.

[0023] In a fourth aspect, the present application provides a computer program product, comprising a computer program, wherein the computer program, when executed by a processor, implements the steps of the multi-unmanned aerial vehicle cooperative search scheduling method based on deep reinforcement learning.

[0024] Beneficial effects: The present application introduces a hierarchical collaborative decision-making framework based on deep reinforcement learning, which uniformly models the task allocation relationship and individual control behavior in the multi-UAV system, enabling the system to effectively learn the collaborative strategy of the UAV cluster under the conditions of dynamic task generation and continuous environmental change, avoiding the dependence of traditional centralized planning methods on complete global information and static environment assumptions. By combining global task situation with local UAV state, the present application can adaptively match dynamic task priority and heterogeneous UAV capabilities under the conditions of limited communication and resource constraints, thereby improving the rationality and execution efficiency of overall task scheduling. By introducing a global feature aggregation mechanism and joint reward modeling method in the collaborative decision-making process, the present application can maintain a stable strategy learning process in complex multi-UAV distribution and high-coupling decision-making scenarios, more fully depicting the influence of collaborative relationships, resource competition and spatial risk distribution between UAVs on decision-making behavior, thereby effectively alleviating the common non-stationarity and strategy oscillation problems in multi-agent reinforcement learning, improving the convergence stability and long-term execution reliability of the collaborative strategy. At the execution level, the present application introduces energy constraints and risk avoidance constraints to UAV control actions, enabling the generated control strategy to be directly applied to the actual UAV system, reducing the exposure of UAVs in high-risk areas while ensuring task completion, and improving the safety and robustness of the system in disaster scenarios. Compared with the prior art, the present application reduces the dependence on artificial rule design and static models, improves the adaptability of the multi-UAV system under different task densities, environmental scales and risk evolution conditions, and is beneficial to improving the deployability and overall performance of the multi-UAV collaborative search system in complex scenarios such as disaster monitoring and emergency rescue. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 A multi-UAV collaborative search framework in an embodiment of the present application is shown in the figure.

[0026] Figure 2 A multi-UAV dynamic collaborative search overall algorithm framework in an embodiment of the present application is shown in the figure.

[0027] Figure 3 A comparison of task completion rates of different methods in a multi-UAV collaborative search scenario in an embodiment of the present application is shown in the figure.

[0028] Figure 4 A comparison of average response times of different methods in an emergency task scenario in an embodiment of the present application is shown in the figure.

[0029] Figure 5 A comparison of residual energy ratios of different methods in a multi-UAV collaborative search process in an embodiment of the present application is shown in the figure.

[0030] Figure 6A comparison diagram of task reassignment times of different methods in the multi-unmanned aerial vehicle cooperative search process in the embodiments of the present application. DETAILED DESCRIPTION

[0031] In order to make the technical problems to be solved by the present application, technical solutions and beneficial effects clearer, the present application will be further described in detail below with reference to the drawings and embodiments.

[0032] In Figure 1 , a multi-unmanned aerial vehicle cooperative search and task scheduling framework diagram provided by an embodiment of the present application is shown. The framework is used to describe the overall processing flow of cooperative search and task execution of a multi-unmanned aerial vehicle system in a dynamic task and risk environment through hierarchical decision-making and deep reinforcement learning. The framework mainly includes a multi-unmanned aerial vehicle cluster, a dynamic task environment, a risk assessment module, and a cooperative decision-making module based on deep reinforcement learning, and the modules interact with each other through state information and decision results to jointly constitute a complete cooperative search system.

[0033] Specifically, the system first constructs a multi-unmanned aerial vehicle cooperative search environment, spatially models the search area, and dynamically generates tasks with different locations, types, and priority attributes in the environment. At the same time, a risk modeling mechanism is introduced to depict the potential danger level in the search area, which is used to reflect the characteristics of the risk changing with time and space in disaster scenarios such as forest fires. Multiple unmanned aerial vehicles are deployed in the environment, each unmanned aerial vehicle has corresponding attributes such as location, energy state, role capability, and communication range, which are used to perform subsequent search and task response operations.

[0034] At the cooperative decision-making level, the present application adopts a hierarchical decision-making structure to divide the functions of task scheduling and individual control. Among them, the scheduling layer generates task allocation decisions based on global environment state information, including unmanned aerial vehicle state set, task set and risk distribution information, to determine the execution unmanned aerial vehicle corresponding to each dynamic task; the control layer outputs continuous control actions based on the local observation information of the unmanned aerial vehicle itself, to guide the unmanned aerial vehicle to complete path planning, target approach and risk avoidance operations. Through the above hierarchical structure, the coordination and unity between global cooperation and individual autonomy are realized.

[0035] In the training phase, the present application trains the scheduling layer and control layer strategies jointly or in stages based on the reinforcement learning framework of centralized training and distributed execution. Through continuous interaction with the environment, the system updates the strategies according to the task completion, risk cost, energy consumption, and cooperative stability, so that the cooperative decision-making model gradually learns the cooperative search strategy that adapts to the dynamic task and complex environmental changes. In the execution phase, each unmanned aerial vehicle makes control decisions independently according to the trained strategy, thereby completing cooperative search and task execution without relying on global communication.

[0036] Through the above multi-unmanned aerial vehicle cooperative search framework, the application can realize stable group cooperation in a dynamic, multi-task and high-risk environment, improve task response efficiency and search coverage capability, and provide a feasible technical implementation scheme for application scenarios such as disaster monitoring and emergency rescue.

[0037] In Figure 2 , the multi-unmanned aerial vehicle dynamic cooperative search overall algorithm framework described in the embodiment of the application is shown, realizing a closed-loop process from environment state construction and dynamic task generation, to scheduling layer task assignment, control layer continuous action decision, and joint training and performance evaluation.

[0038] The following will be described in combination with Figure 2 The multi-unmanned aerial vehicle cooperative search scheduling method based on deep reinforcement learning described in the embodiment of the application is further described in detail. Specifically, the method comprises the following steps:

[0039] Step (1), a multi-unmanned aerial vehicle-dynamic task-risk distribution environment model is established, and a state representation for the scheduling layer and the control layer is constructed; comprising the following specific steps:

[0040] (1a) Establish a spatial environment model for multi-unmanned aerial vehicle cooperative search. In the embodiment of the application, a two-dimensional search area is constructed as a working space for multi-unmanned aerial vehicles to perform cooperative search and task response, and the search area is denoted as

[0041]

[0042] and the continuous space is discretized and divided into a plurality of regular grid units for recording risk intensity, coverage, and task distribution and other environmental statistical information. The position of any unmanned aerial vehicle i at time t is denoted as p i,t =(x i,t ,y i，t ), and the grid unit where it is located is determined by a spatial mapping function, thereby realizing unified representation between continuous motion space and discrete environmental information. Through the above environment modeling method, the system can effectively describe the overall situation of the search area while maintaining the continuous control accuracy of the unmanned aerial vehicle.

[0043] (1b) Establish a dynamic task pool model and define task attributes. In the search environment, the system dynamically generates tasks according to a preset rule to form a task pool that changes over time, and the task pool is defined as

[0044] where each task τ k includes at least a task position p k , a task type, a task priority prio kattributes such as task duration, required drone capability, and task generation time. The task priority is used to depict the urgency of the task and serves as an important reference factor in the scheduling decision. The task is removed from the task pool after completion, and the uncompleted task continues to participate in the subsequent scheduling process, thus forming a dynamic and multi-task decision-making background.

[0045] (1c) Establish a risk distribution model and define a risk query mechanism. To depict the potential risk factors in the disaster environment, a spatial risk distribution function is constructed in the search area to represent the risk intensity of different spatial positions. The risk distribution is formed by superimposing multiple risk sources, and its continuous form can be expressed as:

[0046]

[0047] and the final risk value is obtained through normalization processing The risk value of the current position or the adjacent area of the drone can be queried in real time during the execution of the search and task process. The risk information is used to guide path planning, risk avoidance, and dynamic adjustment of task priority in the task scheduling stage, so as to reflect the safety constraints in the overall decision-making process.

[0048] (1d) Establish a drone individual state model, motion model, and energy constraint. The system includes multiple drones, and each drone has an independent individual state. The state of the i-th drone at time t is defined as:

[0049] s i,t =<p i,t, ,θ i,t ,v i,t ,e i,t ,role i >

[0050] where θ i，t represents the heading angle, v i，t represents the speed, e i，t ∈[0, 1] represents the remaining energy ratio, and role i represents the role capability attribute of the drone.

[0051] The drone uses a continuous action space for control, and the control action is used to determine the flight direction and movement amplitude. Based on the action, the drone position is updated according to the preset kinematic model, and the energy consumption is calculated according to the flight distance or speed to ensure that the drone meets the energy constraint and safety operation requirements during task execution.

[0052] (1e) Establish task arrival judgment, task execution progress and completion condition. In order to judge whether the UAV enters the task execution state, the spatial distance between the UAV and the task is defined. When the distance between the UAV and the task position is less than the preset threshold, it is judged that the UAV arrives at the task area and enters the execution state. The task execution progress accumulates gradually with time, and the accumulation rate is related to the task priority and the UAV execution efficiency. When the task progress reaches the completion threshold, the task is judged to be completed, and it is removed from the task pool, and the task completion time and the execution UAV information are recorded for subsequent performance evaluation.

[0053] (1f) Build global state and local observation representation. In order to support the collaborative decision-making framework of centralized training and distributed execution, the present application simultaneously builds global state representation and local observation representation. The global state at least includes the state set of all UAVs, the current task pool information and the risk distribution description, which is used for centralized task assignment decision-making by the scheduling layer; The local observation is independently obtained by a single UAV, which at least includes the state of the UAV itself, the neighborhood task information, the neighborhood UAV state and the local risk perception result, which is used for the control layer to generate continuous control actions in the distributed execution stage. Through the above state construction method, the coordination and unity between the overall system collaboration and individual autonomous control are realized.

[0054] Step (2), build a hierarchical collaborative decision-making structure and realize dynamic task scheduling of multiple UAVs; including the following specific steps:

[0055] (2a) Build a hierarchical collaborative decision-making framework and adopt a centralized training and distributed execution mechanism. In the embodiment of the present application, the problem of multi-UAV collaborative search and task scheduling is modeled as a hierarchical multi-agent Markov decision process. The system is composed of a scheduling layer and a control layer, wherein the scheduling layer is responsible for dynamic task allocation decision-making at the system level, and the control layer is responsible for continuous motion control of the UAV at the individual level. In the training stage, the centralized training and distributed execution framework is adopted, so that the scheduling layer can learn the strategy based on the global state, and the control layer only depends on the local observation in the execution stage and runs independently, so as to balance the global coordination and execution feasibility.

[0056] Let the global state at time t be:

[0057]

[0058] Where s i,t is the state of the i-th UAV, is the dynamic task pool, represents the risk distribution. The scheduling layer strategy is represented as:

[0059]

[0060] Where is the task allocation action.

[0061] (2b) Determine the scheduling decision period and construct the scheduling layer input state. To avoid the instability of the system caused by task reallocation at each environment step, the embodiment introduces a discrete decision period mechanism in the scheduling layer. Let the scheduling decision period be T s , the scheduling action only triggers the update when t mod T s = 0, and the last scheduling result remains unchanged at other times. In this way, high-frequency continuous control and low-frequency discrete scheduling are separated, effectively alleviating the non-stationary problem in multi-agent decision-making.

[0062] (2c) Generate scheduling decisions based on the matching relationship between UAVs and tasks. At the scheduling decision moment, the scheduling layer evaluates the matching relationship between the UAV set

[0063] and the task set . Define the matching score between the ith UAV and the kth task as:

[0064]

[0065] where the function f(·) considers the distance between the current position of the UAV and the position of the task, the priority of the task, the remaining energy of the UAV, and the risk cost, etc. Based on the matching score, a binary assignment variable

[0066] x i.k ∈{0,1}

[0067] is generated to represent whether the task τ k is assigned to the UAV i. The scheduling result needs to satisfy the following constraint conditions:

[0068]

[0069] The above constraints ensure that the task assignment has feasibility in execution load and role capability.

[0070] (2d) Introduce task locking and reallocation suppression mechanism. In dynamic task scenarios, tasks may frequently trigger reallocation due to environmental changes. To improve the stability of scheduling, the embodiment introduces a task locking mechanism in the scheduling layer. Let the task τ k be assigned at time , then within the locking time window T lock , it satisfies:

[0071]

[0072] unless the UAV energy is insufficient or the task urgency changes significantly. At the same time, the system counts the number of task reallocations: ​

[0073]

[0074] Punish frequent re-allocation behavior in reward design.

[0075] (2e) Construct a scheduling layer reward function and define an optimization objective. To guide the scheduling layer to learn a reasonable task allocation strategy, a system layer scheduling reward function is constructed:

[0076]

[0077] wherein represents the task completion and response income, represents the risk cost, represents the energy consumption cost, represents the task re-allocation penalty term, λ r , λ e , λ s is a weight coefficient. The scheduling layer takes maximizing the long-term cumulative return as the optimization objective:

[0078]

[0079] to learn a stable and efficient task scheduling strategy in the process of multiple rounds of interaction.

[0080] (2f) Implement the collaborative constraints and information feedback between the scheduling layer and the control layer. The task allocation result generated by the scheduling layer is input to the control layer as a constraint condition for limiting the target task of the unmanned aerial vehicle; the control layer updates the position, energy and task progress of the unmanned aerial vehicle during execution, and feeds back the execution result to the global state S t for the next scheduling decision. Through this closed-loop feedback mechanism, collaborative optimization between scheduling decision and individual execution behavior is achieved.

[0081] (2g) Support the scheduling strategy learning mode of phased training and joint training. In different embodiments, the scheduling layer strategy can be trained separately first to have stable task assignment capability, and then the control layer is introduced for joint training; or the scheduling layer and the control layer can be trained simultaneously, so that the scheduling strategy can fully consider the dynamic feedback in the individual execution process, thereby improving the overall collaborative performance of the system in a complex dynamic environment.

[0082] Step (3), construct a continuous control strategy of the control layer and realize joint training with the scheduling layer; including the following specific steps:

[0083] (3a) Constructing a UAV control layer decision model based on local observations. In this embodiment of the invention, the control layer is used to generate continuous control actions based on the local observation information of a single UAV under the task assignment constraints given by the scheduling layer, so as to achieve behaviors such as search coverage, task approach, task execution, and risk avoidance. For the i-th UAV, the local observation at time t is defined as:

[0084]

[0085] Where S i,t Indicates the drone's own status. This indicates mission information within the drone's perception range. Represents the set of states of neighboring drones. This represents the risk value corresponding to the current location. The control layer strategy takes the local observations as input and outputs the continuous control actions of the UAV at the current moment:

[0086] a i，t =π ctrl (o i,t )

[0087] In this way, the control layer strategy can autonomously generate execution actions based on the local situation without relying on global information, thus meeting the requirements of distributed execution.

[0088] (3b) Define the continuous action space and establish the UAV motion update model. The control layer uses the continuous action space to describe the motion control behavior of the UAV, and its actions are as follows:

[0089] a i,t =(φ i,t ,ρ i,t )

[0090] Where φ i,t ∈[0, 2π) represents the flight direction angle, ρ i,t ∈[0,1] represents the velocity ratio. Based on the above control actions, the UAV's position is updated according to the following motion model:

[0091]

[0092] Simultaneously, the remaining energy state is updated based on changes in the UAV's flight distance or speed to ensure that energy constraints are met during control. This continuous control modeling approach enables the UAV to smoothly execute search and task approach behaviors, avoiding path jitter issues caused by discrete actions.

[0093] (3c) Introducing a control execution mechanism with task guidance and risk constraints. When a UAV is assigned a specific task by the scheduling layer, the control layer introduces task guidance and risk constraints during the action generation process. The spatial distance between the UAV and the assigned task is defined as:

[0094] d i，k (t)=||p i,t -p k ||2

[0095] And set the task proximity reward in the reward function:

[0096]

[0097] This is used to guide the drone gradually towards the mission location. Simultaneously, to prevent the drone from entering high-risk areas, a risk penalty is introduced into the control layer reward:

[0098]

[0099] The above design enables the control layer to dynamically balance efficiency and security during task execution.

[0100] (3d) Define the task execution progress model and task completion reward. When the UAV enters the task execution radius and meets the execution conditions, the task enters the execution state, and its execution progress accumulates gradually over time. Define task τ. k The execution progress is π k (t)∈[0,1], and its update method is as follows:

[0101] π k (t+1)=π k (t)+Δπ0·(1+α·prio k )

[0102] Where Δπ0 is the basic schedule increment, α is the priority adjustment coefficient, and prio k Task priority. When π k When (t)≥1, the task is considered complete, and a completion reward is given in the control layer reward:

[0103]

[0104] This is to encourage drones to prioritize tasks with a high degree of urgency.

[0105] (3e) Construct the control layer immediate reward function and define the optimization objective. Based on the above task guidance, risk constraints, and energy consumption factors, construct the control layer immediate reward function:

[0106]

[0107] in The term β1, β2, and β3 represent the energy consumption penalty for the drone, and are weighting coefficients. The control layer strategy aims to maximize the long-term cumulative return.

[0108]

[0109] This guides the drone to learn stable and efficient motion control strategies in complex dynamic environments.

[0110] (3f) Implement a joint training mechanism for the scheduling and control layers. In the joint training mode, the system-level scheduling reward and the individual control reward are weighted and fused to construct a joint reward function:

[0111]

[0112] in λ represents the reward for the scheduling layer, and λ is the weighting coefficient. Through this joint reward mechanism, the scheduling layer can perceive the actual execution effect of the control layer when generating task allocation decisions, thereby reducing unreasonable scheduling and improving the overall collaborative performance of the system.

[0113] (3g) Supports both standalone and joint training modes for the control layer. In different implementations, the control layer policy can be trained alone to verify its execution capability under a given scheduling result; or it can be jointly trained with the scheduling layer policy, enabling the two policies to converge collaboratively under a unified optimization objective. By comparing the changing trends of the control layer rewards under different training modes, the improvement effect of the joint training mechanism on the overall system performance and stability can be evaluated.

[0114] The following simulation experiment uses Python and compares the multi-UAV cooperative search and scheduling method based on deep reinforcement learning described in this invention with existing benchmark methods to verify the performance improvement effect of this invention in dynamic environments.

[0115] exist Figure 3 The simulation results demonstrate the changes in task completion rate with the number of training rounds during the training process, as shown in the embodiments of the present invention. The simulation results show that as training progresses, the overall task completion rate exhibits a continuous upward trend followed by a gradual stabilization. This indicates that the joint optimization mechanism of the scheduling and control layers employed in this invention can continuously improve the rationality of task allocation and the effectiveness of execution strategies during interactive training, thereby enhancing the system's overall ability to complete dynamic tasks. Compared to methods using only a single scheduling or control strategy, this invention achieves a higher task completion rate under the same training conditions, demonstrating the enhanced effect of end-to-end collaborative training in improving task execution efficiency and convergence stability.

[0116] exist Figure 4The paper presents statistical results of the average response time for emergency tasks after training convergence in embodiments of the present invention, where emergency tasks are defined as high-priority tasks with a priority not lower than a preset threshold. Simulation results show that the method proposed in this invention maintains a low and stable level in terms of emergency task response time, indicating that the adopted scheduling strategy can quickly select UAVs with suitable locations and matching capabilities to execute the task after task generation, thereby shortening the time from task generation to the UAV's first entry into the task radius. Compared with the comparative methods, the present invention has a better or comparable response speed under the same experimental conditions, reflecting the dominant role of the scheduling layer in the emergency task allocation stage, and also showing that the joint training mechanism does not weaken the system's rapid response capability to emergency tasks due to the increased complexity of the control layer.

[0117] exist Figure 5 The paper presents a comparison of the average remaining energy ratio of each UAV at the end of the mission cycle, based on embodiments of the present invention. Simulation results show that, compared to the comparative method, the present invention can maintain a higher level of remaining energy under the same experimental conditions, indicating that the proposed collaborative mechanism between the scheduling layer and the control layer effectively improves energy utilization efficiency during mission execution. On the one hand, the scheduling layer comprehensively considers the remaining energy state of the UAVs during the mission allocation phase, avoiding the concentration of tasks on individuals with insufficient energy; on the other hand, the control layer generates smoother and more reasonable flight trajectories through learned continuous control strategies, reducing the additional energy consumption caused by unnecessary high-speed maneuvers and frequent turns. The above results demonstrate that the present invention achieves a more balanced and efficient energy management effect while ensuring mission completion efficiency.

[0118] exist Figure 6 The paper presents a comparison of the average number of task reassignments during task execution using embodiments of the present invention. Simulation results show that the present invention has a lower task reassignment frequency under the same experimental conditions, indicating that the proposed method has a significant advantage in scheduling decision stability. This is because the present invention introduces a task locking mechanism and an allocation benefit improvement threshold in the scheduling layer, triggering reassignment only when the new allocation scheme is a significant improvement over the current scheme, thereby effectively avoiding frequent adjustments caused by local benefit fluctuations. Simultaneously, the control layer execution effect acts on the scheduling layer through a feedback mechanism, enabling the scheduler to make more reasonable decisions in the initial allocation stage. The above results demonstrate that the present invention effectively reduces scheduling oscillations and communication overhead while ensuring system flexibility, which is beneficial for the stable operation of the system in complex dynamic environments.

[0119] Based on the description of the present invention, those skilled in the art should readily recognize that the multi-UAV collaborative search and dynamic task scheduling method based on hierarchical collaborative decision-making proposed in this invention can achieve effective collaboration between the scheduling layer and the control layer in complex dynamic environments. It can complete the joint optimization of task allocation and continuous control without relying on manual rules or static allocation strategies, and maintain a high task completion rate, a fast emergency task response speed, and good energy utilization efficiency in multi-task, high-risk, and uncertain scenarios.

[0120] Based on the same concept, this invention also discloses a corresponding multi-UAV cooperative search and scheduling system, comprising: a scene modeling and risk field generation module, used to establish a multi-UAV cooperative search model including multiple UAVs, a dynamic task pool, and a risk distribution environment; a central scheduling and graph structure construction module, used to receive global state information output by the scene modeling and risk field generation module and construct a bipartite graph structure composed of UAV nodes and task nodes; a UAV observation processing and state coding module, used to collect local observation information for each UAV; a deep reinforcement learning training module, used to construct a multi-agent control network based on an actor-critic framework, and train the continuous action strategy of the UAVs using a global or centralized critic network and a distributed actor network structure; and a cooperative control and online execution module, used to call the convergence policy network obtained by the deep reinforcement learning training module during the execution phase.

[0121] This invention also discloses a computer system, including a memory, a processor, and a computer program that can run thereon, wherein the program, when executed, implements all the steps of the above-described multi-UAV cooperative search and scheduling method based on deep reinforcement learning.

[0122] This invention also discloses a computer program product, which, when executed by a processor, implements the steps of the above-described multi-UAV cooperative search and scheduling method based on deep reinforcement learning.

[0123] The relevant program code of this invention can be executed on a general-purpose computer, a special-purpose computing device or other programmable device, and the remaining parts not described in detail can be implemented by those skilled in the art based on common sense.

Claims

1. A multi-UAV cooperative search and scheduling method based on deep reinforcement learning, characterized in that, Includes the following steps: A multi-UAV collaborative search model is established, which includes multiple UAVs, a dynamic task pool, and a risk distribution environment. By dividing the search area into grids, deploying risk sources, and constructing task generation rules, UAVs can perform search and task response in scenarios with spatial heterogeneity and environmental dynamics. Energy consumption parameters, role attributes, perception range, and motion models are configured for each UAV, and an environmental state set that can be used to describe the global situation is established for subsequent scheduling and control strategy invocation. Local observation information is collected for each UAV, including its position, speed, remaining energy, status of neighboring UAVs, attributes of nearby tasks, and local risk patches. All UAV states, task sets, and risk patterns are integrated into a global state vector, and a central scheduler is constructed to execute task assignment. A graph-based central scheduling module is used to infer the matching relationship between UAVs and tasks. Based on the UAV role capabilities, task priority, spatial distribution, distance cost and reachability constraints, task-UAV matching results are generated, and assignment is completed under competitive conditions to achieve resource optimization in a multi-task environment. According to the assigned task, the UAV generates continuous actions through deep reinforcement learning control strategies, including movement direction, flight speed and local avoidance strategies, in order to achieve path planning, risk avoidance, task approach and execution, enabling the UAV to form a stable trajectory and adaptive planning capability in a dynamic environment. A joint reward mechanism is constructed to evaluate task progress, task completion, priority response, energy consumption, risk exposure, area coverage and coordination behavior with weighted values. The scheduling strategy and the UAV control strategy are jointly trained to improve the overall performance of multiple agents when they cooperate to execute tasks. During the execution phase, the UAV performs dynamic trajectory planning based on the trained deep reinforcement learning policy network and responds to new tasks. At the same time, the central scheduler updates task assignments in real time according to environmental changes, enabling the UAV to achieve efficient search, reasonable allocation and collaborative execution under the conditions of task generation, risk changes and energy constraints, thereby improving task completion rate, resource utilization and system stability.

2. The multi-UAV cooperative search and scheduling method based on deep reinforcement learning according to claim 1, characterized in that, The method further includes a risk field modeling step, which constructs a continuous risk distribution by superimposing multiple risk sources in the environment, so that the probability of the task occurring is consistent with the risk area. The risk field is represented by the following formula: These represent the risk source intensity, center location, and diffusion scale, respectively. Through normalization, the risk is mapped to the weight of task generation, so that high-risk areas have higher task density. Furthermore, the risk gradient is used to influence the drone's risk avoidance behavior, so that the drone's path planning and energy consumption strategy are more in line with environmental constraints. The risk field can also update weights over time to simulate scenarios such as fire spread, disaster diffusion, and changes in regional dynamic hazard levels, thereby making mission generation and UAV strategy spatiotemporally relevant.

3. The multi-UAV cooperative search and scheduling method based on deep reinforcement learning according to claim 1, characterized in that, The method includes a central scheduler construction step, which achieves global scheduling by establishing a bipartite graph structure of UAV nodes and task nodes; projecting UAV and task features onto a unified hidden space, and constructing a graph attention network to extract relationships between nodes; the attention weights are defined by the following formula: Node features updated as follows: Where h i denoted by node features, and W represents the mapping matrix; the reachability, capability matching, priority consistency and resource competition relationship between UAV and task are expressed through a multi-head attention mechanism, thereby forming a globally consistent decision embedding and achieving more reliable task scheduling output.

4. The multi-UAV cooperative search and scheduling method based on deep reinforcement learning according to claim 1, characterized in that, The method includes a task allocation step, which combines differentiable allocation probability generation with a discrete executable assignment strategy; for task-UAV pairing, a scoring matrix S = [s ij The soft assignment probability is calculated using the Gumbel-Softmax method: Where g ij Let Gumbel noise be τ, and τ be the temperature parameter. During the inference phase, a unique task assignment is generated using a greedy strategy based on probability sorting and priority constraints, ensuring that each task is executed by at most one drone. Further filtering is performed based on role constraints and distance constraints, where the distance constraint satisfies: D(U i ,T j )<D max , This ensures that the assignment results remain feasible in terms of physical accessibility, mission requirements, and the capabilities of the drone.

5. The multi-UAV cooperative search and scheduling method based on deep reinforcement learning according to claim 1, characterized in that, The method includes a UAV local observation encoding step, which encapsulates the multi-source information of the UAV into a fixed-dimensional vector. The UAV local observation vector includes the UAV's normalized position, remaining energy, role category embedding, local risk patch, neighboring UAV status, and nearby task attributes. The local risk patch obtains a fixed-scale sub-region feature matrix through pruning and flattens it into a vector. The neighboring UAV status is composed of relative distance, speed, and role label. Nearby task attributes include task type, priority, and relative position. The above observation processing method enables the UAV control strategy to maintain action stability in partially observable environments, while adapting to environmental uncertainties caused by task sparsity, uneven risk distribution, and communication delays.

6. The multi-UAV cooperative search and scheduling method based on deep reinforcement learning according to claim 1, characterized in that, The method includes a deep reinforcement learning control strategy, which optimizes the drone's actions through an actor-critic network structure; the critic network is updated using a target value function. y t =r t +γQ′(s t+1 ,a t+1 ), And by minimizing the error: L=(Q(s t ,a t )-y t ) 2 Optimize the critic parameters; the actor network updates the action policy based on the critic gradient, so that the output action is: a t =π θ (s t ), The exploration is achieved by adding time-dependent noise; the UAV performs continuous motion according to the updated strategy to achieve task approach, path obstacle avoidance, energy consumption optimization and trajectory smoothing control.

7. The multi-UAV cooperative search and scheduling method based on deep reinforcement learning according to claim 1, characterized in that, The method improves learning stability and global synergy by constructing a multi-branch reward structure. The reward consists of task progress, task completion, priority response, energy consumption penalty, risk penalty, boundary penalty, and redistribution penalty, where the redistribution penalty is defined based on changes in task affinity: ΔA=A new -IN old , A penalty is triggered when ΔA < ε to avoid frequent task jumps; the boundary penalty is generated based on the number of steps the drone continuously stays at the boundary of the area to ensure that the drone maintains effective search coverage; the above reward structure enables the drone control decision and scheduling strategy to maintain a balance between long-term benefits and immediate behavior, thereby improving the overall system's collaborative efficiency and robustness.

8. An apparatus for implementing the multi-UAV cooperative search and scheduling method based on deep reinforcement learning as described in claims 1 to 7, characterized in that, include: The scenario modeling and risk field generation module is used to establish a multi-UAV collaborative search model that includes multiple UAVs, a dynamic task pool, and a risk distribution environment. It divides the search area into grids, deploys risk sources, and configures task generation rules to construct a search scenario with spatial heterogeneity, task randomness, and environmental dynamics. Based on this, it forms a set of environmental states to describe the global situation. The central scheduling and graph structure construction module is used to receive global state information output by the scenario modeling and risk field generation module, construct a bipartite graph structure composed of UAV nodes and task nodes, introduce a graph attention network to jointly model UAV capabilities, task priorities, spatial distance, reachability constraints and resource competition relationships, and output the matching relationship and assignment results between multiple UAVs and multiple tasks to achieve global optimized task scheduling. The UAV observation processing and state coding module is used to collect local observation information for each UAV, including position, speed, remaining energy, status of neighboring UAVs, attributes of nearby tasks, and local risk patches. The module normalizes, splices, and encodes the above information to generate a fixed-dimensional local observation vector, and combines it with the central scheduling results to form a state input that can be called by deep reinforcement learning control strategies. The deep reinforcement learning training module is used to build a multi-agent control network based on the actor-critic framework. It adopts a global or centralized critic network and a distributed actor network structure to train the continuous action strategy of the UAV. By defining a composite reward function that includes task progress, task completion, priority response, energy consumption penalty, risk penalty and redistribution penalty, and combining experience replay, target network update and federated parameter sharing mechanism, the strategy of each UAV is iteratively optimized. The collaborative control and online execution module is used to call the convergent policy network obtained by the deep reinforcement learning training module during the execution phase, so that each UAV can generate dynamic trajectories and action decisions based on the current local observation and central scheduling results, realizing path planning, risk avoidance and task approach; at the same time, it responds to environmental changes and new task generation requests, and links the central scheduling and control strategy to complete the efficient search, reasonable allocation and collaborative execution process of multiple UAVs in dynamic task scenarios.

9. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-7.

Images