Multi-UAV collaborative computing offloading and resource allocation method and system for emergency search and rescue in sudden disasters

Abstract

The present invention discloses a multi-UAV collaborative computing offloading and resource allocation method and system for emergency search and rescue in sudden disasters, including: building a multi-UAV assisted three-dimensional computing offloading network architecture for sudden emergency disaster search and rescue scenarios; taking into account the offloading fairness of search and rescue equipment, the load balancing of UAVs, and the reduction of UAV energy consumption, establishing a system utility function, and establishing a deep reinforcement learning model for the search and rescue equipment that determines task offloading with the goal of maximizing the utility function; training the deep reinforcement learning model based on an improved multi-agent dual-delay deep deterministic policy gradient algorithm, introducing OU noise and priority cloning learning mechanisms, and learning and optimizing UAV trajectory planning and task offloading decisions for search and rescue equipment. The present invention can address the collaborative computing offloading needs of UAVs as MEC servers in disaster scenarios, and achieve the goals of ensuring the fairness of search and rescue equipment services and reducing UAV energy consumption through reasonable and efficient computing offloading decisions and resource allocation.

Description

Technical Field

[0001] The present invention relates to a computation offloading technology, and in particular to a UAV-assisted computation offloading and resource allocation method and system. More specifically, the present invention relates to a multi-UAV collaborative computation offloading and resource allocation method and system for emergency disaster search and rescue scenarios. Background Art

[0002] With the rapid development of wireless communication technology and mobile computing, drones are increasingly being used in emergency communications, broadband connectivity, environmental monitoring, and other fields. Especially in areas without internet infrastructure or where communication networks are damaged, mobile edge computing (MEC) systems onboard drones can provide computational offload services for ground-based search and rescue equipment, meeting real-time and low-latency requirements. Drones can be equipped with MEC servers to form a flexible mobile perception platform. This configuration not only ensures a high-quality service experience for search and rescue equipment but also allows for flexible adaptation to various practical application scenarios. In complex scenarios with multiple search and rescue devices, where their mission requirements change dynamically, the allocation of computing resources and decision-making regarding task offloading become particularly critical.

[0003] Currently, MEC systems on drones are widely used for broadband connectivity and emergency communications in areas without internet access. However, drones have limited onboard energy consumption, and providing long-term, efficient computing services for search and rescue equipment within this energy constraint has become a hot research topic. Furthermore, the mission requirements of search and rescue equipment are dynamically changing, and ensuring fairness in search and rescue equipment services and load balancing across drones amidst competition for resources among multiple search and rescue devices is also a current research challenge. Summary of the Invention

[0004] Purpose of the invention: The purpose of the present invention is to propose a multi-UAV collaborative computing offloading and resource allocation method and system for emergency search and rescue in sudden natural disasters. For the scenario of multi-UAV collaborative search and rescue in sudden natural disasters (such as earthquakes), the optimization goal is to take into account the fairness of search and rescue equipment offloading, load balancing of UAVs, and maximization of the system utility function of energy consumption. In the face of disaster environments, a good balance is achieved between complexity and performance. UAVs can dynamically adjust resource allocation according to the needs of ground rescue missions to provide the best service quality.

[0005] Technical solution: To achieve the above-mentioned purpose, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides a multi-UAV collaborative computing offloading and resource allocation method for sudden disaster emergency search and rescue, comprising the following steps:

[0007] Build a multi-UAV-assisted 3D computing offload network architecture for emergency disaster search and rescue scenarios, where UAVs provide computing services for search and rescue equipment, which moves within the disaster area and generates computing tasks.

[0008] Based on the search and rescue equipment's task offloading decision variables for each drone, the search and rescue equipment's offloading ratio parameter in each time slot is determined, thereby measuring the fairness of task offloading between the search and rescue equipment. Based on the drone's resource allocation decision variables for each search and rescue equipment, the drone's resource allocation ratio parameter is determined, thereby measuring the degree of load balancing between the drones. Taking into account the fairness of search and rescue equipment offloading, the load balancing of drones, and the reduction of drone energy consumption, a system utility function is established, and a deep reinforcement learning model is established for the search and rescue equipment that determines task offloading with the goal of maximizing the utility function.

[0009] Based on the improved Multi-Agent Twin Delayed Deep Deterministic Policy Gradients (IMATD3) algorithm to train a deep reinforcement learning model, the Ornstein-Uhlenbeck (OU) noise and priority clonal learning mechanism were introduced to learn and optimize the trajectory planning of drones and the task offloading decisions of search and rescue equipment.

[0010] Furthermore, the system includes multiple rescue equipment and multiple drones. The system's operating time includes multiple consecutive time slots, and the duration of each time slot can be dynamically adjusted according to the urgency of the rescue mission; the movement of drones in the system cannot exceed the boundary of the target disaster area, the flight altitude does not exceed the upper limit, the distance between drones flying between adjacent time slots does not exceed the maximum distance, and the distance between drones is not less than the safe distance.

[0011] Furthermore, the fairness of task offloading between search and rescue devices is expressed as follows:

[0012] ;

[0013] in For search and rescue equipment n In the time slot t Unloading ratio parameter, N The number of rescue equipment, U is the number of drones, Time slot t Search and rescue equipment n drones u The task offloading decision variable; the load balancing degree between UAVs is expressed as follows:

[0014] ;

[0015] in For drones u In the time slot t The resource allocation ratio parameter, Time slot t drones u Search and rescue equipment n The resource allocation decision variables.

[0016] Furthermore, in the deep reinforcement learning model, each drone represents an intelligent agent, and the drone's observation space includes the coordinates of all search and rescue equipment and all drones, the target point coordinates, the drone's remaining energy consumption, as well as the task unloading status of the search and rescue equipment and the drone's load status; the drone's action space includes flight angle, flight distance, range coverage, and resource allocation.

[0017] Furthermore, in the deep reinforcement learning model, the rewards include the remaining energy consumption and fairness rewards, arrival rewards and wrong action penalties; in the time slot t , residual energy consumption and fairness reward is defined as ,in, To ensure fairness in task offloading between search and rescue equipment, is the load balancing degree among drones, and is the weight, is the remaining energy consumption of the drone; arrival reward Expressed as ,in, 、 are the UAV and target locations respectively, represents the Euclidean distance, M 1 is a constant greater than 0. M 2 is a constant used to prevent the denominator from being 0. When the drone flies out of the target disaster area, fails to complete the mission within the time limit, runs out of energy, or collides, it will receive an error action penalty.

[0018] Furthermore, when training the deep reinforcement learning model based on the improved multi-agent double-delay deep deterministic policy gradient algorithm, partial observation states are obtained for each drone, and actions are selected based on the greedy strategy and policy network. At the same time, the OU noise mechanism is used to add noise perturbations to the action selection. The penalty for incorrect actions is calculated according to the selected action and environmental state, and it is judged whether the drone collides or flies out of the target disaster area. If a collision occurs or the drone flies out of the target disaster area, it returns to the initial position. Partial observation states at the next moment are obtained, and for each search and rescue device, a selection is made based on the minimum energy consumption greedy strategy. For each drone, the reward value is calculated. The global state at the next moment is obtained, and the experience at the current moment is stored in the experience pool.

[0019] Furthermore, when training the deep reinforcement learning model based on the improved multi-agent dual-delay deep deterministic policy gradient algorithm, for each drone, batch samples are taken out from the experience pool according to priority, and the loss functions of the two value networks are calculated, and then the value network parameters are updated. The policy network parameters are updated based on the deterministic policy gradient ascent method, and the target network parameters are adjusted. The clone learning mechanism is used to clone some parameters of the current policy network to the target network.

[0020] In a second aspect, the present invention provides a multi-UAV collaborative computing offloading and resource allocation system for emergency search and rescue of sudden disasters, comprising:

[0021] A scenario construction module is used to build a multi-UAV-assisted 3D computing offload network architecture for emergency disaster search and rescue scenarios, where UAVs provide computing services for search and rescue equipment, which moves within the disaster area and generates computing tasks;

[0022] The problem modeling module is used to determine the offloading ratio parameter of the search and rescue equipment in each time slot based on the search and rescue equipment's task offloading decision variables for each drone, thereby measuring the fairness of task offloading between the search and rescue equipment. Based on the drone's resource allocation decision variables for each search and rescue equipment, the resource allocation ratio parameter is determined to measure the degree of load balancing between the drones. Taking into account the offloading fairness of the search and rescue equipment, the load balancing of the drones, and the reduction of drone energy consumption, a system utility function is established. A deep reinforcement learning model is then built for the search and rescue equipment that determines task offloading, with the goal of maximizing the utility function.

[0023] The reinforcement learning module is used to train deep reinforcement learning models based on an improved multi-agent double-delay deep deterministic policy gradient algorithm, introduce OU noise and priority clonal learning mechanisms, and learn and optimize the trajectory planning of drones and the task offloading decisions of search and rescue equipment.

[0024] In a third aspect, the present invention provides a computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the multi-UAV collaborative computing offloading and resource allocation method for emergency search and rescue of sudden disasters are implemented.

[0025] In a fourth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the multi-UAV collaborative computing offloading and resource allocation method for sudden disaster emergency search and rescue.

[0026] Beneficial Effects: This invention addresses the challenges faced by drones in emergency disaster scenarios, including limited onboard energy consumption and the difficulty ensuring service fairness and load balancing among multiple search and rescue equipment competing for resources. By doing so, it proposes a multi-UAV collaborative computation offloading and resource allocation method. Taking into account the influence of factors such as the drone's flight direction, altitude, and distance, the system maximizes the system utility function by controlling the drone's trajectory and the search and rescue equipment's task offloading decisions. The system utility function balances offloading fairness for the search and rescue equipment, load balancing for the drone, and energy consumption. An IMATD3-based algorithm is employed to solve the drone's path planning problem. The optimal policy is dynamically learned through interaction between the policy network and the environment, enabling efficient decision-making in a continuous action space. An OU noise mechanism is introduced into the IMATD3 algorithm to enhance the agent's exploration capabilities in dynamic disaster environments, helping drones better adapt and make decisions in complex and changing disaster environments. Furthermore, a priority cloning learning mechanism is incorporated to learn high-value experience samples. By cloning some policy network parameters into the target network, the learning process is accelerated, improving the algorithm's stability and convergence speed.

[0027] In summary, the present invention can effectively improve the execution efficiency of multi-UAV collaborative rescue missions, ensure the service fairness of search and rescue equipment, and optimize the energy consumption and load balancing of UAVs. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Schematic diagram of a scenario of a multi-UAV collaborative computing offloading and resource allocation method for sudden disaster emergency search and rescue according to an embodiment of the present invention.

[0029] Figure 2 Detailed flow chart of the multi-UAV collaborative computing offloading and resource allocation method for sudden disaster emergency search and rescue according to an embodiment of the present invention.

[0030] Figure 3 This is a graph of the reward results of the IMATD3 algorithm in an embodiment of the present invention and the comparison algorithm under different task amounts.

[0031] Figure 4 This is a graph showing the reward results of the IMATD3 algorithm in an embodiment of the present invention and the comparison algorithm under different maximum computing resources of drones.

[0032] Figure 5 This is a diagram showing a fair comparison of search and rescue equipment between the IMATD3 algorithm in an embodiment of the present invention and the comparison algorithm in different time slots.

[0033] Figure 6 This is a comparison chart of drone load balancing results between the IMATD3 algorithm in an embodiment of the present invention and the comparison algorithm in different time slots. DETAILED DESCRIPTION

[0034] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] exist Figure 1 In this paper, the scenario of multi-UAV collaborative computing offloading and resource allocation for emergency search and rescue in sudden disasters is described. It can be seen that multiple UAVs act as MEC servers to provide services for search and rescue equipment within their coverage area. Figure 1 The embodiment of the present invention discloses a multi-UAV collaborative computing offloading and resource allocation method for sudden disaster emergency search and rescue. First, a multi-UAV assisted three-dimensional computing offloading network architecture for sudden disaster emergency search and rescue scenarios is established, in which UAVs provide computing services for search and rescue equipment, and the search and rescue equipment moves in the disaster area and generates computing tasks. Then, based on the task offloading decision variables of the search and rescue equipment for each UAV, the offloading ratio parameter of the search and rescue equipment in each time slot is determined, thereby measuring the fairness of task offloading among the search and rescue equipment. Based on the resource allocation decision variables of the UAV for each search and rescue equipment, the resource allocation ratio parameter of the UAV is determined, thereby measuring the degree of load balancing among the UAVs. Taking into account the offloading fairness of the search and rescue equipment, the load balancing of the UAVs, and the reduction of UAV energy consumption, a system utility function is established, and a deep reinforcement learning model is established for the search and rescue equipment that determines the task offloading with the goal of maximizing the utility function. Finally, the deep reinforcement learning model is trained based on an improved multi-agent double-delay deep deterministic policy gradient algorithm, and the OU noise and priority cloning learning mechanism are introduced to learn and optimize the trajectory planning of the UAV and the task offloading decision of the search and rescue equipment.

[0036] This embodiment addresses the collaborative computing offloading problem of drones acting as MEC servers in emergency disaster search and rescue scenarios. By optimizing the drone's flight trajectory and the rescue equipment's task offloading decisions, it addresses the challenge of providing long-term computing services to ground-based rescue equipment within limited airborne energy consumption. This embodiment balances offloading fairness for rescue equipment, load balancing for drones, and energy optimization to maximize the system's utility function. It employs an improved multi-agent double-delayed deep deterministic policy gradient (IMATD3) algorithm, dynamically adapting to task demands and resource competition through centralized training and distributed execution. It utilizes an OU noise mechanism to enhance the drone's agent exploration capabilities in complex and dynamic disaster environments, avoiding local optimal solutions. Furthermore, it incorporates a priority cloning learning mechanism to learn high-value experience samples. By cloning some policy network parameters, it accelerates the learning process and improves the algorithm's stability and convergence speed. This method significantly improves service fairness and load balancing for rescue equipment while effectively reducing energy consumption, enabling it to better address the complex demands of dynamic scenarios involving multiple rescue devices.

[0037] exist Figure 2 In this paper, a detailed process of a multi-UAV collaborative computing offloading and resource allocation method for emergency search and rescue in sudden disasters is described. The path planning of the UAV and the task offloading decision of the search and rescue equipment are controlled through dynamic learning through interaction with the environment.

[0038] The following combination Figure 2 A multi-UAV collaborative computing offloading and resource allocation method for emergency search and rescue of sudden disasters is further described in detail. Specifically, the multi-UAV collaborative computing offloading and resource allocation method for emergency search and rescue of sudden disasters includes the following steps:

[0039] Step (1) is to build a multi-UAV assisted three-dimensional computing offloading network architecture for emergency disaster search and rescue scenarios. Multiple UAVs act as MEC servers to provide services to search and rescue equipment within their coverage area. The system can use directional antenna technology and OFDM technology to effectively overcome the communication challenges brought by the complex electromagnetic environment in the disaster area. The UAVs deployed in the system are equipped with computing modules, which can provide computing services to the search and rescue equipment through the MEC servers on board. The search and rescue equipment moves in the disaster area and generates computing tasks. The computing tasks can be partially processed locally and partially offloaded to the UAV edge server.

[0040] Step (2), establish the N Search and rescue equipment and U The communication, latency, and energy consumption models of the two drones are used to build a joint computing migration and resource allocation model, including:

[0041] (2a), establish a communication model for three-dimensional computation offloading of multi-UAV assisted search and rescue, the system includes N Key rescue equipment such as emergency communication terminals and U drones, search and rescue equipment and drones are assembled separately and The running time of the system consists of T consecutive time slots, which can be represented by the set The duration of each time slot is , can be dynamically adjusted according to the urgency of the rescue mission, and the position of each drone can be used Indicates that, , D Indicates that when the drone moves in the target disaster area, and The boundary range on the coordinates ensures that the movement of the drone cannot exceed the boundary of the target disaster area. In addition, , H max Indicates the maximum flight altitude of the drone, ensuring that the drone's flight altitude is within the upper limit. The maximum flight distance of the drone between adjacent time slots isd max To avoid collisions between drones, u and drones The distance between , d min Indicates the safe distance between drones.

[0042] (2b), the communication channel between the search and rescue equipment and the UAV can be modeled as a combination of large-scale fading and small-scale fading. n With drones u Channel gain between It can be expressed as ;in, represents the distance between the search and rescue equipment n and the drone u, Indicates the channel gain when the reference distance is 1m; l is the path loss index, which represents the rate at which the signal attenuates with distance during propagation; Represents the small-scale fading coefficient, which is usually modeled as Rice fading. n and drones u Signal to Noise Ratio (SNR) of the link between It can be expressed as ;in, Indicates search and rescue equipment n To the drone u The transmit power of the transmission signal, Represents the background noise of the communication link, which is the Gaussian noise power. For search and rescue equipment, the actual data transmission rate of its data It can be expressed as ;in, B Indicates the channel bandwidth of the link. Average transmission rate It can be expressed as ;in, , represents the ratio of channel gain to noise power; is the decay factor approximated by logistic regression.

[0043] (2c), establish a time delay model, and implement a partial offloading mechanism for the task, where one part can be executed on the search and rescue equipment, and the other part can be offloaded to the drone for calculation, and the two parts are processed in parallel. Indicates that each unloading node is o To indicate that when o =0, indicating that the task is executed locally. When , it means that the task is offloaded to the MEC server for processing. The decision variables of the search and rescue equipment can be expressed as , Each search and rescue device has a corresponding rescue task that needs to be processed and calculated. Indicates that, Indicates time slot t Search and rescue equipment n The amount of data, Indicates the number of CPU revolutions required to calculate the task. and When Partially uninstalled; and When , it means that the task is completely offloaded to the UAV for processing; and o =0 means the task is processed locally; otherwise, .

[0044] When the search and rescue equipment task is offloaded to the corresponding drone, the transmission delay Indicates the time it takes for the search and rescue equipment to transmit part of the task to the drone, calculating the delay It represents the time required for the UAV to process the search and rescue equipment mission, which can be expressed as

[0045] ;

[0046] in, Indicates time slot t Search and rescue equipment n drones u The decision variables, Indicates time slot t drones u Search and rescue equipment n The resource allocation decision variables.

[0047] Each task offloads computations required in a time slot Therefore, the time for uninstalling part It can be expressed as

[0048] ;

[0049] The local computation time of a task can be expressed as

[0050] ;

[0051] in, Represents the local computing resources of the search and rescue equipment. Similarly, the local computing part also needs to be completed within a time slot, that is, .

[0052] Therefore, the completion delay of each search and rescue equipment mission needs to meet .

[0053] (2d), establish energy consumption model, computing resources provided by drones for search and rescue equipment It can be expressed as ;in, , f max The maximum computing resources provided by the UAV for the search and rescue equipment. The computing energy consumption of the UAV can be expressed as ;in, is a coefficient related to the drone hardware, The propulsion power of the UAV during flight can be expressed as

[0054] ;

[0055] in, Indicates the flight speed of the drone. represents the tip speed of the drone rotor, K 1 to K 5 is a constant related to the aerodynamics and weight of the drone. It can be expressed as .

[0056] The total energy consumption of each drone in each time slot It can be expressed as .

[0057] The remaining energy consumption of all UAVs in time slot t can be expressed as ;in, represents the remaining energy consumption of the UAV in time slot t, Represents the initial energy consumption of each UAV.

[0058] In order to allow each search and rescue device to enjoy the offloading service fairly and avoid some search and rescue devices being in the task offloading state for a long time while other search and rescue devices cannot get the service, the offloading ratio parameter of the search and rescue device in each time slot is defined as , which can be expressed as .

[0059] Construct an offloading balance function for each search and rescue device to measure the balance of offloading tasks between devices, which can be expressed as .

[0060] Define the resource allocation ratio parameters of drones , which is used to measure the computing resource share allocated by each UAV to the search and rescue equipment during the entire mission. It is a dynamically adjusted indicator in the task allocation process and can be expressed as .

[0061] In order to measure the load balance between drones, a load balancing function for each drone is constructed, which can be expressed as .

[0062] In order to take into account the fairness of unloading search and rescue equipment, load balancing of drones and reduction of drone energy consumption, a system utility function is established, which can be expressed as ;in, and The weight parameter representing the offloading fairness of search and rescue equipment and the load balancing of UAVs satisfies .

[0063] (2e), in summary, the following objective function and constraints can be established:

[0064] ;

[0065] in, represents the movement of the drone, represents the task offloading decision. The objective function is the defined system utility function. Constraint C1 ensures that the flight range of the UAV is limited to the three-dimensional spatial boundary of the target disaster area. H min represents the minimum safe altitude for UAV flight; constraint C2 ensures that there will be no collision between the drones; constraints C3 and C4 are the coverage constraints between the UAV and the search and rescue equipment. The position of each search and rescue equipment on the ground can be Indicates that, , represents the angle between the horizontal plane where the UAV is located and the search and rescue equipment; the constraint condition C5 represents the signal-to-noise ratio constraint, which ensures the communication quality between the search and rescue equipment and the UAV. It represents the minimum SNR, ensuring the transmission of critical data such as vital signs; constraint C6 stipulates that the task of the search and rescue equipment must be completed within a time slot; constraint C7 limits the computing resources of the drone; constraint C8 indicates that the remaining energy consumption of the drone must be less than the original energy consumption.

[0066] In step (3), each search and rescue device obtains the UAV location, computing resource usage, and mission information as the basis for establishing the system utility function.

[0067] Step (4) takes into account the load balance between drones, the fairness of search and rescue equipment unloading, the load balance of drones, and the reduction of drone energy consumption, establishes a system utility function, and establishes a deep reinforcement learning model for the search and rescue equipment that determines task unloading with the goal of maximizing the utility function, including the following specific steps:

[0068] (4a), based on the decision of drone trajectory planning and search and rescue equipment task offloading in mobile edge computing, a multi-agent Markov decision process is constructed, where each drone represents an agent. The partial observation space of each drone is composed of the search and rescue equipment, drone and target point coordinates in all systems, and the remaining energy consumption of the drone. It can be expressed as ,in, Indicates the target point location, represents the remaining energy consumption of the UAV from the beginning to time slot t, .

[0069] (4b) The task unloading of the search and rescue equipment and the load of the UAV are also included in the state space of the system. , which can be expressed as .

[0070] The action space of each drone It includes flight angle, flight distance, range coverage and resource allocation. The flight angle includes horizontal angle and vertical angle. Each action is normalized and the action space can be expressed as ;in, Represents the normalized parameter of the vertical direction angle of the UAV; Represents the normalized parameter of the horizontal direction angle; represents the normalized parameter of flight distance; represents the normalized coverage parameter, which can be calculated as

[0071] ;

[0072] in represents the normalized resource allocation parameter, which can be calculated as .

[0073] (4c), the reward function is divided into three parts: remaining energy consumption and fairness reward, arrival reward and wrong action penalty. For a single drone, more attention is paid to the remaining energy consumption in each time slot, which helps to observe the drone state, remaining energy consumption and fairness reward in each time slot. is defined as .

[0074] Designing a suitable arrival reward for each drone can ensure that the drone can reach the designated target area. Since the drone may continue to operate for dozens to hundreds of time slots during flight, and in actual operation, the drone can only intermittently receive signals and determine whether it has reached the destination, its arrival reward is usually very sparse. Since the initial strategy is randomly generated, the probability of reaching the target is close to zero. In order to solve the problem of sparse rewards, the arrival reward is designed. , which can be expressed as ;in, M 1 is a constant greater than 0. M 2 is a constant used to prevent the denominator from being zero.

[0075] When the drone flies out of the target disaster area, fails to complete the mission within the time limit, runs out of energy, or collides, the system will impose corresponding penalties based on the degree of violation. The penalty factor is set to .

[0076] (4d), considering the remaining energy consumption of the UAV and the fairness reward, arrival reward and wrong action penalty, the reward function is designed as .

[0077] The sum of all drone rewards Expressed as .

[0078] Step (5) is to train a deep reinforcement learning model based on IMATD3, add the OU noise process, enhance the cruising capability of the UAV in a dynamic disaster environment through time-related random perturbations, and adopt a priority cloning learning mechanism to intelligently copy and prioritize high-value experience samples to accelerate strategy convergence. The specific steps include the following:

[0079] (5a) Start the environment simulator and initialize the environment and each agent's network parameters, including the policy network, value network, and their corresponding target network. Obtain the global environment state.

[0080] (5b), set the training cycle m and the maximum training cycle M, and repeat the following steps until the training is completed. At each time step, obtain the global environment state at the current moment, obtain the partial observation state of each drone, and calculate the state of the drone based on the training cycle m. Greedy policy and policy network selection action At the same time, the OU noise mechanism is used to add noise perturbations to the action selection, so that the UAV can better explore the environment during the decision-making process, avoid falling into the local optimal solution, and enhance the intelligent agent's exploration ability in the dynamic disaster environment. and environmental status Calculate the penalty for incorrect actions and determine whether the drone collides or flies out of the target disaster area; if a collision or flight occurs, return to the initial position.

[0081] (5c), obtain the partial observation state at the next moment For each search and rescue device, a selection is made based on the minimum energy consumption greedy strategy. For each drone, the reward value is calculated .

[0082] (5d), get the global state at the next moment , and store the current moment's experience into the experience pool.

[0083] (5e) For each drone, a mini-batch of samples is extracted from the experience pool according to priority, and the loss functions of the two value networks are calculated to update the value network parameters. The policy network parameters are updated based on a deterministic policy gradient ascent method, and the target network parameters are adjusted. A priority-based experience cloning learning mechanism is incorporated. This priority mechanism allows the drone to focus more on experience samples with larger errors during training, thereby accelerating the learning process. Furthermore, to further improve the stability of the algorithm, a cloning learning mechanism is employed to clone some parameters of the current policy network to the target network. This not only preserves the stability of the target network but also ensures continuous optimization of the policy network by updating some parameters.

[0084] (5f) During the network update phase at each time step, a mini-batch of samples is extracted from the experience pool according to the sample priority. For each sample, the Q value of the two value networks is calculated. The target Q value is calculated as the minimum of the Q values output by the two target value networks.

[0085] (5g), the value network parameters are optimized by minimizing the error between the Q value output by the value network and the target Q value.

[0086] (5h), based on the output of the first online value network, the policy network parameters are updated using a deterministic policy gradient ascent method.

[0087] (5i), the target network parameters are gradually adjusted through the soft update mechanism.

[0088] Step (6), in the execution phase, each drone obtains corresponding actions based on the trained action network according to the environmental information it observes. After completing the task unloading, the system will distribute rewards based on the task completion status and update the drone's observation space in real time, thereby further promoting the execution of subsequent actions and dynamic adjustment of the state. Specifically, it includes the following steps:

[0089] (6a), get the global environment state at the current moment Set the execution cycle w and the maximum execution cycle W, and for each drone, obtain its partial observation state , and output actions based on the trained strategy network ,During the execution process, the OU noise mechanism is still retained to deal with the uncertainty in ,emergency situations.

[0090] (6b), perform the action , complete the search and rescue equipment calculation and unloading task, and calculate the reward value . Get the observation status of the next moment , and repeat the above process until the task is completed.

[0091] (6c), after the execution is completed, the cumulative reward value is counted and output .

[0092] exist Figure 3 The reward results for different algorithms provided by the present invention under different task loads are described in the following figure. Simulations and comparisons are also conducted with three other algorithms: the Multi-Agent Soft Actor-Critic (MASAC) algorithm, a deep reinforcement learning algorithm based on a randomized strategy and maximum entropy mechanism; the Traditional Convex Optimization (TRAMD) algorithm, which uses a traditional convex optimization algorithm to plan the flight of a UAV from its starting point to its destination; and the Random algorithm, in which each UAV randomly selects its flight direction and distance in each time slot. The results show that the IMATD3 algorithm demonstrates the best performance and adaptability. Taking a 15KB task as an example, the IMATD3 algorithm's reward is 88% higher than MASAC, 194% higher than TRAMDS, and 545% higher than Random. This fully demonstrates the superiority and stability of the IMATD3 algorithm in high-task load scenarios, and its ability to effectively balance task allocation and UAV energy consumption as data volume increases.

[0093] exist Figure 4 The reward graph for different algorithms and maximum computing resources for different drones, as provided by the embodiments of the present invention, shows that the advantages of the IMATD3 algorithm gradually become more apparent as computing resources increase. Even with the minimum computing resources, IMATD3 still performs best. For example, with a drone with a maximum computing resource of 6GHz, the HMATD3 algorithm's reward is 61% higher than MASAC, 132% higher than TRAMD, and 447% higher than Random, demonstrating the IMATD3 algorithm's ability to maximize computing resource utilization.

[0094] exist Figure 5In the figure, a comparison result of search and rescue equipment fairness indicators of different algorithms in different time slots provided by an embodiment of the present invention is described. The IMATD3 algorithm always maintains the highest search and rescue equipment fairness in all time slots. Taking the critical rescue period of time slot 30 as an example, the search and rescue equipment fairness of the IMATD3 algorithm is 21% higher than that of MASAC, 48% higher than that of TRAMD, and 149% higher than that of Random, which fully demonstrates the superiority of the IMATD3 algorithm in search and rescue equipment resource scheduling.

[0095] exist Figure 6 In the figure, a comparison result of the load balancing indicators of drones of different algorithms at different time slots provided by an embodiment of the present invention is described. At different time slots, the IAMTD3 algorithm uses the exploration strategy of OU noise enhancement to make the load balancing indicator always higher than other algorithms. Taking time slot 20 as an example, the load balancing indicator of the IAMTD3 algorithm is improved by 29% compared with MASAC, 101% compared with TRAMD, and 136% compared with Random, which fully demonstrates the excellent performance of the IMATD3 algorithm in load balancing and can achieve efficient resource scheduling in multi-drone collaborative tasks.

[0096] Based on the description of the present invention, it should not be difficult for those skilled in the art to see that the present invention proposes a multi-UAV collaborative computing offloading and resource allocation method for emergency search and rescue in sudden disasters, which can effectively improve the fairness of search and rescue equipment services, achieve load balancing of UAVs and reduce energy consumption.

[0097] Based on the same inventive concept, an embodiment of the present invention also discloses a multi-UAV collaborative computing offloading and resource allocation system for sudden disaster emergency search and rescue, including: a scenario construction module for building a multi-UAV assisted three-dimensional computing offloading network architecture for sudden disaster emergency search and rescue scenarios, wherein UAVs provide computing services for search and rescue equipment, and the search and rescue equipment moves within the disaster area and generates computing tasks; a problem modeling module for determining the offloading ratio parameter of the search and rescue equipment in each time slot based on the task offloading decision variables of the search and rescue equipment for each UAV, thereby measuring the fairness of task offloading among the search and rescue equipment; determining the resource allocation ratio parameter of the UAV based on the resource allocation decision variables of the UAV for each search and rescue equipment, thereby measuring the degree of load balancing among the UAVs; taking into account the offloading fairness of the search and rescue equipment, the load balancing of the UAVs, and the reduction of UAV energy consumption, establishing a system utility function, and establishing a deep reinforcement learning model for the search and rescue equipment that determines the task offloading with the goal of maximizing the utility function; a reinforcement learning module for training the deep reinforcement learning model based on an improved multi-agent dual-delay deep deterministic policy gradient algorithm, introducing OU noise and priority cloning learning mechanisms, and learning and optimizing the trajectory planning of the UAV and the task offloading decision of the search and rescue equipment.

[0098] An embodiment of the present invention also discloses a computer system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the multi-UAV collaborative computing offloading and resource allocation method for sudden disaster emergency search and rescue are implemented.

[0099] An embodiment of the present invention further discloses a computer program product, including a computer program, which, when executed by a processor, implements the steps of the multi-UAV collaborative computing offloading and resource allocation method for sudden disaster emergency search and rescue.

[0100] The program code for implementing the inventive method can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that the program code, when executed by the processor or controller, causes the steps of the inventive method to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as an independent software package and partially on a remote machine, or completely on a remote machine or server. The present invention is not described in detail herein, and all of these are known techniques to those skilled in the art.

Claims

1. A multi-UAV collaborative computing offloading and resource allocation method for emergency search and rescue of sudden disasters, characterized by: The steps include: Build a multi-UAV-assisted 3D computing offload network architecture for emergency disaster search and rescue scenarios, where UAVs provide computing services for search and rescue equipment, which moves within the disaster area and generates computing tasks. According to the decision variables of the search and rescue equipment for each UAV's task offloading, the offloading ratio parameter of the search and rescue equipment in each time slot is determined, and then the fairness of task offloading between the search and rescue equipment is measured. ; According to the resource allocation decision variables of the drone to each search and rescue equipment, the resource allocation ratio parameters of the drone are determined, and then the load balancing degree between the drones is measured. ; Consider the fairness of unloading search and rescue equipment , the load balancing degree of drones and reducing drone energy consumption , establish the system utility function , a deep reinforcement learning model is established for the search and rescue equipment that determines the task offloading with the goal of maximizing the utility function; For search and rescue equipment n In the time slot t Unloading ratio parameter, N The number of rescue equipment, U is the number of drones, Time slot t Search and rescue equipment n drones u The task offloading decision variables; For drones u In the time slot t The resource allocation ratio parameter, Time slot t drones u Search and rescue equipment n The resource allocation decision variables are f max Indicates the maximum computing resources provided by the drone to the search and rescue equipment; and is the weight, T represents the number of time slots; Based on the improved multi-agent double-delay deep deterministic policy gradient algorithm to train the deep reinforcement learning model, the Ornstein-Uhlenbeck noise and priority clonal learning mechanism are introduced to learn and optimize the trajectory planning of drones and the task offloading decision of search and rescue equipment.

2. The method for collaborative computing offloading and resource allocation of multiple UAVs for emergency search and rescue of sudden disasters according to claim 1 is characterized in that: The system includes multiple rescue equipment and multiple drones. The system's operating time includes multiple consecutive time slots, and the duration of each time slot can be dynamically adjusted according to the urgency of the rescue mission; the movement of drones in the system cannot exceed the boundaries of the target disaster area, the flight altitude does not exceed the upper limit, the distance between drones flying in adjacent time slots does not exceed the maximum distance, and the distance between drones is not less than the safe distance.

3. The method for collaborative computing offloading and resource allocation of multiple UAVs for emergency search and rescue of sudden disasters according to claim 1 is characterized in that: In the deep reinforcement learning model, each drone represents an intelligent agent. The drone's observation space includes the coordinates of all search and rescue equipment and all drones, the coordinates of the target point, the drone's remaining energy consumption, the task offloading status of the search and rescue equipment, and the drone's load status; the drone's action space includes flight angle, flight distance, range coverage, and resource allocation.

4. The method for collaborative computing offloading and resource allocation of multiple UAVs for emergency search and rescue of sudden disasters according to claim 1 is characterized in that: In the deep reinforcement learning model, the rewards include the remaining energy consumption and fairness reward, arrival reward and wrong action penalty; in the time slot t , residual energy consumption and fairness reward is defined as ,in, is the remaining energy consumption of the drone; arrival reward Expressed as ,in, 、 are the UAV and target locations respectively, represents the Euclidean distance, M 1 is a constant greater than 0. M 2 is a constant used to prevent the denominator from being 0. When the drone flies out of the target disaster area, fails to complete the mission within the time limit, runs out of energy, or collides, it will receive an error action penalty.

5. The method for collaborative computing offloading and resource allocation of multiple UAVs for emergency search and rescue of sudden disasters according to claim 4 is characterized in that: When training a deep reinforcement learning model based on an improved multi-agent dual-delay deep deterministic policy gradient algorithm, partial observation states are obtained for each drone. While selecting actions based on a greedy strategy and policy network, the Ornstein-Uhlenbeck noise mechanism is used to add noise perturbations to the action selection. Based on the selected action and the environmental state, penalties for incorrect actions are calculated, and it is determined whether the drone has collided or flown out of the target disaster area. If a collision or flight out of the target disaster area occurs, the drone returns to its initial position. Partial observation states are obtained at the next moment, and for each search and rescue device, a selection is made based on the minimum energy consumption greedy strategy. For each drone, a reward value is calculated. Get the global state of the next moment and store the experience of the current moment in the experience pool.

6. The multi-UAV collaborative computing offloading and resource allocation method for sudden disaster emergency search and rescue according to claim 1 is characterized in that: When training a deep reinforcement learning model based on the improved multi-agent dual-delay deep deterministic policy gradient algorithm, for each drone, batch samples are taken from the experience pool according to priority, and the loss functions of the two value networks are calculated. The value network parameters are then updated, the policy network parameters are updated based on the deterministic policy gradient ascent method, and the target network parameters are adjusted. The clone learning mechanism is used to clone some parameters of the current policy network to the target network.

7. A multi-UAV collaborative computing offloading and resource allocation system for emergency search and rescue of sudden disasters, characterized by: include: A scenario construction module is used to build a multi-UAV-assisted 3D computing offload network architecture for emergency disaster search and rescue scenarios, where UAVs provide computing services for search and rescue equipment, which moves within the disaster area and generates computing tasks; The problem modeling module is used to determine the offloading ratio parameter of the search and rescue equipment in each time slot based on the task offloading decision variables of the search and rescue equipment to each drone, and then measure the fairness of task offloading between the search and rescue equipment. ; According to the resource allocation decision variables of the drone to each search and rescue equipment, the resource allocation ratio parameters of the drone are determined, and then the load balancing degree between the drones is measured. ; Consider the fairness of unloading search and rescue equipment , the load balancing degree of drones and reducing drone energy consumption , establish the system utility function , a deep reinforcement learning model is established for the search and rescue equipment that determines the task offloading with the goal of maximizing the utility function; For search and rescue equipment n In the time slot t Unloading ratio parameter, N The number of rescue equipment, U is the number of drones, Time slot t Search and rescue equipment n drones u The task offloading decision variables; For drones u In the time slot t The resource allocation ratio parameter, Time slot t drones u Search and rescue equipment n The resource allocation decision variables are f max Indicates the maximum computing resources provided by the drone to the search and rescue equipment; and is the weight, T represents the number of time slots; The reinforcement learning module is used to train deep reinforcement learning models based on an improved multi-agent double-delay deep deterministic policy gradient algorithm. It introduces the Ornstein-Uhlenbeck noise and priority clonal learning mechanism to learn and optimize the trajectory planning of drones and the task offloading decisions of search and rescue equipment.

8. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by a processor, the steps of the multi-UAV collaborative computing offloading and resource allocation method for sudden disaster emergency search and rescue according to any one of claims 1 to 6 are implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the multi-UAV collaborative computing offloading and resource allocation method for sudden disaster emergency search and rescue according to any one of claims 1 to 6 are implemented.

Images