A method and system for cooperative reasoning of a drone swarm based on deep reinforcement learning

Abstract

The application provides a kind of unmanned aerial vehicle group cooperative inference method and system based on deep reinforcement learning, wherein the method comprises dividing the total flight time of the unmanned aerial vehicle group into multiple time periods, and the unmanned aerial vehicle group receives multiple inference tasks in each time period;Obtain the number of inference tasks received by the unmanned aerial vehicle group in the target time period and the time interval of the adjacent two inference tasks;In the case that the total time delay of the unmanned aerial vehicle group completing the target inference task is minimum, each inference task received by the unmanned aerial vehicle group in the target time period is divided into multiple inference subtasks;The multiple inference subtasks of the target inference task in the target time period are distributed to the unmanned aerial vehicle group to complete the unmanned aerial vehicle group cooperative inference.The application optimizes the task division strategy by deploying a deep neural network model on the unmanned aerial vehicle group and optimizing the task division strategy through deep reinforcement learning, multi-unmanned aerial vehicle cooperative inference, and improves the task execution efficiency.

Description

Technical Field

[0001] This invention belongs to the field of UAV swarm collaborative reasoning technology, and particularly relates to a method and system for UAV swarm collaborative reasoning based on deep reinforcement learning. Background Technology

[0002] Compared to traditional IoT devices, drones offer significant advantages, including lower cost, greater maneuverability, and stronger data collection capabilities. A major challenge in drone edge intelligence applications lies in processing the massive amounts of data collected by these aircraft in real time using advanced deep learning technologies. With the continuous development of artificial intelligence, deep neural networks are being used to process massive amounts of data. While deep neural networks excel at efficiently processing data, their complex structure and superior performance require substantial resource investment.

[0003] Typical deep neural network models consist of multiple layers with thousands of interconnected neurons, requiring significant computational and memory resources. However, the size and cost constraints of drones, along with their limited power supplies, result in relatively low computational capabilities. Traditionally, two approaches have been used to address this issue. One is to simplify complex models using model compression techniques, deploying lightweight models on the drone to accommodate its limited resources. While model compression reduces computational burden and inference latency, it leads to decreased inference accuracy. The other traditional approach is to transmit all source data back to the ground for processing; however, this introduces high communication latency and high costs.

[0004] To process this data in real-time and with high precision on UAVs, some researchers have proposed using cloud servers or edge servers to assist UAVs in collaborative computing. Generally, existing research can be divided into three categories: cloud server-assisted UAV collaborative reasoning, edge server-assisted UAV collaborative reasoning, and local collaborative reasoning for UAV swarms. Cloud server-assisted and edge server-assisted UAV collaborative reasoning methods perform local data preprocessing and select the type of ground station based on different application scenarios. Nevertheless, these two methods still face relatively high air-to-ground communication latency, data privacy breaches, and weak reliability and scalability because they rely on powerful ground computing equipment.

[0005] In contrast, without transmitting data to the ground, collaborative inference among multiple UAVs in a swarm can fully utilize the computing and storage resources of multiple UAVs to handle more complex operations, thereby reducing execution time and ensuring better fault tolerance. However, current work on UAV swarm collaborative inference rarely considers the characteristics and limitations of UAV networks, such as the unstable communication links between UAVs due to their mobility. Furthermore, most existing work only considers the scenario where multiple inference tasks arrive simultaneously for processing. However, in many UAV applications, such as target detection and recognition, images are captured frame by frame, so multiple inference tasks arrive at different times as a task stream. In conclusion, how to improve the performance of collaborative inference in UAV swarms while considering the characteristics of UAV mobility and the dynamic arrival of inference tasks remains a poorly solved problem. Summary of the Invention

[0006] This invention addresses the shortcomings of existing technologies by providing a method and system for collaborative reasoning in unmanned aerial vehicle (UAV) swarms based on deep reinforcement learning.

[0007] In a first aspect, the present invention provides a method for collaborative reasoning of unmanned aerial vehicle swarms based on deep reinforcement learning, comprising:

[0008] The total flight time of the drone swarm is divided into multiple time periods, and the drone swarm receives multiple inference tasks in each time period.

[0009] Obtain the number of inference tasks received by the drone swarm within the target time period and the time interval between two adjacent inference tasks;

[0010] To minimize the total latency of the drone swarm completing the target inference task, each inference task received by the drone swarm within the target time period is divided into multiple inference subtasks.

[0011] Multiple inference subtasks of the target inference task within the target time period are assigned to a drone swarm to complete drone swarm collaborative inference. Each drone executes only one inference subtask at a time. When one drone receives the target inference task, it executes any one of the inference subtasks in the target inference task and sends the intermediate result to the next drone. If the next drone is idle, it executes the assigned inference subtask of the target inference task. If the next drone is busy, it continues to process the previously assigned inference subtask.

[0012] Furthermore, to minimize the total latency of the UAV swarm completing the target inference task, each inference task received by the UAV swarm within the target time period is divided into multiple inference sub-tasks, including:

[0013] Step 301: Obtain the state space S based on the location of the inference task segmentation point; where each state in the state space represents the segmentation method of all current inference tasks in the drone swarm.

[0014] Step 302: Obtain the action space A based on the changes in the position of the inference task segmentation point, and use a hierarchical method to simplify the action complexity; take the discrete action of increasing or decreasing the number of deep neural network layers performed by the UAV as a large discrete action, and divide the large discrete action into multiple small discrete actions; obtain the initial large discrete action by combining multiple small discrete actions into a time series.

[0015] Step 303: Initialize weights, network parameters, and replay experience pool;

[0016] Step 304: Restore the environment and initial state;

[0017] Step 305: Select the action a with the highest score from action space A according to the ε-greedy strategy;

[0018] Step 306: Calculate the next state S' reached after using action a and the reward function R for action a, R = 2·T S′ -T S -T initial ;T initial T represents the total delay in the initial state. S T represents the total latency under the current inference task segmentation strategy state; S' The total latency is the time required to obtain the next reasoning task segmentation strategy state after the action is performed.

[0019] Step 307: Store the current state, the selected action, the reward function, and the next state as a set of data in the replay experience pool, and randomly select multiple sets of data from the replay experience pool as training data.

[0020] Step 308: Calculate the current Q-value and the target Q-value using a deep reinforcement learning neural network, and calculate the gradient function based on the target Q-value and the current Q-value, and update the weight matrix and bias vector parameters of the network; where the Q-value represents the expected reward value obtained in each subsequent state after taking action a in a given state;

[0021] Step 309: Repeat steps 305-308 until the preset step threshold is reached or the ε-greedy strategy effect converges to a stable value.

[0022] Furthermore, the allocation of multiple inference subtasks of the target inference task within the target time period to the UAV swarm to complete UAV swarm collaborative inference includes:

[0023] The latency of the target UAV executing the target inference subtask is calculated using the following formula:

[0024]

[0025] Among them, t i,n,comp The latency for the nth UAV to execute the i-th inference subtask; p i,n′ p is the n'th split point of the i-th reasoning subtask; i,n′-1 c is the (n'-1)th split point of the i-th reasoning subtask; i,l The computational cost of the l-th layer of the deep neural network performing the i-th inference subtask; f n Let $\mathbf{n}$ be the computational cost per second for the $n$-th drone.

[0026] The transmission delay of the intermediate results after the target UAV completes the target inference subtask is calculated using the following formula:

[0027]

[0028] Among them, t i,n,trans The transmission delay for transmitting intermediate results after the nth UAV completes the i-th inference subtask; w pi,n′ For the dividing point p i,n′ The output data size at this location is in bytes; ρ n Let be the data transmission rate of the nth drone during flight;

[0029] Calculate the total latency for the target UAV to complete the target inference subtask using the following formula:

[0030] t i,n =t i,n,comp +t i,n,trans ;

[0031] in,

[0032]

[0033] Among them, t i,n The total latency for the nth drone to complete the i-th inference subtask; T i,n The time t represents the time taken for the nth drone to complete the i-th inference subtask. d This is the time interval between two adjacent inference tasks received within the target time period.

[0034] Secondly, the present invention provides a drone swarm collaborative reasoning system based on deep reinforcement learning, comprising:

[0035] The time segmentation module is used to divide the total flight time of the drone swarm into multiple time periods, and the drone swarm receives multiple inference tasks in each time period.

[0036] The acquisition module is used to acquire the number of inference tasks received by the drone swarm within the target time period and the time interval between two adjacent inference tasks.

[0037] The inference task partitioning module is used to divide each inference task received by the UAV swarm within the target time period into multiple inference subtasks while minimizing the total latency of the UAV swarm completing the target inference task.

[0038] The subtask allocation module is used to allocate multiple inference subtasks of the target inference task within the target time period to the drone swarm to complete the drone swarm collaborative inference. Each drone executes only one inference subtask at a time. When one drone receives the target inference task, it executes any one of the inference subtasks in the target inference task and sends the intermediate result to the next drone. If the next drone is idle, it executes the inference subtask of the assigned target inference task. If the next drone is busy, it continues to process the previously assigned inference subtask.

[0039] Furthermore, the reasoning task partitioning module includes:

[0040] The first acquisition unit is used to acquire the state space S based on the location of the inference task segmentation point; where each state in the state space is the segmentation method of all current inference tasks in the drone swarm.

[0041] The second acquisition unit is used to acquire the action space A based on the changes in the position of the segmentation point of the inference task, and to simplify the action complexity using a hierarchical method; the discrete action of increasing or decreasing the number of deep neural network layers performed by the UAV is taken as a large discrete action, and the large discrete action is divided into multiple small discrete actions; by combining multiple small discrete actions into a time series, the initial large discrete action is obtained.

[0042] The initialization unit is used to initialize weights, network parameters, and the replay experience pool.

[0043] The recovery unit is used to restore the environment and its initial state.

[0044] The selection unit is used to select the highest-scoring action a from the action space A according to the ε-greedy policy;

[0045] The first calculation unit is used to calculate the next state S' reached after using action a and the reward function R of action a, R = 2·T S′ -T S -T initial ;T initial T represents the total delay in the initial state. S T represents the total latency under the current inference task segmentation strategy state; S'The total latency is the time required to obtain the next reasoning task segmentation strategy state after the action is performed.

[0046] The storage unit is used to store the current state, the selected action, the reward function, and the next state as a set of data in the replay experience pool, and randomly draw multiple sets of data from the replay experience pool as training data.

[0047] The second computational unit is used to calculate the current Q-value and the target Q-value through a deep reinforcement learning neural network, and to calculate the gradient function and update the weight matrix and bias vector parameters of the network based on the target Q-value and the current Q-value; where the Q-value represents the expected reward value obtained in each subsequent state after taking action a in a given state;

[0048] The loop unit is used to repeatedly execute the operations of the selection unit, the first calculation unit, the storage unit, and the second calculation unit until the preset number of steps threshold is reached or the ε-greedy strategy effect converges to a stable value.

[0049] Furthermore, the subtask allocation module includes:

[0050] The third calculation unit is used to calculate the latency of the target UAV performing the target inference subtask according to the following formula:

[0051]

[0052] Among them, t i,n,comp The latency for the nth UAV to execute the i-th inference subtask; p i,n′ p is the n'th split point of the i-th reasoning subtask; i,n′-1 c is the (n'-1)th split point of the i-th reasoning subtask; i,l The computational cost of the l-th layer of the deep neural network performing the i-th inference subtask; f n Let $\mathbf{n}$ be the computational cost per second for the $n$-th drone.

[0053] The fourth calculation unit is used to calculate the transmission delay of the intermediate results after the target UAV completes the target inference subtask, according to the following formula:

[0054]

[0055] Among them, t i,n,trans Transmission delay for transmitting intermediate results after the nth UAV completes the i-th inference subtask; For the dividing point p i,n′ The output data size at this location is in bytes; ρ n Let be the data transmission rate of the nth drone during flight;

[0056] The fifth calculation unit is used to calculate the total delay for the target UAV to complete the target inference subtask according to the following formula:

[0057] t i,n =t i,n,comp +t i,n,trans ;

[0058] in,

[0059]

[0060] Among them, t i,n The total latency for the nth drone to complete the i-th inference subtask; T i,n The time t represents the time taken for the nth drone to complete the i-th inference subtask. d This is the time interval between two adjacent inference tasks received within the target time period.

[0061] Thirdly, the present invention provides a computer device including a processor and a memory; wherein, when the processor executes a computer program stored in the memory, it implements the steps of the unmanned aerial vehicle swarm cooperative reasoning method based on deep reinforcement learning described in the first aspect.

[0062] Fourthly, the present invention provides a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, it implements the steps of the unmanned aerial vehicle swarm cooperative reasoning method based on deep reinforcement learning described in the first aspect.

[0063] This invention provides a method and system for collaborative inference among UAV swarms based on deep reinforcement learning. The method includes dividing the total flight time of the UAV swarm into multiple time periods, with the UAV swarm receiving multiple inference tasks within each time period; obtaining the number of inference tasks received by the UAV swarm within a target time period and the time interval between two adjacent inference tasks; dividing each inference task received by the UAV swarm within the target time period into multiple inference sub-tasks while minimizing the total latency of the UAV swarm in completing the target inference task; and allocating the multiple inference sub-tasks of the target inference task within the target time period to the UAV swarm to complete collaborative inference among the UAV swarms. This invention improves task execution efficiency by deploying a deep neural network model on the UAV swarm and optimizing the task partitioning strategy through deep reinforcement learning, enabling multi-UAV collaborative inference. Attached Figure Description

[0064] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 A flowchart illustrating a collaborative reasoning method for unmanned aerial vehicle swarms based on deep reinforcement learning, provided as an embodiment of the present invention;

[0066] Figure 2 An application scenario diagram of the UAV swarm collaborative reasoning method based on deep reinforcement learning provided in the embodiments of the present invention;

[0067] Figure 3 This is an example diagram of a task execution mode provided in an embodiment of the present invention;

[0068] Figure 4 This is a graph showing the performance comparison results of different algorithms under different computing capabilities provided in the embodiments of the present invention;

[0069] Figure 5 This is a schematic diagram of the structure of a drone swarm collaborative reasoning system based on deep reinforcement learning, provided as an embodiment of the present invention. Detailed Implementation

[0070] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0071] In one embodiment, such as Figure 1 As shown, this embodiment of the invention provides a method for collaborative reasoning of unmanned aerial vehicle swarms based on deep reinforcement learning, including:

[0072] Step S1: Divide the total flight time of the drone swarm into multiple time periods, and the drone swarm receives multiple inference tasks in each time period.

[0073] In this step, a pre-trained deep neural network model for task inference is deployed on each drone in the drone swarm. The drones in the swarm are sequentially numbered, and the inter-drone transmission rate is predicted based on the swarm's route, bandwidth, transmit power, channel gain, and noise power. The deep neural network model includes an input layer, hidden layers, and an output layer; in this embodiment, "layer" refers to the layer structure in the neural network, assuming a neural network has L layers.

[0074] Step S2: Obtain the number of inference tasks received by the drone swarm within the target time period and the time interval between two adjacent inference tasks.

[0075] Step S3: Under the condition of minimizing the total delay of the drone swarm completing the target inference task, divide each inference task received by the drone swarm within the target time period into multiple inference sub-tasks.

[0076] In this step, the initial deep reinforcement learning algorithm (DQN, Deep Q-Network) is adjusted to obtain the optimal segmentation method for all tasks and minimize the total latency for the drone swarm to complete all inference tasks.

[0077] Exemplarily, it includes:

[0078] Step 301: Obtain the state space S based on the location of the inference task segmentation point; where each state in the state space represents the segmentation method of all current inference tasks in the drone swarm.

[0079] Step 302: Obtain the action space A based on the changes in the position of the inference task segmentation point, and use a hierarchical method to simplify the action complexity; take the discrete action of increasing or decreasing the number of deep neural network layers performed by the UAV as a large discrete action, and divide the large discrete action into multiple small discrete actions; obtain the initial large discrete action by combining multiple small discrete actions into a time series.

[0080] Step 303: Initialize weights, network parameters, and replay experience pool.

[0081] Step 304: Restore the environment and initial state.

[0082] Step 305: Select the action a with the highest score from action space A according to the ε-greedy strategy.

[0083] Step 306: Calculate the next state S' reached after using action a and the reward function R for action a, R = 2·T S′ -T S -T initial ;T initial T represents the total delay in the initial state. S T represents the total latency under the current inference task segmentation strategy state; S' The total latency is the time required to perform the action and then proceed to the next reasoning task segmentation strategy state.

[0084] Step 307: Store the current state, the selected action, the reward function, and the next state as a set of data in the replay experience pool, and randomly select multiple sets of data from the replay experience pool as training data.

[0085] Step 308: Calculate the current Q-value and the target Q-value using a deep reinforcement learning neural network, and calculate the gradient function based on the target Q-value and the current Q-value, and update the weight matrix and bias vector parameters of the network; where the Q-value represents the expected reward value obtained in each subsequent state after taking action a in a given state;

[0086] Step 309: Repeat steps 305-308 until the preset step threshold is reached or the ε-greedy strategy effect converges to a stable value.

[0087] Step S4: Assign multiple inference subtasks of the target inference task within the target time period to the drone swarm to complete drone swarm collaborative inference.

[0088] like Figure 2 As shown, in this embodiment of the invention, the drone swarm consists of multiple drones with limited resources, each equipped with an onboard embedded computer. The limited resources of the drones include the computing power of the onboard embedded computer, memory size, communication bandwidth between drones, and battery capacity. Each drone executes only one inference subtask at a time. When one drone receives a target inference task, it executes any one of the inference subtasks within the target inference task and sends the intermediate result to the next drone. If the next drone is idle, it executes the assigned inference subtask of the target inference task; if the next drone is busy, it continues processing the previously assigned inference subtask. Each stage includes computation, transmission, and waiting, and the data transmission rate and remaining resources between drones change over time. Therefore, it is necessary to dynamically adjust the computational load and change the model slices when each drone executes each task.

[0089] like Figure 3 As shown, for each task, each drone retrieves the unfinished task from the previous drone and executes it, then waits for the next task to arrive. The completion time of each task includes computation time and data transmission time. The algorithm in this embodiment of the invention optimizes the partitioning of each computational task based on the drone's own state, channel conditions, available resources, and the number of tasks through deep reinforcement learning, thereby obtaining the best task partitioning method and the minimum total inference latency.

[0090] Each inference task is divided into multiple inference subtasks according to the layer structure of the deep neural network model. Each inference subtask is executed once in the order of the drone number, and each drone executes only one inference subtask at a time.

[0091] For example, the latency of the target UAV performing the target inference subtask is calculated according to the following formula:

[0092]

[0093] Among them, t i,n,comp The latency for the nth UAV to execute the i-th inference subtask; p i,n′ p is the n'th split point of the i-th reasoning subtask; i,n′-1 c is the (n'-1)th split point of the i-th reasoning subtask;i,l The computational cost of the l-th layer of the deep neural network performing the i-th inference subtask; f n Let be the computational load per second for the nth drone.

[0094] The transmission delay of the intermediate results after the target UAV completes the target inference subtask is calculated using the following formula:

[0095]

[0096] Among them, t i,n,trans Transmission delay for transmitting intermediate results after the nth UAV completes the i-th inference subtask; For the dividing point p i,n′ The output data size at this location is in bytes; ρ n Let be the data transmission rate of the nth drone during flight;

[0097] Calculate the total latency for the target UAV to complete the target inference subtask using the following formula:

[0098] t i,n =t i,n,comp +t i,n,trans .

[0099] in,

[0100]

[0101] Among them, t i,n The total latency for the nth drone to complete the i-th inference subtask; T 1,1 The time it takes for the first drone to complete its first inference subtask; t 1,1 The total latency for the first drone to complete the first inference subtask; T 1,n The time taken for the nth drone to complete the first inference subtask; (i-1)t d Let p be the arrival time of the i-th reasoning task; max(p,q) is the maximum value between p and q; T i,n The time t represents the time taken for the nth drone to complete the i-th inference subtask. d This is the time interval between two adjacent inference tasks received within the target time period.

[0102] Figure 4 The graph shows the performance comparison results of different algorithms provided in the embodiments of the present invention under different computing capabilities, where the horizontal axis represents the device type and the vertical axis represents the inference latency. Figure 4Five benchmark designs are used to partition and distribute inference tasks among UAVs with different configurations. In this scenario, embodiments of the present invention use five UAVs to collaboratively execute 30 inference tasks of VGG-16, using two devices with different computing capabilities. It can be observed that despite the different computing capabilities of the devices, the method provided by the embodiments of the present invention achieves better performance than other methods. Compared with the best method in the benchmarks, CoDNN, the method provided by the embodiments of the present invention reduces latency by 2.78% and 7.61% in high-computation-capability and low-computation-capability scenarios, respectively. In high-computation-capability scenarios, the latency of the method provided by the embodiments of the present invention is on average 17.56% lower than other methods. In low-computation-capability scenarios, the average reduction reaches 20%. Therefore, it can be seen that the method provided by the embodiments of the present invention has the lowest total latency for completing inference tasks regardless of whether high-computation-capability or low-computation-capability devices are used, giving it a certain advantage over other methods.

[0103] The UAV swarm collaborative reasoning method based on deep reinforcement learning provided in this invention improves task execution efficiency by deploying a deep neural network model on the UAV swarm and optimizing the task partitioning strategy through deep reinforcement learning, enabling multi-UAV collaborative reasoning.

[0104] Based on the same inventive concept, this invention also provides a UAV swarm collaborative reasoning system based on deep reinforcement learning. Since the principle of this system in solving problems is similar to that of the UAV swarm collaborative reasoning method based on deep reinforcement learning, the implementation of this system can refer to the implementation of the UAV swarm collaborative reasoning method based on deep reinforcement learning, and the repeated parts will not be described again.

[0105] In another embodiment, the UAV swarm cooperative reasoning system based on deep reinforcement learning provided in this invention, such as... Figure 5 As shown, it includes:

[0106] The time division module 10 is used to divide the total flight time of the drone swarm into multiple time periods, and the drone swarm receives multiple inference tasks in each time period.

[0107] The acquisition module 20 is used to acquire the number of inference tasks received by the drone swarm within the target time period and the time interval between two adjacent inference tasks.

[0108] The inference task partitioning module 30 is used to divide each inference task received by the UAV swarm within the target time period into multiple inference subtasks while minimizing the total latency of the UAV swarm completing the target inference task.

[0109] The subtask allocation module 40 is used to allocate multiple inference subtasks of the target inference task within the target time period to the UAV swarm to complete the UAV swarm collaborative inference. Each UAV executes only one inference subtask at a time. When one UAV receives the target inference task, it executes any one of the inference subtasks in the target inference task and sends the intermediate result to the next UAV. If the next UAV is in an idle state, it executes the inference subtask of the assigned target inference task. If the next UAV is in a busy state, it continues to process the previously assigned inference subtask.

[0110] For example, the reasoning task partitioning module includes:

[0111] The first acquisition unit is used to acquire the state space S based on the location of the inference task segmentation point; where each state in the state space is the segmentation method of all current inference tasks in the UAV swarm.

[0112] The second acquisition unit is used to acquire the action space A based on the changes in the position of the segmentation point of the inference task, and to simplify the action complexity using a hierarchical method; the discrete action of increasing or decreasing the number of deep neural network layers performed by the UAV is taken as a large discrete action, and the large discrete action is divided into multiple small discrete actions; by combining multiple small discrete actions into a time series, the initial large discrete action is obtained.

[0113] The initialization unit is used to initialize weights, network parameters, and the replay experience pool.

[0114] The recovery unit is used to restore the environment and its initial state.

[0115] The selection unit is used to select the highest-scoring action a from the action space A according to the ε-greedy policy.

[0116] The first calculation unit is used to calculate the next state S' reached after using action a and the reward function R of action a, R = 2·T S′ -T S -T initial ;T initial T represents the total delay in the initial state. S T represents the total latency under the current inference task segmentation strategy state; S' The total latency is the time required to perform the action and then proceed to the next reasoning task segmentation strategy state.

[0117] The storage unit is used to store the current state, the selected action, the reward function, and the next state as a set of data in the replay experience pool, and randomly draw multiple sets of data from the replay experience pool as training data.

[0118] The second computational unit is used to calculate the current Q-value and the target Q-value through a deep reinforcement learning neural network, and to calculate the gradient function and update the weight matrix and bias vector parameters of the network based on the target Q-value and the current Q-value; where the Q-value represents the expected reward value obtained in each subsequent state after taking action a in a given state;

[0119] The loop unit is used to repeatedly execute the operations of the selection unit, the first calculation unit, the storage unit, and the second calculation unit until the preset number of steps threshold is reached or the ε-greedy strategy effect converges to a stable value.

[0120] For example, the subtask allocation module includes:

[0121] The third calculation unit is used to calculate the latency of the target UAV performing the target inference subtask according to the following formula:

[0122]

[0123] Among them, t i,n,comp The latency for the nth UAV to execute the i-th inference subtask; p i,n′ p is the n'th split point of the i-th reasoning subtask; i,n′-1 c is the (n'-1)th split point of the i-th reasoning subtask; i,l The computational cost of the l-th layer of the deep neural network performing the i-th inference subtask; f n Let be the computational load per second for the nth drone.

[0124] The fourth calculation unit is used to calculate the transmission delay of the intermediate results after the target UAV completes the target inference subtask, according to the following formula:

[0125]

[0126] Among them, t i,n,trans Transmission delay for transmitting intermediate results after the nth UAV completes the i-th inference subtask; For the dividing point p i,n′ The output data size at this location is in bytes; ρ n Let be the data transmission rate of the nth drone during flight.

[0127] The fifth calculation unit is used to calculate the total delay for the target UAV to complete the target inference subtask according to the following formula:

[0128] t i,n =t i,n,comp +t i,n,trans .

[0129] in,

[0130]

[0131] Among them, t i,n The total latency for the nth drone to complete the i-th inference subtask; T i,n The time t represents the time taken for the nth drone to complete the i-th inference subtask. d This is the time interval between two adjacent inference tasks received within the target time period.

[0132] For more detailed information on the working process of each of the above modules, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0133] In another embodiment, the present invention provides a computer device including a processor and a memory; wherein the processor executes a computer program stored in the memory to implement the steps of the above-described deep reinforcement learning-based UAV swarm collaborative reasoning method.

[0134] For more detailed information on the above methods, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0135] In another embodiment, the present invention provides a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, it implements the steps of the above-described deep reinforcement learning-based UAV swarm collaborative reasoning method.

[0136] For more detailed information on the above methods, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0137] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The systems, devices, and storage media disclosed in the embodiments are described simply because they correspond to the methods disclosed in the embodiments; relevant details can be found in the method section.

[0138] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0139] The present invention has been described in detail above with reference to specific embodiments and exemplary examples; however, these descriptions should not be construed as limiting the present invention. Those skilled in the art will understand that various equivalent substitutions, modifications, or improvements can be made to the technical solutions and embodiments of the present invention without departing from the spirit and scope of the invention, and all such modifications and improvements fall within the scope of the present invention. The scope of protection of the present invention is defined by the appended claims.

Claims

1. A method for collaborative reasoning among unmanned aerial vehicle (UAV) swarms based on deep reinforcement learning, characterized in that, include: The total flight time of the drone swarm is divided into multiple time periods, and the drone swarm receives multiple inference tasks in each time period. Obtain the number of inference tasks received by the drone swarm within the target time period and the time interval between two adjacent inference tasks; To minimize the total latency of the drone swarm completing the target inference task, each inference task received by the drone swarm within the target time period is divided into multiple inference subtasks. Multiple inference subtasks of the target inference task within the target time period are assigned to a drone swarm to complete drone swarm collaborative inference. Each drone executes only one inference subtask at a time. When one drone receives the target inference task, it executes any one of the inference subtasks in the target inference task and sends the intermediate result to the next drone. If the next drone is idle, it executes the assigned inference subtask of the target inference task. If the next drone is busy, it continues to process the previously assigned inference subtask.

2. The UAV swarm cooperative reasoning method based on deep reinforcement learning according to claim 1, characterized in that, To minimize the total latency of the drone swarm completing the target inference task, each inference task received by the drone swarm within the target time period is divided into multiple inference sub-tasks, including: Step 301: Obtain the state space S based on the location of the inference task segmentation point; where each state in the state space represents the segmentation method of all current inference tasks in the drone swarm. Step 302: Obtain the action space A based on the changes in the position of the inference task segmentation point, and use a hierarchical method to simplify the action complexity; take the discrete action of increasing or decreasing the number of deep neural network layers performed by the UAV as a large discrete action, and divide the large discrete action into multiple small discrete actions; obtain the initial large discrete action by combining multiple small discrete actions into a time series. Step 303: Initialize weights, network parameters, and replay experience pool; Step 304: Restore the environment and initial state; Step 305: Select the action a with the highest score from action space A according to the ε-greedy strategy; Step 306: Calculate the next state S' reached after using action a and the reward function R for action a, R = 2·T S′ -T S -T initial ;T initial T represents the total delay in the initial state. S T represents the total latency under the current inference task segmentation strategy state; S' The total latency is the time required to obtain the next reasoning task segmentation strategy state after the action is performed; Step 307: Store the current state, the selected action, the reward function, and the next state as a set of data in the replay experience pool, and randomly select multiple sets of data from the replay experience pool as training data. Step 308: Calculate the current Q-value and the target Q-value using a deep reinforcement learning neural network, and calculate the gradient function based on the target Q-value and the current Q-value, and update the weight matrix and bias vector parameters of the network; where the Q-value represents the expected reward value obtained in each subsequent state after taking action a in a given state; Step 309: Repeat steps 305-308 until the preset step threshold is reached or the ε-greedy strategy effect converges to a stable value.

3. The UAV swarm cooperative reasoning method based on deep reinforcement learning according to claim 1, characterized in that, The method of allocating multiple inference subtasks of the target inference task within the target time period to the UAV swarm to complete UAV swarm collaborative inference includes: The latency of the target UAV executing the target inference subtask is calculated using the following formula: Among them, t i,n,comp The latency for the nth UAV to execute the i-th inference subtask; p i,n′ p is the n'th split point of the i-th reasoning subtask; i,n′-1 c is the (n'-1)th split point of the i-th reasoning subtask; i,l The computational cost of the l-th layer of the deep neural network performing the i-th inference subtask; f n Let $\mathbf{n}$ be the computational cost per second for the $n$-th drone. The transmission delay of the intermediate results after the target UAV completes the target inference subtask is calculated using the following formula: Among them, t i,n,trans The transmission delay for transmitting intermediate results after the nth UAV completes the i-th inference subtask; w pi,n′ For the dividing point p i,n′ The output data size at this location is in bytes; ρ n Let be the data transmission rate of the nth drone during flight; Calculate the total latency for the target UAV to complete the target inference subtask using the following formula: t i,n =t i,n,comp +t i,n,trans ; in, Among them, t i,n The total latency for the nth drone to complete the i-th inference subtask; T i,n The time t represents the time it takes for the nth drone to complete the i-th inference subtask. d This is the time interval between two adjacent inference tasks received within the target time period.

4. A collaborative reasoning system for unmanned aerial vehicle swarms based on deep reinforcement learning, characterized in that, include: The time segmentation module is used to divide the total flight time of the drone swarm into multiple time periods, and the drone swarm receives multiple inference tasks in each time period. The acquisition module is used to acquire the number of inference tasks received by the drone swarm within the target time period and the time interval between two adjacent inference tasks. The inference task partitioning module is used to divide each inference task received by the UAV swarm within the target time period into multiple inference subtasks while minimizing the total latency of the UAV swarm completing the target inference task. The subtask allocation module is used to allocate multiple inference subtasks of the target inference task within the target time period to the drone swarm to complete the drone swarm collaborative inference. Each drone executes only one inference subtask at a time. When one drone receives the target inference task, it executes any one of the inference subtasks in the target inference task and sends the intermediate result to the next drone. If the next drone is idle, it executes the inference subtask of the assigned target inference task. If the next drone is busy, it continues to process the previously assigned inference subtask.

5. The UAV swarm cooperative reasoning system based on deep reinforcement learning according to claim 4, characterized in that, The reasoning task partitioning module includes: The first acquisition unit is used to acquire the state space S based on the location of the inference task segmentation point; where each state in the state space is the segmentation method of all current inference tasks in the drone swarm. The second acquisition unit is used to acquire the action space A based on the changes in the position of the segmentation point of the inference task, and to simplify the action complexity using a hierarchical method; the discrete action of increasing or decreasing the number of deep neural network layers performed by the UAV is taken as a large discrete action, and the large discrete action is divided into multiple small discrete actions; by combining multiple small discrete actions into a time series, the initial large discrete action is obtained. The initialization unit is used to initialize weights, network parameters, and the replay experience pool. The recovery unit is used to restore the environment and its initial state. The selection unit is used to select the highest-scoring action a from the action space A according to the ε-greedy policy; The first calculation unit is used to calculate the next state S' reached after using action a and the reward function R of action a, R = 2·T S′ -T S -T initial ;T initial T represents the total delay in the initial state. S T represents the total latency under the current inference task segmentation strategy state; S' The total latency is the time required to obtain the next reasoning task segmentation strategy state after the action is performed; The storage unit is used to store the current state, the selected action, the reward function, and the next state as a set of data in the replay experience pool, and randomly draw multiple sets of data from the replay experience pool as training data. The second computational unit is used to calculate the current Q-value and the target Q-value through a deep reinforcement learning neural network, and to calculate the gradient function and update the weight matrix and bias vector parameters of the network based on the target Q-value and the current Q-value; where the Q-value represents the expected reward value obtained in each subsequent state after taking action a in a given state; The loop unit is used to repeatedly execute the operations of the selection unit, the first calculation unit, the storage unit, and the second calculation unit until the preset number of steps threshold is reached or the ε-greedy strategy effect converges to a stable value.

6. The UAV swarm cooperative reasoning system based on deep reinforcement learning according to claim 4, characterized in that, The subtask allocation module includes: The third calculation unit is used to calculate the latency of the target UAV performing the target inference subtask according to the following formula: Among them, t i,n,comp The latency for the nth UAV to execute the i-th inference subtask; p i,n′ p is the n'th split point of the i-th reasoning subtask; i,n′-1 c is the (n'-1)th split point of the i-th reasoning subtask; i,l The computational cost of the l-th layer of the deep neural network performing the i-th inference subtask; f n Let $\mathbf{n}$ be the computational cost per second for the $n$-th drone. The fourth calculation unit is used to calculate the transmission delay of the intermediate results after the target UAV completes the target inference subtask, according to the following formula: Among them, t i,n,trans The transmission delay for transmitting intermediate results after the nth UAV completes the i-th inference subtask; w pi,n′ For the dividing point p i,n′ The output data size at this location is in bytes; ρ n Let be the data transmission rate of the nth drone during flight; The fifth calculation unit is used to calculate the total delay for the target UAV to complete the target inference subtask according to the following formula: t i,n =t i,n,comp +t i,n,trans ; in, Among them, t i,n The total latency for the nth drone to complete the i-th inference subtask; T i,n The time t represents the time it takes for the nth drone to complete the i-th inference subtask. d This is the time interval between two adjacent inference tasks received within the target time period.

7. A computer device, characterized in that, It includes a processor and a memory; wherein, when the processor executes the computer program stored in the memory, it implements the steps of the deep reinforcement learning-based UAV swarm collaborative reasoning method according to any one of claims 1-3.

8. A computer-readable storage medium, characterized in that, Used to store computer programs; when the computer programs are executed by a processor, they implement the steps of the deep reinforcement learning-based UAV swarm collaborative reasoning method as described in any one of claims 1-3.

Images