Low-altitude intelligent networking dynamic collaborative reasoning method based on multi-agent reinforcement learning

Abstract

The application discloses a low-altitude intelligent networking dynamic collaborative reasoning method based on multi-agent reinforcement learning, relates to the technical field of low-altitude intelligent networks and edge artificial intelligence, and comprises the following steps: preloading a light model and a complex model on each unmanned aerial vehicle (UAV), deploying a complete complex model by means of a ground station, and constructing an air-ground integrated intelligent reasoning system. By introducing an enhanced multi-agent deep reinforcement learning algorithm, each UAV can dynamically select a model type, determine a model segmentation point, and reasonably allocate bandwidth and ground computing resources based on the state of the UAV, network conditions and task characteristics during task execution, so that multi-DNN flow heterogeneous resource-aware collaborative reasoning is realized. The application can effectively improve reasoning accuracy and reduce average delay under different device performance, bandwidth conditions and task density, has good system scalability and adaptability, and solves the problems of limited single-machine processing capacity, restricted communication resources and inefficient model switching.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of low-altitude intelligent networking and edge artificial intelligence, and particularly relates to a low-altitude intelligent networking dynamic collaborative reasoning method based on multi-agent reinforcement learning. BACKGROUND

[0002] With the development of low-altitude economy and artificial intelligence technology, unmanned aerial vehicles have the potential to transform from traditional data collection to intelligent perception and autonomous decision-making. The deep integration of the two promotes the formation of a low-altitude intelligent networking (LAIN) technology system with intelligent and collaborative as its core features, becoming an important infrastructure for the intelligent transformation of urban airspace.

[0003] In practical applications, multiple unmanned aerial vehicles often need to collaborate to perform intensive and real-time perception and decision-making tasks. For example, full-area image coverage and analysis in disaster scenarios, multi-view target recognition in urban high-altitude traffic monitoring, and intelligent warning in forest fire monitoring. These tasks usually involve multiple concurrent image / video DNN inference streams and require both accuracy and response speed of the inference results.

[0004] Traditional intelligent inference deployment modes mainly include the following two types: 1. Deploying a pruned and compressed lightweight model on each unmanned aerial vehicle locally, which can complete inference independently on the device. This solution can respond quickly with low latency, but the model's ability is limited, making it difficult to guarantee high accuracy in complex scenarios. 2. Sending raw data to the ground station via a wireless link, and completing the inference task on a high-performance server. This approach has higher accuracy, but it consumes a large amount of bandwidth and has significant communication delays, especially in dynamic scenarios, which can easily cause response timeouts or link congestion.

[0005] To balance the advantages and disadvantages of the above solutions, researchers have proposed a model partitioning + air-ground collaborative inference mechanism, which divides complex neural network models according to their hierarchical structure, with the front part deployed on the unmanned aerial vehicle for execution and the back part handled by the ground station. Through intermediate feature uploading, the mechanism reduces communication load and improves execution efficiency. This mechanism improves the adaptability of the collaborative inference system to some extent.

[0006] However, most of the existing research focuses on single-arch unmanned aerial vehicle performing single task or single model division and scheduling problem, which is difficult to adapt to the complexity of concurrent collaborative execution of "multi-unmanned aerial vehicle-multi-task flow". The main limitations are as follows: first, most of the schemes rely on preset division strategy, and the model division position is fixed, which cannot dynamically switch the reasoning path according to the task type, real-time load and link condition. Second, in the multi-task environment, the current method lacks fine-grained modeling of the computing resources, communication bandwidth and ground station processing capacity of the unmanned aerial vehicle, and it is difficult to realize efficient resource matching and dynamic scheduling. Third, the current system mostly adopts centralized task scheduling mode, which is easy to form a bottleneck in high-density task or multi-machine deployment environment, and lacks a multi-agent collaborative control strategy. Fourth, the existing method is difficult to learn the optimal model selection, reasoning division and resource allocation strategy based on the global perspective, which leads to the difficulty in optimizing the overall reasoning accuracy and delay at the same time. SUMMARY

[0007] The purpose of the present application is to provide a low-altitude intelligent networking dynamic collaborative reasoning method based on multi-agent reinforcement learning, which solves the problems existing in the prior art.

[0008] To achieve the above purpose, the present application provides a low-altitude intelligent networking dynamic collaborative reasoning method based on multi-agent reinforcement learning, comprising the following steps:

[0009] Step 100, deploying a lightweight neural network model and a complex neural network model on each unmanned aerial vehicle, and deploying a complex neural network model on a ground station;

[0010] Step 200, when receiving a reasoning task request, in each time slot, all unmanned aerial vehicles receiving the task request output reasoning decisions based on a multi-agent reinforcement learning model;

[0011] The reasoning decisions include model selection of each unmanned aerial vehicle, division point of the complex neural network model, allocation value of the communication bandwidth and allocation value of the ground station computing resources;

[0012] The model selection includes a lightweight neural network model and a complex neural network model;

[0013] Step 300, each unmanned aerial vehicle allocates communication bandwidth and ground station computing resources according to the reasoning decisions, and executes corresponding reasoning according to the reasoning decisions.

[0014] Further, both the lightweight neural network model and the complex neural network model use a deep neural network model to perform reasoning, and the deep neural network model includes an input layer, a hidden layer and an output layer.

[0015] Further, the unmanned aerial vehicle executes corresponding reasoning according to the reasoning decisions, which includes:

[0016] The unmanned aerial vehicle adopts a lightweight neural network model to perform an inference task alone, in which the communication bandwidth in the inference decision is a specific value, and the rest of the values are 0, and the unmanned aerial vehicle completes all the inference locally and returns the final result to the ground station;

[0017] The unmanned aerial vehicle adopts a complex neural network model and a ground station to cooperatively perform an inference task, divides the complex neural network model into an initial layer and a remaining layer according to a division point of the complex neural network model, processes the initial layer on the unmanned aerial vehicle, and transmits intermediate features generated by the initial layer to the ground station, where the remaining layer is completed.

[0018] Further, the multi-agent reinforcement learning algorithm is an enhanced multi-agent deep deterministic policy gradient algorithm, and the problem is constructed as a multi-agent Markov decision process, so that the unmanned aerial vehicles can jointly learn a real-time policy, and specifically includes:

[0019] Each unmanned aerial vehicle is an agent, inputs its own state and historical action information, outputs a four-dimensional action according to a policy space: model type selection, model division position, bandwidth allocation value, and computing resource request value; all agents share a centralized evaluator for joint training to achieve joint optimization; the learning goal is to maximize a system-level reward function defined as: inference accuracy weighted minus inference delay weighted; a decentralized strategy is executed in the inference process, and distributed decision-making is realized based on local observation.

[0020] Further, the policy space, state space, reward function, and training method of the multi-agent reinforcement learning model are as follows:

[0021] The policy space of the multi-agent reinforcement learning model is defined as: ;

[0022] The model selection variable of the i-th unmanned aerial vehicle, the division point of the complex neural network model, the communication bandwidth variable, and the ground station computing resource variable are represented by and respectively, and the specific descriptions are as follows:

[0023] : used for deciding whether to adopt a complex model; represents that the i-th unmanned aerial vehicle selects a local lightweight neural network model to perform independent inference, represents that the i-th unmanned aerial vehicle adopts a complex neural network model to cooperatively perform inference with the ground station;

[0024] : represents the maximum value of the division point of the complex neural network model, represents that when The division point of the complex neural network model, that is, the i-th unmanned aerial vehicle uploads the intermediate features to the ground station for continuing to execute the remaining layers after the inference task is executed by the i-th unmanned aerial vehicle after the β-th layer of the complex neural network model;

[0025] : represents the maximum value of the communication bandwidth resource, represents the communication bandwidth resource allocated for the i-th unmanned aerial vehicle;

[0026] : represents the maximum value of the ground station computing resource, represents the ground station computing resource allocated for the i-th unmanned aerial vehicle;

[0027] The state space of the multi-agent reinforcement learning model is defined as:

[0028] ;

[0029] represents the state space, represents the total number of unmanned aerial vehicles executing the inference task;

[0030] ;

[0031] represents the state of the i-th unmanned aerial vehicle at time t, represents the received inference request of the i-th unmanned aerial vehicle at time t, represents the on-board computing capability of the i-th unmanned aerial vehicle at time t, represents the on-board memory state of the i-th unmanned aerial vehicle at time t, represents the distance between the i-th unmanned aerial vehicle and the GS (ground station) at time t, represents the decision vector of the i-th unmanned aerial vehicle at time t-1;

[0032] The reward function of the multi-agent reinforcement learning model is the average of all local reward functions, and the specific local reward function is:

[0033] ;

[0034] wherein represents the local reward function of the i-th unmanned aerial vehicle at time t; and respectively represent the normalized inference accuracy and delay of the i-th unmanned aerial vehicle at time t, and respectively represent the normalized inference accuracy weight and the normalized delay weight;

[0035] The training method is that after performing the joint action, the agent receives the global reward and observes the next joint state; each transition tuple is stored in a priority experience replay buffer for policy learning; when enough experience is collected, a small batch is sampled and the network is updated; training is performed by minimizing the time difference loss.

[0036] Further, the unmanned aerial vehicle needs to meet the following constraint conditions when performing corresponding reasoning according to the inference decision:

[0037] ;

[0038] ;

[0039] ;

[0040] ;

[0041] ;

[0042] wherein is the total delay of the ith unmanned aerial vehicle at time t, denotes the duration of each time slot, denotes the total number of unmanned aerial vehicles performing the reasoning task, denotes the total number of time slots, , , and respectively represent the model selection of the ith unmanned aerial vehicle at time t, the division point of the complex neural network model, the allocated communication bandwidth resource and the allocated ground station computing resource, denotes the computational complexity of the jth deep neural network layer, denotes the onboard computing capacity of the ith unmanned aerial vehicle, denotes the memory requirement of the jth deep neural network layer, denotes the maximum storage capacity of the ith unmanned aerial vehicle.

[0043] Further, the output reasoning decision is to maximize the utility weighted sum of the average delay and the average accuracy of multiple task executions, and the objective function of the optimal problem is defined, and the calculation formula is:

[0044] ;

[0045] wherein, and are respectively the average normalized reasoning accuracy weight and the delay weight, and respectively represent the average normalized reasoning accuracy and delay of all unmanned aerial vehicles and time steps, and are defined as:

[0046] ;

[0047] ;

[0048] wherein represents the normalized inference accuracy of the ith UAV at time t, represents the normalized inference delay of the ith UAV at time t.

[0049] Further, when the UAV selects a lightweight neural network model to perform the task inference alone, the total delay is composed of the cumulative calculation time of all layers and the small transmission delay of sending the final output to the ground station, and the calculation formula is as follows:

[0050] ;

[0051] wherein, is the total delay of the ith UAV at time t when performing the inference task alone, is the maximum number of layers of the lightweight neural network model; is the amount of calculation required for the UAV to perform the inference task of the jth layer; Pi is the calculation capability of the ith UAV, represents the output data size of the lightweight neural network model, represents the constant nominal speed of the light transmission.

[0052] Further, when the UAV and the ground station are selected to jointly perform the inference task, the target layer of the complex neural network model is the smallest sub-task execution unit, and the total delay includes the calculation and processing of the UAV side to the β[t] layer, the transmission process of the intermediate feature map, and the calculation and processing of the ground station side to the remaining layers, and the calculation formula is as follows:

[0053] ;

[0054] wherein, is the total delay of the ith UAV at time t when performing the inference task with the ground station, is the data size of the output of the inference task of the jth layer performed by the ith UAV at time t; is the data size of the output of the inference task of the jth layer performed by the ith UAV at time t; is the signal transmission rate of the ith UAV at time t; and respectively represent the calculation delay of the UAV and the ground station performing the inference task, is the transmission delay of the ground station receiving the input data.

[0055] Further, the allocation of communication bandwidth and ground station computing resources according to the inference decision includes:

[0056] The communication bandwidth allocation adopts an orthogonal frequency division multiple access method, divides the total bandwidth into multiple orthogonal sub-frequency bands, and allocates the multiple orthogonal sub-frequency bands to different unmanned aerial vehicles, avoids communication interference in the system, dynamically adjusts the allocation proportion according to the channel quality of each unmanned aerial vehicle, and ensures the efficiency of the middle feature transmission; the ground station allocates corresponding computing resources within the maximum processing capacity according to the model divided by each unmanned aerial vehicle, and avoids the increase of inference delay caused by resource competition. Finally, the optimal strategy is obtained by using the reinforcement learning strategy.

[0057] Therefore, the low-altitude intelligent networking dynamic collaborative inference method based on multi-agent reinforcement learning has the following beneficial effects:

[0058] Firstly, the low-altitude intelligent networking dynamic collaborative inference method based on multi-agent reinforcement learning is different from the traditional data back transmission inference or single machine local inference mode, and the method proposed in the application deploys a deep learning model in a multi-machine air-ground system, and improves the task execution efficiency through air-ground collaborative inference.

[0059] Secondly, the low-altitude intelligent networking dynamic collaborative inference method based on multi-agent reinforcement learning adjusts the node types and model layer calculation amount of the participating nodes based on the real-time perception of the unmanned aerial vehicle and the task processing state, makes an optimized decision under the precision limit, and finally realizes low-delay collaborative inference, so that the system can more efficiently complete the inference task.

[0060] The technical solutions of the application will be further described in detail below with reference to the drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 A flowchart of a low-altitude intelligent networking dynamic collaborative inference method based on multi-agent reinforcement learning provided for an embodiment of the application;

[0062] Figure 2 A low-altitude intelligent networking dynamic collaborative inference scene diagram provided for an embodiment of the application;

[0063] Figure 3 A performance comparison result diagram of an algorithm under different task numbers when using AlexNet and VGG16 models provided for an embodiment of the application;

[0064] Figure 4 A performance comparison result diagram of an algorithm under different task numbers when using AlexNet and ResNet152 models provided for an embodiment of the application;

[0065] Figure 5A comparison chart of algorithm performance under different numbers of unmanned aerial vehicles when using AlexNet and ResNet152 models is provided for an embodiment of the present application.

[0066] Figure 6 A comparison chart of algorithm performance under different numbers of unmanned aerial vehicles when using AlexNet and ResNet152 models is provided for an embodiment of the present application.

[0067] Figure 7 A comparison chart of algorithm performance under different device computing capabilities when using AlexNet and VGG16 models is provided for an embodiment of the present application.

[0068] Figure 8 A comparison chart of algorithm performance under different device computing capabilities when using AlexNet and ResNet152 models is provided for an embodiment of the present application.

[0069] Figure 9 A comparison chart of algorithm performance under different total bandwidths when using AlexNet and VGG16 models is provided for an embodiment of the present application.

[0070] Figure 10 A comparison chart of algorithm performance under different total bandwidths when using AlexNet and ResNet152 models is provided for an embodiment of the present application.

[0071] Figure 11 A structure diagram of a low-altitude intelligent networking dynamic collaborative reasoning system based on multi-agent reinforcement learning is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0072] The following detailed description of embodiments of the application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0073] Please refer to Figure 1 The low-altitude intelligent networking dynamic collaborative reasoning method based on multi-agent reinforcement learning comprises the following steps:

[0074] Step 100, deploying a lightweight neural network model and a complex neural network model on each unmanned aerial vehicle, and deploying a complex neural network model on a ground station;

[0075] As Figure 2As shown, the embodiment of the application is composed of multiple UAVs and a ground station. The UAVs handle on-board computation, while the ground station supports more complex tasks, both using DNN models to perform inference. The system supports two model types: a small, lightweight model for small, less complex tasks requiring lower accuracy, and a large, complex model for larger, more complex tasks requiring higher accuracy, but with increased latency. We pre-deploy a small model and a large model on the UAVs. The small model runs independently on the UAVs, while the large model is partitioned and executed collaboratively by the UAVs and the ground station according to the task requirements.

[0076] At step 200, when receiving the inference task request, all UAVs receiving the task request output inference decisions based on the multi-agent reinforcement learning model in each time slot;

[0077] The inference decisions include model selection of each UAV, partition points of the complex neural network model, allocation values of the communication bandwidth, and allocation values of the computing resources of the ground station.

[0078] The model selection includes a lightweight neural network model and a complex neural network model.

[0079] Specifically, taking the UAVs as air nodes and the ground station as a ground node, after receiving the inference task request, the UAVs evaluate their own states and available communication resources, perceive the channel conditions of air-ground transmission under the constraints of the task completion time threshold and other system constraints, and further learn the allocation of communication resources and ground computing resources through an algorithm. In detail, the inference task is searched for an optimized decision through an enhanced multi-agent deep deterministic policy gradient (MADDPG) algorithm according to the model layer and the dynamic multi-DNN task flow, combining the statistical evaluation results and the perception results. The total delay of the entire task flow under different decision sets (including model selection, partition points, bandwidth allocation, and computing resource scheduling) is evaluated under the channel conditions at the current UAV location. Based on the reward values (i.e., “accuracy weighted minus delay weighted”) corresponding to the total delay and the inference accuracy under different value conditions, the multi-agent reinforcement learning model is trained to make the strategy gradually converge. According to the decision sets of all tasks obtained, the types of nodes participating (UAV local or air-ground collaboration) and the allocated model computation are adjusted: if the UAV local inference is selected, the preloaded lightweight model is called to complete the inference independently without model partitioning; if the ground station is needed for collaborative processing, the complex model is partitioned into two sub-tasks according to the partition points of the decision, the UAV executes the front model and transmits the intermediate features, and the ground station executes the remaining rear model to jointly complete the inference task.

[0080] At step 300, each UAV allocates the communication bandwidth and the computing resources of the ground station according to the inference decisions, and executes the corresponding inference according to the inference decisions.

[0081] The corresponding inference performed by the UAV according to the inference decision comprises:

[0082] (1) The UAV performs the inference task by using a lightweight neural network model alone, in which case the communication bandwidth in the inference decision is a specific value, and the rest of the values are 0, and the UAV completes all the inference locally and returns the final result to the ground station;

[0083] (2) The UAV performs the inference task in cooperation with the ground station by using a complex neural network model, and divides the complex neural network model into an initial layer and a remaining layer according to a division point of the complex neural network model, the initial layer is processed on the UAV, and the intermediate features generated are transmitted to the ground station, where the remaining layer is completed.

[0084] Both the lightweight neural network model and the complex neural network model use a deep neural network model to perform inference, and the deep neural network model comprises an input layer, a hidden layer and an output layer; the layer in this embodiment refers to the layer structure in the deep neural network.

[0085] The strategy space, state space, reward function and training method of the multi-agent reinforcement learning model are as follows:

[0086] The strategy space of the multi-agent reinforcement learning model is defined as: ;

[0087] respectively represent the model selection variable of the i-th UAV, the division point of the complex neural network model, the communication bandwidth variable and the ground station computing resource variable, and the specific explanations are as follows:

[0088] : used for deciding whether to use a complex model; represents that the i-th UAV selects a local lightweight neural network model for independent inference, represents that the i-th UAV uses a complex neural network model to perform inference in a cooperative manner with the ground station;

[0089] : represents the maximum value of the division point of the complex neural network model, represents the division point of the complex neural network model when , that is, when the i-th UAV performs an inference task, the inference task is performed by the i-th UAV after the first layer of the complex neural network model, and the intermediate features are uploaded to the ground station for further execution of the remaining layer;

[0090] : represents the maximum value of the communication bandwidth resource, represents the communication bandwidth resource allocated for the i-th UAV;

[0091] denotes the maximum of ground station computing resources, denotes the ground station computing resources allocated for the ith UAV.

[0092] The state space of the multi-agent reinforcement learning model is defined as:

[0093]

[0094] denotes the state space, denotes the total number of UAVs performing inference tasks;

[0095]

[0096] denotes the state of the ith UAV at time t, denotes the received inference requests of the ith UAV at time t, denotes the onboard computing capability of the ith UAV at time t, denotes the onboard memory status of the ith UAV at time t, denotes the distance between the ith UAV and the ground station (GS) at time t, denotes the decision vector of the ith UAV at time t-1.

[0097] The reward function of the multi-agent reinforcement learning model is the average of all local reward functions, and the specific local reward function is:

[0098]

[0099] wherein denotes the local reward function of the ith UAV at time t; and denotes the normalized inference accuracy and delay of the ith UAV at time t, and denotes the normalized inference accuracy weight and normalized delay weight.

[0100] The training method is: after performing a joint action, the agent receives the global reward and observes the next joint state; each transition tuple is stored in a priority experience replay buffer for policy learning; when enough experiences are collected, a small batch is sampled and the network is updated; training is performed by minimizing the time difference loss.

[0101] ​​​​Furthermore, this invention takes maximizing the utility-weighted sum of the average latency and average accuracy of multiple task executions as the optimization objective, and defines the objective function of the optimization problem as follows:

[0102] ;

[0103] in and These are the average normalized inference accuracy weight and the average normalized latency weight, respectively. and The average normalized inference accuracy and latency for all drones and time steps are defined as follows:

[0104] ;

[0105] ;

[0106] in Let represent the standardized inference accuracy of the i-th drone at time t. Let represent the standardized inference delay of the i-th drone at time t.

[0107] When a drone executes its reasoning based on a decision-making process, it must satisfy the following constraints, expressed as follows:

[0108] ;

[0109] ;

[0110] ;

[0111] ;

[0112] ;

[0113] in Let be the total delay of the i-th drone at time t. This indicates the duration of each time slot. This represents the total number of drones performing reasoning tasks. Indicates the total number of time slots. , , and Let $\mathbf{i}$ represent the model selection, the partition point of the complex neural network model, the allocated communication bandwidth resources, and the allocated ground station computing resources for the $i$ UAV at time $t$, respectively. This represents the computational complexity of the j-th deep neural network layer. This represents the onboard computing power of the i-th drone. memory requirement of the jth deep neural network layer, maximum storage capacity of the ith UAV.

[0114] The first constraint specifies that the task of the UAV must be completed within the time slot duration. To ensure fairness of resource allocation, the second constraint requires that the total communication bandwidth allocated to all UAVs in each time slot does not exceed the system-wide limit. The third constraint guarantees that the computational workload allocated to the ground station does not exceed its maximum processing capacity. Similarly, the fourth constraint constrains the computational load allocated to each UAV when executing complex models, ensuring that it does not exceed their respective capabilities. Finally, the last constraint enforces that the memory usage of each UAV does not exceed the on-board memory limit.

[0115] In the first case, when the UAV chooses to perform the task inference alone using a lightweight neural network model, the total delay consists of the cumulative computation time of all layers and the small transmission delay of sending the final output to the ground station, which is calculated as follows:

[0116] ;

[0117] where, total delay of the ith UAV at time t when performing the inference task alone, Lmaxis the maximum number of layers of the lightweight neural network model; Cj is the computational load required by the UAV to perform the inference task of the jth layer; P; is the computational capacity of the ith UAV, denotes the output data size of the lightweight neural network model, denotes the constant nominal rate of the lightweight transmission.

[0118] In the second case, when the UAV and the ground station are chosen to perform the inference task together, the target layer of the complex neural network model is the smallest sub-task execution unit, and the total delay includes the computation of the first β[t] layers at the UAV end, the transmission process of the intermediate feature map, and the computation of the remaining layers at the ground station end, which is calculated as follows:

[0119] ;

[0120] ;

[0121] where, total delay of the ith UAV at time t when performing the inference task with the ground station, data size of the output of the inference task of the jth layer performed by the ith UAV at time t; data size of the output of the inference task of the jth layer performed by the ith UAV at time t; signal transmission rate of the ith UAV at time t; and respectively represent the computing delay of the unmanned aerial vehicle and the ground station performing the inference task, is the transmission delay of the ground station receiving input data, is the channel gain, is the noise power.

[0122] In the first and second cases in the embodiment, only one of the two cases will occur, that is, the first case and the second case will not occur at the same time.

[0123] Further, the allocation of communication bandwidth and ground station computing resources according to the inference decision includes:

[0124] The communication bandwidth allocation adopts an orthogonal frequency division multiple access method to divide the total bandwidth into multiple orthogonal sub-frequency bands and allocate them to different unmanned aerial vehicles to avoid communication interference in the system, and dynamically adjust the allocation ratio according to the channel quality of each unmanned aerial vehicle to ensure the efficiency of intermediate feature transmission; the ground station allocates corresponding computing resources within the maximum processing capacity according to the model divided by each unmanned aerial vehicle, to avoid an increase in inference delay caused by resource competition. Finally, the optimal strategy is obtained using a reinforcement learning strategy.

[0125] Figure 3 、 Figure 4 is the system reward of the FECI and three baseline algorithms in the embodiment under different numbers of tasks. Among them, Figure 3 the complex model in is VGG16, Figure 4 the complex model in is ResNet152. Under all task loads from 10 to 50 tasks, FECI obtains the highest reward, and its advantage becomes more obvious as the task density increases. For example, in the case of 50 tasks, the average reward of FECI using VGG16 reaches 117.01, which is 6.3%, 14.2% and 33.4% higher than that of random selection, DMS and average bandwidth allocation (Aver-band), respectively. This performance improvement reflects the cooperative optimization capability of FECI in model selection, inference partitioning, bandwidth allocation and ground resource scheduling. In contrast, DMS has limited adaptability due to its fixed search strategy; random selection has a performance decline as the load increases due to its lack of task awareness; and average bandwidth allocation is limited by the uniform bandwidth allocation method and cannot cope with the heterogeneity of tasks. When the complex model is upgraded to ResNet152, Figure 4), all methods benefit from improved feature extraction, resulting in higher rewards. Even under this more demanding setting, FECI maintains the highest performance, outperforming the average bandwidth allocation (Aver-band), DMS, and random selection (Rand-select) by 11.52%, 28.35%, and 33.6% on average, respectively. This result confirms that FECI can effectively cope with increased computational and communication burdens by adjusting its decision-making process in real-time, achieving a robust balance between accuracy and latency.

[0126] Figure 5 、 Figure 6 We investigate how the system reward varies with the number of UAVs involved in the collaborative reasoning process. Figure 5 We compare the system reward under different numbers of UAVs, ranging from 3 to 9. FECI consistently achieves the highest reward across all configurations, demonstrating its strong scalability and robustness in distributed task management. On average, it outperforms random selection (Rand-select), DMS, and average bandwidth allocation (Aver-band) by 14.67%, 6.35%, and 14.45%, respectively. As the number of UAVs increases, competition for bandwidth and ground computing resources becomes more intense, leading to a slight decrease in overall system performance. However, FECI maintains the most stable reward trajectory, reflecting its exceptional adaptability to dynamic resource conditions. As task complexity increases, random selection (Rand-select) exhibits a more significant performance decline due to its unguided decision-making approach, highlighting its limitations under complex coordination requirements. Meanwhile, average bandwidth allocation (Aver-band) strictly enforces equal bandwidth allocation, demonstrating increasingly lower efficiency in large-scale UAV clusters, exposing its shortcomings in flexible resource scheduling. Notably, as the UAV fleet size expands, the performance advantage of FECI becomes more pronounced, indicating its strong ability to manage increasingly complex systems. In Figure 6 , when using ResNet152 instead of VGG16 as the complex model, the system reward of all methods increases significantly, confirming the advantages of deeper neural network architectures. Under different UAV configurations, FECI consistently outperforms all baseline methods. On average, it achieves 24.09%, 13.49%, and 15.2% improvements compared to random selection (Rand-select), DMS, and average bandwidth (Aver-band), respectively. These results are generally consistent with those observed in Figure 5 . However, unlike Figure 5Unlike other baselines, random selection significantly lags behind randomized model selection when applied in conjunction with ResNet152. This is primarily due to the higher risk associated with randomized model selection as model complexity increases, potentially leading to greater costs in terms of latency and accuracy. Overall, these findings further confirm the resilience and effectiveness of FECI in collaborative inference scenarios involving high system loads and complex model requirements.

[0127] In addition to the number of drones, Figure 7 , Figure 8 In addition, we also evaluated how heterogeneous airborne hardware affects algorithm performance, highlighting the adaptability of each method to resource differences. Figure 7 The evaluation was conducted with different computing capabilities (300). 10 6 560 10 6 1140 10 6 The system rewards for drones with embedded devices (multiplications per second) and using VGG16 as a complex model were compared. All methods showed an overall upward trend in rewards as device computing power increased, indicating that stronger local processing power helps alleviate latency and resource bottlenecks. FECI achieved the most significant and consistent gains across all settings, with average improvements of 8.79%, 3%, and 10.55% compared to Rand-select, DMS, and Aver-band, respectively. These results highlight the adaptability of FECI under different computing conditions. Notably, on high-performance devices such as the 1140M platform, FECI continues to achieve significant performance improvements, while Aver-band and Rand-select show only minor improvements. This difference stems from FECI's ability to dynamically adjust model partitioning and resource scheduling strategies, thus more effectively utilizing available computing power. In contrast, Aver-band's static bandwidth allocation and Rand-select's random decision-making fail to fully utilize available device computing resources. Figure 8 While keeping the device configuration unchanged, the evaluation was expanded by replacing the complex model from VGG16 to ResNet152. FECI maintained strong adaptability and stable performance as model complexity and system load increased. Figure 7Similarly, it consistently outperforms the baseline methods in terms of system rewards across all hardware platforms, and its effectiveness improves as the computational capacity increases. Baseline methods like DMS cannot achieve this level of fine-grained balancing due to their lack of adaptive coordination capabilities under mixed resource constraints. Notably, random selection (Rand-select) suffers from significant performance degradation due to the mismatch between randomly selected models and resource availability, which exacerbates the latency penalty. In summary, FECI demonstrates superior resource awareness and scheduling intelligence under heterogeneous hardware conditions, further validating its suitability for cooperative drone systems that span across different platform constraints.

[0128] Figure 9 、 Figure 10 The system performance under a range of bandwidth settings is investigated to reflect the variability of air-ground links in real-world scenarios. Figure 9 The impact of air-ground communication bandwidth ranging from 5MHz to 25MHz on system rewards is demonstrated under the AlexNet+VGG16 model setting. As the bandwidth increases, the rewards of all algorithms generally show an upward trend, indicating that enhanced communication capacity helps alleviate transmission latency and improve inference efficiency. FECI consistently achieves the highest rewards, increasing significantly from 21.91 to 26.736 across all bandwidth levels. On average, FECI outperforms random selection (Rand-select), DMS, and average bandwidth (Aver-band) by 14.22%, 5.93%, and 11.19%, respectively. This performance indicates that FECI can effectively utilize available bandwidth through dynamic resource scheduling. In contrast, average bandwidth (Aver-band) and random selection (Rand-select) exhibit a relatively flat trend, reflecting their inability to adapt to bandwidth changes due to static or random strategies. Although DMS benefits to some extent from increased bandwidth, its limited adaptability under constrained conditions results in inferior performance compared to FECI. Figure 10 This comparison is extended using ResNet152 as a complex model. Higher computational demands amplify the system's sensitivity to bandwidth availability, making communication capacity a more decisive factor in overall performance. Compared to the VGG16-based setting, the reward values of all methods increase, but FECI still maintains a clear advantage. Its rewards improve by nearly 25% from 5MHz to 25MHz, significantly outperforming the baseline methods. Averaged across the entire bandwidth range, FECI outperforms random selection (Rand-select), DMS, and average bandwidth (Aver-band) by 22.55%, 4.59%, and 16.9%, respectively. These results further demonstrate the advantages of FECI in adapting to bandwidth-constrained environments, especially under high-load inference tasks. Its bandwidth-aware design ensures efficient coordination and stable performance in dynamic multi-drone cooperative scenarios.

[0129] The unmanned aerial vehicle air-ground collaborative reasoning method based on adaptive switching of lightweight and complex models provided by the embodiment of the application selects a model reasoning mode by evaluating the resources and communication states of the unmanned aerial vehicle and the ground station, selects lightweight model alone reasoning or divides the high-precision complex neural network model and collaboratively reasons with the ground station, reduces the problems of high processing delay of complex models and insufficient precision of lightweight models, and improves the efficiency of the system in executing a reasoning task. The method provided by the embodiment of the application also uses a deep reinforcement learning algorithm, can quickly collect an optimal solution, and enables the air-ground system to dynamically select a suitable strategy to execute a computing task according to the state of the system and application requirements, thereby realizing collaborative reasoning with high energy efficiency and low delay.

[0130] Based on the same inventive concept, the embodiment of the application also provides a low-altitude intelligent networking dynamic collaborative reasoning system based on multi-agent reinforcement learning. Since the principle of solving problems by the system is similar to the air-ground dynamic collaborative reasoning method for low-altitude intelligent networking, the implementation of the system can be referred to the implementation of the unmanned aerial vehicle air-ground collaborative reasoning method based on adaptive switching of lightweight and complex models, and the repeated parts will not be described herein.

[0131] In another embodiment, the low-altitude intelligent networking dynamic collaborative reasoning method system based on multi-agent reinforcement learning provided by the embodiment of the application, as shown in Figure 11 , includes:

[0132] A model deployment module 10 is configured to deploy two specified trained deep learning models on the unmanned aerial vehicle and the ground station.

[0133] A strategy evaluation module 20 is configured to detect and collect low-altitude reasoning tasks. After receiving a reasoning task request, the unmanned aerial vehicle evaluates the system state, available resources, and channel conditions for air-ground transmission.

[0134] A task decision module 30 is configured to select whether the unmanned aerial vehicle uses a small lightweight deep neural network model alone to perform reasoning or allocates the task to the unmanned aerial vehicle and the ground station to collaboratively perform the reasoning task according to the evaluation result.

[0135] A task execution module 40 is configured to execute the task reasoning process according to the selected model reasoning mode.

[0136] The more specific working processes of the above-mentioned modules can be referred to the corresponding contents disclosed in the foregoing embodiments, which will not be described herein.

[0137] In another embodiment, the application provides a computer device including a processor and a memory. When the processor executes a computer program saved in the memory, the steps of the above-mentioned unmanned aerial vehicle group collaborative reasoning method based on model segmentation are implemented.

[0138] In another embodiment, the present application provides a computer readable storage medium, for storing a computer program; the computer program is executed by a processor to implement the steps of the above-mentioned low-altitude intelligent networking oriented air-ground dynamic collaborative reasoning method.

[0139] Those skilled in the art will understand that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer usable program code. The solutions in the embodiments of the present application can be implemented in various computer languages, such as object-oriented programming languages Java and interpreted scripting language JavaScript, etc.

[0140] The present application is described with reference to flowcharts and / or block diagrams according to the methods, devices (systems) and computer program products of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that implements the functions specified in the flowcharts and / or block diagrams.

[0141] These computer program instructions can also be stored in a computer readable memory that can direct the computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce a product including instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that implements the functions specified in the flowcharts and / or block diagrams.

[0142] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide a process for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that implements the functions specified in the flowcharts and / or block diagrams.

[0143] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application but not to limit it, and although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can still be modified or replaced equivalently, and these modifications or equivalent replacements should not make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present application.

Claims

1. A dynamic collaborative reasoning method for low-altitude intelligent networks based on multi-agent reinforcement learning, characterized in that, Includes the following steps: Step 100: Deploy a lightweight neural network model and a complex neural network model on each drone, and deploy a complex neural network model on the ground station; Step 200: Receive the reasoning task request and output the reasoning decision; In each time slot, all UAVs that receive inference task requests output inference decisions based on a multi-agent reinforcement learning model; Reasoning and decision-making include: model selection for each UAV, partitioning points for complex neural network models, allocation of communication bandwidth, and allocation of ground station computing resources; Model selection includes lightweight neural network models and complex neural network models; Step 300: Each UAV allocates communication bandwidth and ground station computing resources according to the reasoning decision, and executes the corresponding reasoning according to the reasoning decision; The multi-agent reinforcement learning algorithm is an enhanced multi-agent deep deterministic policy gradient algorithm. It constructs the problem as a multi-agent Markov decision process, enabling UAVs to jointly learn real-time policies. Specifically, it includes: Each drone is an intelligent agent, inputting its own state and historical action information, and outputting four-dimensional actions according to the policy space: model type selection, model partitioning position, bandwidth allocation value, and computing resource request value; all intelligent agents share a centralized evaluator for joint training to achieve joint optimization; the learning objective is to maximize the system-level reward function, defined as: inference accuracy weighted minus inference latency weighted; during inference, a decentralized strategy is executed, and distributed decision-making is achieved based solely on local observations; The allocation of communication bandwidth and ground station computing resources based on reasoning decisions includes: The communication bandwidth allocation adopts the orthogonal frequency division multiple access method, which divides the total bandwidth into multiple orthogonal sub-frequency bands and allocates them to different UAVs. At the same time, the allocation ratio is dynamically adjusted according to the channel quality of each UAV. The ground station allocates corresponding computing resources within its maximum processing capacity according to the post-model computing requirements of each UAV. Finally, the optimal strategy is obtained by using reinforcement learning strategy.

2. The dynamic collaborative reasoning method for low-altitude intelligent networks based on multi-agent reinforcement learning according to claim 1, characterized in that, Both lightweight neural network models and complex neural network models use deep neural network models to perform inference, which include input layers, hidden layers, and output layers.

3. The low-altitude intelligent network dynamic collaborative reasoning method based on multi-agent reinforcement learning according to claim 2, characterized in that, The drone performs the following reasoning based on its reasoning decisions: The drone uses a lightweight neural network model to perform inference tasks independently. In this case, the communication bandwidth in the inference decision is a specific value, and the rest are 0. The drone completes all inference locally and transmits the final result back to the ground station. The UAV uses a complex neural network model and a ground station to collaboratively perform inference tasks. The inference task is divided into an initial layer and a remaining layer according to the division points of the complex neural network model. The initial layer is processed on the UAV, and the intermediate features generated are transmitted to the ground station, where the remaining layer is completed.

4. The dynamic collaborative reasoning method for low-altitude intelligent networks based on multi-agent reinforcement learning according to claim 3, characterized in that, The policy space, state space, reward function, and training method of the multi-agent reinforcement learning model are as follows: The policy space of a multi-agent reinforcement learning model is defined as: ; Let $\mathbf{i}$ represent the model selection variable, the partition point of the complex neural network model, the communication bandwidth variable, and the ground station computing resource variable for the $i$-th UAV, respectively, as explained below: Used to decide whether to adopt a complex model; This indicates that the i-th drone selects a local lightweight neural network model for independent inference. This indicates that the i-th UAV uses a complex neural network model and performs inference in a collaborative manner with the ground station; : This represents the maximum value of the split point in a complex neural network model. Indicates when The partitioning point of the complex neural network model is as follows: when the i-th UAV is performing the inference task, after the β-th layer of the complex neural network model, the i-th UAV uploads the intermediate features to the ground station to continue executing the remaining layers. : This represents the maximum value of communication bandwidth resources. This represents the communication bandwidth resources allocated to the i-th UAV; : This represents the maximum value of the ground station's computing resources. This represents the ground station computing resources allocated to the i-th UAV; The state space of a multi-agent reinforcement learning model is defined as follows: ； Representing the state space, This represents the total number of drones performing inference tasks; ； This represents the state of the i-th drone at time t. Let represent the inference request received by the i-th drone at time t. This represents the onboard computing power of the i-th UAV at time t. This represents the onboard memory state of the i-th drone at time t. This represents the distance between the i-th UAV and the ground station at time t. Let represent the decision vector of the i-th drone at time t-1; The reward function of a multi-agent reinforcement learning model is the average of all local rewards. The specific local reward function is as follows: ； in Let represent the local reward function of the i-th drone at time t; and Let represent the normalized inference accuracy and latency of the i-th drone at time t, respectively. and These represent the normalized inference accuracy weight and the normalized latency weight, respectively. The training method is as follows: after performing a joint action, the agent receives a global reward and observes the next joint state; each transition tuple is stored in a priority experience replay buffer for policy learning; when enough experience is collected, a small batch is sampled and the network is updated; training is performed by minimizing the temporal difference loss.

5. The dynamic collaborative reasoning method for low-altitude intelligent networks based on multi-agent reinforcement learning according to claim 4, characterized in that, When a drone executes a reasoning decision based on inference, it must satisfy the following constraints: ； ； ； ； ； in Let be the total delay of the i-th drone at time t. This indicates the duration of each time slot. This represents the total number of drones performing reasoning tasks. Indicates the total number of time slots. , , and Let represent the model selection, the partition point of the complex neural network model, the allocated communication bandwidth resources, and the allocated ground station computing resources for the i-th UAV at time t, respectively. This represents the computational complexity of the j-th deep neural network layer. This represents the onboard computing power of the i-th drone. This represents the memory requirement of the j-th deep neural network layer. This represents the maximum storage capacity of the i-th drone.

6. The low-altitude intelligent network dynamic collaborative reasoning method based on multi-agent reinforcement learning according to claim 5, characterized in that, When making output inference decisions, the optimization objective is to maximize the utility-weighted sum of the average latency and average accuracy of multiple task executions. The objective function of the optimization problem is defined and its calculation formula is as follows: ； in, and These are the average normalized inference accuracy weight and the average normalized latency weight, respectively. and Let represent the average normalized inference accuracy and latency for all drones and time steps, respectively, as expressed below: ； ； in Let represent the standardized inference accuracy of the i-th drone at time t. Let represent the standardized inference delay of the i-th drone at time t.

7. The dynamic collaborative reasoning method for low-altitude intelligent networks based on multi-agent reinforcement learning according to claim 6, characterized in that, When a drone selects a lightweight neural network model for task inference alone, the total latency consists of the cumulative computation time of all layers and the small transmission latency of sending the final output to the ground station, as calculated in the following formula: ； in, Let be the total latency of the i-th drone performing an inference task alone at time t. The number of layers is the maximum number of layers in a lightweight neural network model; The computational cost required for the UAV to perform the j-th level inference task; Let i be the computing power of the i-th drone. This indicates the output data size of the lightweight neural network model. This indicates the constant nominal speed of a light-duty transmission.

8. The dynamic collaborative reasoning method for low-altitude intelligent networks based on multi-agent reinforcement learning according to claim 7, characterized in that, When both UAVs and ground stations are selected to perform inference tasks, the target layer of the complex neural network model is the smallest subtask execution unit. The total latency includes the computation and processing of the first β[t] layers by the UAV, the transmission of intermediate feature maps, and the computation and processing of the remaining layers by the ground station. The calculation formula is as follows: ； in, Let be the total latency of the i-th UAV when it collaborates with the ground station to perform inference tasks at time t. For the i-th drone to execute the first time at time t The size of the data output by the layered inference task; Let be the signal transmission rate of the i-th drone at time t; and These represent the computational latency of the drone and the ground station performing inference tasks, respectively. Transmission delay for ground station to receive input data.

Images