Unmanned aerial vehicle RAN slicing method based on multi-head attention enhancement knowledge distillation

Abstract

Due to the dynamic deployment characteristic, the unmanned aerial vehicle small base station is difficult to acquire enough data for model training. Therefore, according to the unmanned aerial vehicle RAN slicing method based on multi-head attention enhancement knowledge distillation, a ground base station large model is adopted to train an unmanned aerial vehicle-mounted RAN slicing small model, and the RAN slicing small model is used for RAN slicing of an unmanned aerial vehicle-mounted small base station. The training step comprises a multi-teacher knowledge distillation KD stage and a segmentation federal distillation SFD stage. And dynamically calculating the adaptive weight of each teacher by multiple attention deployed in mobile edge computing MEC, and realizing weighted integration of soft and hard label loss of each teacher model. By constructing global distillation loss, the student model is updated. In the SFD stage, clustering and segmentation strategies of heterogeneous perception are provided, and the model performance is further optimized through the SFD, so that the overall guiding ability of teachers to students is enhanced.

Description

Technical Field

[0001] This invention is an application of artificial intelligence technology in the field of communication technology, specifically a UAV RAN slicing method based on multi-head attention-enhanced knowledge distillation. Background Technology

[0002] As UAV-assisted communication technology evolves towards higher density and greater dynamism, resource allocation conflicts and interference coordination failures arising from the dynamic networking of heterogeneous UAV swarms pose significant challenges to Radio Allocation (RAN). The models carried by UAVs exhibit model heterogeneity due to differences in computing and storage capabilities. Low-cost UAVs with limited computing power can only deploy lightweight models, while high-computing-power UAVs can run complex models. On the one hand, traditional transfer learning (TL) methods, which rely on simple and efficient isomorphic transfer, are suitable for scenarios with minimal domain differences and abundant resources. This approach struggles to adapt to the model heterogeneity of UAV swarms, resulting in limited training performance. On the other hand, existing flow processing (FL) methods typically assume model isomorphism to facilitate model aggregation and parameter distribution. However, this assumption is ill-suited to real-world UAV scenarios, leading to decreased solution usability. Therefore, there is an urgent need to construct customized collaborative mechanisms for heterogeneous UAV swarms to accommodate resource constraints and model differences, and to fully leverage the knowledge value of high-capacity nodes.

[0003] Knowledge distillation (KD) is an effective model compression and performance optimization method. The basic principle is to transfer the implicit knowledge (such as feature distribution and decision boundary) of the high-complexity "teacher model" to the lightweight "student model" and use the professional knowledge of the teacher model to guide the training of the student model. KD provides a solution for model updating and collaborative optimization of model heterogeneous UAV swarms[1]. For a single UAV, multi-teacher distillation enhances the generalization and scene adaptability of the lightweight model by aggregating the heterogeneous knowledge of multiple base stations. By using knowledge representation (such as soft label and feature embedding) to replace model gradient transfer, federated distillation (FD) realizes collaborative optimization of UAV swarms across architecture models. It reduces the computational cost while avoiding interaction with the original data and realizes global knowledge sharing. Although it has many potentials, there are many challenges in the design of KD and FD for model heterogeneous UAVs, considering dynamic deployment, limited computing power and reliance on battery power.

[0004] 1) Optimization of knowledge transfer efficiency. Due to the heterogeneity of airborne models, there is a semantic gap in the knowledge representation space, which leads to cross-model feature alignment deviation and knowledge transfer efficiency decay. Traditional static feature alignment methods lack attention mechanisms and are difficult to adaptively capture the semantic relationship between models, which exacerbates the loss of knowledge. Fixed weight allocation method [2] cannot adjust the knowledge contribution of different models in real time, which easily leads to resource utilization imbalance. Li et al. [3] used a multi-scale feature attention mechanism to deal with the problem of insufficient early feature utilization in self-distillation and improved the knowledge transfer efficiency. Hierarchical multi-attention mechanism [4] alleviated the problem of insufficient deep knowledge utilization caused by single attention. Although it achieved multi-granularity feature alignment, it did not consider the adaptive adjustment of attention weights of different models and could not adapt to heterogeneous models. Liu et al. [5] used dynamic weight to filter key areas in the graph and prioritized the alignment of key teacher and student features. Although this method alleviated the feature weakening and cross-scale feature mismatch of small target detection in UAV scenarios, it could not support online knowledge transfer. Therefore, how to design a dynamic attention mechanism to align heterogeneous model features and improve knowledge transfer efficiency needs further research.

[0005] 2) Joint optimization of collaboration opportunities and heterogeneous models. Traditional clustering strategies are based on homogeneous models and cluster according to a single index. However, in real-world scenarios, the number of homogeneous model nodes may be small and their locations may be scattered. This not only exacerbates cross-cluster communication overhead but also limits the knowledge transfer ability of high-capacity nodes to low-computing-power nodes, resulting in a double decrease in the convergence speed and accuracy of the global model. The clustering method based on improved k-means and spectral clustering [6] models UAV nodes as graph structures to construct heterogeneous attribute graphs and capture multidimensional heterogeneous correlations. To adapt to topological changes, the dynamic clustering and path optimization method based on hybrid Q-learning [7] can adjust the clustering structure in real time, but the computational overhead is difficult to meet the real-time requirements of UAVs in high-maneuverability scenarios. In response to the problem of redundancy of multi-scale target features, Sun et al. [8] proposed feature similarity clustering, but did not consider extreme scale differences and real-time clustering overhead. It is evident that it is essential to construct a clustering method that comprehensively considers model morphology and knowledge sharing efficiency to optimize heterogeneous model training.

[0006] 3) Cost optimization of heterogeneous federated training. Multi-UAV collaborative training for heterogeneous models often suffers from bottlenecks such as poor gradient compatibility, high communication overhead, and unbalanced resource utilization. Existing distillation methods [9] reduce the amount of global communication data through local teacher nodes, but do not consider the contribution weight of local models, thus weakening the guidance ability of global teacher models. To this end, Aghli et al.

[10] proposed a selective layer weight pruning and KD joint compression method to balance the high deployment efficiency and low communication cost of complex model training. Through adaptive uniform cross-model voxel features, the FASD framework

[11] realizes heterogeneous model distillation, alleviating the bottlenecks of real-time performance and resource utilization in traditional feature extraction. However, feature alignment is limited by the cross-model semantic gap, which weakens the robustness of distillation. Cheng et al.

[12] proposed an attention mechanism oriented towards local-global frequency to alleviate the problem of knowledge heterogeneity and inefficient integration in multi-teacher distillation, but did not consider the overhead of model training. Based on this, the design of a KD method that takes into account both training cost and attention mechanism needs further exploration. Summary of the Invention

[0007] To address the heterogeneity of models in large-scale dynamic deployment scenarios of UAVs, this invention proposes a distillation-driven UAV (base station) resource slicing method to support the collaborative optimization of heterogeneous airborne models.

[0008] This invention presents a UAV RAN slicing method based on multi-head attention-enhanced knowledge distillation. In large-scale UAV collaborative training scenarios where a large ground-based model empowers a small airborne model, a UAV-borne small model is deployed in the UAV small base station as the RAN slice model. Resources are allocated to the slices based on the resource requirements predicted by the RAN slice model. The training of the UAV-borne RAN slice model includes two stages:

[0009] Phase 1, Multi-Teacher Knowledge Distillation (KD): A large model deployed across multiple ground base stations covering drones forms a teacher model team, while the smaller models deployed on drones are student models; the teacher model team guides the student models in their learning; each teacher model is a heterogeneous model.

[0010] The Transformer multi-head attention deployed on the mobile edge computing MEC dynamically calculates the adaptive weights of each teacher model, realizing the weighted integration of the soft and hard label losses of each teacher model; the student model is updated by constructing a global distillation loss.

[0011] Phase 2, Segmented Federated Distillation (SFD): The student model is segmented using a heterogeneous perception clustering and segmentation strategy; and the model performance is further optimized through SFD to enhance the global guidance capability of the teacher model for the student model.

[0012] The main innovations of this invention include:

[0013] 1) A multi-teacher distillation method based on multi-head attention. Multiple ground base stations act as a heterogeneous team of teacher models, guiding the student model carried by a drone to complete rapid offline fine-tuning through distillation. This method aims to overcome the coverage limitations of a single base station by fusing the soft and hard label losses of multiple base stations to construct a global distillation loss, thereby improving the cross-scene generalization ability of the student model.

[0014] 2) Model segmentation strategy based on heterogeneous clustering. Unlike traditional clustering strategies that only consider a single indicator (such as geographical location, data distribution, etc.), this strategy comprehensively considers the heterogeneity and locational relationships of UAV and ground base station models to better adapt to heterogeneous models and improve knowledge transfer efficiency. Based on task requirements and model complexity, model segmentation points are designed to ensure that the segmentation of shallow models minimizes the impact on deep models. This enables student models to acquire shallow knowledge while ensuring model scalability and efficiency.

[0015] 3) SFD method based on multi-head attention. The multi-head attention mechanism is used to measure the contribution weights of heterogeneous student models. The teacher model updates its model according to the weights to enhance its global guidance capability for the student models. Through multiple rounds of SFD, the student models continuously learn knowledge from the global teacher model, improving their environmental adaptability. Attached Figure Description

[0016] Figure 1 This represents a dynamic RAN slicing method that adapts to window size.

[0017] Figure 2 This describes a multi-teacher online distillation scenario assisted by MEC.

[0018] Figure 3 This indicates the details of multi-teacher distillation coordinated by MEC.

[0019] Figure 4 This indicates that multiple teachers are engaged in online distillation.

[0020] Figure 5 This represents clustering and model segmentation in SFD.

[0021] Figures 6(a) and 6(b) show the clustering and segmentation of the heterogeneous model, respectively, where: Figure 6(a) corresponds to cluster 1 and Figure 6(b) corresponds to cluster 2.

[0022] Figure 7 This indicates the SFD workflow.

[0023] Figure 8 This represents a detailed drawing of the SFD (Self-Device Rendering).

[0024] Figure 9 This indicates the impact of clustering methods on prediction errors.

[0025] Figures 10(a) to 10(f) show the variation of prediction error with communication rounds and the probability density function of the error for the three slices, respectively. Specifically: Figure 10(a) shows the variation of MSE of slice 1 with communication rounds; Figure 10(b) shows the variation of MSE of slice 2 with communication rounds; Figure 10(c) shows the variation of MSE of slice 3 with communication rounds; Figure 10(d) shows the probability density function of prediction error for slice 1; Figure 10(e) shows the probability density function of prediction error for slice 2; and Figure 10(f) shows the probability density function of prediction error for slice 3.

[0026] Figure 11 This indicates the impact of the number of resource blocks on performance isolation.

[0027] Figure 12 This indicates the impact of model segmentation on training costs. Detailed Implementation

[0028] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0029] 1 Overview

[0030] Due to their dynamic deployment characteristics, small drone base stations often struggle to collect enough data for model training. Even after initial model optimization, changes in the surrounding environment can cause model parameters to gradually become ineffective during movement.

[0031] To address this issue, this invention proposes a UAV RAN slicing method based on Transformer multi-head attention-enhanced knowledge distillation, comprising two stages: multi-teacher knowledge distillation (KD) and split-federated distillation (SFD). The aim is to construct a design paradigm where ground base station teacher models "feed back" to UAV student models, reducing the burden of training UAV small base station models. In the multi-teacher distillation stage, multiple ground base stations covering the UAV form a teacher team to guide UAV learning. Transformer multi-head attention deployed on Mobile Edge Computing (MEC) dynamically calculates the adaptive weights of each teacher, achieving weighted integration of the soft and hard label losses of each teacher. The student model is updated by constructing a global distillation loss. In the SFD stage, heterogeneous perception clustering and segmentation strategies are developed. Furthermore, SFD is used to optimize model performance, enhancing the teacher's global guidance capability for the student.

[0032] 2 System Model and Problem Description

[0033] 2.1 Network Scenarios

[0034] This invention considers a large-scale UAV collaborative training scenario where a large ground-based model empowers a small airborne model. In this scenario, heterogeneous models are deployed on different types of UAVs to predict resource requirements. The large model is deployed at ground base stations. The MEC controller accesses multiple ground base stations and coordinates data from these stations to assist in model training.

[0035] As an aerial small base station, the drone can be deployed and moved over a wide area as needed. When the drone enters the coverage area of ​​the ground base station, it can complete the handshake and inquiry with the ground base station through the 5GNR protocol

[13] . The ground base station refreshes the drone connection list periodically, and when a drone needs to leave the area to perform other tasks, it will issue a notice to release the connection.

[0036] 2.2 Problem Modeling and Solution Framework

[0037] The physical resources of each drone (small base station) are pooled and divided into multiple slices and a resource-sharing area. Each slice supports a type of customized service. Based on resource requirements, each drone dynamically determines the slice window length and resource allocation using a predictive model-driven method. Figure 1 drones in For example, suppose time is dynamically divided into multiple slice windows, each containing a certain number of discrete time slots.

[0038] Assuming a drone The total number of resource blocks (RBs) held is These resources are virtualized as There are service slices, and the slice set is denoted as . At the start of each window, the service slices and shared areas' Resource Blocks (RBs) are reallocated and remain so until the end of the window. At the end of each window, the drone... Calculate the resource requirements and shared area size for each slice in the next window. Arriving tasks are scheduled to different slices based on their attributes. Each slice allocates resources to tasks centered around a time slot. Within the window... Initially, drones slices The number of RBs allocated to the shared area is expressed as and The sum of the two cannot exceed the total number of RBs held by the drone, satisfying the condition that...

[0039] (1)

[0040] This is the set of slice windows for drone k. When slice resources are insufficient, the RBs in the shared area can be temporarily occupied.

[0041] In dynamic resource-sharing area optimization strategies, the Sliding Window Pressure Index (SWPI) is used to calculate the number of resource blocks (RBs) in the shared area. When the SWPI of the slice window increases, the number of RBs in the shared area should dynamically increase to alleviate resource contention pressure. Assume the slice window... resource pressure is ,recent The average pressure of each slice window is The variance of pressure fluctuation is The number of RBs in the shared area is then calculated as follows:

[0042] (2)

[0043] in, Adjust the weighting coefficient for fluctuations. This represents the standard deviation of historical stress. Based on equation (2), when Higher than When the shared area's capacity is linearly expanded; when Higher than This means that during sudden fluctuations, the expansion rate of the shared area will be increased. However, the maximum capacity of the shared area will not exceed [a certain limit]. 15%. Conversely, the shared area will shrink to achieve a balance between resource buffering and utilization efficiency.

[0044] Performance isolation ensures that the resource consumption of each slice does not interfere with the service provision of other slices, which is a necessary condition for the coexistence of multiple slices

[14] . The resource allocation strategy at the beginning of each slice window determines the quality of performance isolation of each slice within that window.

[0045] Assuming in time slot drones upper slice The number of RB requirements is The number of RBs occupied by the shared area is... The performance isolation quality at that moment is defined as...

[0046] (3)

[0047] From equation (3), we can see that and The smaller the difference between them ( The higher the prediction accuracy, the higher the performance isolation quality. If ,slice No need to occupy This will meet current needs and achieve the highest level of isolation quality. );like The shared area occupies all One RB is not enough to meet the requirements for slicing. Service demands, performance isolation is broken ( );like ,slice The need to temporarily occupy shared area resources to meet current demands weakens performance isolation. ).

[0048] drones The average isolation quality of the upper slice is expressed as Apply it to The long-term slicing performance isolation optimization problem for all windows in the process is modeled as follows:

[0049] (4)

[0050] question The essence is to use drones Centered on the resource constraints, the number of RBs in the shared area is dynamically allocated according to equation (2). At the same time, determine the resource allocation for each slice. ) and slice window length ( This is to achieve the optimal performance isolation quality.

[0051] The subsequent sections 3 and 4, based on the Transformer multi-head attention mechanism, design knowledge distillation KD and federated distillation FD methods for single-UAV and multi-UAV collaborative scenarios, respectively, to solve the problem. .

[0052] 3. Transformer-enhanced multi-teacher distillation

[0053] Upon reaching a new area, a drone may be covered by multiple ground base stations. Drones have limited computing resources, making it difficult to support long-term, complex online training. Simultaneously, the heterogeneous models carried by the drone exhibit differences in knowledge representation, leading to decreased knowledge transfer efficiency. To address this, this section proposes a Transformer-enhanced multi-teacher distillation method. The drone and ground base stations establish teacher-student and collaborative relationships through a handshake. Under the coordination of the MEC (Multi-Teacher Distillation Center), large models on adjacent base stations form teacher groups to help the drone achieve low-cost, rapid online model updates.

[0054] by Figure 2 Let's take an example to explain the proposed method. Ground base stations 1 and 2, covering UAV 3, are selected as teachers, and UAV 3 is the student. To reduce the cost of model fine-tuning, the student model carried by UAV 3 is uploaded to the MEC controller via the base stations. The multi-head attention mechanism deployed in the Transformer of the MEC is used to calculate the contribution weights of each teacher model to generate a global distillation loss and update the student model. The updated student model is then fed back to UAV 3 via the original path.

[0055] 3.1 Teacher Selection

[0056] In the Transformer-enhanced multi-teacher distillation process, the MEC controller needs to appropriately select the coverage drones (students). This section utilizes historical data stored by each teacher to ensure the effectiveness and representativeness of knowledge transfer. It employs a multi-base station method based on KL divergence and its local data selection approach to ensure that task-relevant and valid data is selected for multi-teacher distillation.

[0057] KL divergence

[15] serves as the standard for selecting the target domain (i.e., the base station participating in model fine-tuning). The smaller (larger) its value, the more similar (different) the distribution of historical data provided by the UAV and the base station is. Based on this standard, similar data is used as much as possible for model fine-tuning to reduce the risk of overfitting. Assuming the UAV... and base stations The data distribution is as follows: and ( ).

[0058] (5)

[0059] In equation (5), and It is a drone and base stations The hidden representation of the model. (And drones) Connected base stations KL divergence

[0060] (6)

[0061] in, for Information entropy; for and Cross-entropy; It represents the set of all possible distributions.

[0062] Coverage drones The set and cardinality of ground base stations are respectively represented as and The set of target domain base stations selected to participate in model fine-tuning is as follows:

[0063] (7)

[0064] Among them This is the KL divergence threshold. (Base station) Choice and Fine-tuning of similar historical data This portion of the data accounts for 10%

[0065] (8)

[0066] in, It is negatively correlated with the amount of data selected.

[0067] Based on the above method, assuming that the set of ground base stations (teachers) selected by MEC and the cardinality are respectively and The proportion of selected data is

[0068] (9)

[0069] Assuming a drone upper slice exist The actual resource demand vector for each time slot is represented as: So, drones The actual number of RBs for all slices is

[0070] (10)

[0071] drones upper slice exist The number of RBs required for the wheel is predicted to be

[0072] (11)

[0073] in, These are the parameters for the student model.

[0074] Next, the loss function Used to measure drones upper slice Prediction error of the required number of RBs. (UAV) The loss function for the upper slice can be expressed as:

[0075] (12)

[0076] The student model uses gradient descent. Updated to

[0077] (13)

[0078] in, It is the learning rate updated in the student model. Indicates drone The gradient of the loss function.

[0079] 3.2 Online Distillation Based on Multi-Head Attention

[0080] drones After the student model was uploaded to the MEC controller by the base station, online distillation was completed under the guidance of the teaching team. In the... Wheels, ground base stations superior The resource requirements of a slice are represented as ,slice exist The actual resource demand vector for each time slot is represented as: Ground base station (Teacher Model) Slice exist The number of RBs predicted in each round is

[0081] (14)

[0082] in, For teachers The model parameters. Based on this, A slice in The number of RBs required for round prediction is

[0083] (15)

[0084] in, This is a soft tag. Input into drone The student model on the above, the output is represented as

[0085] (16)

[0086] like Figure 3 As shown, the goal of distillation is to ensure the output of the student model during training. ) and corresponding hard tags and soft tags ( and The total training loss consists of two parts: soft-label loss and hard-label loss. Assume... and These are the soft-label loss and the hard-label loss, respectively.

[0087] (17)

[0088] The loss value is determined by the mean square error between the predicted values ​​of the student model and the teacher model and the actual value of the student model.

[0089] Within the considered multi-teacher distillation framework, the Transformer's multi-head attention mechanism measures the contribution of each teacher model to the student model through the dot product similarity of the query and key, thus focusing the model on highly relevant information. For example... Figure 4 As shown, TE will Encode it into an input sequence. This input sequence is then stacked through multiple layers to generate vectors about the Query, Key, and Value. , and .

[0090] (18)

[0091] Through the multi-head attention mechanism, teachers The attention weights are

[0092] (19)

[0093] Based on equation (19), The attention weight vector of Chinese teachers is represented as follows The optimization objective of multi-teacher distillation is to minimize the following total training loss.

[0094] (20)

[0095] Among them, parameters is a weighting coefficient used to adjust the weight between soft label loss and hard label loss, thereby controlling the student model's attention to hard and soft labels during training. Based on equation (20), the student model, through gradient descent, Updated to

[0096] (twenty one)

[0097] in, It is the learning rate for model training. This represents the gradient of the multi-teacher distillation loss function.

[0098] The details of multi-teacher distillation training are summarized in Algorithm 1, which is executed on the MEC controller. Under the multi-head attention mechanism, The attention weights of the secondary teachers are calculated (see lines 3-5). The total training loss of the multi-teacher distillation process consists of the soft and hard label losses and the attention weights (see line 6). Gradient descent is used to update the student model and obtain... (See line 7). By using dynamic weight allocation through multi-head attention and unified weight calculation via MEC, this training reduces distillation time and lowers the cost of online training for UAVs.

[0099]

[0100] 4 Transformer-enhanced split-fed distillation

[0101] After completing multi-teacher distillation, the drones can decide whether to stay or leave. For neighboring drones that remain, this section designs a Transformer-enhanced partitioned federated distillation (SFD) method to further enhance model performance through knowledge transfer from the global teacher model. (Continuing...) Figure 1 The scenes shown, such as Figure 5 As shown, the scheme includes three steps: heterogeneous perception clustering, selection of model segmentation points, and distillation-driven segmentation federated training.

[0102] 4.1 Clustering and Segmentation in Heterogeneous Sensing

[0103] In multi-UAV collaborative scenarios, the models carried by the UAVs (airborne models) exhibit heterogeneity. This heterogeneity means that model parameters or intermediate features cannot be directly fused, and traditional clustering methods based on the isomorphism assumption are unsuitable for heterogeneous federated aggregation. To address this, this section designs a heterogeneous perception-based clustering method that comprehensively considers both UAV location and airborne model morphology, aiming to balance federated aggregation opportunities with reduced collaborative training costs. Ground base stations and UAV base stations serve as cluster heads and cluster members, respectively.

[0104] The complexity of the airborne model is measured by FLOPs (Floating Point Operations), which is positively correlated with the model complexity

[16] . Assume that the set and cardinality of the UAVs residing in adjacent areas are respectively... and Drones and base stations The positions are respectively represented as and Taking into account both location information and model complexity, the drone... The feature vectors are constructed as

[0105] (twenty two)

[0106] in, Indicates drone The model's FLOPs. Assumptions Used to measure drones With base station The weighted relationship between (centroids) is expressed as:

[0107] (twenty three)

[0108] in, Used to measure drones and The FLOPs differences of the model. Assuming clusters The collection of drones in the middle is Based on equation (23), cluster head The selection was determined to be

[0109] (twenty four)

[0110] As shown in Figure 6, drones with small FLOP differences (meaning similar model shapes) and close locations are grouped into clusters. In order to improve the efficiency of federated aggregation and the energy of UAVs. The clustering process of heterogeneous perception is summarized as Algorithm 2. According to equations (23) and (24), each UAV is assigned to the cluster with the closest location and the most similar model.

[0111]

[0112] The goal of model segmentation is to minimize the impact of shallow models on deep models while maintaining model performance. The selection of segmentation points is determined by the FLOPs of each member within the cluster. For example... Figure 5 As shown, the selection of segmentation points must ensure consistency in the shallow structure of the student model (number of layers in the aerial model), i.e., parameter dimension matching. This allows ground base stations to handle complex deep feature extraction, while drones primarily perform lightweight shallow computation tasks.

[0113] Assuming a drone The stacking model contains Layer, number The FLOPs of a layer are represented as Then from the 1st floor to the 2nd floor The FLOPs of the layer (segmentation layer) are

[0114] (25)

[0115] To measure the impact of reducing the number of model layers on the overall model complexity, layer splitting was performed. The model complexity gradient is

[0116] (26)

[0117] in, The FLOPs represent the complete student model. The result of equation (26) reflects the segmentation layer. Compared to the first The FLOPs difference rate of each layer. To ensure the optimal model complexity ratio among different UAVs, the layers are segmented. The selection must satisfy the following conditions:

[0118] (27)

[0119] in, It is a balancing factor among FLOPs to balance the model complexity among drones and improve knowledge utilization. It is balance Adjustment factor for impact.

[0120] 4.2 Model Segmentation and Local Iteration

[0121] To reduce the cost of collaborative training, a federated segmentation method is constructed. (UAV) The carried Bi-LSTM model, i.e. It is divided into an airborne and a ground-based model

[17] , represented as and , which correspond to the shallow and deep layers of the original model, respectively. Keep it locally. Transferred to the base station acting as the cluster head The two split models transmit backpropagated gradients wirelessly.

[0122] Assuming in time slot The resource prediction value of forward LSTM is The resource prediction value of the inverse LSTM is Therefore, the resource prediction value of Bi-LSTM is

[0123] (28)

[0124] in, This represents the combination of two hidden states.

[0125] drones upper slice In the window The number of resource blocks required is predicted to be

[0126] (29)

[0127] This section uses MSE loss, i.e. The loss of the model output is measured

[18] . Substitution (29), UAV upper slice In the window The required number of RBs is predicted to be

[0128] (30)

[0129] based on and loss function Used to measure drones upper slice Prediction error of the required number of RBs. Regarding... The loss function is expressed as

[0130] (31)

[0131] make and represent and The loss function. and The summation is represented as

[0132] (32)

[0133] This process uses source domain data and newly acquired data carried by the drone to... Forward propagation. The resulting intermediate features, also known as shredded data, are transmitted to the base station. Base station Perform forward and backward propagation to update And it transmits the gradient back. Upon receiving the gradient, the drone transmits it via backpropagation. The aerial model has been updated to

[0134] (33)

[0135] in, It is a drone The learning rate. A complete forward and backward propagation process described above is called a local iteration.

[0136] Subsequently, based on the local iterative process of the above segmentation model, the drone... The aerial model (shallow layer) has been updated to The ground model (deep layer) has been updated to .

[0137] 4.3 Distillation-Driven Split Federated Training

[0138] by Figure 5 Taking cluster 2 as an example, after updating the shallow model, the teacher receives... The system represents the knowledge of each student and uses the dot product similarity between the query and the key to measure the contribution weight of different students. In this process, such as... Figure 7 As shown, unlike in Section 3, in order to optimize SFD, the Transformer's TE is integrated into the ground base station instead of being deployed in the MEC to enhance the model's generalization ability.

[0139] TE will Encode it into an input sequence. This input sequence is then stacked multiple times to generate... , and . and As a query and keywords The input is fed into the Transformer's multi-head attention to measure the importance of each position in the input sequence. (Drone) The attention weights are

[0140] (34)

[0141] Based on equation (34), It is updated using gradient descent.

[0142] (35)

[0143] in, For students The loss function for model training, For learning rate, This represents the gradient of the loss function.

[0144] like Figure 8 As shown, the total training loss of SFD consists of soft-label loss and hard-label loss. and Composed of, represented as

[0145] (36)

[0146] Based on equation (17), the optimization objective of online distillation is to minimize equation (36). This is a parameter that adjusts the loss weights for soft and hard labels. Based on equation (32), Updated to

[0147] (37)

[0148] in, It is the learning rate for model updates. This represents the gradient of the SFD loss function.

[0149] The ground base station undertakes the training task of SFD, and the relevant details are summarized in Algorithm 3. This involves traversing... drones, base stations Receive the knowledge representation of UAVs within the cluster. TE will Encoded as (See line 2). Multi-head attention mechanisms are achieved through… and The dot product similarity is used to measure the contribution weight of each student within the cluster, and the teacher model is updated based on the weights (see rows 3-5). Through online distillation, the student model is updated to... (See lines 6-7)

[0150]

[0151] 5. Simulation Experiments and Result Analysis

[0152] For performance evaluation, this invention designed a simulation experiment on a high-performance server. The server was configured with an Intel Core i9-14900K processor, 64GB DDR5 5200MHz memory, a 4TB PCIe 4.0 solid-state drive, an ASUS PRIME Z790-PWIFID5 motherboard, and two Gigabyte RTX 4090 24GB WindForce graphics cards.

[0153] To simulate inter-node collaboration, heterogeneous stacked LSTM models (single-layer, two-layer, and three-layer LSTMs) for each drone and a large model of the ground base station, along with related data, are encapsulated in independent Docker containers. Each LSTM layer has 64 hidden neurons and one linear layer that maps features to predictions. These containers run on the same physical server. The containers interact through a virtual network, simulating message passing and collaboration in a real network environment. This setup allows us to flexibly adjust the number of nodes to simulate large-scale, distributed multi-drone collaborative networks.

[0154] In order to analyze the impact of different strategies on the overall performance, the proposed methods are divided into three categories as shown in Table 1. Proposed-1 retains only the Transformer-enhanced multi-teacher distillation (corresponding to Section 3) and omits the Transformer-enhanced SFD; Proposed-2's clustering only considers the UAV position, and Proposed-3's clustering comprehensively considers position information and model morphology (corresponding to Section 4.1); Proposed-3 and Proposed-4 include two stages: Transformer-enhanced multi-teacher distillation and FD, the difference being that Proposed-4 adopts a model segmentation strategy (corresponding to Section 3 and Section 4). This experiment selects the five methods listed in Table 2 as the baseline methods (clustering only considers the UAV position and does not have model segmentation). Baseline-1 and 2 belong to the traditional KD; Baseline-3 and 5 both consist of two stages: KD and FD, the difference being that Baseline-5 adopts a position-based clustering method; Baseline-4 includes two stages: KD and FL. The method proposed by Thammawichai et al.

[19] was used to quantify the communication cost, including the energy consumption of parameter transmission and reception. Time cost was used as an important indicator to reflect computational cost

[20] . The dataset used was communication data from Trentito province, Italy, which included data services, voice calls and other services

[21] , to simulate the arrival of sliced ​​tasks. SGD was selected as the optimizer. The detailed parameter settings are shown in Table 3.

[0155] Table 1 Classification of the proposed methods

[0156]

[0157] Table 2 Classification of Benchmark Methods

[0158]

[0159] Table 3 Experimental Parameter Settings

[0160]

[0161] 5.1 The impact of clustering in heterogeneous sensing on prediction error

[0162] This group of experiments examines the impact of the clustering methods described in Section 4.1 (Proposed-3) and the location-based clustering method (Proposed-2) on collaborative training. The mean of the MSE is used to measure the error between the predicted and actual resource demand values. Figure 9The performance comparison of two clustering methods on three slices is presented, and the results are the average of multiple outcomes. Compared to Proposed-2, which only considers location, Proposed-3's clustering method benefits from joint optimization of multi-dimensional metrics, taking into account both location proximity and model similarity. Proposed-3 achieves performance gains of 4.3%, 11.1%, and 12.5% ​​on the three slices, respectively, verifying the effect of heterogeneous feature fusion on improving the accuracy of resource demand prediction.

[0163] 5.2 Resource Forecasting Accuracy Analysis

[0164] In this experiment, the mean value of MSE is used to measure the error between the predicted and actual RB demand values ​​within a slice window. Figures 10(a) to 10(b) (c) By comparing the prediction errors of the proposed method and the baseline method, the variation of the prediction error of the required RB for each slice with the number of communication rounds is shown. As the number of communication rounds increases, the prediction error of each slice gradually converges to a certain interval. Compared with the baseline method, the proposed method, through Transformer multi-head attention, enables students to learn the teacher's knowledge more accurately, further accelerating the convergence speed of the model. Compared with Baseline-1 and 2, Proposed-1 breaks through the upper limit of the teaching ability of a single teacher by integrating the knowledge advantages of multiple teacher models. While maintaining lightweight design, it improves the generalization ability and robustness of the student model, achieving performance gains of 11.12%-22.13%, 20.93%-35.27%, and 6.97%-25.32% on the three slices, respectively. Compared to Baseline-3 and 5, Proposed-3, through Transformer-enhanced multi-teacher distillation, improves the model's adaptability to heterogeneous data distributions, achieving performance gains of 7.20%-19.13%, 5.01%-9.43%, and 4.97%-7.91% on the three slices, respectively. Proposed-4, based on model segmentation, and Proposed-3 without segmentation were used to examine the impact of model segmentation on prediction accuracy. For ease of observation, the local feature maps of Proposed-4 and Proposed-3 are placed in each subplot of Figure 10(a)-(c). As can be seen from the figures, the trends of the two are almost identical in the indicated communication rounds. This phenomenon indicates that the adopted model segmentation method can guarantee resource prediction accuracy.

[0165] Figures 10(d) to 10(f) shows the probability density function of the prediction error. For slice 1, the probability of Proposed-3 having a prediction error less than 0.2 is approximately 69%, while that of Baseline-5 is only 51%; for slice 2, the probabilities of Proposed-3 and Baseline-5 having prediction errors less than 0.1 are approximately 93% and 89%, respectively; for slice 3, the probability of Proposed-3 having a prediction error less than 0.1 is approximately 88%, while that of Baseline-5 is approximately 75%. This indicates that Proposed-3 can accurately capture changes in resource demand.

[0166] 5.3 Impact of Resource Block Quantity on Performance Isolation

[0167] This experiment investigated the impact of the total number of Restricted Blocks (RBs) held by a drone on the slicing performance isolation effect. All drones had the same number of available RBs. Figure 11 As shown, in the initial stage, the performance isolation quality of each method rapidly improves as resource pressure eases. Among them, the performance isolation quality of Proposed-4 is the best. In all compared methods, the performance isolation quality of the proposed method is higher than the baseline method. This phenomenon indicates that the knowledge fusion method based on the Transformer multi-head attention mechanism promotes model performance improvement. When the UAV's resources are sufficient, the performance isolation quality gradually approaches an ideal state. Compared to Baseline-1 and 2, which use the ordinary distillation method, Proposed-1 shows a performance improvement of 16.4% and 7.27% in slice performance isolation quality, respectively. Compared to Baseline-3, Proposed-2 shows a performance improvement of 8.96% in slice performance isolation quality. Compared to Baseline-4 and 5, Proposed-4 shows a performance improvement of 5.28% in slice performance isolation quality. This is due to the accurate prediction of slice resource requirements.

[0168] 5.4 The impact of model segmentation on the cost of collaborative training

[0169] Figure 12 This paper presents comparative experimental results on the communication and computation costs in collaborative training. Model segmentation effectively reduces the number of parameters transmitted and lowers communication latency. Although the Transformer's multi-head attention mechanism increases computational overhead, its dynamic weight fusion mechanism effectively improves model convergence efficiency. Specifically, Proposed-4 achieves a total training cost of 705J, representing a 25% and 23.2% reduction compared to Proposed-2 and Proposed-3, respectively. The experiments show that the reduction in communication overhead outweighs the increase in computational overhead, validating the synergistic optimization effect of model segmentation and the multi-head attention mechanism. Figures 10(a) to 10(b)The convergence curve analysis of 0(f) shows that this method reduces the cost of collaborative training while ensuring the accuracy of resource demand prediction.

[0170] 6. Summary

[0171] To support UAV RAN slicing in heterogeneous airborne model scenarios, this invention proposes a Transformer-enhanced KD method. By using ground base station teacher models to guide UAV student models, the UAV small base stations can alleviate the model training burden. For a single UAV, a Transformer-enhanced multi-teacher distillation method is designed, covering multiple ground base stations as multiple teachers to guide UAV learning. For UAV swarms, a Transformer-enhanced SFD method is designed. Building upon the clustering and segmentation of heterogeneous perception, SFD further enhances the global guidance capability of base stations for UAVs. Simulation results show that, compared with benchmark methods, this invention can reduce the cost of collaborative UAV training while maintaining the performance isolation effect of slicing. The proposed method has standardized deployment characteristics and heterogeneous scenario expansion capabilities, is not limited by the number of UAVs or their movement trajectories, and can be further applied to scenarios such as data acquisition, distributed content caching, and content distribution.

[0172] References

[0173] [1] Huang Zhenhua, Yang Shunzhi, Lin Wei, Ni Juan, Sun Shengli, Chen Yunwen, Tang Yong. A review of knowledge distillation research [J]. Chinese Journal of Computers, 2022, 45(03):624-653

[0174] [2]Wu S, Luo X, Liu J, et al. Knowledge distillation with adaptedweight[J]. Statistics, 2025: 1-28.

[0175] [3]Li L, Su W, Liu F, et al. Knowledge fusion distillation: Improvingdistillation with multi-scale attention mechanisms[J]. Neural Processing Letters, 2023, 55(5): 6165-6180.

[0176] [4]Gou J, Sun L, Yu B, et al. Hierarchical multi-attention transferfor knowledge distillation[J]. ACM Transactions on Multimedia Computing,Communications and Applications, 2023, 20(2): 1-20.

[0177] [5]Liu B Y, Chen H X, Huang Z, et al. Zoominnet: A novel small objectdetector in drone images with cross-scale knowledge distillation[J]. RemoteSensing, 2021, 13(6): 1198.

[0178] [6]Xue W, Jing L I U, Jiani S U N, et al. Spectral clustering-basedenergy-efficient resource allocation algorithm in heterogeneous cellularultra-dense network[J]. Journal on Communication / Tongxin Xuebao, 2021, 42(7).

[0179] [7]Mughal F R, He J, Zhu N, et al. An intelligent Hybrid‐Q Learningclustering approach and resource management within heterogeneous clusternetworks based on reinforcement learning[J]. Transactions on EmergingTelecommunications Technologies, 2024, 35(4): e4852.

[0180] [8]Sun J, Gao H, Yan Z, et al. Lightweight UAV object-detectionmethod based on efficient multidimensional global feature adaptive fusion andknowledge distillation[J]. Electronics, 2024, 13(8): 1558.

[0181] [9]Kaimakamidis A, Mademlis I, Pitas I. Collaborative KnowledgeDistillation via a Learning-by-Education Node Community[J]. arXiv preprintarXiv:2410.00074, 2024.

[0182]

[10] Aghli N, Ribeiro E. Combining weight pruning and knowledgedistillation for cnn compression[C] / / Proceedings of the IEEE / CVF conferenceon computer vision and pattern recognition. 2021: 3191-3198.

[0183]

[11] Yu R, Zhao R, Li J, et al. Unleashing the Potential of Mamba:Boosting a LiDAR 3D Sparse Detector by Using Cross-Model KnowledgeDistillation[J]. arXiv preprint arXiv:2409.11018, 2024.

[0184]

[12] Cheng X, Zhou J. LGFA-MTKD: Enhancing Multi-Teacher KnowledgeDistillation with Local and Global Frequency Attention[J]. Information, 2024,15(11): 735.

[0185]

[13] Lin X, Li J, Baldemair R, et al. 5G new radio: Unveiling theessentials of the next generation wireless access technology[J]. IEEECommunications Standards Magazine, 2019, 3(3): 30-37.

[0186]

[14] He M, Wu H, Zhou C, et al. Resource Slicing with Cross-CellCoordination in Satellite-Terrestrial Integrated Networks[C] / / ICC 2024-IEEEInternational Conference on Communications, 2024: 2501-2506.

[0187]

[15] Cao Z, Zhou Y, Yang A, et al. Deep transfer learning mechanismfor fine-grained cross-domain sentiment classification[J]. ConnectionScience, 2021, 33(4): 911-928.

[0188]

[16] Wang Z, Xie X, Zhao Q, et al. Filter clustering for compressingcnn model with better feature diversity[J]. IEEE Transactions on Circuits andSystems for Video Technology, 2022, 33(12): 7385-7397.

[0189]

[17] Liao Y, Xu Y, Xu H, et al. Mergesfl: Split federated learningwith feature merging and batch size regulation[C] / / 2024 IEEE 40thInternational Conference on Data Engineering (ICDE), 2024: 2054-2067.

[0190]

[18] Kim T, Oh J, Kim N Y, et al. Comparing kullback-leiblerdivergence and mean squared error loss in knowledge distillation[J]. arXivpreprint arXiv:2105.08919, 2021.

[0191]

[19] Thammawichai M, Baliyarasimhuni S P, Kerrigan E C, et al.Optimizing communication and computation for multi-UAV information gatheringapplications[J]. IEEE Transactions on Aerospace and Electronic Systems, 2017,54(2): 601-615.

[0192]

[20] Tian Y, Wan Y, Lyu L, et al. FedBERT: When federated learningmeets pre-training[J]. ACM Transactions on Intelligent Systems and Technology(TIST), 2022, 13(4): 1-26.

[0193]

[21] Barlacchi G, De Nadai M, Larcher R, et al. A multi-source datasetof urban life in the city of Milan and the Province of Trentino[J].Scientific data, 2015, 2(1): 1-15.

[0194]

[22] Liu Y, Zhang W, Wang J. Adaptive multi-teacher multi-levelknowledge distillation[J]. Neurocomputing, 2020, 415: 106-113.

[0195]

[23] Chang X, Lee S Y M, Zhu S, et al. One-teacher and multiple-student knowledge distillation on sentiment classification[C] / / Proceedings ofthe 29th International Conference on Computational Linguistics. 2022: 7042-7052.

[0196]

[24] El Khediri S, Fakhet W, Moulahi T, et al. Improved nodelocalization using K-means clustering for Wireless Sensor Networks[J].Computer Science Review, 2020, 37: 100284.

[0197]

[25] Zhu Z, Hong J, Zhou J. Data-free knowledge distillation forheterogeneous federated learning[C] / / International conference on machinelearning. PMLR, 2021: 12878-12889.

[0198]

[26] Sattler F, Marban A, Rischke R, et al. Communication-efficientfederated distillation[J]. arXiv preprint arXiv:2012.00632,

Claims

1. A method for unmanned aerial vehicle (UAV) RAN slicing based on multi-head attention enhanced knowledge distillation, in a network scenario of large-scale UAV cooperative training enabled by ground base station large models empowering air base small models, a UAV-borne small model is deployed in a UAV small base station as a RAN slicing model, and resources are allocated to the slice according to the resource demand predicted by the RAN slicing model; the UAV small base station is referred to as a UAV; the large model refers to a model deployed in a ground base station, and the small model refers to a model deployed in a UAV base station; In the network scenario: the airborne models deployed by UAV-borne small base stations of different types are heterogeneous models to predict resource demand; the ground base station deploys a large model; the MEC controller accesses multiple ground base stations and coordinates the data of multiple ground base stations to assist in training the small model; the UAV is deployed on demand in a wide area; when the UAV enters the coverage range of the ground base station, the airborne small base station completes handshaking and inquiry with the ground base station through a communication protocol; the ground base station periodically refreshes the connection list of the UAV-borne small base station, and when a UAV needs to leave the coverage area of the ground base station to perform other tasks, the ground base station issues a notice to release the connection; The application is characterized in that The training of the UAV-borne RAN slicing model includes two stages: Stage one, multi-teacher knowledge distillation (KD): the large models deployed by multiple ground base stations covering the UAV form a team of teacher models, and the small model deployed by the UAV is a student model; the team of teacher models guides the student model to learn; each teacher model is a heterogeneous model; The Transformer multi-head attention deployed in the mobile edge computing (MEC) dynamically calculates the adaptive weight of each teacher model, realizes the weighted integration of the soft and hard label losses of each teacher model, and updates the student model by constructing a global distillation loss; Stage two, split federated distillation (SFD): the student model is split by adopting a heterogeneous-aware clustering and splitting strategy; and the model performance is further optimized by SFD to enhance the global guidance ability of the teacher model to the student model.

2. The method of claim 1, wherein the method is characterized in that After the multi-teacher KD is completed, the UAV decides to stay or leave on its own, and for the adjacent UAVs that stay, it enters stage two for processing.

3. The method of claim 1, wherein the method further comprises: In the network scenario, the physical resources on each UAV are pooled and divided into multiple slices and a resource sharing area, and each slice supports a customized service; according to the resource demand, each UAV dynamically decides the slice window length and resource allocation by using the RAN slicing model driven method; For drones Assuming time is dynamically partitioned into multiple slicing windows, each window containing multiple discrete time slots; Assuming a drone The total number of resource blocks RBs held is These resources are virtualized into service slices, the set of slices is denoted by ; At the beginning of each window, the resource blocks (RBs) of each service slice and the shared area are reallocated and maintained until the end of the window; at the end of each window, the UAV calculates the resource requirement of each slice and the size of the shared area for the next window; Arriving tasks are scheduled to different slices according to their attributes, and each slice allocates resources to tasks centered on time slots; In the window At the beginning, the UAV The slice And the number of RBs allocated to the shared area are respectively represented as And The sum of the two does not exceed the total number of RBs held by the UAV Hold, meet (1)， a set of slice windows for the drone k; In the case of insufficient slice resources, the RBs of the sharing area are temporarily occupied; In the dynamic resource sharing area optimization process, the sliding window stress index (SWPI) is used to calculate the number of resource sharing blocks (RBs) in the sharing area; as the SWPI of the slice window increases, the number of RBs in the sharing area should increase dynamically; assuming the slice window... The SWPI value is ,recent The average SWPI of each slice window is The variance of SWPI volatility is The number of RBs in the shared area is then calculated as follows: (2)， wherein, is a volatility correction weight coefficient, represents the historical SWPI standard deviation; Based on formula (2), when higher than , the capacity of the shared area is linearly expanded; when the burst fluctuates, higher than , the expansion range of the shared area will be enhanced, and the maximum capacity of the shared area does not exceed the preset threshold; when not higher than , the shared area will shrink; The resource allocation policy at the beginning of each slice window determines the performance isolation quality of each slice in the window; assuming at time slot t, the UAV The number of RBs required by the upper slice The number of RBs occupied by the shared region The performance isolation quality at time t is defined as​ (3)， The smaller the gap between the values of the formula (3), and the higher the prediction accuracy of the formula (3), and the higher the performance isolation quality. If , slice does not occupy i.e. meets the current demand, the isolation quality reaches the highest level, ; If , the shared zone occupies all RBs and does not meet the service requirement of the slice , the performance isolation is broken ; If , slice needs to temporarily occupy the resources of the shared area to meet the current demand, the performance isolation is weakened ; Drones The average isolation quality of the upper slice is denoted as It is applied to All the windows in the middle, long slice performance isolation optimization problem is modeled as (4)， Problem representing the dynamic allocation of the number of RBs of the shared area according to formula (2) under the resource constraint, with the UAV as the center At the same time, the resource allocation of each slice and the slice window length is determined to optimize the performance isolation quality; Based on the multi-head attention mechanism of the Transformer, the multi-teacher KD and SFD methods are used for solving ; The slice window of the UAV small base station is represented Resource allocation.

4. The method of claim 1, wherein the method is characterized by The student model deployed by the UAV is uploaded to the MEC controller through the ground base station, and then updated; The updated student model is returned to the original UAV.

5. The method of claim 3, wherein the method is characterized by In stage one, the steps of multi-teacher KD are: 1.1) Teacher model selection The teacher model selection method based on KL divergence and local data of multiple ground base stations: KL divergence is used as the standard for target domain selection. The smaller the value of KL divergence, the more similar the historical data distribution provided by the UAV and the corresponding ground base station. The larger the value of KL divergence, the greater the difference in historical data distribution provided by the UAV and the corresponding ground base station. The target domain is the base station participating in the fine-tuning of the student model. 1.2) Online distillation based on multi-head attention The student model carried by the UAV is uploaded to the MEC controller by the ground base station, and the online distillation is completed under the guidance of the teacher model team.

6. The method of claim 5, wherein the method is characterized by In stage one, the UAV and the ground base station establish a teacher-student and collaborative relationship through handshaking. Under the coordination of the MEC, the large models on adjacent ground base stations form a teacher model team to help the UAV small model realize online update. In step 1.1), Assume the drone and the data distribution of the ground base station is and , respectively, (5)， wherein, and are hidden representations of small models of drones and large models of ground stations respectively. Drones Ground base stations connected to the drones The KL divergence is: (6)， wherein, is the information entropy of is and the cross-entropy of represents the set of all possible distributions; Coverage drone The set and cardinality of ground base stations covering the drone are denoted as and ; The target domain base station set selected to participate in the fine-tuning of the UAV small model is: (7)， wherein, is a KL divergence threshold value; Base station selecting and Similar part history data fine-tuning , the proportion of this part of data is (8)， wherein, is negatively correlated with the amount of data selected; Assume that the ground base station set and the cardinality selected by the MEC are and The proportion of selected data is: (9)， Assuming a drone The upper slice In The actual resource requirement vector for one time slot is denoted as ; then, the drone The actual resource block RB number for all the slices on the drone (10)， Drones Upper slice In The number of RBs required for wheel prediction is predicted as (11)， wherein, are student model parameters; Next, a loss function is used to measure the UAV upper slice prediction error of the number of required RBs, the UAV The loss function of the upper slice is represented as: (12)， by gradient descent method, is updated to (13)， wherein, is a learning rate for student model update, represents a drone a gradient of the loss function; In step 1.2), In the During round prediction, ground base stations superior The resource requirements of a slice are represented as ,slice exist The actual resource demand vector for each time slot is represented as: ; Ground base station Upper slice In The number of RBs for wheel prediction is (14)， wherein, a ground base station model parameters of a teacher model; but, A slice in The number of RBs required for round prediction is (15)， wherein, is a soft label; a student model inputted onto the drone a student model output (16)； The goal of distillation is to guarantee that the output of the student model matches the output of the teacher model during training with the corresponding hard labels and soft labels ​ The total training loss is composed of two parts, soft label loss and hard label loss, assuming and are the soft label loss and hard label loss, respectively. (17)， The loss value is determined by the mean square error between the predicted value of the student model and the teacher model and the true value of the student model. The multi-head attention mechanism of Transformer measures the contribution of each teacher model to the student model through the dot product similarity of Query and Key, making the model focus on high correlation information: The embedding representation layer TE of the Transformer encodes an input sequence into vectors for the query Q, the key K, and the value V , and ; (18)， Through the multi-head attention mechanism, the attention weights of the teacher model are (19)， Based on equation (19), The attention weight vector of the teacher model is represented as ; The optimization goal of multi-teacher KD is to minimize the total training loss (20)， wherein the parameter is a weight coefficient for adjusting the weight between the soft label loss and the hard label loss to control the attention of the student model to the hard label and the soft label during the training process. Based on equation (20), the student model is updated by gradient descent method as (21)， wherein, is the learning rate for model training, denotes the gradient of the multi-teacher KD loss function.

7. The method of claim 3, wherein the method further comprises: In stage two, the SFD method enhanced by Transformer is used to further enhance the performance of the airborne small model through the knowledge transfer of the global teacher model. The steps include: 2.1) Heterogeneous-aware clustering The heterogeneous-aware clustering method considers the location of the UAV and the shape of the airborne small model to balance the federal aggregation opportunity and reduce the cost of cooperative training. The ground base station and the UAV small base station are respectively taken as the cluster head and the cluster member. The complexity of the airborne small model is measured by the number of floating-point operations FLOPs, which is positively related to the complexity of the airborne small model. UAVs with small FLOPs difference reflecting close model shape and close location are grouped into the corresponding ground base station cluster. 2.2) Selection of airborne small model segmentation point Determined by the FLOPs of each member in the cluster. The selection of the segmentation point ensures that the shallow structure of the student model is consistent, i.e., the parameter dimension matches, so that the ground base station processes deep feature extraction with high complexity, while the UAV performs lightweight shallow calculation tasks. The shallow structure of the student model is the number of air models. 2.3) Model segmentation and local iteration The airborne small model is segmented into air models and ground models, which correspond to the shallow and deep layers of the original small model respectively. The air model is retained locally, and the ground model is transferred to the ground base station as the cluster head. The two segmented models transmit the gradient of backpropagation through wireless communication. 2.4) Distillation-driven segmented federal training During training, the teacher model receives the knowledge representation of the student models in the cluster, and measures the contribution weight of different students using the dot product similarity of Query and Key. The embedding representation layer TE of Transformer is integrated into the ground base station. The TE encodes the ground truth model into an input sequence, which generates the query, key and value of the UAV after multi-layer stacking; the teacher model and the ground truth model are input into the multi-head attention of the Transformer as the query and key of the ground base station, so as to measure the importance of each position in the input sequence. The total training loss of the SFD is composed of a soft label loss and a hard label loss.

8. The method of claim 7, further comprising: In step 2.1), it is assumed that the set and cardinality of drones staying in the adjacent area are and respectively. Drones and ground base stations are represented as and respectively, considering both the position information and the complexity of the on-board model, the feature vector of the drone is constructed as (22)， wherein, representing a drone FLOPs of the onboard small model; Let The UAV is used to measure the weighted relationship between the base station , expressed as (23)， wherein, FLOPs difference for models for measuring drones and ​ Set of drones in cluster is Based on equation (23), the selection of cluster head i* is determined as (24)； Drones with small difference in FLOPs and close location are grouped into a cluster In the middle; In step 2.2), it is assumed that the stacked model of the UAV contains layers, the FLOPs of the first layer as a split layer are denoted as , and the FLOPs of the first 1 layer to the first layer are (25)， To measure the impact of reducing the number of model layers on the complexity of the full model, the model complexity gradient of the split layers is (26)， wherein, FLOPs representing the complete student model; the result of equation (22) embodies the split layer FLOPs difference rate of the layer compared to the first FLOPs difference rate of the layer compared to the first To ensure the record small model complexity ratio optimal between each UAV, the selection of the segmentation layer satisfies (27)， wherein, is a balancing factor of FLOPs among the UAVs; is an adjustment coefficient of the balancing global balancing target and influences; In step 2.3), the UAV The small model is a bidirectional long short-term memory model Bi-LSTM, denoted as ​ are split into an aerial model and a ground model, denoted as and , respectively, which correspond to the shallow and deep layers of the original model , respectively; are kept locally, are transferred to the base station acting as cluster head ; the two split models and are communicated wirelessly to backpropagate the gradients; Assume that in the time slot , the resource prediction value of the forward LSTM is , the resource prediction value of the reverse LSTM is , then the resource prediction value of the Bi-LSTM is (28)， wherein denotes the combination of two hidden states; Drones Upper slice In the window The number of resource blocks required is predicted as (29)， using the mse loss the loss of the metric model output; the substituted into equation (29), the drone upper slice in the window the number of required RBs is predicted as (30)， Based on and , the loss function is used to measure the prediction error of the number of required RBs on the UAV upper slice ; the loss function for is expressed as (31)， Let and represent and a loss function; and the accumulation is represented as (32)， The process uses source domain data carried by the drone and newly collected data to forward propagate, the resulting intermediate features are passed to the ground base station ; ground base station perform forward and backward propagation to update and back the gradient; after receiving the gradient, the drone updates its aerial model through backward propagation (33)， wherein, is a learning rate of the UAV ​ Then, the aerial model of the UAV , i.e., the shallow model, is updated to , and the ground model, i.e., the deep model, is updated to ; In step 2.4), after updating the shallow model, the teacher model receives the knowledge representation of the student model, and measures the contribution weight of different students using the dot product similarity of Query and Key. TE will encoded into an input sequence that, after being stacked in multiple layers, generates , and ; and as queries and keywords are input into the multi-head attention of the Transformer to measure the importance of each position in the input sequence; Drones the attention weight of (34)， Based on equation (34), is updated by the gradient descent method as (35)， wherein, for the student a loss function for the model training, for the learning rate, denotes the gradient of the loss function; The total training loss of SFD is composed of soft label loss and hard label loss and is represented as (36)， Based on equation (17), the optimization goal of online distillation is to minimize equation (36); where, is a parameter that adjusts the soft label and hard label loss weights; Based on equation (32), is updated to (37)， wherein, is the learning rate for model update, denotes the gradient of the SFD loss function.

Images