Unmanned aerial vehicle small base station ran slicing method based on knowledge distillation

Abstract

A UAV small base station RAN slicing method based on knowledge distillation, in a wireless communication network composed of ground base stations, UAV small base stations and MEC controllers, a machine learning-based RAN slicing model is deployed in the UAV small base station, and resources are dynamically allocated to the slice according to the resource demand of the service. A large model is deployed in the ground base station; the large model in each ground base station uses a federated learning (FL) method to generate a batch of RAN slicing models for deployment in the UAV small base station and is called a small model; the small model is stored in the model cache of the ground base station; the small model is customized for the UAV small base station based on the large model through a knowledge distillation method according to the model and demand of the UAV small base station; the ground base station selects the most suitable small model from the model cache and delivers it to the UAV small base station; the UAV small base station receives and fine-tunes the small model, and then slices the local resources of the UAV small base station according to the decision output by the small model.

Description

Technical Field

[0001] This invention belongs to the field of network technology, specifically a method for RAN slicing of unmanned aerial vehicle (UAV) small base stations based on knowledge distillation. Background Technology

[0002] Radio Access Networks (RAN) slicing is one of the key enabling technologies in 5G networks, providing customized services based on customer needs. Existing technologies employ various RAN slicing methods, all of which divide a shared physical wireless network into multiple isolated logical networks, dynamically and flexibly allocating network resources to users.

[0003] Extending RAN slices to drone small base stations to enable them to support diverse services is a future trend in integrated air-ground networks.

[0004] Deploying machine learning-based RAN slicing models to UAV small base stations allows the UAV collaborative training framework to overcome many limitations of single-machine deployments, achieving better slicing performance isolation while reducing communication costs. However, since model training is performed on the UAV platform, it inevitably incurs computational and storage burdens. On the other hand, the ground base station merely acts as a central controller to assist the UAV small base station, and its advantages, such as strong computational and storage capabilities and the ability to run complex models, are not fully utilized.

[0005] The heterogeneity and mobility of the UAV platform[1] also lead to some challenging problems in deploying machine learning models (RAN slice models) on UAVs, mainly in the following aspects:

[0006] (1) The small base station of the UAV relies on batteries for power and has limited computing and storage capabilities, making it difficult to support long-term and complex model training. Most existing methods start from data augmentation[2] and model parameter compression[3]. The former can solve the problem of weak data acquisition capability of UAV, but brings additional computing costs; the latter optimizes model training by reducing the model size, but still cannot fundamentally alleviate the burden of model training.

[0007] (2) The dynamic deployment characteristics make it difficult for UAVs to collect enough data for model training within the working area. Even if the model reaches the ideal state through training, it will gradually fail due to environmental changes during UAV movement. Pre-planning the path[4] can overcome the impact of mobility on the model, but the UAV's mission execution is inevitably limited by the movement trajectory.

[0008] (3) Model deployment needs to adapt to the system heterogeneity of UAVs. When defining the structure of machine learning models and training models, the differences in computing, storage and endurance of different UAV models must be considered. Existing work mainly achieves model lightweighting through model pruning [5], but the model structure produced by this method is the same and cannot achieve the best matching between the model and different UAV platforms. Summary of the Invention

[0009] Knowledge distillation is an effective way to obtain efficient small models. This method transfers the "knowledge" of complex teacher models to simple student models at the cost of slight performance loss [6], which is very suitable for resource-constrained edge devices such as UAVs. This invention proposes a RAN slicing framework / method for UAV small base stations based on knowledge distillation. Unlike existing solutions, the framework proposed in this invention adopts the design paradigm of "feeding back" the UAV small model (student model) to the large model (teacher model) of the ground base station, which allows the UAV small base station to get rid of the burden brought by model training. Through knowledge distillation, the UAV small base station can obtain a higher model starting point and quickly enter an efficient working state.

[0010] In the efficient integration of knowledge distillation and drones, some specific problems still need to be solved, mainly including the following three aspects:

[0011] (1) The small drone model (student model) must maintain a certain level of freshness to cope with the dynamic changes in the network environment. Therefore, the large model (teacher model) of the ground base station needs to be adjusted according to the network situation. When the ground base station delivers the model to the drone, how to reduce the drone's waiting time is also a problem that needs to be considered. Pre-caching the small model at the ground base station can solve the problem of long drone waiting time, but there will be cases of model cache misses.

[0012] (2) Due to spatial distribution, the amount and quality of data collected by ground base stations vary in different places, resulting in differences in model quality. The performance of UAV small models (student models) constructed by ground base station large models (teacher models) with poor model quality will be limited. Knowledge merging [7] and multi-teacher learning [8] can alleviate the negative impact of poor teacher model quality. The former requires the knowledge of multiple teachers to be used to update the parameters of a single student model [9], which involves knowledge fusion and increases computational complexity; the latter requires a single student model to learn from multiple teachers

[10] , which brings high communication overhead to UAV small base stations.

[0013] (3) There are many types of UAVs and different types of tasks. Different models can support different model structures and complexities. Therefore, the constructed student model needs to have a certain degree of adaptability. Existing knowledge distillation methods for UAV networks only consider a single model

[11] , which cannot achieve the customization of small models.

[0014] To address the above problems, the main solutions of this invention include:

[0015] First, to address problem (1), a federated learning (FL)-driven small model caching and online distillation framework is designed. The ground base station's large model (teacher model) generates a batch of small models after FL based on the latest model and places them in the model cache. When a model request is received from a UAV small base station, the ground base station can select the most suitable small model from the model cache. If the required small model for the UAV is not present in the model cache, the ground base station generates it online.

[0016] Second, to address problem (2), a collaborative training framework for ground base stations based on federated learning (FL) is designed to improve the model quality of ground base stations. Under this framework, adjacent ground base stations act as "teaching and research groups" and jointly train RAN slice models using the FL method. Simultaneously, to improve the generalization of the teacher models, a model aggregation scheme based on an attention mechanism is introduced.

[0017] Third, in response to problem (3), a customized delivery solution for UAV small models based on knowledge distillation is proposed. This solution uses the high-performance model of the ground base station as a foundation and constructs a lightweight model for the UAV small base station through knowledge distillation. The ground large model will customize a small model for the UAV according to the model and requirements. The UAV small base station that receives the small model can fine-tune it according to its own data. Attached Figure Description

[0018] Figure 1 It is a RAN slicing structure for drone small base stations;

[0019] Figure 2 It is a service-oriented resource allocation;

[0020] Figure 3 It is a collaborative training framework;

[0021] Figure 4 It is based on federated learning for base station model training;

[0022] Figure 5 It is a model aggregation based on the attention mechanism;

[0023] Figure 6 It is a drone model customization based on knowledge distillation;

[0024] Figure 7The impact of the number of LSTM layers on model performance;

[0025] Figure 8 This relates to the impact of the number of resource blocks on performance isolation.

[0026] Figure 9 It represents the percentage of time spent on performance isolation;

[0027] Figure 10 It is the error in the resource demand forecast of the slice;

[0028] Figure 11 It compares the prediction errors of two student models with those of the teacher model. Detailed Implementation

[0029] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0030] 1 Overview

[0031] This paper proposes a knowledge distillation-based radio access network (RAN) slicing framework for UAV small base stations, aiming to free UAVs from the burden of training and improve slicing performance and isolation effects.

[0032] On the one hand, federated learning facilitates the training of teacher models deployed in terrestrial base stations, thereby maximizing the quality of teacher models.

[0033] On the other hand, we design a small-model delivery strategy based on knowledge distillation to adapt to the differences in the platform capabilities of drones.

[0034] Simulation results show that, compared with typical distributed machine learning methods, the proposed scheme improves the time ratio of slice performance isolation by 16.2% and 4.8%, respectively, and reduces the communication cost of UAV collaborative training.

[0035] 2 System Model and Problem Description

[0036] 2.1 Network Scenarios

[0037] like Figure 1 As shown, consider a scenario where a large number of drones provide differentiated services to ground users over a wide area. The physical resources on each drone are virtualized into multiple slices, each slice supporting a type of customized service. To cope with the dynamic nature of the network environment, machine learning models (slice models) are deployed on the drones, employing a model-driven approach to schedule slice resources, thereby maintaining service agility and improving resource utilization. Different models of drone small base stations can be dynamically deployed according to preset trajectories or can autonomously determine their movement trajectories.

[0038] 2.2 Small Model Delivery Framework for UAVs

[0039] In this scenario, the MEC controller acts as the central controller, and the ground base station jointly trains a RAN slice model through FL. Then, based on the latest model after each round of FL, the ground base station generates and caches a batch of "pre-trained small models" locally according to the model and requirements of the UAV. Assume that these UAVs belong to different models and have different computing and storage capabilities. When the UAV small base station enters the coverage area of ​​the ground base station, it can establish a connection with the base station through the handshake protocol

[15] . The ground base station periodically refreshes the UAV connection list and deletes invalid UAV association information. The ground base station stores the model list corresponding to different UAV models. When the UAV small base station establishes a connection with the ground base station, the ground base station will query the list for the model related to the UAV model. When a model request is received from the UAV small base station, the ground base station can generate it in real time or select the most suitable small model from the model cache and distribute it to it.

[0040] 2.3 Service-Oriented Dynamic RAN Slicing Framework / Method

[0041] Resource slicing needs to be adaptively adjusted according to fluctuations in resource demand for various services. This section considers a service-oriented resource demand-aware resource slicing scheme, in which the controller on the UAV dynamically allocates resources to slices based on the resource demand of services.

[0042] The following is based on Figure 2 Let's take an example to explain service-oriented resource allocation. Assume that time is divided into multiple slice windows, and each slice window is further divided into multiple discrete time slots. The set of time slots contained in slice window 'a' is represented by T, where the number of elements in the set is denoted as T. a The drone predicts the number of resource blocks required for each slice in the next window a+1 during the operation of the current slice window a. At the start of window a+1, the resources for each slice are reallocated based on the predicted resource requirements. Figure 2 In Within window 'a', the number of resource blocks allocated to each slice remains constant. At the start of each time slot 't' within the slice window, the local controller on the UAV allocates resource blocks for the received tasks. To cope with sudden, instantaneous changes in resource demand, each UAV reserves a portion of resource blocks that can be shared by all slices. In the event of resource shortage, slices can temporarily occupy these shared resource blocks.

[0043] 2.4 Problem Modeling

[0044] Performance isolation is a crucial prerequisite for the coexistence of resource slices, ensuring that overload in one slice does not affect other slices. The resource allocation strategy at the beginning of each slice window largely determines the effectiveness of performance isolation among slices in the next window.

[0045] Assume that drone k has Q resource blocks. k These resource blocks are arranged into M slices, and the set of slices is denoted as . Suppose that in slice window a, the predicted resource requirement of slice m on drone k in the next window a+1 is... If in a time slot The actual resource allocation is Then there is The allocation of slice resources needs to consider the following two situations:

[0046] (1) This means that the actual number of resource blocks required in time slot t is greater than the predicted value within slice window a. In this case, slice m needs to temporarily occupy the reserved shared resource blocks.

[0047] (2) This means that the number of resource blocks required is less than the predicted value, that is, the resources of slice m can meet the current demand.

[0048] Assume that the shared resource block reserved for drone k is Q′. k Regarding the above situation (1), if If all shared resource blocks are insufficient to handle sudden resource demands, then slice performance isolation cannot be guaranteed. Let binary variables... This means that each slice on UAV k in time slot t can maintain performance isolation; otherwise, it is 0.

[0049]

[0050] As shown in equation (1), within window a+1, a higher proportion of slice performance isolation time indicates a more effective resource allocation scheme. Therefore, the corresponding slice performance isolation optimization problem can be described as follows:

[0051]

[0052] During the operation of window a, the predicted number of resource blocks for slice m on drone k in the next window a+1. and real resource needs The smaller the gap, The higher the probability of a value being 1, the better. Therefore, by minimizing the mean square error (MSE) between the predicted and actual values, the problem... It can be transformed into the following optimization problem

[0053]

[0054] question The essence is to reduce prediction errors by optimizing model training under the constraint of total resource availability, so as to maximize the time ratio of resource slicing performance isolation. The following section will explore how to design a knowledge distillation framework to enhance the resource slicing performance of UAVs.

[0055] 3 Design Scheme

[0056] This section uses the following... Figure 3 The collaborative training framework shown is divided into a federated learning layer and a knowledge distillation layer. The former constructs a large ground base station model (teacher model) through federated learning (FL), while the latter customizes the UAV small model (student model) through knowledge distillation. The UAV small base station RAN slicing method designed in this chapter mainly includes the following three steps:

[0057] (1) Training of the large terrestrial base station model (teacher model) based on federated learning. Each terrestrial base station trains its local model. After training, the terrestrial base station uploads its local model parameters to the MEC controller. After receiving the model parameters from each terrestrial base station, the MEC controller performs global model aggregation. Then, the aggregated global model parameters are distributed to the terrestrial base stations for local model updates. See Section 3.1 for details.

[0058] (2) Provision of UAV mini-models (student models) based on knowledge distillation. The ground base station uses its local complex model as a foundation to construct a more lightweight mini-model through knowledge distillation and stores it in its own model cache. The student models generated by knowledge distillation have different structures to adapt to different UAV models and requirements. When a model request is received from the UAV mini-base station, the ground base station searches for a matching model in its model cache based on the UAV model. If a matching mini-model exists, the ground base station delivers the model directly; if no suitable model is cached, the ground base station generates a matching mini-model in real time based on the UAV's requirements and delivers it. See Section 3.2 for details.

[0059] (3) Small model-driven UAV RAN slicing. After receiving the small model, the UAV can fine-tune it using local data, and then slice local resources according to the decision output by the model. See Section 3.3 for details.

[0060] 3.1 Construction of a Large-Scale Model for Ground Base Stations Based on Federated Learning

[0061] The quality improvement of the UAV small model (student model) is highly dependent on the ground base station large model (teacher model). If the quality of adjacent ground base station models is not significantly different, the UAV small base station can directly select the nearest ground base station to establish a connection to save energy. This section designs a collaborative training framework based on federated learning, where adjacent ground base stations jointly train a single model to improve the quality of the teacher model and reduce the model differences between adjacent ground base stations. To maintain the generalization ability of the ground base station large model, an attention mechanism is introduced into the model aggregation.

[0062] like Figure 4 As shown, the main steps of attention-based FL collaborative training include:

[0063] (1) Local model training:

[0064] Assume there are B ground base stations, and each ground base station has local resource requirement data for model training. Let... Let represent the actual resource demand vector of slice m on ground base station b across Z time slots. Furthermore, the actual resource demand matrix for all slices on ground base station b is:

[0065] R b =[R b,1 ,R b,2 ,…,R b,M (4)

[0066] In round z, the number of resource blocks required by base station b for slice m in round z+1 is:

[0067]

[0068] Where f(·) and These represent the prediction model and model parameters for base station b, respectively.

[0069] Next, the loss function is introduced. The prediction error measures the number of resource blocks required for slice m on base station b. The loss function for all resource slices on base station b can be expressed as:

[0070]

[0071] Finally, the local model parameters of base station b are updated using the gradient descent method.

[0072]

[0073] Where η is the learning rate. The loss function represents the loss function relative to w. (z) The gradient.

[0074] (2) Global model aggregation:

[0075] After receiving the model parameters from each ground base station, the MEC controller performs model aggregation to establish a global model. It is worth noting that there are differences in model quality among different ground base stations, which leads to different contributions of each ground base station to the global model. In order to improve the generalization of the global model, this section proposes a model aggregation framework based on the attention mechanism

[12] , such as Figure 5 As shown. The large model of the ground base station adopts a stacked LSTM

[13] structure. Therefore, in the process of global model aggregation, this section applies the attention mechanism to each layer parameter of the model to quantify the contribution of each ground base station local model to the global model in a hierarchical manner.

[0076] Assume that the global model parameters at the MEC controller in round z are represented as w (z) In the (z+1)th round, the set of local model parameters for the B ground base stations is represented as... Attention-based FL model aggregation needs to consider the global model parameter w. (z) With ground base station model parameters W (z+1) The correlation between them. Assume the parameters of layer c in the model of ground base station b are represented as... The parameters of the c-th layer in the global model are represented as make Represents a query. For the corresponding keywords, and The similarity between the two matrices is calculated as the Frobenius norm of the difference between the two matrices, i.e.

[0077]

[0078] The hierarchical attention scores of the local model parameters of B ground base stations can be calculated using equation (8), denoted as [s1,s2,...,s...]. B ], where s b =[s b,1 ,s b,2 ,...,s b,C Then, by normalizing the scores using the softmax function, the hierarchical attention distribution of the local model parameters of the B ground base stations can be obtained, i.e., α = [α1, α2, ..., α]. B ], where α b =[α b,1 ,α b,2 ,...,α b,C The attention distribution was calculated as follows:

[0079]

[0080] Minimize w (z) and W (z+1)The expected distance between them can yield a global model that is relatively close to the local model in parameter space. Specifically, the attention distribution output by equation (9) is used as weights to minimize w. (z) and W (z+1) The distance between them, i.e.

[0081]

[0082] Where σ(·,·) represents the Euclidean distance between the two sets of parameters. Taking the derivative of the objective function of equation (4-8), we obtain...

[0083]

[0084] Finally, the gradient descent algorithm is executed to update the parameters of the global model, and the global model parameters are updated to...

[0085]

[0086] Where ε represents w (z+1) The step size is the number of steps moved in the opposite gradient direction during each iteration. Afterwards, the MEC controller distributes the global model parameters to each ground base station to update the local model of the ground base station. See Algorithm 1 for specific implementation details.

[0087]

[0088] (3) Local model update:

[0089] After the MEC controller completes global model aggregation, it distributes the aggregated global model parameters to each ground base station. Then, based on the global model parameters, the ground base stations update their local models.

[0090] 3.2 Providing a small-scale UAV model based on knowledge distillation

[0091] Due to their strong computing and storage capabilities, coupled with the support of cloud platforms or edge servers, ground base stations can train relatively complex models. However, due to the differences in computing, storage, and endurance capabilities between drones and ground base stations, these models are difficult to directly adapt to drone-based small base stations.

[0092] To address the aforementioned issues, this section proposes a knowledge distillation-based model customization scheme for UAV small base stations. The ground base station first trains a model using its own data, constructing a complex and high-performance teacher model. Then, based on the teacher model, the ground base station uses knowledge distillation to generate a batch of "pre-trained small models" locally, according to the model type and requirements of the UAV small base stations it is connecting to, and stores them in a model cache. These small models have fewer parameters but achieve performance close to the teacher model. Figure 6As shown, the ground base station provides small models with different structures for different drone models.

[0093] The model customization scheme proposed in this section mainly includes the following steps:

[0094] (1) Small model batch generation

[0095] The following describes the construction process of the small-scale drone model (student model). In round z, assume R... b =[R b,1 ,R b,2 ,...,R b,M [This refers to] resource requirement data for M slices on ground base station b, where... Let f(·) be the actual resource demand vector for slice m across Z time slots. For ease of representation, let f(·) denote the prediction model of terrestrial base station b. The large model (teacher model) of terrestrial base station b predicts the number of resource blocks required for slice m in round z+1.

[0096]

[0097] in, These are the model parameters. Based on this, the number of resource blocks required for M slices in round z+1 is obtained as follows:

[0098]

[0099] in, This is a soft tag. Then, R... b Input the data into the drone mini-model to obtain the number of resource blocks required for M slices in round z+1.

[0100]

[0101] The goal of knowledge distillation is to distill the output of a small model during training. With corresponding hard labels and soft labels This matches. Therefore, the total training loss consists of two parts: the original loss and the distillation loss. Assume... The soft loss is calculated from the mean square error between the predicted values ​​of the UAV small model (student model) and the predicted values ​​of the ground base station large model (teacher model). The hard loss is calculated from the mean squared error between the predicted and actual values ​​of the UAV small model (student model), and is expressed as follows:

[0102]

[0103] At this point, the optimization objective is to minimize the total loss function, i.e.

[0104]

[0105] The parameter β is used to adjust the weights of hard loss and soft loss.

[0106] After the above process, the ground base station generates and caches a batch of small models locally. This set of pre-trained small models is represented as... Where n is the number of model layers.

[0107] (2) Model cache update

[0108] To maintain the freshness of the drone mini-model, the ground base station large model needs to update the mini-model cache in real time based on the latest model after FL (Flash Rendering). In round z+1, the ground base station large model generates a new pre-trained mini-model based on the latest model, denoted as...

[0109]

[0110] (3) Model delivery and online distillation

[0111] After establishing a connection with the ground base station, drone k first sends its own model number to the ground base station. The ground base station stores a list of model specifications corresponding to the drone model. When drone k sends a model request to the ground base station, the following two situations may occur:

[0112] (a) If the model f required by UAV k exists in the model cache of the ground base station. n Then it is delivered directly to the drone small base station;

[0113] (b) If the model required by UAV k is not present in the ground base station model cache, a small model is generated online for UAV k according to the requirements of the UAV small base station.

[0114] 3.3 Small Model-Driven UAV RAN Slicing

[0115] When drone k receives model f n Next, fine-tune the model based on local data, and then segment the physical resources based on the model output. This mainly includes the following steps:

[0116] (1) Slice resource prediction: The number of resource blocks required for M slices in window a+1 is predicted by drone k.

[0117]

[0118] (2) Slice resource allocation: At the beginning of window a+1, the UAV reallocates the resources of each slice according to the results of the model output.

[0119] Subsequently, UAV k performs resource scheduling in slicing windows. At the beginning of each slicing window, steps (1) and (2) above are repeated, and resources are re-sliced ​​according to the number of resource blocks required for each slice in the next window as predicted by the model. Other UAVs also follow this pattern for resource slicing.

[0120] 4. Simulation Experiments and Result Analysis

[0121] This section designs a series of simulation experiments to evaluate the performance of the proposed method. To analyze the impact of different functional modules on the overall performance, the proposed scheme is divided into the following two types:

[0122] (1) FedAtt+KD: Default mode, with all preset functions.

[0123] (2) KD: This scheme omits the training of the ground base station model based on federated learning in Section 3.1. Instead, the ground base station directly builds a model for the UAV small base station that it connects with based on its own model.

[0124] All ground base stations use a stacked LSTM model, which includes three LSTM layers (each with 64 hidden neurons) and one linear layer that maps features to predictions. Two types of UAVs are used here, employing a single-layer LSTM model and a double-layer LSTM model, respectively, containing one and two LSTM layers (each with 64 hidden neurons), and one linear layer that maps features to predictions. The dataset uses communication data from Trentito Province, Italy, including services such as data services, voice calls, and SMS

[14] . This chapter uses different types of service data to simulate the resource demand fluctuations of slices. The parameter settings are shown in Table 1.

[0125] For ease of comparison and analysis, this paper selects the following two benchmark methods:

[0126] (1) Baseline Method-1: The model is deployed on each drone for training in a fully distributed manner.

[0127] (2) Benchmark Method-2: The customized scheme for small UAV models is omitted, and all UAVs adopt a single-layer LSTM model.

[0128] Table 1 Experimental parameter settings

[0129]

[0130] 4.1 The impact of model structure on model performance

[0131] To ensure the effectiveness of knowledge distillation, it is necessary to select the teacher model with the most reasonable structure. This experiment examines the impact of the number of LSTM layers in a stacked structure on the performance of the teacher model. Figure 7As shown, the prediction error of the teacher model on the three slices initially decreases with increasing layer number, then gradually increases. The prediction error is lowest when the number of layers is 3. Overfitting may occur when the number of layers is too low or too high, thus affecting model performance. Therefore, this chapter sets the number of LSTM layers in the teacher model to 3.

[0132] 4.2 Time percentage for slice performance isolation

[0133] This experiment investigated the impact of varying the total number of drone resource blocks on the slicing performance isolation effect. All drones had the same number of physical resource blocks. Figure 8 The impact of varying the total number of UAV resource blocks on slice performance isolation is presented. When the number of resource blocks is 50, the slice performance isolation of each method begins to improve rapidly. Specifically, the FedAtt+KD method shows a higher time ratio for slice performance isolation compared to KD and the two baseline methods. When the number of resource blocks is 150, the advantage of the proposed method in terms of performance isolation time ratio becomes even more pronounced.

[0134] Next, we will observe the convergence and performance isolation effect of the proposed method. Figure 9 The variation of slice performance isolation time percentage with the slice window operation is presented when the number of resource blocks is 250. At the start of the third slice window, the slice performance isolation time percentage of the baseline method 1 increases rapidly from 0 until it stabilizes; while FedAtt+KD, KD, and baseline method 2 have time percentages of 70%, 58%, and 61%, respectively, at the start of the window operation. This is because the UAV small models under FedAtt+KD, KD, and baseline method 2 exhibit better performance in the early stages of learning after the knowledge distillation process. Baseline method 1's prediction accuracy for resource requirements is weaker than the other three methods, leading to an unreasonable resource allocation strategy and reduced performance isolation effectiveness. After iterative convergence, compared to baseline methods 1 and 2, FedAtt+KD shows a performance improvement of 16.2% and 4.8% in slice performance isolation time percentage, respectively.

[0135] 4.3 Analysis of the accuracy of resource demand forecasting

[0136] This group of experiments uses MSE to measure the error between the predicted and actual resource demand values. Figure 10The overall prediction error for the resources required for each slice is shown. Because FL-driven knowledge distillation has more stable teacher model performance, FedAtt+KD achieves gains of 3.75%, 6.42%, and 5.37% on the three slices compared to KD. Compared to the baseline method 1, FedAtt+KD achieves performance gains of 8.22%, 14.32%, and 12.29% on the three slices. FedAtt+KD has two main advantages over the baseline method 1: firstly, FedAtt shares knowledge through model aggregation, improving the quality of the teacher model while maintaining its generalization ability; secondly, the knowledge distillation method gives the UAV model a higher starting point. Compared to the baseline method 2, FedAtt+KD achieves performance gains of 2.06%, 4.32%, and 3.29% on the three slices, demonstrating the effectiveness of the model customization scheme.

[0137] Figure 11 The resource demand prediction errors of two student models are compared with those of the teacher model. For example... Figure 11 As shown, the resource demand prediction errors of the two student models are very close to those of the teacher model, indicating that the proposed scheme can construct student models with performance close to that of the teacher model. Although the prediction errors are not significantly different, student model 2 outperforms student model 1. This is because student model 2 has a better model structure than student model 1, resulting in more accurate resource demand predictions. Overall, the proposed scheme enables UAV small base stations to obtain high-performance models that are model-appropriate while relieving them of the burden of model training.

[0138] 5. Conclusion

[0139] This paper proposes a dynamic RAN slicing framework for UAV small base stations based on knowledge distillation. Simulation results show that, compared with benchmark methods, the proposed scheme can fully guarantee the performance isolation effect of slices. The proposed slicing scheme is scalable and can be used to support any type of UAV, without being limited by the number of UAVs or their movement trajectories.

[0140] References

[0141] [1]Barlacchi G, De Nadai M, Larcher R, et al. A multi-source dataset of urban life in the city of Milan and the Province of Trentino[J]. Scientificdata, 2015, 2(1):1-15.

[0142] [2]Dadboud F,Patel V,Mehta V,et al.Single-stage uav detection and classification with yolov5:Mosaic data augmentation and panet[C] / / 2021 17thIEEE International Conference on Advanced Video and Signal Based Surveillance(AVSS).IEEE,2021:1-8.

[0143] [3]Fakhreddine A, Bettstetter C, Hayat S, et al. Handover challenges for cellular-connected drones[C] / / Proceedings of the 5th Workshop on Micro AerialVehicle Networks, Systems, and Applications. 2019:9-14.

[0144] [4]Chen J, Zhang Y, Wu L, et al. An adaptive clustering-based algorithm for automatic path planning of heterogeneous UAVs[J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 23(9): 16842-16853.

[0145] [5]Lin T, Stich SU, Barba L, et al.Dynamic model pruning with feedback[J].arXiv preprint arXiv:2006.07253,2020.

[0146] [6] Huang Zhenhua, Yang Shunzhi, Lin Wei, Ni Juan, Sun Shengli, Chen Yunwen, Tang Yong. A review of knowledge distillation research [J]. Chinese Journal of Computers, 2022, 45(03):624-653.

[0147] [7]Xu D,Ouyang W,Wang X,et al.Pad-net:Multi-tasks guided prediction-and-distillation network for simultaneous depth estimation and scene parsing[C] / / Proceedings of the IEEE Conference on Computer Vision and PatternRecognition.2018:675-684.

[0148] [8]Jin X,Lan C,Zeng W,et al.Uncertainty-aware multi-shot knowledgedistillation for image-based object re-identification[C] / / Proceedings of theAAAI Conference on Artificial Intelligence.2020,34(07):11165-11172.

[0149] [9]Shen C,Wang X,Song J,et al.Amalgamating knowledge towardscomprehensive classification[C] / / Proceedings of the AAAI Conference onArtificial Intelligence.2019,33(01):3068-3075.

[0150]

[10] Jiang L,Wen Z,Liang Z,et al.Long short-term sampledistillation[C] / / Proceedings of the AAAI Conference on Artificial Intelligence.2020,34(04):4345-4352.

[0151]

[11] Luo H,Chen T,Li X,et al.KeepEdge:A knowledge distillationempowered edge intelligence framework for visual assisted positioning in UAVdelivery[J].IEEE Transactions on Mobile Computing,2022.

[0152]

[12] Niu Z,Zhong G,Yu H.A review on the attention mechanism of deeplearning[J].Neurocomputing,2021,452:48-62.

[0153]

[13] Zhu W,Lan C,Xing J,et al.Co-occurrence feature learning forskeleton based action recognition using regularized deep LSTM networks[C] / / Proceedings of the AAAI conference on artificial intelligence.2016,30(1).

[0154]

[14] Thammawichai M,Baliyarasimhuni S P,Kerrigan E C,et al.Optimizingcommunication and computation for multi-UAV information gatheringapplications[J].IEEE Transactions on Aerospace and Electronic Systems,2017,54(2):601-615.

[0155]

[15] Song Q,Zeng Y,Xu J,et al.A survey of prototype and experiment forUAV communications[J].Science China Information Sciences,2021,64(4):1-21。

Claims

1. A knowledge distillation-based RAN slicing method for unmanned aerial vehicle (UAV) small base stations, characterized by: In a wireless communication network consisting of ground base stations, UAV small base stations, and MEC controllers, a RAN slicing model based on machine learning technology is deployed in the UAV small base stations; the RAN slicing model is used to dynamically allocate resources to slices according to the resource requirements of the service. The method for generating, training, and delivering the RAN slice model is as follows: Teacher models are deployed in ground base stations and are referred to as large models. The large models in each ground base station use the federated learning (FL) method to generate a batch of RAN slice models for deployment in UAV small base stations. These RAN slice models are student models and are referred to as small models. The small models are stored in the model cache of the ground base stations. The small models are based on the large models and are customized for UAV small base stations according to their model and requirements through knowledge distillation. When a ground base station receives a model request from a drone small base station, the ground base station selects the most suitable small model from the model cache and delivers it to the drone small base station; if the required small model is not in the model cache, the ground base station generates a small model online for the drone small base station. After receiving the small model, the drone small base station fine-tunes the small model itself, and then allocates the local resources of the drone small base station according to the decision output by the small model. In UAV small base stations, the slicing method driven by a small model is as follows: assuming that time is divided into multiple slicing windows, and each slicing window is divided into multiple discrete time slots; during the operation of the current slicing window a, the UAV small base station uses a small model to predict the number of resource blocks required by each slice in the next window a+1; at the beginning of window a+1, the resources of each slice are reallocated according to the resource requirements predicted by the small model; within window a, the number of resource blocks allocated to each slice remains unchanged. At the beginning of each time slot t within the window, the local controller of the UAV small base station allocates resource blocks for the received tasks; The drone small base station reserves a portion of resource blocks that are shared by various slices; when resources are insufficient, the slice temporarily occupies these shared resource blocks.

2. The RAN slicing method for UAV small base stations based on knowledge distillation according to claim 1, characterized in that: Assume that the drone small base stations in the wireless communication network are of different models and have different computing and storage capabilities; First, the ground base stations jointly train a RAN slice model through FL; then, based on the latest model after each round of FL, the ground base stations generate and cache a batch of pre-trained small models locally according to the model and requirements of the UAV small base stations. When a drone small base station enters the coverage area of ​​a ground base station, the two establish a connection through a handshake protocol. The ground base station periodically refreshes the drone small base station connection list and deletes invalid drone small base station association information. The ground base station stores a list of small models corresponding to different drone small base station models. After a drone small base station establishes a connection with the ground base station, the ground base station queries the list for small models related to that drone small base station model. When a model request is received from a drone small base station, the ground base station generates a model in real time or selects the most suitable small model from the model cache and distributes it to the drone small base station.

3. The RAN slicing method for UAV small base stations based on knowledge distillation according to claim 1 or 2 is characterized in that during the process of each ground base station in the wireless communication network jointly training a RAN slice model through FL: First, each ground base station performs local large model training; then, the ground base station uploads the local large model parameters to the MEC controller; next, after the MEC controller receives the large model parameters from each ground base station, it performs global model aggregation; finally, the aggregated global model parameters are distributed to each ground base station for updating the local large model of the ground base station.

4. The RAN slicing method for UAV small base stations based on knowledge distillation according to claim 3, characterized in that: The training steps for a large-scale ground base station model include: 1.1) Local model training: Assume there are B ground base stations, and each ground base station has local resource requirement data for large model training; make Let represent the actual resource demand vector of slice m on ground base station b across Z time slots; further, the actual resource demand matrix for all slices on ground base station b is: R b =[R b,1 ,R b,2 ,...,R b,m ] (1) During the z-th iteration of training, the number of resource blocks required by the predicted slice m of ground base station b in the z+1th iteration is: Where f(·) and These represent the large model and model parameters of ground base station b, respectively; Next, the loss function is introduced. The prediction error measures the number of resource blocks required for slice m on terrestrial base station b; the loss function for all resource slices on terrestrial base station b is expressed as follows: Finally, the local model parameters of ground base station b are updated using the gradient descent method. in, This represents the local model parameters of ground base station b, where η is the learning rate. The loss function represents the loss function relative to w. (z) The gradient; 1.2) Global model aggregation: After receiving the model parameters from various ground base stations, the MEC controller uses an attention-based model aggregation method to aggregate the models and establish a global model, specifically: The large model of the ground base station adopts a stacked LSTM structure. During the global model aggregation process, the attention mechanism is applied to the parameters of each layer of the model to quantify the contribution of each ground base station's local model to the global model in a hierarchical manner. Suppose that during the z-th iteration of training, the global model parameters at the MEC controller are represented as w. (z) In the (z+1)th round, the set of local model parameters for the B ground base stations is represented as... Attention-based FL model aggregation considers global model parameters w (z) With ground base station model parameters W (z+1) The correlation between them; assuming the parameters of layer c in the local model of ground base station b are represented as The parameters of the c-th layer in the global model are represented as make Represents a query. For the corresponding keywords, and The similarity between the two matrices is calculated as the Frobenius norm of the difference between the two matrices, i.e. The hierarchical attention scores of the local model parameters of B ground base stations are calculated using equation (5), and are represented as [s1,s2,…,s…]. B ], where s b =[s b,1 ,s b,2 ,...,s b,C Then, by normalizing the scores using the softmax function, the hierarchical attention distribution of the local model parameters for the B ground base stations is obtained, i.e., α = [α1, α2, ..., α]. B ], where α b =[α b,1 ,α b,2 ,...,α b,C Attention distribution was calculated as follows: Minimize w (z) and W (z+1) The expected distance between them is used to obtain a global model that is closer to the local model in parameter space. Specifically, the attention distribution output by equation (6) is used as the weight to minimize w. (z) and W (z+1) The distance between them, i.e. Where σ(·,·) represents the Euclidean distance between the two sets of parameters; by differentiating the objective functions of equations (1) to (5), we obtain Finally, the gradient descent algorithm is executed to update the parameters of the global model, and the global model parameters are updated to... Where ε represents w (z+1) The step size in the opposite gradient direction during each iteration; The MEC controller then distributes the global model parameters to various ground base stations; 1.3) Local model update: Each ground base station updates its local model based on the global model parameters.

5. The RAN slicing method for UAV small base stations based on knowledge distillation according to claim 4, characterized in that: The method for customizing small models based on knowledge distillation is as follows: First, the ground base station trains a local model based on its own data to build a large model. Then, based on the large model, the ground base station uses knowledge distillation to generate a batch of pre-trained small models locally according to the model and requirements of the drone small base stations it connects to, and stores them in the model cache. The large model is the teacher model, and the small model is the student model; The specific steps include: 2.1) Batch generation of small models In the z-th iteration of training, assume R b =[R b,1 ,R b,2 ,…,R b,M [This refers to] resource requirement data for M slices on ground base station b, where... Let f(·) be the actual resource demand vector of slice m in Z time slots; let f(·) represent the prediction model of ground base station b; the large model of ground base station b predicts the number of resource blocks required by slice m in round z+1. in, These are the model parameters; based on this, the number of resource blocks required for M slices in round z+1 is obtained as follows. in, This is a soft tag; afterwards, R... b Input the data into the drone mini-model to obtain the number of resource blocks required for M slices in round z+1. The goal of knowledge distillation is to distill the output of a small model during training. With corresponding hard labels and soft labels If they match, then the total training loss consists of two parts: the original loss and the distillation loss; assuming... The soft loss is calculated from the mean squared error between the predictions of the small model and the predictions of the large model. The hard loss is calculated from the mean squared error between the predicted and actual values ​​of the small model, and is expressed as follows: At this point, the optimization objective is to minimize the total loss function, i.e. Among them, parameter β is used to adjust the weights of hard loss and soft loss; After the above process, the ground base station generates and caches a batch of small models locally. This set of pre-trained small models is represented as... Where n is the number of model layers; 2.2) Model cache update The large model updates the small model cache in real time based on the latest model after FL; in the (z+1)th round, the large model generates new pre-trained small models based on the latest model, which are represented as follows. 2.3) Small-scale model delivery and online distillation After the UAV small base station k establishes a connection with the ground base station, it first sends its own model number to the ground base station; the ground base station stores a list of model specifications corresponding to the UAV small base station model; when the UAV small base station k sends a model request to the ground base station, the following two situations occur: (a) If the model cache of the ground base station contains the model f required by the UAV small base station k n Then it is delivered directly to the drone small base station; (b) If the model required by UAV small base station k is not present in the ground base station model cache, a small model is generated online according to the needs of the UAV small base station.

6. The RAN slicing method for UAV small base stations based on knowledge distillation according to claim 5, characterized in that: The method for small-model-driven UAV RAN slicing is as follows: When the drone base station k receives model f n Then, the system first fine-tunes the data based on the local data from the UAV small base station, and then segments the physical resources based on the prediction results output by the small model, including the following steps: 3.1) Slice Resource Prediction: The UAV small base station k predicts the number of resource blocks required for M slices in window a+1 using a small model. This represents the predicted number of resource blocks in slice M on the drone small base station k in the next window a+1; 3.2) Slice resource allocation: At the beginning of window a+1, the UAV small base station reallocates the resources of each slice according to the results output by the small model; The UAV small base station k performs resource scheduling in a periodic slice window; at the beginning of each slice window, steps 3.1) and 3.2) are repeated to re-divide resources according to the number of resource blocks required by each slice in the next window as predicted by the small model.

Images