Unmanned aerial vehicle model pruning method and system based on federal reinforcement learning

Abstract

The invention discloses an unmanned aerial vehicle model pruning method and system based on federal reinforcement learning, and relates to the federal learning and reinforcement learning technology, and the method comprises the steps that in a pruning round, a server uses a reinforced generalization pruning method (RGP) to prune a global model, and a pruned model is obtained; the server performs screening to obtain clients participating in pre-training of the current round, and issues the pruned model or the global model to the clients of the current round; the client receives the pruned model or global model sent by the server, and performs fine tuning by using local data; and the client transmits the fine-tuned model to the server, and a sparse model aggregation method (FedSA) is used in the server for model aggregation to obtain a new global model. According to the method, the training effect of federal learning at the edge end with limited resources can be optimized, and the complexity of the training process is reduced while the federal model effect is kept.

Description

Technical Field

[0001] The present application relates to the field of federated learning and reinforcement learning technology, and in particular to a method and system for pruning a drone model based on federated reinforcement learning. Background Art

[0002] The rise of the Internet of Things has connected a large number of sensor devices to the network, generating a large amount of data. Due to the sensitivity of the data, centralized training on a central server poses a risk of privacy leakage.

[0003] Federated learning was born to solve the data privacy issues brought about by centralized training. However, there are challenges in conducting local training of federated learning on resource-constrained edge devices. First, edge devices have limited computing power and storage capacity, while traditional deep neural networks contain a large number of parameters, which require huge computing and memory during training. This results in a long training time for federated learning on edge devices. Second, edge devices have limited communication bandwidth and cannot withstand a large amount of parameter transmission, which in turn affects the stability of the federation process. Summary of the invention

[0004] The embodiments of the present application provide a method and system for pruning a drone model based on federated reinforcement learning, so as to optimize the training effect of federated learning at the resource-constrained edge end, thereby reducing the complexity of the training process while maintaining the effect of the federated model.

[0005] The embodiment of the present application provides a UAV model pruning method based on federated reinforcement learning, which is applied to a UAV device under a federated learning architecture. The federated learning architecture includes a server and multiple client nodes, wherein a server data database module, a server model trainer module, and a node manager module are arranged on the server, and a client model database, a client data database, and a client model trainer are arranged on the client node. The UAV device, as a client node, realizes the collection of image and location information based on GPS and image sensors. The UAV model pruning method includes:

[0006] In the pruning round, the server uses the reinforcement generalization pruning method (RGP) to prune the global model to obtain the pruned model;

[0007] The server selects the clients participating in this round of pre-training and sends the pruned model or the global model to the clients of this round;

[0008] The client receives the pruned model or global model sent by the server and fine-tunes it using local data;

[0009] The client transmits the fine-tuned model to the server, where the server uses the Federated Sparse Averaging (FedSA) method to aggregate the model and obtain a new global model.

[0010] Optionally, it also includes initializing model parameters on the server in the first round, and training the model using server data in the model trainer to obtain an initial global model.

[0011] Optionally, the pruning round specifically includes:

[0012] The agent selects an action to execute based on the current state and performs unstructured pruning on the global model based on the selected action;

[0013] The server fine-tunes the pruned model using the server data in the enhanced pruning environment;

[0014] The server evaluates the generalization of the fine-tuned model in the enhanced pruning environment and calculates the learning distance of the model, which is defined as the l2 distance between the fine-tuned model and its initial global model, satisfying: D(W finetune )=||W init -W finetune ||2, where W init ∈R d is the initial global model, W finetune ∈R d is the fine-tuned model;

[0015] Using the generalized reward mechanism, the reward for this action is calculated based on the learning distance, and the reinforcement learning model is updated based on the reward value.

[0016] Optionally, using a generalized reward mechanism, the reward for this action based on the learning distance calculation includes:

[0017] Describe UAV model pruning based on federated reinforcement learning as one-dimensional scene exploration;

[0018] Sampling within the set range, taking the highest generalization point as the initial state point of reinforcement learning, setting the pruning rate of the initial state point to p0, and defining the model of this point as

[0019] Calculate the reward function R(a) and use the reward function to encourage the agent to move to increase the pruning rate until it reaches the set end point.

[0020] Optionally, the reward function R(a) is calculated to satisfy:

[0021]

[0022] Among them, uplimit represents the upper limit of pruning, end represents the end point of the agent, and end satisfies

[0023] Optionally, using a sparse model aggregation method (Federated Sparse Averaging, FedSA) on the server to perform model aggregation includes:

[0024] FedSA is used to analyze the sparsity of the model and obtain the sparse structure of the model. Sparsity represents the percentage of zero weights in all weights of the model, which satisfies Where m represents the number of model layers, τ i represents the number of zero weights in the i-th layer model, M i ,N i Respectively represent the number of rows and columns of the weight matrix of the i-th layer model;

[0025] Perform weighted averaging on non-zero weights to obtain the global model parameter w obtained by round t+1 aggregation t+1 ,satisfy:

[0026]

[0027] Among them, w t+1 represents the global model parameters obtained by the t+1th round of aggregation, represents the local model parameters of client k in the t+1th round of aggregation, M represents the number of clients participating in the aggregation, and m k represents the number of samples of client k in the t+1th round of aggregation, and m represents the total number of samples of M clients.

[0028] An embodiment of the present application also proposes a UAV model pruning system based on federated reinforcement learning, including a processor and a memory, wherein a computer program is stored on the memory, and when the computer program is executed by the processor, the steps of the UAV model pruning method based on federated reinforcement learning as described above are implemented.

[0029] The method of the embodiment of the present application optimizes the training effect of federated learning at the resource-constrained edge, thereby reducing the complexity of the training process while maintaining the effect of the federated model.

[0030] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0032] Figure 1 The overall process of the UAV model pruning method based on federated reinforcement learning in an embodiment of the present application is schematically illustrated. DETAILED DESCRIPTION

[0033] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0034] The embodiment of the present application provides a method for pruning a drone model based on federated reinforcement learning, which is applied to a drone device under a federated learning model architecture. The federated learning architecture includes a server and multiple client nodes, wherein the server is provided with a server data database module, a server model trainer module, and a node manager module, and the client node is provided with a client model database, a client data database, and a client model trainer. The drone device, as a client node, collects image and location information based on GPS and image sensors. In some examples, the drone device is used as an intelligent agent, for example, a DJI Mini-3 model can be used, which is equipped with hardware such as GPS, image sensors, and microcontrollers. The microcontroller is responsible for running the model pruning method. The drone model pruning method, such as Figure 1 The steps shown include:

[0035] In step S101, in the pruning round, the server uses the reinforcement generalization pruning method (RGP) to prune the global model to obtain a pruned federated global model.

[0036] In step S102, the server selects and obtains the clients participating in this round of pre-training, and sends the pruned federated global model or the unpruned federated global model to the clients of this round.

[0037] In step S103, the client receives the pruned federated global model or the unpruned federated global model sent by the server, and performs fine-tuning using local data.

[0038] In step S104, the client transmits the fine-tuned model to the server, and the server uses a sparse model aggregation method (Federated Sparse Averaging, FedSA) to aggregate the model to obtain a new global model.

[0039] The method of the embodiment of the present application optimizes the training effect of federated learning at the resource-constrained edge, thereby reducing the complexity of the training process while maintaining the effect of the federated model.

[0040] In some embodiments, the step S100 is also included, in which the server initializes the model parameters in the first round, and uses the server data to train the model in the model trainer to obtain an initial global model.

[0041] In some embodiments, the pruning round specifically includes:

[0042] The agent selects actions to execute based on the current state and performs unstructured pruning on the federated global model based on the selected actions, that is, pruning individual weights or connections in the neural network. This method can effectively remove redundant parameters without changing the overall structure of the network.

[0043] The server fine-tunes the pruned model using the server data in the enhanced pruning environment;

[0044] The server evaluates the generalization of the fine-tuned model in the enhanced pruning environment and calculates the learning distance of the model, which is defined as the l2 distance between the fine-tuned model and its initial global model, satisfying: D(W finetune )=||W init -W finetune ||2, where W init ∈R d is the initial global model, W finetune ∈R d is the fine-tuned model. The smaller the learning distance, the stronger the generalization of the model and the better the pruning effect.

[0045] Using the generalized reward mechanism, the reward for this action is calculated based on the learning distance, and the reinforcement learning model is updated based on the reward value.

[0046] In some embodiments, using a generalized reward mechanism, the reward for this action based on the learning distance calculation includes:

[0047] Describe UAV model pruning based on federated reinforcement learning as one-dimensional scene exploration;

[0048] Sampling within the set range, taking the highest generalization point as the initial state point of reinforcement learning, setting the pruning rate of the initial state point to p0, and defining the model of this point as

[0049] Calculate the reward function R(a) and use the reward function to encourage the agent to move to increase the pruning rate until it reaches the set end point.

[0050] In some embodiments, the reward function R(a) is calculated to satisfy:

[0051]

[0052] Among them, uplimit represents the upper limit of pruning, end represents the end point of the agent, and end satisfies

[0053] In some embodiments, performing model aggregation using a sparse model aggregation method (Federated Sparse Averaging, FedSA) on a server includes:

[0054] FedSA is used to analyze the sparsity of the model and obtain the sparse structure of the model. Sparsity represents the percentage of zero weights in all weights of the model, which satisfies Where m represents the number of model layers, τ i represents the number of zero weights in the i-th layer model, M i ,N i Respectively represent the number of rows and columns of the weight matrix of the i-th layer model;

[0055] Perform weighted averaging on non-zero weights to obtain the global model parameter w obtained by round t+1 aggregation t+1 ,satisfy:

[0056]

[0057] Among them, w t+1 represents the global model parameters obtained by the t+1th round of aggregation, represents the local model parameters of client k in the t+1th round of aggregation, M represents the number of clients participating in the aggregation, and m k represents the number of samples of client k in the t+1th round of aggregation, and m represents the total number of samples of M clients.

[0058] This application designs a new federated pruning framework based on reinforcement learning. This application uses an enhanced generalization pruning method to implement iterative pruning of the model. First, the generalization of the model is used to guide the reinforcement learning pruning process to improve the generalization ability of the model; then, a pruning method that is common to reinforcement learning is formed through an abstract pruning process. The method of this application also uses a sparse model aggregation method to achieve fusion of only non-zero weights in the model, further improving the sparsity of the model.

[0059] An embodiment of the present application also proposes a UAV model pruning system based on federated reinforcement learning, including a processor and a memory, wherein a computer program is stored on the memory, and when the computer program is executed by the processor, the steps of the UAV model pruning method based on federated reinforcement learning as described above are implemented.

[0060] It should be noted that in the various embodiments of the present application, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0061] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0062] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0063] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A UAV model pruning method based on federated reinforcement learning, characterized in that: The invention is applied to a federated learning architecture using a drone device, wherein the federated learning architecture includes a server and multiple client nodes, wherein the server is provided with a server data database module, a server model trainer module, and a node manager module, and the client node is provided with a client model database, a client data database, and a client model trainer. The drone device, as a client node, realizes the collection of image and location information based on GPS and an image sensor, and the drone model pruning method includes: In the pruning round, the server uses the reinforcement generalization pruning method (RGP) to prune the global model to obtain the pruned model; The server selects the clients participating in this round of pre-training and sends the pruned model or the global model to the clients of this round; The client receives the pruned model or global model sent by the server and fine-tunes it using local data; The client transmits the fine-tuned model to the server, where the server uses the Federated Sparse Averaging (FedSA) method to aggregate the model and obtain a new global model.

2. The UAV model pruning method based on federated reinforcement learning as claimed in claim 1, characterized in that: It also includes initializing model parameters on the server in the first round, and training the model using server data in the model trainer to obtain an initial global model.

3. The UAV model pruning method based on federated reinforcement learning as claimed in claim 2, characterized in that: The pruning rounds specifically include: The agent selects an action to execute based on the current state and performs unstructured pruning on the global model based on the selected action; The server fine-tunes the pruned model using the server data in the enhanced pruning environment; The server evaluates the generalization of the fine-tuned model in the enhanced pruning environment and calculates the learning distance of the model, which is defined as the l2 distance between the fine-tuned model and its initial global model, satisfying: D(W finetune )=||W init -W finetune ||2, where W init ∈R d is the initial global model, W finetune ∈R d is the fine-tuned model; Using the generalized reward mechanism, the reward for this action is calculated based on the learning distance, and the reinforcement learning model is updated based on the reward value.

4. The UAV model pruning method based on federated reinforcement learning as claimed in claim 3, characterized in that: Using the generalized reward mechanism, the rewards for this action are calculated based on the learning distance, including: Describe UAV model pruning based on federated reinforcement learning as one-dimensional scene exploration; Sampling within the set range, taking the highest generalization point as the initial state point of reinforcement learning, setting the pruning rate of the initial state point to p0, and defining the model of this point as Calculate the reward function R(a) and use the reward function to encourage the agent to move to increase the pruning rate until it reaches the set end point.

5. The UAV model pruning method based on federated reinforcement learning as claimed in claim 4, characterized in that: The calculation reward function R(a) satisfies: Among them, uplimit represents the upper limit of pruning, end represents the end point of the agent, and end satisfies 6. The UAV model pruning method based on federated reinforcement learning as claimed in claim 5, characterized in that: Model aggregation using the Federated Sparse Averaging (FedSA) method on the server includes: FedSA is used to analyze the sparsity of the model and obtain the sparse structure of the model. Sparsity represents the percentage of zero weights in all weights of the model, which satisfies Where m represents the number of model layers, τ i represents the number of zero weights in the i-th layer model, M i ,N i Respectively represent the number of rows and columns of the weight matrix of the i-th layer model; Perform weighted averaging on non-zero weights to obtain the global model parameter w obtained by round t+1 aggregation t+1 ,satisfy: Among them, w t+1 represents the global model parameters obtained by the t+1th round of aggregation, represents the local model parameters of client k in the t+1th round of aggregation, M represents the number of clients participating in the aggregation, and m k represents the number of samples of client k in the t+1th round of aggregation, and m represents the total number of samples of M clients.

7. A UAV model pruning system based on federated reinforcement learning, characterized in that: It includes a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the UAV model pruning method based on federated reinforcement learning as described in any one of claims 1 to 6 are implemented.

Images