Unmanned aerial vehicle group real-time image recognition method and system based on asynchronous federated learning

Abstract

The invention discloses an unmanned aerial vehicle group real-time image recognition method and system based on asynchronous federated learning, and the method comprises the steps: carrying out the local recognition model training of each unmanned aerial vehicle, and uploading a trained local model to a central server; calculating a corresponding Shapley value, and introducing an old degree attenuation factor to adjust the weight to obtain an adjusted Shapley value; updating the global model through weighted aggregation based on the adjusted Shapril value; and after the global model is updated, the unmanned aerial vehicle with the highest utility is selected by adopting a confidence upper limit algorithm to participate in subsequent federated learning, and client optimization scheduling in a dynamic environment is realized. The asynchronous federated learning mechanism based on the Shapley value can effectively measure the contribution of each unmanned aerial vehicle to the global model, reduces the influence of outdated updating by introducing an attenuation factor, improves the accuracy of the global model in a dynamic environment, enables the model to better adapt to the complex environment and data change of the unmanned aerial vehicle group, and improves the robustness of the unmanned aerial vehicle group. Therefore, the accuracy of image recognition is improved.

Description

Technical Field

[0001] The present invention belongs to the field of drone and artificial intelligence technology, and in particular relates to a real-time image recognition method and system for drone swarms based on asynchronous federated learning. Background Art

[0002] With the rapid development of drone technology, it plays a key role in computer vision applications, particularly in real-time image recognition tasks. Drones, with their high maneuverability and flexibility, equipped with high-resolution cameras and onboard processors, can capture images from different perspectives, providing more comprehensive and dynamic information for image recognition. These drones have important applications in a wide range of fields, including traffic monitoring, environmental monitoring, and disaster response.

[0003] However, individual drones have significant limitations in computing power and battery life, making it difficult to independently handle complex image recognition tasks in large-scale or dynamic environments. Therefore, the collaboration of multiple drones to form a cluster has emerged, improving the efficiency and accuracy of image recognition by sharing resources and data. However, there are many challenges in the process of cluster collaboration. On the one hand, when transmitting raw image data or large feature sets between drone swarms and central servers, it is often impractical due to limited communication bandwidth and unstable network. This is especially true in time-sensitive application scenarios such as disaster relief or rapid industrial manufacturing, where even a small recognition delay can lead to serious consequences. On the other hand, image data often contains sensitive information, such as private property or personal images, and data privacy issues have become a major obstacle to drone swarm collaboration.

[0004] Federated learning provides an effective approach to addressing data privacy and efficient computing challenges in drone swarm collaboration. By integrating federated learning into drone swarms, distributed computing and efficient data sharing can be achieved, enabling the training and deployment of high-performance, real-time image recognition models while protecting the privacy of each drone's local data. However, existing federated learning methods in drone swarm applications ignore the impact of the dynamic and unstable nature of the drone swarm environment on the learning process, resulting in reduced model recognition accuracy and slower convergence. Specifically, unstable communication channels between drones and central servers increase model upload latency, hindering timely updates of the global model. Furthermore, the dynamic state of drones—such as their ability to freely join or leave missions, dynamic changes in local training due to limited computing power, and time-varying local datasets caused by the continuous acquisition and processing of image data during recognition tasks—can lead to inconsistent quality of local models sent to the central server, slowing convergence and reducing the accuracy of the recognition model. Summary of the Invention

[0005] The purpose of the present invention is to provide a real-time image recognition method and system for drone swarms based on asynchronous federated learning, so as to solve the problems of low model accuracy and slow convergence speed in the existing drone swarm federated learning image recognition technology.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for real-time image recognition of a drone swarm based on asynchronous federated learning, comprising:

[0008] Each drone downloads the latest global image recognition model from the central server, trains a local recognition model, and uploads the trained local model to the central server;

[0009] The central server receives the trained local model uploaded by the drone, calculates the corresponding Shapley value, and introduces the staleness attenuation factor to adjust the weight to obtain the adjusted Shapley value;

[0010] The global model is updated by weighted aggregation based on the adjusted Shapley values;

[0011] After the global model is updated, a time-varying context multi-armed bandit model is constructed, and the drone feature vector is input into the Gaussian random walk model to predict the client utility. The confidence upper bound algorithm is used to select the drone with the highest utility to participate in subsequent federated learning, realizing client optimization scheduling in a dynamic environment.

[0012] Furthermore, each drone downloads the latest global image recognition model from the central server, including:

[0013] Drone n first downloads the latest version of the image recognition model ω(t) from the central server. The trained model extracts features from the image through convolution, activation, and pooling operations, and performs image recognition based on these features.

[0014] Furthermore, each drone performs local recognition model training and uploads the trained local model to the central server, including:

[0015] The drone uses its local data n Perform h rounds of local training and get The local training of each round is expressed as follows:

[0016]

[0017] Among them, λ>0 represents the learning rate, is the loss function on the dataset D n After completing local training, drone n updates its local model Transmit to the central server and convert τmax Set to the maximum tolerated outdated global round, the drone is in the maximum tolerated round τ max Otherwise, the server will reject the connection and terminate the upload.

[0018] Furthermore, the central server receives the trained local model uploaded by the drone and calculates the corresponding Shapley value, including:

[0019] Under asynchronous federated learning, each drone contributes its local model update to form a cooperative game and construct a cooperative game. in represents the set of participating drones, Is to reflect the model in the collection only The value function of the accuracy when the drone is involved, the calculation process is:

[0020]

[0021] where Q(.) returns the utility score of the model, is a collection The federated learning model obtained after t rounds of asynchronous updates of the drones in the game; the cooperative game is decomposed into multiple small cooperative games, each of which evaluates the contribution of the drones during each round of asynchronous aggregation; each game in represents the performance improvement of the model on the test dataset and is defined as:

[0022]

[0023] The Shapley value measures the impact of a user joining or leaving the game and calculates the contribution by averaging the marginal contributions of all possible coalitions, with n for different subsets The marginal contribution is expressed as:

[0024]

[0025] Take all subsets The average marginal contribution of , we get the following Shapley value:

[0026]

[0027] Furthermore, the introduction of the staleness attenuation factor to adjust the weight to obtain the adjusted Shapley value includes:

[0028] The staleness decay factor ω is introduced when UAV n uploads its latest model update ω during the t′th global round n (t), ω n The staleness δ of (t) nDefined as |t′-t|; as the staleness increases, the staleness decay factor decreases exponentially, using the staleness decay factor This factor is applied to the Shapley value, and the adjusted Shapley value is:

[0029]

[0030] Furthermore, updating the global model through weighted aggregation based on the adjusted Shapley value includes:

[0031] The first step is to determine the contribution of each UAV to the updated global model ω(t′), expressed as θ(t′) = θ h (t′),...,θ n (t′),...,θ N (t′); the second is to use θ(t′) as the weight to update the global model:

[0032]

[0033] This process is iterated until the global loss function converges to ω * ; The global loss function is expressed as follows:

[0034]

[0035] Furthermore, after the global model is updated, a time-varying context multi-armed bandit model is constructed, and the drone feature vector is input into the Gaussian random walk model to predict the client utility. The confidence upper bound algorithm is used to select the drone with the highest utility to participate in subsequent federated learning, thereby achieving client optimization scheduling in a dynamic environment, including:

[0036] In the Shapley-based asynchronous federated learning framework, client utility consists of two components: statistical utility quantifying the model contribution of drone data and system utility reflecting the improvement in convergence speed;

[0037] Statistical utility:

[0038]

[0039] System Utility:

[0040]

[0041] in Count the disconnections that occurred due to timeout before round t;

[0042] Total utility:

[0043]

[0044] The client selection problem is reformulated as a contextual MAB problem, where each drone n is regarded as an arm with client utility as the reward; the utility parameter is defined as the feature vector x n (t):

[0045]

[0046] This feature vector changes over time and its value remains unknown before the end of round t; the central server maintains a time-varying coefficient vector w n , which represents the contribution of each feature to utility prediction; the expected utility of drone n is modeled as:

[0047]

[0048] where ∈ n The variance is The zero-mean Gaussian noise of is:

[0049]

[0050] The central server uses the upper confidence limit (UCB) algorithm to calculate the score of each drone and selects the drone with the highest UCB score for federated learning:

[0051]

[0052] A Gaussian random walk model with Markov process is used to capture w n Drift;

[0053]

[0054] in Represents the static component, capturing the average contribution of each context feature, the static component Modeling using a conjugate prior distribution:

[0055]

[0056] in is the prior mean, which represents the belief about the average contribution of drone context features to utility; is the covariance matrix, introducing a Gaussian random walk modeled as a Markov process:

[0057]

[0058] where η n (t) represents the drift value, which is affected by the previous drift η n (t-1) and Gaussian noise vector the impact of; The scaling factor in determines the importance of each element in the drift vector Δ(t), which is expressed as:

[0059]

[0060] in is the average of the scaling factors, is the covariance matrix; the parameters drawn from an inverse gamma distribution;

[0061] The expected client utility of drone n in round t is given by:

[0062]

[0063] where x n (t) is the feature vector of drone n in round t;

[0064] In round t, the central server collects feature vector x from drone n n (t);

[0065] Model the parameters using a Gaussian distribution:

[0066]

[0067] where the mean and covariance are updated as:

[0068]

[0069] Where G(.) is the Kalman gain function, and in the t-1 round, the parameter random variable The sufficient statistics for make and infer and This is equivalent to inferring the distribution v n ; Assume that Σ′, μ′, α′, and β′ are sufficient statistics for round t and are updated based on new observations at time t:

[0070]

[0071] Round t, and v n The sampling process is summarized as follows:

[0072]

[0073] Based on sufficient statistical data from the previous t-1 rounds, w n (t-1) Sample from the following:

[0074]

[0075] in:

[0076]

[0077] The estimated utility is expressed as:

[0078]

[0079] In this expression, the first term represents the expected utility, while the second term captures the uncertainty in the estimate and is weighted by the exploration parameter λ.

[0080] In a second aspect, the present invention provides a real-time image recognition system for drone swarms based on asynchronous federated learning, comprising:

[0081] The local training module is used for each drone to download the latest global image recognition model from the central server, each drone performs local recognition model training, and uploads the trained local model to the central server;

[0082] The Shapley value calculation module is used by the central server to receive the trained local model uploaded by the drone, calculate the corresponding Shapley value, and introduce the staleness attenuation factor to adjust the weight to obtain the adjusted Shapley value;

[0083] A global update module for updating the global model through weighted aggregation based on the adjusted Shapley value;

[0084] The dynamic client selection module is used to build a time-varying context multi-armed bandit model after the global model is updated. The drone feature vector is input into the Gaussian random walk model to predict the client utility. The confidence upper bound algorithm is used to select the drone with the highest utility to participate in subsequent federated learning, realizing client optimization scheduling in a dynamic environment.

[0085] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method for real-time image recognition of a swarm of drones based on asynchronous federated learning are implemented.

[0086] In a fourth aspect, the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the method for real-time image recognition of drone swarms based on asynchronous federated learning.

[0087] Compared with the prior art, the present invention has the following technical effects:

[0088] The asynchronous federated learning mechanism based on Shapley value of the present invention can effectively measure the contribution of each drone to the global model. By introducing a decay factor to reduce the impact of outdated updates, the accuracy of the global model in a dynamic environment is improved, and the model can better adapt to the complex environment and data changes of the drone swarm, thereby improving the accuracy of image recognition.

[0089] The client selection strategy based on the time-varying context multi-armed bandit can accurately capture the dynamic state of the drone by constructing a dynamic utility model and adopting an effective selection algorithm, and select drones with high update contribution and low update latency to participate in learning, thereby accelerating the convergence speed of federated learning, reducing training time and resource consumption, and improving the overall performance and efficiency of the system.

[0090] The drone swarm image recognition system of the present invention, through the collaborative work between drones and a central server, combined with the above-mentioned innovative learning and selection methods, can effectively cope with the instability and dynamics of the drone swarm environment while ensuring data privacy, and achieve efficient and accurate real-time image recognition. It has important application value in the fields of traffic monitoring, environmental monitoring, disaster response, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Figure 1 It is an asynchronous federated learning graph for real-time image recognition in drone swarms.

[0092] Figure 2 Performance comparison in unstable environments, (a) is unstable air-to-ground communication, (b) is time-varying UAV local dataset, (c) is dynamic UAV local training time, and (d) is dynamic UAV swarm.

[0093] Figure 3 Accuracy versus communication rounds on Fashion-MNIST, (a) when γ = 0.1, (b) when γ = 1.

[0094] Figure 4 Comparison of recognition accuracy and communication rounds between CSS-AsyFL and other baselines on different datasets, (a) when γ = 0.1, (b) when γ = 1 for CIFAR-10.

[0095] Figure 5 is the number of communication rounds required to achieve the object recognition accuracy of CSS-AsyFL and other baselines on different datasets.

[0096] Figure 6 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0097] The present invention will be further described below with reference to the accompanying drawings:

[0098] Example 1, please refer to Figure 6 The present invention provides a real-time image recognition method for a drone swarm based on asynchronous federated learning, comprising:

[0099] Each drone downloads the latest global image recognition model from the central server, trains a local recognition model, and uploads the trained local model to the central server;

[0100] The central server receives the trained local model uploaded by the drone, calculates the corresponding Shapley value, and introduces the staleness attenuation factor to adjust the weight to obtain the adjusted Shapley value;

[0101] The global model is updated by weighted aggregation based on the adjusted Shapley values;

[0102] After the global model is updated, a time-varying context multi-armed bandit model is constructed, and the drone feature vector is input into the Gaussian random walk model to predict the client utility. The confidence upper bound algorithm is used to select the drone with the highest utility to participate in subsequent federated learning, realizing client optimization scheduling in a dynamic environment.

[0103] In Example 2, the present invention provides a method for real-time image recognition of a drone swarm based on asynchronous federated learning, specifically comprising:

[0104] See also Figure 1 , where a group of drones is denoted as Deployed in various geographical areas to collect images and perform real-time image recognition tasks within a specified time frame. This paper proposes a real-time image recognition method for drone swarms based on asynchronous federated learning, which includes the following steps:

[0105] S1. Model download and image recognition task execution. Drone n first downloads the latest version of the image recognition model ω(t) from the central server. To adapt to its specific mission scenario, the drone retrains the model to meet accuracy requirements. The retrained model extracts features from the image through operations such as convolution, activation, and pooling, and performs image recognition based on these features.

[0106] S2, local recognition model training and uploading. After completing the task, the drone uses its free time and resources to use its local data D n Perform h rounds of local training and get The local training of each round is expressed as follows:

[0107]

[0108] Among them, λ>0 represents the learning rate, is the loss function on the dataset D nAfter completing local training, drone n updates its local model To avoid long model upload time due to unstable communication, τ max Set to the maximum tolerated outdated global rounds. The drone must be within the maximum tolerated rounds τ max Upload the local model within the server, otherwise the server will reject the connection and terminate the upload;

[0109] S3, global recognition model update. Once drone n completes local training and uploads the model The global model may have been updated several times, up to round t′, where t′ ≥ t. Integrating into the global model, the central server faces two challenges: first, determining the contribution of each drone to the updated global model ω(t′), expressed as θ(t′) = θ h (t′),...,θ n (t′),...,θ N (t′); The second problem is to update the global model using θ(t′) as weights:

[0110]

[0111] This process is iterated until the global loss function converges to ω * The global loss function is expressed as follows:

[0112]

[0113] S4. Shapley-based asynchronous update mechanism. Shapley values are used to measure each drone's contribution to the overall model, enhancing the impact of high-quality local updates, which is crucial for accurate and timely image recognition. To address the issue of outdated updates, the present invention modifies the Shapley values to reduce the impact of outdated contributions. The adjusted Shapley values are then used as aggregation weights to dynamically adjust the impact of each drone on the global model, ultimately improving accuracy and ensuring convergence.

[0114] S401. Value function based on cooperative games. Shapley value is a concept in cooperative game theory that measures the fair contribution of individuals participating in the game by considering all possible combinations of participants and fairly assigning value to each participant. In the context of asynchronous federated learning, each drone contributes its local model update to form a cooperative game, whose goal is to improve the overall model performance. Construct a cooperative game in represents the set of participating drones, Is to reflect the model in the collection only The value function of the accuracy when the drone is involved, the calculation process is:

[0115]

[0116] where Q(.) returns the utility score of the model, i.e. its performance on the test set, is a collection The federated learning model is obtained after t rounds of asynchronous updates of the drones in the game. The present invention further decomposes the cooperative game into multiple smaller cooperative games, each of which evaluates the contribution of the drones during each round of asynchronous aggregation. in represents the performance improvement of the model on the test dataset and is defined as:

[0117]

[0118] S402. Evaluate the contribution of drones in asynchronous aggregation based on Shapley value. Shapley value measures the impact of users joining or leaving the game and calculates their contribution by averaging the marginal contributions of all possible coalitions. The marginal contribution is expressed as:

[0119]

[0120] To calculate the overall contribution of drone n, take all subsets The average marginal contribution of , we get the following Shapley value:

[0121]

[0122] S403, staleness attenuation factor. In order to deal with the problem of outdated updates due to unstable air-to-ground communication, the present invention introduces a staleness attenuation factor when UAV n uploads its latest model update ω during the t′th global round. n (t), ω n The staleness δ of (t) n Defined as |t′-t|. As the staleness increases, the staleness attenuation factor decreases exponentially. The staleness attenuation factor used in this invention is This factor is applied to the Shapley value to reduce the impact of outdated updates while maintaining fairness in contribution evaluation. The adjusted Shapley value is given by:

[0123]

[0124] S5. Client Selection. In the Shapley-based asynchronous federated learning framework, client utility consists of two components: statistical utility (quantifying the model contribution of drone data) and system utility (reflecting the improvement in convergence speed).

[0125] Statistical utility:

[0126]

[0127] System Utility:

[0128]

[0129] in Counts disconnects that occurred due to timeouts t rounds ago.

[0130] Total utility:

[0131]

[0132] S501, time-varying contextual multi-armed bandit. This paper reformulates the client selection problem as a contextual MAB problem. In this framework, each drone n is considered as an arm, with client utility as the reward. This paper defines the utility parameter as the feature vector x n (t):

[0133]

[0134] This feature vector varies with time and its value remains unknown until the end of round t. The central server maintains a time-varying coefficient vector w n , which represents the contribution of each feature to utility prediction. Therefore, the expected utility of drone n is modeled as:

[0135]

[0136] where ∈ n The variance is Zero-mean Gaussian noise. Therefore:

[0137]

[0138] S502. The central server uses the upper confidence limit (UCB) algorithm to calculate the score of each drone and selects the drone with the highest UCB score for federated learning:

[0139]

[0140] In most existing contextual MAB works, w nis considered unknown but fixed over time, and the algorithm aims to estimate its true value through continuous online learning. However, in the context of asynchronous federated learning in unstable environments, the elements of the UAV feature vector are time-varying, making w n The static assumption of w becomes an invalid method. Therefore, a Gaussian random walk model with Markov process is used to capture w n Drift.

[0141]

[0142] in represents the static component, capturing the average contribution of each context feature, while Δ(t) models the dynamic drift due to environmental fluctuations. Modeling using a conjugate prior distribution:

[0143]

[0144] in is the prior mean, representing the belief about the average contribution of drone context features to utility. is the covariance matrix, which encodes the uncertainty in the prior belief. To model the dynamic drift Δ(t), the present invention introduces a Gaussian random walk modeled as a Markov process:

[0145]

[0146] where η n (t) represents the drift value, which is affected by the previous drift η n (t-1) and Gaussian noise vector v~ This random walk captures the unpredictable fluctuations in the drone’s utility that arise from time-varying contextual conditions, such as communication channel conditions and drone state. The scaling factor in determines the importance of each element in the drift vector Δ(t), which is expressed as:

[0147]

[0148] in is the average of the scaling factors, is the covariance matrix. is drawn from an inverse gamma distribution.

[0149] This random walk approach enables the model to effectively adapt to the dynamic nature of drone usage. By incorporating a dynamic drift term, the proposed model captures variability and enhances utility prediction. Therefore, the expected client utility of drone n in round t is given by:

[0150]

[0151] where x n (t) is the feature vector of drone n in round t. By integrating static and dynamic components, this method achieves accurate and adaptable drone client utility prediction.

[0152] S503, Client Selection. In round t, the central server collects feature vector x from drone n. n (t). However, before the completion of round t of federated learning, the actual utility U of drone n n (t) is still unknown. To estimate the expected utility of drone selection in round t, the parameter set

[0153] Time-varying drift η n (t) cannot be observed directly and evolves over time, so a Gaussian distribution is used to model this parameter:

[0154]

[0155] where the mean and covariance are updated as:

[0156]

[0157] where G(.) is the Kalman gain function that adjusts the estimated drift based on the observed utility.

[0158] In round t-1, the parameter random variable The sufficient statistics for make and Therefore, it is inferred and This is equivalent to inferring the distribution v n Assume that Σ′, μ′, α′, and β′ are sufficient statistics for round t and are updated based on new observations at time t:

[0159]

[0160] Round t, and v n The sampling process is summarized as follows:

[0161]

[0162] Based on sufficient statistical data from the previous t-1 rounds, w n (t-1) Sample from the following:

[0163]

[0164] in:

[0165]

[0166] The estimated utility is expressed as:

[0167]

[0168] In this expression, the first term represents the expected utility, while the second term captures the uncertainty in the estimate and is weighted by the exploration parameter λ.

[0169] The convergence of the proposed Shapley-based asynchronous federated learning framework is analyzed. First, the following assumptions are introduced:

[0170] 1. L-smooth: A differentiable function F is L-smooth if, for any x,y, the following inequality holds: Where L>0.

[0171] 2. μ-convexity: A differentiable function F is a μ-weakly convex function if, for any x, y, we have: Where μ≥0.

[0172] 3. Bounded gradient: The stochastic gradient norm of each drone is uniformly bounded, i.e.

[0173] Considering these assumptions, we now propose the following theorem, which proves that the proposed federated learning algorithm converges after a sufficient number of global updates:

[0174] Assume that assumptions 1, 2, and 3 hold for the global objective function F(ω), and each UAV performs at least l min and up to l max Local training rounds. After j rounds of global updates, the convergence rate meets the following bounds:

[0175]

[0176] Among them, l j It's another drone j The number of local rounds executed during round l, It is the maximum contribution value (adjusted Shapley value) among all drones that upload local models.

[0177] The result of this theorem shows that the convergence rate depends on several key factors, including the total number of updates T, the bounded gradient norm ρ, the maximum contribution and the staleness parameter τ. The derived bounds show that the gradient norm decreases as the global update proceeds, demonstrating the convergence of the proposed asynchronous federated learning method.

[0178] Describe the implementation process and experimental comparison to adequately support the technical solution and demonstrate its beneficial effects. The data should be presented in a table format.

[0179] This section describes the simulation setup used to evaluate the proposed Shapley-based asynchronous federated learning framework with a fine-grained client selection strategy for real-time image recognition in a swarm of drones. 1) Base Setup: The federated learning environment is established using PyTorch. A central server is deployed on a high-performance workstation (Dell PowerEdge T630 equipped with 2× GTX 4090Ti GPUs) and coordinates N = 20 drones, each cruising at an altitude ranging from 80 to 100 meters. The channel power gain g0 between the drones and the central server is initially set to 1.42×10-4, reflecting realistic air-to-ground communication conditions.

[0180] To evaluate the performance of the proposed framework, two image recognition tasks were designed using convolutional neural networks (CNNs) to classify the Fashion-MNIST and CIFAR-10 datasets. For Fashion-MNIST, a CNN with convolutional, pooling, and fully connected layers was used. For the more complex CIFAR-10 dataset, a larger CNN with greater capacity was employed to handle the more challenging dataset. To simulate the real-world scenario where data is often non-independent and non-identically distributed (non-IID) across clients, a Dirichlet distribution was used to generate non-IID drone data. Specifically, each drone's local data is drawn from a Dirichlet distribution D~Dir(γ,p), where p is a ten-dimensional vector with all elements equal to 1, and γ controls the level of non-IID. When γ→0, each drone only stores data for one randomly selected class (highly non-IID). Conversely, when γ→∞, each drone accesses data from all classes, approximately achieving an independent and identical distribution. Imagine. 2) Volatile Environment Simulation: To thoroughly test the robustness of the proposed framework, this experiment simulates a volatile environment that introduces challenges commonly encountered in the real world, including unreliable air-to-ground communications, dynamically changing drone swarms, time-varying local datasets, and fluctuating local training conditions.

[0181] To evaluate the performance of the proposed framework, this experiment compares it with two well-established asynchronous federated learning frameworks (AsyFL and OCSF-AsyFL). These benchmarks highlight the effectiveness of the proposed framework in handling unstable environments.

[0182] AsyFL: AsyFL is a traditional asynchronous federated learning framework. In this approach, drones perform local model updates asynchronously, and the server aggregates the models immediately after receiving them, without waiting for all drones to complete local training.

[0183] OCSF-AsyFL: Online Client Selection with Fairness Consideration for Asynchronous Federated Learning (OCSF-AsyFL) is an enhanced version of AsyFL that incorporates a fairness-aware online client selection algorithm. This framework takes into account the availability of drones and long-term fairness constraints, ensuring that all drones contribute to the global model regularly. OCSF-AsyFL: Online Client Selection with Fairness Consideration for Asynchronous Federated Learning (OCSF-AsyFL) is an enhanced version of AsyFL that incorporates a fairness-aware online client selection algorithm. This framework takes into account the availability of drones and long-term fairness constraints, ensuring that all drones contribute to the global model regularly.

[0184] To further verify the effectiveness of the proposed algorithm, this experiment conducts ablation experiments by integrating two drone client selection strategies into the Shapley value-based asynchronous federated learning framework (S-AsyFL). These experiments help isolate the contributions of different components within the proposed framework.

[0185] OCSF: An online client selection algorithm that considers fairness, as described in the OCSF-AsyFL framework. This algorithm ensures that all drones, regardless of their availability or computing power, have an equal opportunity to contribute to the global model over time.

[0186] UCBC: Upper Confidence Bound-based Client Selection (UCBC) is a widely used algorithm for the contextual multi-armed bandit problem. UCBC balances exploration and exploitation by selecting clients based on estimated utility and uncertainty, aiming to maximize the long-term reward during joint training.

[0187] The asynchronous update mechanism based on Shapley values is abbreviated as S-AsyFL (Shapley value-based asynchronous federated learning). The proposed accelerated asynchronous federated learning scheme is combined with S-AsyFL and abbreviated as CSS-AsyFL (Time-varying Contextual multi-armed bandits-based client selection for Shapley value-based asynchronous federated learning). In addition, to explore the impact of the exploration factor λ on the proposed CSS-AsyFL, three λ values are set: 0.01, 0.5, and 1, representing low, medium, and high levels of exploration preference, respectively.

[0188] Performance comparison in unstable environments. Figure 2 As shown in , this experiment uses the non-IID CIFAR-10 dataset to evaluate the convergence performance of four asynchronous federated learning schemes in an unstable environment. Setting (γ=1). Figure 2 In (a), it is observed that unstable air-to-ground communications caused by simulated mobility, interference, and channel fading significantly affect the convergence performance of the learning schemes. CSS-AsyFL achieves the best convergence performance, followed by SAsyFL, while the two baseline schemes (AsyFL and OCSFAsyFL) perform poorly. The superior performance of S-AsyFL is attributed to the Shapley value, which effectively measures the contribution of each UAV to the global model. CSS-AsyFL further enhances this by selecting UAVs that are better suited to handling communication instability, improving the overall convergence speed and stability of the global model. Figure 2 In (b), 2(c) and 2(d), the convergence performance is compared under dynamic UAV conditions, such as changing availability and computing power. Figure 2 (a) Consistency: CSS-AsyFL outperforms S-AsyFL, which in turn outperforms OCSF-AsyFL and AsyFL. This performance hierarchy is due to two reasons: the Shapley value helps amplify the contribution of high-quality local updates, and the time-varying context multi-armed bandit (MAB) model in CSSAsyFL accurately captures the dynamic state of each drone, selecting the drone with the highest quality updates in each round.

[0189] To further verify the effectiveness of S-AsyFL and CSSAsyFL, this experiment simulates an unstable asynchronous federated learning environment by combining all four unstable factors. Experiments are conducted on Fashion-MNIST and CIFAR-10, and the recognition accuracy and convergence speed of different schemes are compared under various non-IID conditions. Figure 3 As shown, S-AsyFL significantly improves the recognition accuracy compared with AsyFL and OCSF-AsyFL, especially in the case of highly non-independent and identically distributed (NIDs). By leveraging the Shapley value, S-AsyFL amplifies the contribution of high-quality local updates, thereby accelerating the convergence of the global model. CSS-AsyFL combines this with a drone selection strategy based on the time-varying context MAB model. Experimental results show that the moderate exploration factor (λ=0.05) in CSS-AsyFL enhances the selection of drones that provide higher quality updates, thereby improving the recognition accuracy and convergence speed of the model. This improvement is particularly evident in the case of highly non-independent and identically distributed settings, where CSS-AsyFL achieves the highest recognition accuracy with minimal fluctuations. On the more challenging CIFAR-10, as Figure 3As shown, all algorithms exhibit lower recognition accuracy and slower convergence compared to Fashion-MNIST. However, S-AsyFL consistently outperforms the baseline, highlighting the effectiveness of the Shapley value in amplifying high-quality updates. CSS-AsyFL further improves convergence and recognition accuracy by adaptively selecting drones with stable updates in unstable environments. Specifically, setting λ = 0.5 proves to be most effective, achieving recognition accuracy close to 0.7 in later communication rounds, significantly exceeding the baseline method.

[0190] In addition to CSS-AsyFL, this experiment also integrates two other client selection strategies, OCSF and UCBC, into the asynchronous update mechanism based on Shapley value (S-AsyFL) to perform ablation experiments. Figure 4 As shown in (a), all algorithms achieve high recognition accuracy on Fashion-MNIST. However, CSS-AsyFL with λ = 0.5 shows the most stable and highest recognition accuracy of 0.98 with the least fluctuation. In contrast, OCSF and UCBC show more significant fluctuations, reaching accuracies of around 0.92 and 0.90, respectively. This indicates that both OCSF and UCBC are less stable in unstable environments. Similarly, Figure 4 As shown in (b), while the recognition accuracy of all schemes on CIFAR-10 decreased, CSS-AsyFL maintained the highest and most stable recognition accuracy, peaking at approximately 0.64. In contrast, the recognition accuracy of OCSF-AsyFL and UCBC-AsyFL fluctuated significantly, reaching approximately 0.58 and 0.56, respectively. These results clearly demonstrate that CSS-AsyFL outperforms other methods, particularly on the noni.id dataset. This superiority stems from its ability to adaptively perceive fluctuations and accurately select drones with high-quality updates, resulting in better recognition accuracy and faster convergence. In contrast, UCBC, as a static MAB algorithm, fails to consider the impact of unstable communications and dynamic drone states on model quality. OCSF, as an online selection strategy, relies solely on the current drone state and lacks a comprehensive evaluation of historical drone performance.

[0191] Figure 5The number of communication rounds required for each scheme to achieve a target accuracy of 0.8 on the Fashion-MNIST dataset and 0.55 on the CIFAR-10 dataset was compared. CSS-AsyFL consistently required fewer rounds, significantly outperforming the baseline method, which required several times more rounds to achieve the same recognition accuracy. Notably, UCBC consistently required the most communication rounds to achieve object recognition accuracy. Real-world image recognition tasks often involve a large number of drones. By analyzing the convergence performance under varying numbers of drones, the scalability and practicality of the proposed schemes were effectively evaluated. Table 1 summarizes the number of communication rounds required for different schemes to achieve object recognition accuracy on CIFAR-10. Regardless of the number of drones, CSS-AsyFL (λ = 0.5) consistently achieved object recognition accuracy with the fewest communication rounds. This highlights that balancing exploration and exploitation is the most effective choice in dynamic and heterogeneous data environments. In summary, CSS-AsyFL is more suitable for real-world image recognition tasks and effectively addresses the challenges posed by unstable environments and data heterogeneity.

[0192] Table 1. Number of communication rounds required to achieve target accuracy on CIFAR-10 (γ = 0.1).

[0193]

[0194] In another embodiment of the present invention, a drone swarm real-time image recognition system based on asynchronous federated learning is provided, which can be used to implement the above-mentioned drone swarm real-time image recognition method based on asynchronous federated learning. Specifically, the system includes:

[0195] The local training module is used for each drone to download the latest global image recognition model from the central server, each drone performs local recognition model training, and uploads the trained local model to the central server;

[0196] The Shapley value calculation module is used by the central server to receive the trained local model uploaded by the drone, calculate the corresponding Shapley value, and introduce the staleness attenuation factor to adjust the weight to obtain the adjusted Shapley value;

[0197] A global update module for updating the global model through weighted aggregation based on the adjusted Shapley value;

[0198] The dynamic client selection module is used to build a time-varying context multi-armed bandit model after the global model is updated. The drone feature vector is input into the Gaussian random walk model to predict the client utility. The confidence upper bound algorithm is used to select the drone with the highest utility to participate in subsequent federated learning, realizing client optimization scheduling in a dynamic environment.

[0199] The module division in the embodiments of the present invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in various embodiments of the present invention may be integrated into a single processor, exist physically as separate modules, or two or more modules may be integrated into a single module. The integrated modules may be implemented in either hardware or software functional modules.

[0200] In another embodiment of the present invention, a computer device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the real-time image recognition method of a drone swarm based on asynchronous federated learning.

[0201] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory). The computer-readable storage medium is a memory device in a computer device, used to store programs and data. It is understood that the computer-readable storage medium herein may include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides storage space, which stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for being loaded and executed by a processor. These instructions may be one or more computer programs (including program code). It should be noted that the computer-readable storage medium herein may be a high-speed RAM memory or a non-volatile memory, such as at least one disk drive. The processor may load and execute the one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the method for real-time image recognition of drone swarms based on asynchronous federated learning in the above-mentioned embodiment.

[0202] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0203] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0204] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0205] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0206] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A real-time image recognition method for drone swarms based on asynchronous federated learning, characterized by: include: Each drone downloads the latest global image recognition model from the central server, trains a local recognition model, and uploads the trained local model to the central server; The central server receives the trained local model uploaded by the drone, calculates the corresponding Shapley value, and introduces the staleness attenuation factor to adjust the weight to obtain the adjusted Shapley value; The global model is updated by weighted aggregation based on the adjusted Shapley values; After the global model is updated, a time-varying context multi-armed bandit model is constructed, and the drone feature vector is input into the Gaussian random walk model to predict the client utility. The confidence upper bound algorithm is used to select the drone with the highest utility to participate in subsequent federated learning, realizing client optimization scheduling in a dynamic environment.

2. The method for real-time image recognition of drone swarms based on asynchronous federated learning according to claim 1 is characterized in that: Each drone downloads the latest global image recognition model from the central server, including: Drone n first downloads the latest version of the image recognition model ω(t) from the central server. The trained model extracts features from the image through convolution, activation, and pooling operations, and performs image recognition based on these features.

3. The method for real-time image recognition of drone swarms based on asynchronous federated learning according to claim 2 is characterized in that: Each drone performs local recognition model training and uploads the trained local model to the central server, including: The drone uses its local data m Perform h rounds of local training and get The local training of each round is expressed as follows: Among them, λ>0 represents the learning rate, is the loss function on the dataset D n After completing local training, drone n updates its local model Transmit to the central server and convert τ max Set to the maximum tolerated outdated global round, the drone is in the maximum tolerated round τ max Otherwise, the server will reject the connection and terminate the upload.

4. The method for real-time image recognition of drone swarms based on asynchronous federated learning according to claim 1, characterized in that: The central server receives the trained local model uploaded by the drone and calculates the corresponding Shapley value, including: Under asynchronous federated learning, each drone contributes its local model update to form a cooperative game and construct a cooperative game. in represents the set of participating drones, Is to reflect the model in the collection only The value function of the accuracy when the drone is involved, the calculation process is: where Q(.) returns the utility score of the model, is a collection The federated learning model obtained after t rounds of asynchronous updates of the drones in the game; the cooperative game is decomposed into multiple small cooperative games, each of which evaluates the contribution of the drones during each round of asynchronous aggregation; each game in represents the performance improvement of the model on the test dataset and is defined as: The Shapley value measures the impact of a user joining or leaving the game and calculates the contribution by averaging the marginal contributions of all possible coalitions, with n for different subsets The marginal contribution is expressed as: Take all subsets The average marginal contribution of , we get the following Shapley value:

5. The method for real-time image recognition of drone swarms based on asynchronous federated learning according to claim 1, characterized in that: The introduction of the staleness attenuation factor to adjust the weight to obtain the adjusted Shapley value includes: The staleness decay factor is introduced when UAV n is ′ Upload its latest model update ω during the next global round n (t), ω n The staleness δ of (t) n Defined as |t ′ -t|; As the staleness increases, the staleness attenuation factor decreases exponentially. Use the staleness attenuation factor This factor is applied to the Shapley value, and the adjusted Shapley value is:

6. The method for real-time image recognition of drone swarms based on asynchronous federated learning according to claim 1, characterized in that: The updating of the global model by weighted aggregation based on the adjusted Shapley value includes: The first step is to determine the updated global model ω(t ′ ) is expressed as θ(t ′ )=θ h (t ′ ),...,θ n (t ′ ),...,θ N (t ′ ); the second is to use θ(t ′ ) as weights to update the global model: This process is iterated until the global loss function converges to ω * ; The global loss function is expressed as follows:

7. The method for real-time image recognition of drone swarms based on asynchronous federated learning according to claim 1, characterized in that: After the global model is updated, a time-varying context multi-armed bandit model is constructed. The drone feature vector is input into the Gaussian random walk model to predict the client utility. The confidence upper bound algorithm is used to select the drone with the highest utility to participate in subsequent federated learning, realizing client optimization scheduling in a dynamic environment, including: In the Shapley-based asynchronous federated learning framework, client utility consists of two components: statistical utility quantifying the model contribution of drone data and system utility reflecting the improvement in convergence speed; Statistical utility: System Utility: in Count the disconnections that occurred due to timeout before round t; Total utility: The client selection problem is reformulated as a contextual MAB problem, where each drone n is regarded as an arm with client utility as the reward; the utility parameter is defined as the feature vector x n (t): This feature vector changes over time and its value remains unknown before the end of round t; the central server maintains a time-varying coefficient vector ω n , which represents the contribution of each feature to utility prediction; the expected utility of drone n is modeled as: where ∈ n The variance is The zero-mean Gaussian noise of is: The central server uses the upper confidence limit (UCB) algorithm to calculate the score of each drone and selects the drone with the highest UCB score for federated learning: A Gaussian random walk model with Markov process is used to capture ω n Drift; in Represents the static component, capturing the average contribution of each context feature, the static component Modeling using a conjugate prior distribution: in is the prior mean, which represents the belief about the average contribution of drone context features to utility; is the covariance matrix, introducing a Gaussian random walk modeled as a Markov process: where η n (t) represents the drift value, which is affected by the previous drift η n (t-1) and Gaussian noise vector the impact of; The scaling factor in determines the importance of each element in the drift vector Δ(t), which is expressed as: in is the average of the scaling factors, is the covariance matrix; the parameters drawn from an inverse gamma distribution; The expected client utility of drone n in round t is given by: where x n (t) is the feature vector of drone n in round t; In round t, the central server collects feature vector x from drone n n (t); Model the parameters using a Gaussian distribution: where the mean and covariance are updated as: Where G(.) is the Kalman gain function, and in the t-1 round, the parameter random variable The sufficient statistics for make and infer and This is equivalent to inferring the distribution of ν n ; Assume that Σ′, μ′, α′, and β′ are sufficient statistics for round t and are updated based on new observations at time t: Round t, and ν n The sampling process is summarized as follows: Based on sufficient statistical data from the previous t-1 rounds, w n (t-1) Sample from the following: in: The estimated utility is expressed as: In this expression, the first term represents the expected utility, while the second term captures the uncertainty in the estimate and is weighted by the exploration parameter λ.

8. A real-time image recognition system for drone swarms based on asynchronous federated learning, characterized by: include: The local training module is used for each drone to download the latest global image recognition model from the central server, each drone performs local recognition model training, and uploads the trained local model to the central server; The Shapley value calculation module is used by the central server to receive the trained local model uploaded by the drone, calculate the corresponding Shapley value, and introduce the staleness attenuation factor to adjust the weight to obtain the adjusted Shapley value; A global update module for updating the global model through weighted aggregation based on the adjusted Shapley value; The dynamic client selection module is used to build a time-varying context multi-armed bandit model after the global model is updated. The drone feature vector is input into the Gaussian random walk model to predict the client utility. The confidence upper bound algorithm is used to select the drone with the highest utility to participate in subsequent federated learning, realizing client optimization scheduling in a dynamic environment.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method for real-time image recognition of a swarm of drones based on asynchronous federated learning are implemented as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for real-time image recognition of a swarm of drones based on asynchronous federated learning are implemented.

Images