An online federated learning system and method for UAV visual target detection based on collaborative reasoning

Abstract

The present invention discloses an online federated learning system and method for unmanned aerial vehicle (UAV) visual target detection based on collaborative reasoning. The system comprises: a UAV control module transmits control instructions to the UAV optical pod to control the UAV optical pod to collect images from different perspectives; a UAV collaborative reasoning module generates online samples through multi-machine collaborative reasoning; a UAV client module performs local model training; a UAV server module obtains a global model through model aggregation and performs target detection on image data collected by the UAV optical pod; and a ground station transmits control instructions to the UAV control module to control the UAV's flight parameters and monitors and demonstrates target detection results in real time. This system achieves real-time expansion of the target detection model training set, enabling the UAV to perform more accurate target detection in a real-world environment while effectively avoiding the loss of the collected images to be annotated that are transmitted back to the ground station.

Description

Technical Field

[0001] The present invention belongs to the technical field of drone target detection, and specifically relates to an online federated learning system and method for drone visual target detection based on collaborative reasoning. Background Art

[0002] Federated learning is a new type of distributed machine learning, first proposed by Google in 2016. Federated learning offloads data storage and model training tasks from traditional machine learning to local users. A centralized server aggregates models by receiving only model updates, not data updates, enabling multi-user data sharing while ensuring user privacy. It has been widely used in healthcare, finance, and the military.

[0003] Applying federated learning to target detection tasks within drone swarms can effectively leverage the performance advantages of drones. Compared to traditional base stations, which serve as information transmission media, drones offer greater flexibility, responsiveness, and a three-dimensional perspective that is independent of location. By using drone nodes to capture target images at varying altitudes and angles, they provide a large number of samples for training target recognition models. Federated learning can collaboratively leverage sample data from multiple drone nodes to rapidly train a highly accurate and versatile target detection model.

[0004] Extensive research and experimentation has been conducted on the optimization of federated learning algorithms and the application of target detection. However, there is currently little research on federated learning systems for drone target detection. The application of drone target detection still faces the following challenges:

[0005] 1. Difficulty adapting to dynamic and changing environments: Due to the deviation between drone training samples and test samples in real-world environments, that is, the migration error between the source domain and the target domain, the object detection model that performs well during training is difficult to apply in real-world application environments. For example, in foggy weather, the blurred vision makes it difficult for drones to detect targets.

[0006] 2. Limited training samples: For some detection targets, due to resource constraints and other issues, there are not enough samples to support the federated learning of the model, such as a new type of combat tank appearing on the battlefield. The only option is to rely on drones to re-collect images during the target detection process;

[0007] 3. Sample production is time-consuming: The drone transmits the real-time collected images back to the ground station, and the process of manual image annotation and sample production requires a lot of communication time and human resources.

[0008] The root cause of these issues is that drones struggle to acquire high-quality target samples in real time and adjust the target detection model online during target detection. Therefore, for drone target detection, online federated learning based on collaborative inference data is required to support real-time model updates and optimization. Summary of the Invention

[0009] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned existing technologies and provide an online federated learning system and method for UAV visual target detection based on collaborative reasoning. By installing a Jetson NX embedded computer on the UAV to add edge computing nodes, the images collected by the UAV in real time are used to produce online learning samples with high accuracy through multi-machine collaborative reasoning using a high-precision complex model (Fast-RCNN) that is difficult to execute on a single machine, thereby providing real-world sample data for model training in real time. Moreover, based on the original model, a lightweight small model (Yolov4-Tiny) suitable for target detection in the current environment is trained through online federated learning. This realizes the real-time expansion of the target detection model training set, enabling the UAV to perform more accurate target detection in the actual environment. At the same time, the UAV's autonomous generation of new samples also effectively avoids the loss of the UAV transmitting the collected images to be labeled back to the ground station.

[0010] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0011] An online federated learning system for UAV visual target detection based on collaborative reasoning, comprising a ground station, a UAV optical pod, a UAV control module, a UAV collaborative reasoning module for online federated learning, a UAV client module, and a UAV server module;

[0012] The drone control module is used to transmit control instructions to the drone optical pod to control the acquisition frame rate, shooting angle, and zoom factor of the drone optical pod, thereby acquiring images from different perspectives;

[0013] The UAV collaborative reasoning module is used to add edge computing nodes by equipping the UAV with a Jetson NX embedded computer, and to generate online samples from real-time collected images through multi-machine collaborative reasoning using a high-precision complex model that is difficult to execute on a single machine;

[0014] The drone client module is used to receive online samples produced by the drone collaborative reasoning module and perform local model training;

[0015] The drone server module is used to receive the local model of the drone client module, obtain a global model through model aggregation, and perform target detection on the image data collected by the drone optical pod;

[0016] The ground station communicates with the drone control module and the drone server through a wireless network, and is used to transmit control instructions to the drone control module, control the flight trajectory, flight speed, and hovering time of the drone, and monitor and demonstrate the target detection results obtained by the drone server module in real time.

[0017] To optimize the above technical solutions, specific measures taken also include:

[0018] The above-mentioned drone collaborative reasoning module deploys a collaborative reasoning program. By splitting the high-precision complex model that is difficult for a single drone to execute into multiple sub-models and assigning them to each drone node, each drone collaborative reasoning module will reason on the sub-model in sequence when performing image reasoning, and finally complete the sample production.

[0019] The above-mentioned drone client module is provided with a sample preprocessing module for compressing, quantizing, and storing training samples before training.

[0020] The above-mentioned drone server module is provided with a model weighting module for weighted aggregation of local models according to the number of samples in the training set of each drone client module.

[0021] The aforementioned UAV collaborative reasoning module uses the Fast-RCNN model for collaborative reasoning;

[0022] The UAV client module and the UAV server module use the Yolov5-Tiny model for target detection.

[0023] The above-mentioned drone client module is provided with a label conversion module, which is used to convert the label data of the received collaborative reasoning samples from the upper left corner X, upper left corner Y, lower right corner X, and lower right corner Y of the Fast-RCNN model to the center point X ratio, center point Y ratio, length ratio, and width ratio of the Yolov4-Tiny model.

[0024] An online federated learning method for UAV visual target detection based on collaborative reasoning, the method comprising the following specific steps:

[0025] S1: The ground station transmits control parameters to the drone control module to guide the drone's flight, and the drone control module transmits the relevant parameters to the drone's optical pod for real-time image acquisition;

[0026] S2: The drone optical pod transmits the collected real-time images to the drone collaborative reasoning module, which then generates online samples.

[0027] S3: The drone client module receives the online samples produced by the drone collaborative reasoning module and adds them to the local training set for local model training;

[0028] S4: The drone server module receives the local models uploaded by multiple drone client modules, obtains the global model through model aggregation, and transmits the global model to the drone client module to start a new round of local model training;

[0029] S5: Repeat steps S3 and S4 to implement online federated learning until the model converges;

[0030] S6: The drone optical pod transmits the collected real-time video to the drone server module. The drone server module uses the global model obtained by online federated learning as the target detection model for target detection.

[0031] S7: The UAV server module transmits the target detection results to the ground station, which then displays the results.

[0032] The present invention has the following beneficial effects:

[0033] The present invention collects images in real time through a drone pod; performs collaborative reasoning on the collected images through a drone cluster to produce online learning samples; the client node trains a local model in combination with the online learning samples; the server node aggregates the local models uploaded by each client node to produce a global model; the server node uses the global model to perform target detection and returns the detection results to the ground station. The main difference between the present invention and existing online federated learning is that it can use drone clusters to collaboratively produce online learning samples and perform online adjustments to the target detection model based on the real-time collected images. The present invention performs online federated learning by producing high-precision samples in real time, which can effectively improve the accuracy of target detection in dynamic environments. In addition, the method of producing online learning samples through collaborative reasoning of drone clusters effectively avoids the time of sending the collected images back to the ground station for annotation.

[0034] Federated learning enables data sharing between drone clusters, improving the versatility of target detection models. Compared to traditional federated learning systems, online federated learning based on collaborative reasoning data can add new samples in real time during target detection and make timely adjustments to target detection models based on actual environmental requirements, improving the adaptability of drones to multi-dynamic environments. The accuracy and convergence of drone target detection models are also improved. The creation of online samples through multi-machine collaborative reasoning effectively reduces the communication time required for drones to transmit image data and the labor consumption required to manually annotate images and create samples.

[0035] This paper uses different target detection models for collaborative reasoning and federated learning. Collaborative reasoning uses the high-precision, complex Fast-RCNN model, which requires the coordinated computation of multiple nodes to ensure high accuracy. Federated learning uses the lightweight, single-step Yolov5-Tiny model, which can quickly complete target detection tasks. This approach can leverage drone swarms to collaboratively create online learning samples, allowing for online adjustments to the target detection model based on real-time imagery.

[0036] The UAV collaborative reasoning module of the present invention deploys a collaborative reasoning program. By splitting the high-precision complex model that is difficult for a single UAV to execute into multiple sub-models and assigning them to each UAV node, each UAV collaborative reasoning module reasons on the sub-model in sequence when performing image reasoning, and finally completes sample production.

[0037] The annotation conversion module is used to convert the annotation data of the received collaborative reasoning samples from the upper left corner X, upper left corner Y, lower right corner X, and lower right corner Y of the Fast-RCNN model to the center point X ratio, center point Y ratio, length ratio, and width ratio of the Yolov4-Tiny model;

[0038] The drone client module is equipped with a sample preprocessing module, which is used to compress, quantize, and store training samples before training, eliminating the time loss of loading samples during local training and improving GPU utilization.

[0039] The drone server module is equipped with a model weighting module, which is used to perform weighted aggregation on the local model according to the number of training samples of the local node, effectively alleviating the negative impact of data heterogeneity between nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a structural diagram of the online federated learning system for UAV visual target detection based on collaborative reasoning of the present invention.

[0041] Figure 2 It is a curve diagram of the change of the target detection model loss value during the model training process of an embodiment of the present invention.

[0042] Figure 3 2 is a graph showing the mAP change of the target detection model during the model training process of an embodiment of the present invention;

[0043] The figures are marked as follows: 1-ground station, 2-UAV optical pod, 3-UAV control module, 4-UAV collaborative reasoning module, 5-UAV client module, 6-UAV server module. DETAILED DESCRIPTION

[0044] The present invention will now be described in further detail with reference to the accompanying drawings.

[0045] It should be noted that the terms such as "upper", "lower", "left", "right", "front", "back", etc. cited in the invention are only for the convenience of description and are not used to limit the scope of implementation of the present invention. Changes or adjustments to their relative relationships should be regarded as the scope of implementation of the present invention without substantially changing the technical content.

[0046] The present invention carries a Jetson NX embedded computer on a multi-rotor drone platform as an edge computing node, and the drone cluster jointly builds an online federated learning system. Specifically, Figure 1 As shown in the figure, the online federated learning system for UAV visual target detection based on collaborative reasoning of the present invention includes a UAV collaborative reasoning module 4 for online federated learning, a UAV client module 5, and a UAV server module 6. To further achieve real-time target detection of UAVs in the air, the invented system also includes a ground station 1, a UAV control module 3, and a UAV optical pod 2.

[0047] The UAV collaborative reasoning module 4 deploys a collaborative reasoning system. By splitting the high-precision complex model that is difficult to execute on a single machine into multiple sub-models and assigning them to each node, each collaborative reasoning module 4 uses this to reason on the sub-model in sequence when performing image reasoning, and finally completes the sample production.

[0048] The drone collaborative reasoning module 4, drone client module 5, and drone server module 6 form an online federated learning system. The drone client module 5 receives online samples generated by the drone collaborative reasoning module and performs local training. The drone server module 6 receives the local model transmitted by the drone client module and aggregates the model to obtain a global model.

[0049] The ground station 1 and the UAV control module 3 are connected via a wireless network to form a communication system. The ground station 1 transmits control instructions to the UAV control module 3 to control the UAV's flight trajectory, flight speed, hovering time and other parameters;

[0050] The drone control module 3 and the drone optical pod 2 form an image acquisition system. The drone control module 3 transmits control instructions to the drone optical pod 2 to control the optical pod's acquisition frame rate, shooting angle, zoom factor and other control parameters, thereby acquiring images from different perspectives.

[0051] The drone optical pod 2 and the drone server module 6 constitute a target detection system. The drone optical pod 2 transmits the acquired data to the drone server module 6, which processes the data using the target detection model to achieve target detection.

[0052] The ground station 1 and the drone server 6 are connected via a wireless network to form a detection result demonstration system. The drone server module 6 transmits the target detection results back to the ground station 1, and the ground station 1 monitors and demonstrates the detection results in real time.

[0053] The object detection process of the online federated learning system based on collaborative reasoning includes the following specific steps:

[0054] S1, the ground station 1 transmits control parameters to the drone control module 3 to guide the drone flight, and the drone control module 3 transmits the relevant parameters to the drone optical pod 2 to realize image acquisition;

[0055] S2, the UAV optical pod 2 transmits the collected real-time image to the UAV collaborative reasoning module 4, which produces an online sample;

[0056] S3, the drone client module 5 receives the online sample transmission produced by the drone collaborative reasoning module 4 and adds it to the local training set for local model training;

[0057] S4: The drone server module 6 receives the local models uploaded by multiple drone client modules 5, obtains a global model through model aggregation, and transmits the global model to the drone client module 5 to start a new round of local model training;

[0058] S5, repeat steps S3 and S4 to implement online federated learning until the model converges;

[0059] S6, the drone optical pod 2 transmits the collected real-time video to the drone server module 6, and the drone server module 6 performs target detection with the help of the global model obtained by online federated learning;

[0060] S7, the UAV server module 6 transmits the target detection results to the ground station 1, and the ground station 1 displays the results.

[0061] This embodiment realizes sample data sharing between drones through online federated learning, and uses a large amount of sample data to quickly train a target detection model with high accuracy and strong convergence. Its main implementation method is shown in steps S3, S4, and S5.

[0062] In step S3, the UAV client module 5 performs local model training on the local training set, which includes multiple clients C K ={1,2,…,K}. Each client k∈C K All equipped with local data sets in is the i-th data of customer sample K, It is a collection of corresponding annotations.

[0063] In addition, for local model training, this system defines a vector w to represent the parameters related to the federated learning model and defines a loss function To quantify the data sample With its mark The error between .

[0064] Then the value of the local loss function of client k is defined as

[0065] Therefore, the global loss function is expressed as in Indicates the total amount of local data of all clients.

[0066] Therefore, the main goal of local training is to find an optimal global model w to minimize the global loss function F(θ).

[0067] For example, a sample preprocessing module can be added on the basis of the drone client module 5. By compressing, quantizing and other preprocessing the training samples before training, and caching the processed samples in the local memory, they can be directly called during the local training process, effectively eliminating the time loss of loading samples during the local training process and improving the utilization rate of the GPU.

[0068] In step S4, the UAV server module 6 performs a weighted aggregation operation on the local model, where w t,k represents the kth client uploading the updated local model in round t, w t Represents the global model that the server transmits to the client in round t. The gradient calculated by client k during local training Update the local model by gradient ​​denoted as w t,k =w t-1 -μg k , where μ is the learning rate of local training; the aggregated model in the server aggregation process is expressed as

[0069] The online federated learning system repeats steps S3 and S4 according to this process to ensure that the global model of the server and the local model of the client are up to date until the global loss function F(θ) reaches the threshold to obtain the final model.

[0070] For example, a model weighting module can be added to the drone server module 6. Based on the number of samples in the training set of each drone client module, the corresponding weight is added during the model aggregation process of the drone server module to prevent data offset in the global model of the drone client module 5 with too many or too few samples, effectively alleviating the negative impact of data heterogeneity.

[0071] This embodiment uses collaborative reasoning to accurately annotate images captured by a drone's optical gondola. To ensure inference accuracy, the Jetson NX embedded computer, used as an edge computing node on the drone, offers advantages such as portability and compact size, but its computing resources are relatively limited. This embodiment employs multi-machine collaborative reasoning to generate online samples. The drone collaborative reasoning module splits complex, high-precision models, which are difficult to execute on a single machine, into multiple sub-models based on a scheduling scheme. Starting from the initial drone node, after the current node completes inference on a sub-model, the inference results are passed to subsequent nodes according to the scheduling scheme. Subsequent nodes continue the inference process until the last node completes the sample annotation. After completing inference on a sub-model, even if the inference task is not fully completed, each node will continue to process the next task without interruption. This is equivalent to performing inference on multiple tasks simultaneously, effectively improving execution efficiency while ensuring inference accuracy.

[0072] To ensure high precision of collaborative reasoning, this embodiment uses Fast-RCNN as the collaborative reasoning model. In addition, in order to meet the requirements for detection speed during drone target detection, it is also necessary to ensure the accuracy of target detection. This embodiment uses Yolov4-Tiny as the target detection model. When performing target detection, this model directly completes the tasks of feature extraction, candidate box regression and classification in a branchless convolutional network. Compared with two-step target detection algorithms such as Fast-RCNN, which skip the extraction of candidate boxes, the single-step network structure makes the target detection speed of the Yolov4-Tiny model about 10 times higher than that of the two-step target detection algorithm.

[0073] This embodiment uses the Visdrone2021 drone vision dataset as the original training sample. The dataset was collected and compiled by the AISKYEYE team. The dataset contains more than 10,000 drone vision image samples, which were collected and photographed by drone cameras of different models in different scenes and various weather and lighting conditions. It covers a wide range of aspects, including location, environment, objects (cars, pedestrians, trucks, etc.) and density data information.

[0074] The real-time collected images are used as online learning samples, and online federated learning is performed in synchronous and asynchronous ways, namely

[0075] Synchronization: Online learning samples are prepared before federated learning begins, so that the original training samples and the newly added online learning samples can be trained synchronously.

[0076] Asynchronous: After the target detection model is trained using the original training samples, online learning samples are added to further optimize the model based on the original target detection model.

[0077] The final trained model performance is as follows Figure 2 and 3 shown. Figure 2 and Figure 3 The following are the model loss value change curve and mAP change curve after 20 rounds of federated learning iterations, where the horizontal axis is the number of iterations and the vertical axis is the loss value and mAP. Figure 2 As shown in the figure, as the number of iterations increases, the loss value tends to be stable, smooth, and decreases slightly. The synchronous online federated learning can maintain a relatively high convergence speed at the beginning of training, while the convergence speed of asynchronous federated learning is significantly improved after adding online learning samples. The final loss value converges to a value that is basically consistent with the synchronous online federated learning. Compared with traditional federated learning, the training method of this embodiment reduces the loss value by about 30%.

[0078] like Figure 3 As shown in the figure, mAP reflects the recognition accuracy of the object detection model. With the increase in the number of iterations, the mAP of synchronous online federated learning increases rapidly, while the mAP growth rate of asynchronous online federated learning also increases slightly after adding online learning samples. Due to the limited number of iterations, the mAP of the object detection model trained is slightly lower than that of synchronous online federated learning. However, due to traditional online federated learning, compared with traditional federated learning, synchronous and asynchronous online federated learning in this embodiment improve mAP by 34% and 18%, respectively.

[0079] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions based on the principles of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. An online federated learning system for UAV visual target detection based on collaborative reasoning, comprising a ground station (1), a UAV optical pod (2), and a UAV control module (3), characterized in that: It also includes a drone collaborative reasoning module (4), a drone client module (5) and a drone server module (6) for online federated learning; The drone control module (3) is used to transmit control instructions to the drone optical pod (2), controlling the acquisition frame rate, shooting angle, and zoom factor of the drone optical pod (2), thereby acquiring images from different perspectives; The UAV collaborative reasoning module (4) is used to add edge computing nodes by equipping the UAV with a Jetson NX embedded computer, and to generate online samples from real-time collected images by using a high-precision complex model that is difficult to execute on a single machine through multi-machine collaborative reasoning; The drone client module (5) is used to receive online samples produced by the drone collaborative reasoning module (4) and perform local model training; The drone server module (6) is used to receive the local model of the drone client module (5), obtain a global model through model aggregation, and perform target detection on image data collected by the drone optical pod (2); The ground station (1) communicates with the drone control module (3) and the drone server module (6) via a wireless network, and is used to transmit control instructions to the drone control module (3), control the flight trajectory, flight speed, and hovering time of the drone, and monitor and demonstrate the target detection results obtained by the drone server module (6) in real time; The UAV collaborative reasoning module (4) deploys a collaborative reasoning program, which splits a high-precision complex model that is difficult for a single UAV to execute into multiple sub-models and assigns them to each UAV node. When performing image reasoning, each UAV collaborative reasoning module (4) sequentially reasons on the sub-models, and finally completes sample production; The UAV collaborative reasoning module (4) uses the Fast-RCNN model to perform collaborative reasoning; The drone client module (5) and the drone server module (6) use the Yolov5-Tiny model to perform target detection. The drone server module (6) is provided with a model weighting module for weighted aggregation of local models according to the number of samples in the training set of each drone client module; the drone client module (5) is provided with a label conversion module for converting the label data of the received collaborative reasoning sample from the upper left corner X, upper left corner Y, lower right corner X, and lower right corner Y of the Fast-RCNN model to the center point X ratio, center point Y ratio, length ratio, and width ratio of the Yolov4-Tiny model; An online federated learning method for UAV visual target detection based on collaborative reasoning implemented by the system includes: 1): The ground station (1) transmits control parameters to the drone control module (3) to guide the drone to fly, and the drone control module (3) transmits the relevant parameters to the drone optical pod (2) for real-time image acquisition; 2): The UAV optical pod (2) transmits the collected real-time image to the UAV collaborative reasoning module (4), and the UAV collaborative reasoning module (4) produces online samples; 3): The UAV client module (5) receives the online samples produced by the UAV collaborative reasoning module (4) and adds them to the local training set for local model training; 3) The UAV client module (5) performs local model training on the local training set, which includes multiple clients C K = {1, 2, ..., K}, each client k∈C K All equipped with local data sets in is the i-th data of customer sample K, is the corresponding annotation collection; in addition, for local model training, this system defines a vector w to represent the parameters related to the federated learning model and defines a loss function To quantify the data sample With its mark The error between them; then the value of the local loss function of client k is defined as The global loss function is expressed as in Represents the total amount of local data of all clients; the main goal of local training is to find an optimal global model w to minimize the global loss function F(θ); 4): The drone server module (6) receives the local models uploaded by multiple drone client modules (5), obtains the global model through model aggregation, and transmits the global model to the drone client module (5) to start a new round of local model training; 4) The UAV server module (6) performs a weighted aggregation operation on the local model, where w t,k represents the kth client uploading the updated local model in round t, w t Represents the global model that the server transmits to the client in round t, and the gradient calculated by client k during local training Update the local model by gradient ​​denoted as w t,k =w t-1 -μg k , where μ is the learning rate of local training; the aggregated model in the server aggregation process is expressed as 5): Repeat 3) and 4) to implement online federated learning until the model converges; 6): The drone optical pod (2) transmits the collected real-time video to the drone server module (6), and the drone server module (6) uses the global model obtained by online federated learning as a target detection model to perform target detection; 7): The UAV server module (6) transmits the target detection results to the ground station (1), and the ground station (1) displays the results.

2. The online federated learning system for UAV visual target detection based on collaborative reasoning according to claim 1 is characterized in that: The UAV client module (5) is provided with a sample pre-processing module for compressing, quantizing, and storing training samples before training.

Images