Unmanned aerial vehicle aerial image processing method, system and device and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and particularly provides an unmanned aerial vehicle aerial image processing method, system and device and a storage medium, and the method comprises the steps: obtaining an aerial image, generating a label for the aerial image, and storing the aerial image with the label into a sample set; generating a verification label for the aerial image in the sample set by using a generator model, generating a loss value of the verification label by using a discriminator model, and correcting an original label of the aerial image based on the verification label and the loss value; based on the corrected aerial image tags, counting the number of various tags, and marking the tag category of which the number does not reach a preset number threshold value as a to-be-expanded tag category; and generating an image copy for the aerial image of which the tag category is to be expanded, setting the tag of the original aerial image as the tag of the corresponding image copy, and converting an image style for the image copy. According to the invention, optimization of unmanned aerial vehicle data quality is realized.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a method, system, device and storage medium for processing aerial images taken by unmanned aerial vehicles. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology, drone technology has been widely used and developed in many fields such as agricultural monitoring, environmental investigation, logistics distribution, disaster relief, film and television shooting, etc. With its unique flexibility and versatility, drones can efficiently collect a large amount of image and video data by carrying various advanced sensors and high-resolution cameras, providing valuable resources for subsequent data analysis and decision support.

[0003] However, in practical applications, data collected by drones often encounter a series of quality issues, which directly affect the subsequent data processing and the performance of machine learning models. One of the most prominent problems is data noise, especially label noise - that is, errors or inconsistencies in the process of annotating data. In addition to label noise, data collected by drones may also face the problem of class imbalance. This means that the number of data samples in some categories is much larger than that in other categories. This imbalance will cause machine learning models to favor the majority category and ignore the minority category, which in turn affects the overall generalization ability of the model. In addition, since drones usually work in different geographical locations and environments, the data they collect may have significant differences in statistical characteristics. This heterogeneity of data makes it difficult for the model to capture the common features of all categories during training, thus affecting the generalization ability and robustness of the model. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method, system, device and storage medium for processing aerial images of unmanned aerial vehicles to solve the above-mentioned technical problems.

[0005] In a first aspect, the present invention provides a method for processing aerial images taken by a drone, comprising: Acquire an aerial image, generate a label for the aerial image, and save the labeled aerial image into a sample set; Generate verification labels for aerial images in the sample set using the generator model, generate loss values ​​for the verification labels using the discriminator model, and correct the original labels of the aerial images based on the verification labels and the loss values; Based on the corrected aerial image labels, the number of various labels is counted, and the label categories whose number does not reach the preset number threshold are marked as the label categories to be expanded; An image copy is generated for the aerial image of the to-be-expanded label category, and the label of the original aerial image is set as the label of the corresponding image copy, so as to convert the image style of the image copy.

[0006] In an optional implementation, acquiring an aerial image, generating a label for the aerial image, and then saving the labeled aerial image to a sample set includes: Monitoring the update status of the storage file of the aerial image, and extracting the updated aerial image from the storage file based on the update status; Use annotation tools to generate labels for the extracted aerial images and save the labeled aerial images to the sample collection.

[0007] In an optional implementation, a generator model is used to generate verification labels for aerial images in a sample set, and a discriminator model is used to generate loss values ​​for the verification labels, and the original labels of the aerial images are corrected based on the verification labels and the loss values, including: Generating verification labels for the aerial images using a generator model, wherein the generator model includes a graph neural network and a classifier; The discriminator model generates a loss value of the verification label relative to the label of the aerial image; The label loss values ​​of all aerial images are passed as input features to the Gaussian mixture model to identify the potential distribution model of the loss value, each distribution represents a data category; Soft-classify the labels according to the loss values ​​and generate soft-classified labels; Calculate the average value of each Gaussian distribution and select the distribution with the highest average value to mark the noise data, so as to establish a noise sample set and determine the proportion of noise samples in the entire data set; The aerial images that need to be relabeled are determined based on the preset trust threshold Trust and cleaning threshold c, and the labels of the aerial images that need to be relabeled are updated to the corresponding verification labels.

[0008] In an optional embodiment, the discriminator model generates a loss value of the verification label relative to the label of the aerial image, including: Convert labels into semantic vectors; The loss value of the verification label relative to the label of the aerial image is calculated based on a preset loss function, and the loss function is: ; in, represents all data instances of the kth drone, And represents the drone group; ; Among them, the vectorA=(A 1 ,A 2 ,…，A n ） , is the semantic vector of the verification label of the i-th aerial image; vector B= (B1, B2, …, Bn) , is the semantic vector of the label of the i-th aerial image.

[0009] In an optional implementation, generating an image copy for an aerial image of a to-be-expanded label category, setting the label of the original aerial image as the label of the corresponding image copy, and converting the image style of the image copy includes: generating a plurality of image copies for the aerial image; Image style conversion is performed on the multiple image copies respectively, and the image style conversion includes rotation, scaling, blurring, brightness adjustment, and fusion with template images, and the template images include rain template images and snow template images.

[0010] In an optional embodiment, the method further comprises: Deploy the model to be trained on multiple drones; Under the federated learning framework, the models to be trained are deployed on multiple drones based on pre-prepared data sets.

[0011] In an optional implementation, training the to-be-trained model deployment of multiple drones based on a pre-prepared data set under a federated learning framework includes: The server initializes the model parameters according to the type of task to be solved, and distributes the initialized model parameters to each drone; The server assigns an ID or identifier to the drones participating in the training; The drone uses the received model parameters to perform model training on the local data set. After the training is completed, the drone obtains the updated model parameters, packages these updated parameters and sends them back to the server. The server receives updated parameters from all participating drones and executes an aggregation algorithm to update the global model parameters in an average or weighted average manner; The aggregated global model parameters are sent to the drone again to start a new round of local training, with multiple iterations until the set convergence conditions are met or the number of iterations reaches the predetermined number of iterations.

[0012] In a second aspect, the present invention provides a drone aerial image processing system, comprising: An image acquisition module is used to acquire aerial images, generate labels for the aerial images, and save the labeled aerial images into a sample set; A label correction module is used to generate verification labels for aerial images in the sample set using the generator model, and to generate loss values ​​of the verification labels using the discriminator model, and to correct the original labels of the aerial images based on the verification labels and the loss values; A quantity statistics module is used to count the number of various labels based on the corrected aerial image labels, and mark the label categories whose number does not reach a preset quantity threshold as the label categories to be expanded; The sample expansion module is used to generate image copies for the aerial images of the to-be-expanded label categories, and to set the labels of the original aerial images as the labels of the corresponding image copies, so as to convert the image styles of the image copies.

[0013] In a third aspect, a device is provided, including: A memory, used for storing a drone aerial image processing program; The processor is used to implement the steps of the drone aerial image processing method provided in the first aspect when executing the drone aerial image processing program.

[0014] In a fourth aspect, a computer-readable storage medium is provided, on which a drone aerial image processing program is stored. When the drone aerial image processing program is executed by a processor, the steps of the drone aerial image processing method provided in the first aspect are implemented.

[0015] The beneficial effect of the present invention lies in that the drone aerial image processing method, system, device and storage medium provided by the present invention optimize the drone data quality, improve the model training efficiency and performance, and enhance the robustness and security of the system through a series of steps such as data collection and annotation, data preprocessing, data set division, AC-GAN model training, model deployment, drone data collection, data label verification and correction, data enhancement decision, and federated learning initialization and local model training.

[0016] In addition, the invention has a reliable design principle, a simple structure and a very broad application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention.

[0019] Figure 2 is another schematic flow chart of a method according to an embodiment of the present invention.

[0020] Figure 3 The figure is a schematic flowchart of the model training of the method according to an embodiment of the present invention.

[0021] Figure 4 is a schematic block diagram of a system according to an embodiment of the present invention.

[0022] Figure 5 A schematic diagram of the structure of a device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0025] The key terms appearing in the present invention are explained below.

[0026] As an emerging distributed machine learning paradigm, the federated learning framework aims to solve the data silo problem and allow different institutions or devices to share model updates rather than original data, thereby protecting user privacy and data security. Compared with traditional centralized storage methods, federated learning has the following advantages: i) Traditional methods usually require data to be stored in a central server, which not only increases the risk of data leakage, but also may lead to the abuse of sensitive information. In contrast, federated learning allows data to remain on the local device and only uploads model updates, thereby effectively avoiding the risk of data leakage; ii) In the centralized storage solution, a large amount of raw data needs to be transmitted over the network to the central server for processing. This not only consumes a lot of bandwidth resources, but may also be affected by network latency. In the federated learning framework, since only model updates are transmitted, communication costs and network latency are greatly reduced.

[0027] AC-GAN, or Auxiliary Classifier Generative Adversarial Network, is an improved model based on the traditional Generative Adversarial Network (GAN). The following is a detailed introduction to the AC-GAN model: AC-GAN combines the concept of conditional generative adversarial network (CGAN) and improves the performance and diversity of the model by introducing auxiliary classifiers. Its architecture mainly includes generator, discriminator and auxiliary classifier.

[0028] The drone aerial image processing method provided in the embodiment of the present invention is executed by a computer device, and accordingly, the drone aerial image processing system runs in the computer device.

[0029] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention. Figure 1 The execution subject can be a drone aerial image processing system. According to different requirements, the order of the steps in the flowchart can be changed, and some can be omitted.

[0030] like Figure 1 As shown, the method includes: S1. Acquire an aerial image, generate a label for the aerial image, and then save the aerial image with the label to a sample set.

[0031] First, we need to collect a large number of aerial images. These images can be obtained by drones or other high-altitude shooting equipment. After obtaining the images, we need to generate one or more labels for each image. These labels usually describe the key content or features in the image, such as urban landscapes, natural landscapes, building types, weather conditions, etc. Label generation can be based on manual annotation or use some automated image recognition technology. After completing label generation, we save the labeled aerial images into a sample collection, which will be used for subsequent model training and testing.

[0032] S2. Generate verification labels for the aerial images in the sample set using the generator model, and generate loss values ​​for the verification labels using the discriminator model, and correct the original labels of the aerial images based on the verification labels and the loss values.

[0033] The concept of Generative Adversarial Networks (GANs) is introduced, specifically AC-GAN (Generative Adversarial Networks with Auxiliary Classifiers) or similar models. The generator model tries to generate images that are similar to real aerial images and generates verification labels for them. The discriminator model is responsible for distinguishing between generated images and real images and evaluating the accuracy of the generated verification labels. By calculating the loss value between the verification label and the true label (or the label we think is true), we can evaluate the accuracy of the true label and correct the original label accordingly. This process can be an iterative process. Through multiple training and adjustments, we can gradually improve the accuracy and consistency of the labels.

[0034] S3. Based on the corrected aerial image labels, count the number of various labels, and mark the label categories whose number does not reach a preset quantity threshold as label categories to be expanded.

[0035] The corrected aerial image labels are counted to understand the quantity distribution of each label category. If the quantity of a label category is lower than the preset quantity threshold, we mark it as a label category to be expanded. This quantity threshold can be set according to specific application scenarios and requirements. The purpose of marking the label category to be expanded is to provide guidance for subsequent data enhancement or label expansion work.

[0036] S4. Generate an image copy for the aerial image of the to-be-expanded label category, and set the label of the original aerial image as the label of the corresponding image copy, thereby converting the image style of the image copy.

[0037] Perform image enhancement or style transfer on the aerial images of the label category to be augmented. This can be achieved through a variety of methods, such as image rotation, scaling, cropping, color adjustment, etc. In addition, we can also use deep learning techniques to generate image copies that are visually different from the original images but retain key features and label information. In this way, we can increase the number of images in the label category to be augmented while enriching the diversity and style of the images. When generating image copies, we set the label of the original aerial image to the label of the corresponding image copy to ensure consistency and accuracy of the label. Finally, we can perform style transfer on the image copies to further increase the diversity and beauty of the images.

[0038] In an embodiment of the present invention, based on step S1, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0039] S101. Update of storage files of monitoring aerial images: In real applications, aerial images are usually stored in specific folders or databases. These storage locations may be populated with new image files over time.

[0040] To ensure that our sample collection always contains the latest aerial images, we need to set up a monitoring mechanism to detect updates to the storage files.

[0041] This can be done by periodically checking a folder for new files, monitoring a database's change log, or using the file system's event notifications.

[0042] S102. Extracting updated aerial images from the storage file based on the update status: Once updates to the storage files are detected, we need to extract new aerial images from these files.

[0043] The extraction process may involve operations such as file reading, decoding, and image preprocessing.

[0044] The extracted new images will be used for subsequent label generation and sample set updates.

[0045] S103. Generate labels for the extracted aerial images using annotation tools: For the new aerial images extracted, we need to use annotation tools to generate labels for them.

[0046] The annotation tool can be a manual annotation software or an automated image recognition system.

[0047] The generation of labels should be based on key content or features in the image, such as urban landscapes, natural landscapes, building types, etc.

[0048] In an embodiment of the present invention, based on step S2, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0049] S201. Generate verification labels for aerial images using a generator model, wherein the generator model includes a graph neural network and a classifier.

[0050] Building on this, we can describe in more detail how to use a generator model to generate labels for aerial images, especially when this generator model combines a graph neural network (GNN) and a classifier. Here are the details: Generator Model Overview: In this scenario, the generator model is designed to automatically generate labels for aerial images. This model combines the functions of graph neural networks (GNNs) and classifiers to extract features from images and generate accurate labels based on these features.

[0051] The role of graph neural networks (GNNs): Graph neural networks are particularly suitable for processing data with graph structures. In the case of aerial images, we can regard objects, regions, or features in the image as nodes in the graph and represent the relationship between them through edges. GNNs are able to capture the complex relationships between these nodes and edges, thereby extracting deeper features in the image. These features are crucial for subsequent label generation.

[0052] Integration of classifiers: After GNN, we integrated a classifier. The task of this classifier is to predict the labels of aerial images based on the features extracted by GNN. The classifier can be a traditional machine learning model, such as support vector machine (SVM), decision tree, etc., or a deep learning model, such as a variant of convolutional neural network (CNN). The choice of classifier should be determined according to the specific application scenario and the characteristics of the data set. In this embodiment, the classifier uses a support vector machine.

[0053] The process of generating labels: When an aerial image is input into the generator model, it is first fed into the GNN for processing. The GNN extracts features from the image and represents these features as nodes and edges in the graph. Then, these features are passed to the classifier. The classifier predicts the label of the image based on these features. Finally, the generator model outputs one or more labels that describe the key content or features in the image.

[0054] In summary, using a generator model that combines graph neural networks and classifiers to generate labels for aerial images is an effective method. This method can automatically extract image features and generate accurate labels, providing strong support for subsequent image analysis and applications.

[0055] S202. The discriminator model generates a loss value of the verification label relative to the label of the aerial image.

[0056] Convert labels into semantic vectors; The loss value of the verification label relative to the label of the aerial image is calculated based on a preset loss function, and the loss function is: ； in, represents all data instances of the kth drone, And represents the drone group; ; Among them, the vector A=(A 1 ,A 2 ,…，A n ） , is the semantic vector of the verification label of the i-th aerial image; vectorB= （B 1 ,B 2 ,…，B n ） , is the semantic vector of the label of the i-th aerial image.

[0057] S203. Pass the label loss values ​​of all aerial images as input features to the Gaussian mixture model to identify the potential distribution model of the loss values, each distribution represents a data category.

[0058] The loss values ​​of all instances will be passed as input features to the Gaussian Mixture Model (GMM). The goal of this stage is to identify the potential distribution patterns of the loss values, each of which may represent a group or category in the dataset that reflects different noise levels and data quality.

[0059] S204. Soft-classify the labels according to the loss values ​​to generate soft-classified labels.

[0060] GMM soft-classifies the data instances based on the loss value and generates soft-classification labels as follows: = {𝑝𝑟𝑒d 𝑐𝑡(𝐺𝑀𝑀( , 𝑐𝑜𝑚𝑝𝑜𝑛𝑒𝑛gna𝑠= 2)}, ∀ ∈ , 𝑘∈𝒦}; Among them, components represents the number of Gaussian distributions, and components=2 is intended to capture the loss value distribution of noisy samples and non-noise samples. The number corresponds to the value of components.

[0061] S205. Calculate the average value of each Gaussian distribution, and select the distribution with the highest average value to mark the noise data, so as to establish a noise sample set, and determine the proportion of the noise samples in the entire data set accordingly.

[0062] The average values ​​of each Gaussian distribution are calculated, and the distribution with the highest average value is selected to mark the noise data. This is used to establish a noise sample set, and the proportion of noise samples in the entire data set, i.e., the noise level, is determined accordingly.

[0063] In a specific example, suppose we have a dataset of 1,000 aerial images, each of which has been labeled as one of "city", "forest" or "river". However, due to some errors in the labeling process, there are some noisy labels in the dataset. Our goal is to use a Gaussian mixture model (GMM) to identify these noisy labels and build a collection of noisy samples.

[0064] Step 1: Average calculation Data preparation: We first use the discriminator model to calculate the loss value between the verification label and the self-label corresponding to each aerial image.

[0065] GMM training: Using these loss values ​​as input features, we train a Gaussian mixture model (GMM) assuming the existence of 3 Gaussian distributions (components=3) to capture the underlying distribution patterns in the data.

[0066] Average calculation: After training, GMM will output the parameters of each Gaussian distribution, including mean, variance, and weight. We choose the Gaussian distribution with the highest mean as the representative of the noise data.

[0067] Assume that the means of the three Gaussian distributions output by GMM are =0.5, =1.2, =2.0. At most, we choose the third Gaussian distribution as the distribution of noise data.

[0068] Step 2: Noise sample set construction Data distribution: Based on the output of GMM, we assign each data instance in the dataset to the most likely Gaussian distribution. Specifically, we calculate the probability of each data instance belonging to each Gaussian distribution and assign the instance to the distribution with the highest probability.

[0069] Noise sample screening: Data instances assigned to the third Gaussian distribution (i.e., the distribution with the highest mean) are considered potential noise data. We collect these instances into the noise sample set.

[0070] Assume that there are 200 images assigned to the third Gaussian distribution, then these 200 images constitute our noise sample set.

[0071] Step 3: Noise Level Assessment Proportional calculation: We calculate the proportion of the noise sample set in the entire dataset to assess the noise level. In this example, the noise level is 200 / 1000=20%.

[0072] Quality Assessment: By evaluating the noise level, we can understand the quality and reliability of the dataset. In this example, due to the high noise level (20%), we may need further data cleaning or label correction work to improve the quality of the dataset.

[0073] S206. Determine the aerial images that need to be re-labeled in the end according to the preset trust threshold Trust and the cleaning threshold c, and update the labels of the aerial images that need to be re-labeled to the corresponding verification labels.

[0074] The data instances that need to be relabeled are determined based on the preset trust threshold Trust and cleaning threshold c. Finally, the discriminator E is used to relabel these data instances, that is, the labels in the noisy data set are adjusted based on the output of the discriminator. The basic assumption of this process is that the discriminator of AC-GAN can effectively identify and correct label noise because it has been trained on high-quality data. The relabeling process is as follows: ={( , ), = 𝑎𝑟𝑔𝑚𝑎igh(𝐸( )), ∀ ∈( [1 ∶ 𝑠𝑖𝑧𝑒])}.

[0075] In a specific example, the label update process is as follows: Trust threshold: We set a trust threshold to determine whether the label of a data instance is reliable enough. For example, we can set the trust threshold to 0.8, which means that only when the predicted probability of the discriminator E for a data instance exceeds 0.8, we consider the label to be reliable.

[0076] Cleaning threshold (c): We set a cleaning threshold to determine the number of data instances that need to be relabeled. For example, we can set the cleaning threshold to 0.2, indicating that we plan to relabel 20% of the samples in the dataset.

[0077] Data screening: First, we screen out all data instances with label reliability lower than 0.8 according to the trust threshold. These instances will be regarded as potential samples that need to be relabeled.

[0078] Discriminator evaluation: Next, we use the discriminator E to evaluate these potential samples that need to be relabeled. The discriminator E outputs the probability distribution of each sample belonging to different categories.

[0079] Label adjustment: According to the output of the discriminator E, we adjust the labels in the noisy dataset. Specifically, we select the category with the highest probability as the new label and update it to the corresponding data instance.

[0080] Performance Evaluation: During the re-labeling process, we use methods such as cross-validation to evaluate the performance of the discriminator E. This ensures that the discriminator we use in the re-labeling process is accurate and reliable.

[0081] Data update: Finally, we update the re-labeled labels to the corresponding aerial images. This ensures the accuracy and consistency of the dataset and provides strong support for subsequent analysis and application.

[0082] In an embodiment of the present invention, based on step S4, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0083] S401. Generate multiple image copies First, we need to generate multiple copies of the aerial images for the categories of labels to be augmented. This can be achieved in a variety of ways, including but not limited to: Image rotation: Rotate the original image at a certain angle (such as 90 degrees, 180 degrees, 270 degrees, etc.) to generate a new image copy.

[0084] Image scaling: Changing the size of the original image to generate copies of different sizes. This helps the model learn the features of images at different sizes.

[0085] Image cropping: Crop different regions or image blocks of different sizes from the original image as new image copies. This helps the model learn the local features of the image.

[0086] In addition, other image enhancement techniques, such as color adjustment, contrast adjustment, etc., can be combined to generate more diverse image copies.

[0087] S402. Image Style Transfer After generating multiple copies of the image, we need to transfer the image style. This includes rotation (although it has been mentioned when generating copies, it is emphasized again here as part of style transfer), scaling (also mentioned when generating copies), and various other processing methods, such as blurring, brightness adjustment, and fusion with template images.

[0088] Blurring: Blurring an image to simulate effects such as different focal lengths or motion blur, and generating copies of the image with different degrees of blur.

[0089] Brightness adjustment: Change the brightness of an image to generate copies of the image at different brightness levels. This helps the model learn the characteristics of images under different lighting conditions.

[0090] Fusion with template images: The generated image copy is fused with a specific template image. These template images may include rain template images, snow template images, etc., to simulate different weather or environmental effects.

[0091] Raining template image: The generated image copy is merged with the raining template image to simulate the image effect in rainy weather.

[0092] Snowing template image: Similarly, the generated image copy is merged with the snowing template image to simulate the image effect in snowy weather.

[0093] In this way, we can generate a large number of copies of images with different styles and features, thus greatly enriching the image dataset. This not only helps improve the generalization ability of the model, but also enables the model to better adapt to image recognition tasks in different scenarios and conditions.

[0094] Please refer to Figure 2 In order to facilitate the understanding of the present invention, the following is a further description of the UAV aerial image processing method provided by the present invention based on the principle of the UAV aerial image processing method of the present invention and the process of processing the UAV aerial image in the embodiment.

[0095] Step 1: Data collection and labeling. The server obtains drone data through multiple channels, which can include data collected in actual flight missions, data generated in simulation environments, and publicly available data sets (for example, obtained from scientific research institutions, government open resources, or other legal channels).

[0096] The collected data should cover various scenarios that the drone may encounter when performing missions, including different weather conditions (sunny, rainy, foggy, etc.), various terrain features (plains, mountains, urban environments, etc.), different lighting conditions during the day and at night, and multiple target object types (people, vehicles, buildings, vegetation, etc.).

[0097] In order to ensure data quality, the data needs to be initially screened to exclude those images that are blurry, severely occluded or incomplete.

[0098] Step 2: Data preprocessing. The purpose of data preprocessing is to ensure that the collected raw data is suitable for AC-GAN model training. Through a series of preprocessing steps, the quality and consistency of the data are improved, laying the foundation for subsequent model training and evaluation. The preprocessing process mainly includes the following: De-duplication and noise removal: Identify and remove duplicate data in a dataset and use filtering techniques (such as median filtering) to remove noise from images.

[0099] Format unification: Ensure that all image data uses a unified file format, such as JPEG or PNG; Dataset integrity verification: Check the amount of data in each category to meet the standard of non-independent and identically distributed.

[0100] Step 3: Dataset division: Divide the preprocessed dataset into training set, validation set, and test set to ensure that each subset can fully represent the characteristics of the entire dataset and provide a suitable benchmark for model training and evaluation.

[0101] Step 4: Initialize the AC-GAN model on the server side and train it using the training set data in the standard dataset. At the same time, use the validation set to regularly evaluate the model performance to ensure that the model does not overfit.

[0102] Step 5: Model deployment. After completing the training of the AC-GAN model, the next step is to deploy it to the drone device so that the model can run smoothly on the drone. Considering the limited computing resources of the drone, the CPU and memory resources are reasonably allocated to the AC-GAN model.

[0103] Step 6: UAV data collection. UAV data collection is an important part of the entire workflow, which involves multiple aspects such as UAV flight mission planning, data collection, and storage.

[0104] Determine the collection target: clarify the type of data required, such as terrain mapping, crop monitoring, or urban planning. Determine the specific area and scope of data collection. The collection target is closely related to the federated learning task; Flight path design: Design the flight route to cover the required collection area, plan the starting point, end point, flight altitude, speed and other parameters; Safety considerations: confirm whether there are no-fly zones or restricted-fly zones within the flight area, plan emergency landing points and develop emergency evacuation plans; Data logging: Use appropriate storage media (such as SD cards) to ensure secure storage of collected data.

[0105] Step 7: Data label verification and correction. The quality of data labels directly affects the performance of machine learning models. Using the AC-GAN model can help us filter and correct wrong labels in private datasets collected by drones, thereby improving the quality of the overall dataset. The following is the main process of this step: Predict labels: Use pre-trained classifiers to predict labels in each drone’s private dataset; Compare to true labels: Compare the predicted labels with the true labels. Identify data samples where the predicted labels are inconsistent with the true labels; Anomaly detection: Pay extra attention to samples with low predicted probability or low confidence, which may contain wrong labels or difficult-to-classify border cases; Correct labels: Correct labels that are confirmed to be erroneous (the actual labels that are inconsistent with the predicted labels and whose predicted probability is higher than the preset threshold).

[0106] In specific implementation, after the drone receives the AC-GAN model, for each local data instance, the discriminant component in AC-GAN is used to calculate its loss value relative to the actual label:

[0107] here, represents all data instances of the kth drone, And represents the drone group; The loss values ​​of all instances will be passed as input features to the Gaussian Mixture Model (GMM). The goal of this stage is to identify the potential distribution pattern of the loss values. Each distribution may represent a group or category in the data set, which reflects different noise levels and data quality. GMM soft-classifies the data instances based on the loss values ​​and generates soft classification labels as follows: = {𝑝𝑟𝑒d 𝑐𝑡(𝐺𝑀𝑀( , 𝑐𝑜𝑚𝑝𝑜𝑛𝑒𝑛gna𝑠= 2)}, ∀ ∈ , 𝑘∈𝒦} Among them, components represents the number of Gaussian distributions, and components=2 is intended to capture the loss value distribution of noisy samples and non-noise samples. The number corresponds to the value of components; Next, the average values ​​of each Gaussian distribution are calculated, and the distribution with the highest average value is selected to label the noise data, so as to establish a noise sample set, and determine the proportion of noise samples in the entire data set, that is, the noise level; then, the data instances that need to be relabeled are determined based on the pre-set trust threshold Trust and cleaning threshold c; finally, the discriminator E is used to relabel these data instances, that is, the labels in the noise data set are adjusted according to the output of the discriminator; the basic assumption of this process is that the discriminator of AC-GAN can effectively identify and correct label noise because it has been trained on high-quality data; the relabeling process is as follows: ={( , ), = 𝑎𝑟𝑔𝑚𝑎igh(𝐸( )), ∀ ∈( [1 ∶ 𝑠𝑖𝑧𝑒])} Step 8: Data augmentation decision. Data augmentation is a commonly used technique to increase the diversity and quantity of training data, thereby improving the generalization ability of machine learning models. After completing the data label verification and correction, the next step is to evaluate whether the corrected data set has class imbalance or insufficient sample size problems, and decide whether data augmentation operations are needed based on this. The following is the main process of this step: Dataset Evaluation Statistical analysis: Count the number of samples in each category, calculate the mean and standard deviation of the category distribution, and whether it is the minority class or the majority class; Visualization: Use histograms, pie charts and other charts to display the quantity distribution of samples in each category and observe whether there is a significant category imbalance.

[0108] Decision Making Evaluation requirements: If the number of samples for some classes is much lower than that for other classes, there may be a class imbalance problem.

[0109] Formulate strategy: If there is a problem of class imbalance, use AC-GAN's generator to expand samples of minority categories to achieve the purpose of generating more training samples.

[0110] Step 9: Federated learning initialization and local model training. Federated learning is a distributed machine learning framework that allows multiple clients (such as mobile devices, edge computing nodes, etc.) to collaborate on training models without sharing original data. In this scenario, we will explore how to implement federated learning in a network that includes multiple drones as clients. Figure 3 As shown, the following are the detailed steps and processes: Server-side initialization Depending on the type of task to be solved, the server initializes the model parameters and prepares to distribute these parameters to each drone; the server assigns IDs or identifiers to the drones participating in the training and selects a suitable scheduling strategy.

[0111] Send the initial model to each drone The server sends the initialized model parameters to each participating drone through a secure channel. Each drone receives the same version of the initial model.

[0112] Drones for local training Each drone uses the received model parameters to train the model on the local data set. After the training is completed, the updated model parameters are obtained, packaged and sent back to the server; Step 10: Global model aggregation and iteration. The server aggregates the model parameters of all drones to form a global model, and then forms the final global model through multiple rounds of iterative training. The following are the detailed steps and processes: Parameter aggregation The server receives updated parameters from all participating drones. The server executes an aggregation algorithm (such as FedAvg) to update the global model parameters in an average or weighted average manner.

[0113] Multiple rounds of iteration The server sends the aggregated global model parameters to the drone again to start a new round of local training. This process will continue for multiple rounds until certain convergence conditions are met or the predetermined number of iterations is reached.

[0114] End Condition When the model reaches a predetermined convergence criterion or the communication reaches a specified number of rounds, training stops.

[0115] In some embodiments, the drone aerial image processing system may include multiple functional modules composed of computer program segments. The computer programs of each program segment in the drone aerial image processing system may be stored in a memory of a computer device and executed by at least one processor to perform (see Figure 1 Description) Functions for processing drone aerial images.

[0116] In this embodiment, the drone aerial image processing system can be divided into multiple functional modules according to the functions it performs, such as Figure 4As shown. The functional modules of the system 400 may include: an image acquisition module 410, a label correction module 420, a quantity statistics module 430 and a sample expansion module 440. The module referred to in the present invention refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0117] An image acquisition module is used to acquire aerial images, generate labels for the aerial images, and save the labeled aerial images into a sample set; A label correction module is used to generate verification labels for aerial images in the sample set using the generator model, and to generate loss values ​​of the verification labels using the discriminator model, and to correct the original labels of the aerial images based on the verification labels and the loss values; A quantity statistics module is used to count the number of various labels based on the corrected aerial image labels, and mark the label categories whose number does not reach a preset quantity threshold as the label categories to be expanded; The sample expansion module is used to generate image copies for the aerial images of the to-be-expanded label categories, and to set the labels of the original aerial images as the labels of the corresponding image copies, so as to convert the image styles of the image copies.

[0118] Optionally, as an embodiment of the present invention, acquiring an aerial image, generating a label for the aerial image, and then saving the aerial image with the label to a sample set includes: Monitoring the update status of the storage file of the aerial image, and extracting the updated aerial image from the storage file based on the update status; Use annotation tools to generate labels for the extracted aerial images and save the labeled aerial images to the sample collection.

[0119] Optionally, as an embodiment of the present invention, a generator model is used to generate verification labels for aerial images in a sample set, and a discriminator model is used to generate loss values ​​of the verification labels, and the original labels of the aerial images are corrected based on the verification labels and the loss values, including: Generating verification labels for the aerial images using a generator model, wherein the generator model includes a graph neural network and a classifier; The discriminator model generates a loss value of the verification label relative to the label of the aerial image; The label loss values ​​of all aerial images are passed as input features to the Gaussian mixture model to identify the potential distribution model of the loss value, each distribution represents a data category; Soft-classify the labels according to the loss values ​​and generate soft-classified labels; Calculate the average value of each Gaussian distribution and select the distribution with the highest average value to mark the noise data, so as to establish a noise sample set and determine the proportion of noise samples in the entire data set; The aerial images that need to be relabeled are determined based on the preset trust threshold Trust and cleaning threshold c, and the labels of the aerial images that need to be relabeled are updated to the corresponding verification labels.

[0120] Optionally, as an embodiment of the present invention, the discriminator model generates a loss value of the verification label relative to the label of the aerial image, including: Convert labels into semantic vectors; The loss value of the verification label relative to the label of the aerial image is calculated based on a preset loss function, and the loss function is: ; in, represents all data instances of the kth drone, And represents the drone group; ; Among them, the vector A=(A1,A2,…,An) , is the semantic vector of the verification label of the i-th aerial image; vector B= (B1, B2, …, Bn) , is the semantic vector of the label of the i-th aerial image.

[0121] Optionally, as an embodiment of the present invention, generating an image copy for an aerial image of a to-be-expanded label category, setting a label of the original aerial image as a label of the corresponding image copy, and converting an image style for the image copy includes: generating a plurality of image copies for the aerial image; Image style conversion is performed on the multiple image copies respectively, and the image style conversion includes rotation, scaling, blurring, brightness adjustment, and fusion with template images, and the template images include rain template images and snow template images.

[0122] Optionally, as an embodiment of the present invention, it further includes: Deploy the model to be trained on multiple drones; Under the federated learning framework, the models to be trained are deployed on multiple drones based on pre-prepared data sets.

[0123] Optionally, as an embodiment of the present invention, training the to-be-trained model deployment of multiple drones based on a pre-prepared data set under a federated learning framework includes: The server initializes the model parameters according to the type of task to be solved, and distributes the initialized model parameters to each drone; The server assigns an ID or identifier to the drones participating in the training; The drone uses the received model parameters to perform model training on the local data set. After the training is completed, the drone obtains the updated model parameters, packages these updated parameters and sends them back to the server. The server receives updated parameters from all participating drones and executes an aggregation algorithm to update the global model parameters in an average or weighted average manner; The aggregated global model parameters are sent to the drone again to start a new round of local training, with multiple iterations until the set convergence conditions are met or the number of iterations reaches the predetermined number of iterations.

[0124] Figure 5 The drone aerial image processing method provided for the embodiment of the present application can be applied to the device. It can be understood by those skilled in the art that the device structure involved in the embodiment of the present invention does not constitute a limitation on the device, and the device may include more or fewer components than shown, or combine certain components, or arrange different components. In an embodiment of the present invention, the device includes but is not limited to a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present application described and / or required herein.

[0125] The device 500 may include: a processor 510, a memory 520 and a communication unit 530. These components communicate via one or more buses. Those skilled in the art will appreciate that the server structure shown in the figure does not limit the present invention, and it may be a bus structure or a star structure, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0126] The memory 520 may be used to store the execution instructions of the processor 510, and the memory 520 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. When the execution instructions in the memory 520 are executed by the processor 510, the device 500 is enabled to perform some or all of the steps in the following method embodiments.

[0127] The processor 510 is the control center of the storage device, and uses various interfaces and lines to connect various parts of the entire electronic device. It runs or executes software programs and / or modules stored in the memory 520, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of a plurality of packaged ICs with the same or different functions. For example, the processor 510 can include only a central processing unit (CPU). In an embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.

[0128] The communication unit 530 is used to establish a communication channel so that the storage device can communicate with other devices, receive user data sent by other devices or send user data to other devices.

[0129] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program may include some or all of the steps in each embodiment provided by the present invention. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0130] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes, including several instructions for enabling a computer device (which can be a personal computer, a server, or a second device, a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0131] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.

[0132] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or modules, which can be electrical, mechanical or other forms.

[0133] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0134] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0135] Although the present invention has been described in detail with reference to the accompanying drawings and in combination with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions shall be within the scope of the present invention. Any person of ordinary skill in the art may easily think of changes or substitutions within the technical scope disclosed by the present invention, and these shall be within the scope of protection of the present invention.

Claims

1. A method for processing drone aerial images, characterized in that: include: Acquire an aerial image, generate a label for the aerial image, and save the labeled aerial image into a sample set; Generate verification labels for aerial images in the sample set using the generator model, generate loss values ​​for the verification labels using the discriminator model, and correct the original labels of the aerial images based on the verification labels and the loss values; Based on the corrected aerial image labels, the number of various labels is counted, and the label categories whose number does not reach the preset number threshold are marked as the label categories to be expanded; An image copy is generated for the aerial image of the to-be-expanded label category, and the label of the original aerial image is set as the label of the corresponding image copy, so as to convert the image style of the image copy.

2. The method according to claim 1, characterized in that Acquiring an aerial image, generating a label for the aerial image, and saving the labeled aerial image to a sample set includes: Monitoring the update status of the storage file of the aerial image, and extracting the updated aerial image from the storage file based on the update status; Use annotation tools to generate labels for the extracted aerial images and save the labeled aerial images to the sample collection.

3. The method according to claim 1, characterized in that The generator model is used to generate verification labels for the aerial images in the sample set, and the discriminator model is used to generate loss values ​​of the verification labels, and the original labels of the aerial images are corrected based on the verification labels and the loss values, including: Generating verification labels for the aerial images using a generator model, wherein the generator model includes a graph neural network and a classifier; The discriminator model generates a loss value of the verification label relative to the label of the aerial image; The label loss values ​​of all aerial images are passed as input features to the Gaussian mixture model to identify the potential distribution model of the loss value, each distribution represents a data category; Soft-classify the labels according to the loss values ​​and generate soft-classified labels; Calculate the average value of each Gaussian distribution and select the distribution with the highest average value to mark the noise data, so as to establish a noise sample set and determine the proportion of noise samples in the entire data set accordingly; The aerial images that need to be relabeled are determined based on the preset trust threshold Trust and cleaning threshold c, and the labels of the aerial images that need to be relabeled are updated to the corresponding verification labels.

4. The method according to claim 1, characterized in that: The discriminator model generates the loss value of the verification label relative to the label of the aerial image, including: Convert labels into semantic vectors; The loss value of the verification label relative to the label of the aerial image is calculated based on a preset loss function, and the loss function is: ； in, represents all data instances of the kth drone, And represents the drone group; ； Among them, the vector A=(A 1 ,A 2 ,…，A n ） , is the semantic vector of the verification label of the i-th aerial image; vector B=(B1, B2,…,Bn) , is the semantic vector of the label of the i-th aerial image.

5. The method according to claim 1, characterized in that Generate an image copy for the aerial image of the label category to be expanded, set the label of the original aerial image to the label of the corresponding image copy, and convert the image style of the image copy, including: generating a plurality of image copies for the aerial image; Image style conversion is performed on the multiple image copies respectively, and the image style conversion includes rotation, scaling, blurring, brightness adjustment, and fusion with template images, and the template images include rain template images and snow template images.

6. The method according to claim 1, characterized in that The method further comprises: Deploy the model to be trained on multiple drones; Under the federated learning framework, the models to be trained are deployed on multiple drones based on pre-prepared data sets.

7. The method according to claim 6, characterized in that In the federated learning framework, the models to be trained are deployed on multiple drones based on the pre-prepared datasets, including: The server initializes the model parameters according to the type of task to be solved, and distributes the initialized model parameters to each drone; The server assigns an ID or identifier to the drones participating in the training; The drone uses the received model parameters to perform model training on the local data set. After the training is completed, the drone obtains the updated model parameters, packages these updated parameters and sends them back to the server. The server receives updated parameters from all participating drones and executes an aggregation algorithm to update the global model parameters in an average or weighted average manner; The aggregated global model parameters are sent to the drone again to start a new round of local training, with multiple iterations until the set convergence conditions are met or the number of iterations reaches the predetermined number of iterations.

8. A drone aerial image processing system, characterized in that: include: An image acquisition module is used to acquire aerial images, generate labels for the aerial images, and save the labeled aerial images into a sample set; A label correction module is used to generate verification labels for aerial images in the sample set using the generator model, and to generate loss values ​​of the verification labels using the discriminator model, and to correct the original labels of the aerial images based on the verification labels and the loss values; A quantity statistics module is used to count the number of various labels based on the corrected aerial image labels, and mark the label categories whose number does not reach a preset quantity threshold as the label categories to be expanded; The sample expansion module is used to generate image copies for the aerial images of the to-be-expanded label categories, and to set the labels of the original aerial images as the labels of the corresponding image copies, so as to convert the image styles of the image copies.

9. A device, characterized in that: include: A memory, used for storing a drone aerial image processing program; A processor is used to implement the steps of the drone aerial image processing method as described in any one of claims 1 to 7 when executing the drone aerial image processing program.

10. A computer-readable storage medium storing a computer program, characterized in that: The readable storage medium stores a UAV aerial image processing program, and when the UAV aerial image processing program is executed by the processor, the steps of the UAV aerial image processing method according to any one of claims 1 to 7 are implemented.

Images