UAV aerial image processing method, system, device and storage medium

Abstract

The present invention relates to the field of artificial intelligence technology, and specifically provides a method, system, device, and storage medium for processing aerial images taken by unmanned aerial vehicles, comprising: acquiring aerial images, generating labels for the aerial images, and then saving the labeled aerial images to a sample set; generating verification labels for the aerial images in the sample set using a generator model, and generating loss values ​​for the verification labels using a discriminator model, and correcting the original labels of the aerial images based on the verification labels and the loss values; counting the number of various labels based on the corrected aerial image labels, and marking label categories whose number does not reach a preset threshold as categories to be expanded; generating image copies for the aerial images in the categories to be expanded, and setting the labels of the original aerial images to the labels of the corresponding image copies, thereby converting the image styles of the image copies. The present invention optimizes the quality of drone data.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a method, system, device and storage medium for processing aerial images taken by unmanned aerial vehicles. Background Art

[0002] In recent years, with the rapid development of artificial intelligence (AI), drone technology has been widely applied and developed in fields such as agricultural monitoring, environmental surveys, logistics and distribution, disaster relief, and film and television production. Drones, with their unique flexibility and versatility, equipped with a variety of advanced sensors and high-resolution cameras, can efficiently collect large amounts of image and video data, providing valuable resources for subsequent data analysis and decision support.

[0003] However, in practical applications, data collected by drones often encounter a series of quality issues, which directly impact subsequent data processing and the performance of machine learning models. One of the most prominent issues is data noise, particularly label noise—incorrect or inconsistent data annotation. In addition to label noise, data collected by drones may also suffer from class imbalance. This means that some classes have far more data samples than others. This imbalance can cause machine learning models to favor the majority class and neglect the minority class, thus impacting the model's overall generalization ability. Furthermore, because drones typically operate in diverse geographic locations and environments, the statistical characteristics of the data they collect may vary significantly. This data heterogeneity makes it difficult for the model to capture common features across all classes during training, thus impacting the model's generalization and robustness. Summary of the Invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method, system, device and storage medium for processing aerial images of unmanned aerial vehicles to solve the above-mentioned technical problems.

[0005] In a first aspect, the present invention provides a method for processing drone aerial images, comprising:

[0006] Acquire an aerial image, generate a label for the aerial image, and save the labeled aerial image into a sample set;

[0007] Generating verification labels for the aerial images in the sample set using the generator model, generating loss values ​​for the verification labels using the discriminator model, and correcting the original labels of the aerial images based on the verification labels and the loss values;

[0008] Based on the corrected aerial image labels, the number of various labels is counted, and label categories whose number does not reach a preset threshold are marked as label categories to be expanded;

[0009] An image copy is generated for the aerial image of the to-be-expanded label category, and the label of the original aerial image is set as the label of the corresponding image copy, thereby converting the image style of the image copy.

[0010] In an optional embodiment, acquiring an aerial image, generating a label for the aerial image, and then saving the labeled aerial image to a sample set includes:

[0011] monitoring an update status of a storage file of the aerial image, and extracting an updated aerial image from the storage file based on the update status;

[0012] Use annotation tools to generate labels for the extracted aerial images and save the labeled aerial images to the sample collection.

[0013] In an optional embodiment, a generator model is used to generate verification labels for aerial images in a sample set, and a discriminator model is used to generate loss values ​​for the verification labels. Based on the verification labels and the loss values, the original labels of the aerial images are corrected, including:

[0014] Generating verification labels for the aerial images using a generator model comprising a graph neural network and a classifier;

[0015] The discriminator model generates a loss value of the verification label relative to the label of the aerial image;

[0016] The label loss values ​​of all aerial images are passed as input features to the Gaussian mixture model to identify the potential distribution model of the loss values, where each distribution represents a data class.

[0017] Soft-classify the labels according to the loss value to generate soft classification labels;

[0018] Calculate the average value of each Gaussian distribution and select the distribution with the highest average value to mark the noise data, thereby establishing a noise sample set and determining the proportion of noise samples in the entire data set;

[0019] The aerial images that need to be relabeled are determined based on the pre-set trust threshold Trust and cleaning threshold c, and the labels of the aerial images that need to be relabeled are updated to the corresponding verification labels.

[0020] In an optional embodiment, the discriminator model generates a loss value of the verification label relative to the aerial image's own label, including:

[0021] Convert labels into semantic vectors;

[0022] The loss value of the verification label relative to the label of the aerial image is calculated based on a preset loss function, and the loss function is:

[0023] ;

[0024] in, represents all data instances of the k-th drone, And represents the drone group;

[0025] ;

[0026] Among them, the vector A=(A 1 ,A 2 ,…，A n ） , is the semantic vector of the verification label of the i-th aerial image; vector B= (B1, B2, …, Bn) , is the semantic vector of the label of the i-th aerial image.

[0027] In an optional embodiment, generating an image copy for an aerial image of a to-be-expanded label category, setting the label of the original aerial image as the label of the corresponding image copy, and converting the image style of the image copy includes:

[0028] generating a plurality of image copies for the aerial image;

[0029] Image style conversion is performed on multiple image copies respectively, and the image style conversion includes rotation, scaling, blurring, brightness adjustment, and fusion with template images. The template images include rain template images and snow template images.

[0030] In an optional embodiment, the method further comprises:

[0031] Deploy the model to be trained on multiple drones;

[0032] Under the federated learning framework, the models to be trained are deployed on multiple drones based on a pre-prepared dataset.

[0033] In an optional embodiment, training the to-be-trained model deployment of multiple drones based on a pre-prepared dataset under a federated learning framework includes:

[0034] The server initializes the model parameters according to the type of task to be solved and distributes the initialized model parameters to each drone;

[0035] The server assigns an ID or identifier to the drones participating in the training;

[0036] The drone uses the received model parameters to train the model on the local dataset. After the training is completed, it obtains the updated model parameters, packages these updated parameters and sends them back to the server.

[0037] The server receives updated parameters from all participating drones and executes an aggregation algorithm to update the global model parameters in an average or weighted average manner;

[0038] The aggregated global model parameters are sent to the UAV again to start a new round of local training, which is repeated multiple times until the set convergence conditions are met or the number of iterations reaches the predetermined number of iterations.

[0039] In a second aspect, the present invention provides a UAV aerial image processing system, comprising:

[0040] An image acquisition module is used to acquire aerial images, generate labels for the aerial images, and save the labeled aerial images into a sample set;

[0041] A label correction module is used to generate verification labels for aerial images in the sample set using the generator model, generate loss values ​​for the verification labels using the discriminator model, and correct the original labels of the aerial images based on the verification labels and the loss values;

[0042] A quantity statistics module is used to count the number of various labels based on the corrected aerial image labels, and mark the label categories whose number does not reach a preset quantity threshold as the label categories to be expanded;

[0043] The sample expansion module is used to generate image copies for aerial images of the to-be-expanded label category, set the labels of the original aerial images as the labels of the corresponding image copies, and convert the image styles of the image copies.

[0044] According to a third aspect, a device is provided, comprising:

[0045] A memory for storing a UAV aerial image processing program;

[0046] The processor is used to implement the steps of the drone aerial image processing method provided in the first aspect when executing the drone aerial image processing program.

[0047] In a fourth aspect, a computer-readable storage medium is provided, on which a drone aerial image processing program is stored. When the drone aerial image processing program is executed by a processor, the steps of the drone aerial image processing method provided in the first aspect are implemented.

[0048] The beneficial effect of the present invention is that the drone aerial image processing method, system, device and storage medium provided by the present invention optimize the drone data quality, improve the model training efficiency and performance, and enhance the robustness and security of the system through a series of steps such as data collection and labeling, data preprocessing, data set division, AC-GAN model training, model deployment, drone data collection, data label verification and correction, data enhancement decision-making, and federated learning initialization and local model training.

[0049] In addition, the present invention has a reliable design principle, a simple structure and a very broad application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0051] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention.

[0052] Figure 2 is another schematic flow chart of a method according to an embodiment of the present invention.

[0053] Figure 3 It is a schematic flowchart of the model training method of an embodiment of the present invention.

[0054] Figure 4 FIG. 4 is a schematic block diagram of a system according to an embodiment of the present invention.

[0055] Figure 5 A schematic structural diagram of a device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0058] The key terms appearing in the present invention are explained below.

[0059] The federated learning framework, an emerging distributed machine learning paradigm, aims to address data silos by allowing different organizations or devices to share model updates rather than raw data, thereby protecting user privacy and data security. Compared to traditional centralized storage methods, federated learning offers the following advantages: i) Traditional methods typically require data to be stored centrally on a central server, which not only increases the risk of data leakage but also potentially leads to the misuse of sensitive information. In contrast, federated learning allows data to remain locally on the device and only upload model updates, effectively avoiding the risk of data leakage. ii) In centralized storage solutions, large amounts of raw data must be transmitted over the network to a central server for processing. This not only consumes significant bandwidth resources but can also be affected by network latency. In the federated learning framework, since only model updates are transmitted, communication costs and network latency are significantly reduced.

[0060] AC-GAN, or Auxiliary Classifier Generative Adversarial Network, is an improved model based on the traditional Generative Adversarial Network (GAN). The following is a detailed introduction to the AC-GAN model:

[0061] AC-GAN combines the concepts of conditional generative adversarial networks (CGANs) and introduces auxiliary classifiers to improve model performance and diversity. Its architecture mainly consists of a generator, a discriminator, and an auxiliary classifier.

[0062] The drone aerial image processing method provided in the embodiment of the present invention is executed by a computer device, and accordingly, the drone aerial image processing system runs in the computer device.

[0063] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention. Figure 1 The execution subject can be a UAV aerial image processing system. According to different needs, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0064] like Figure 1 As shown, the method includes:

[0065] S1. Acquire an aerial image, generate a label for the aerial image, and save the labeled aerial image to a sample set.

[0066] First, we need to collect a large amount of aerial imagery. These images can be acquired using drones or other high-altitude imaging equipment. After acquiring the images, we need to generate one or more labels for each image. These labels typically describe key content or features in the image, such as urban landscapes, natural landscapes, building types, and weather conditions. Label generation can be based on manual annotation or automated image recognition techniques. Once labels are generated, we save the labeled aerial images into a sample collection, which will be used for subsequent model training and testing.

[0067] S2. Generate verification labels for the aerial images in the sample set using the generator model, and generate loss values ​​for the verification labels using the discriminator model, and correct the original labels of the aerial images based on the verification labels and the loss values.

[0068] The concept of generative adversarial networks (GANs) was introduced, specifically AC-GAN (Generative Adversarial Networks with Auxiliary Classifiers) or similar models. The generator model attempts to generate images similar to real aerial images and generates verification labels for them. The discriminator model is responsible for distinguishing between the generated images and real images and evaluating the accuracy of the generated verification labels. By calculating the loss between the verification labels and the true labels (or labels we believe to be true), we can evaluate the accuracy of the true labels and modify the original labels accordingly. This process can be iterative, and through repeated training and adjustments, we can gradually improve the accuracy and consistency of the labels.

[0069] S3. Based on the corrected aerial image labels, count the number of various labels, and mark the label categories whose number does not reach a preset threshold as label categories to be expanded.

[0070] We collect statistics on the corrected aerial image labels to understand the distribution of the number of labels in each category. If the number of labels in a category falls below a preset threshold, we mark it as a category to be expanded. This threshold can be set based on the specific application scenario and requirements. The purpose of marking the category to be expanded is to provide guidance for subsequent data augmentation or label expansion.

[0071] S4. Generate an image copy for the aerial image of the to-be-expanded label category, set the label of the original aerial image as the label of the corresponding image copy, and convert the image style of the image copy.

[0072] Perform image augmentation or style transfer on aerial images of the categories to be augmented. This can be achieved through various methods, such as image rotation, scaling, cropping, and color adjustment. Furthermore, we can use deep learning techniques to generate image copies that are visually different from the originals while retaining key features and label information. This approach increases the number of images in the categories to be augmented while enriching the diversity and style of the images. When generating the copies, we set the label of the original aerial image to the label of the corresponding copy to ensure label consistency and accuracy. Finally, we can perform style transfer on the copy to further increase the diversity and aesthetics of the images.

[0073] In an embodiment of the present invention, based on step S1, a possible embodiment will be given below to illustrate its specific implementation scheme in a non-limiting manner.

[0074] S101. Update of the storage file of the monitoring aerial image:

[0075] In practical applications, aerial images are usually stored in specific folders or databases. These storage locations may add new image files over time.

[0076] To ensure that our sample collection always contains the latest aerial images, we need to set up a monitoring mechanism to detect updates to the stored files.

[0077] This can be achieved by periodically checking a folder for new files, monitoring a database change log, or using file system event notifications.

[0078] S102. Extracting updated aerial images from the storage file based on the update status:

[0079] Once updates to the storage files are detected, we need to extract the new aerial images from these files.

[0080] The extraction process may involve operations such as file reading, decoding, and image preprocessing.

[0081] The extracted new images will be used for subsequent label generation and sample set updates.

[0082] S103. Generate labels for the extracted aerial images using annotation tools:

[0083] For the new aerial images extracted, we need to use annotation tools to generate labels for them.

[0084] The annotation tool can be a manual annotation software or an automated image recognition system.

[0085] Label generation should be based on key content or features in the image, such as urban landscape, natural landscape, building type, etc.

[0086] In an embodiment of the present invention, based on step S2, a possible embodiment will be given below to illustrate its specific implementation scheme in a non-limiting manner.

[0087] S201. Generate verification labels for aerial images using a generator model, where the generator model includes a graph neural network and a classifier.

[0088] Building on this foundation, we can now describe in more detail how to use a generator model to generate labels for aerial images, particularly when this generator model combines a graph neural network (GNN) with a classifier. Here are the details:

[0089] Generator Model Overview: In this scenario, a generator model is designed to automatically generate labels for aerial images. This model combines the capabilities of a graph neural network (GNN) and a classifier to extract features from images and generate accurate labels based on these features.

[0090] The role of graph neural networks (GNNs): Graph neural networks are particularly well-suited for processing data with graph structures. In the context of aerial images, we can view objects, regions, or features in the image as nodes in a graph, with edges representing the relationships between them. GNNs are able to capture the complex relationships between these nodes and edges, thereby extracting deeper features from the image. These features are crucial for subsequent label generation.

[0091] Classifier Integration: Following the GNN, we integrate a classifier. This classifier's task is to predict the labels of aerial images based on the features extracted by the GNN. The classifier can be a traditional machine learning model, such as a support vector machine (SVM) or decision tree, or a deep learning model, such as a convolutional neural network (CNN) variant. The choice of classifier should be determined based on the specific application scenario and the characteristics of the dataset. In this implementation, a support vector machine (SVM) is used as the classifier.

[0092] The label generation process: When an aerial image is input into the generator model, it is first processed by the GNN. The GNN extracts features from the image and represents them as nodes and edges in a graph. These features are then passed to the classifier. The classifier predicts the image's label based on these features. Ultimately, the generator model outputs one or more labels that describe the key content or features in the image.

[0093] In summary, using a generator model that combines a graph neural network and a classifier to generate labels for aerial images is an effective method. This method can automatically extract image features and generate accurate labels, providing strong support for subsequent image analysis and applications.

[0094] S202. The discriminator model generates a loss value of the verification label relative to the label of the aerial image.

[0095] Convert labels into semantic vectors;

[0096] The loss value of the verification label relative to the label of the aerial image is calculated based on a preset loss function, and the loss function is:

[0097] ；

[0098] in, represents all data instances of the k-th drone, And represents the drone group;

[0099] ;

[0100] Among them, the vector A=(A 1 ,A 2 ,…，A n ） , is the semantic vector of the verification label of the i-th aerial image; vector B= （B 1 ,B 2 ,…，B n ） , is the semantic vector of the label of the i-th aerial image.

[0101] S203. Pass the label loss values ​​of all aerial images as input features to the Gaussian mixture model to identify the potential distribution model of the loss values, each distribution representing a data category.

[0102] The loss values ​​of all instances are passed as input features to the Gaussian Mixture Model (GMM). The goal of this stage is to identify the underlying distribution pattern of the loss values, each of which may represent a group or category in the dataset, reflecting different noise levels and data quality.

[0103] S204. Soft-classify the labels according to the loss values ​​to generate soft classification labels.

[0104] GMM soft-classifies the data instances based on the loss value and generates soft classification labels as follows:

[0105] = {𝑝𝑟𝑒d 𝑐𝑡(𝐺𝑀𝑀( , 𝑐𝑜𝑚𝑝𝑜𝑛𝑒𝑛gna𝑠= 2)}, ∀ ∈ , 𝑘∈𝒦};

[0106] Among them, components represents the number of Gaussian distributions, and components=2 is intended to capture the loss value distribution of noisy samples and non-noise samples. The number corresponds to the value of components.

[0107] S205. Calculate the average value of each Gaussian distribution and select the distribution with the highest average value to mark the noise data, thereby establishing a noise sample set and determining the proportion of the noise samples in the entire data set.

[0108] Calculate the average value of each Gaussian distribution and select the distribution with the highest average value to mark the noise data. This will establish a noise sample set and determine the proportion of noise samples in the entire data set, i.e., the noise level.

[0109] As a concrete example, suppose we have a dataset of 1,000 aerial images, each labeled as "city," "forest," or "river." However, due to errors in the labeling process, there are some noisy labels in the dataset. Our goal is to use a Gaussian mixture model (GMM) to identify these noisy labels and build a collection of noisy samples.

[0110] Step 1: Average calculation

[0111] Data preparation:

[0112] We first use the discriminator model to calculate the loss value between the verification label and the self-label corresponding to each aerial image.

[0113] GMM training:

[0114] Using these loss values ​​as input features, we train a Gaussian mixture model (GMM) assuming the existence of 3 Gaussian distributions (components=3) to capture the underlying distribution patterns in the data.

[0115] Average value calculation:

[0116] After training, GMM will output the parameters of each Gaussian distribution, including mean, variance, and weight. We choose the Gaussian distribution with the highest mean as the representative of the noisy data.

[0117] Assume that the means of the three Gaussian distributions output by GMM are =0.5, =1.2, =2.0. At the highest, we choose the third Gaussian distribution as the distribution of noise data.

[0118] Step 2: Noise sample set construction

[0119] Data distribution:

[0120] Based on the output of the GMM, we assign each data instance in the dataset to the most likely Gaussian distribution. Specifically, we calculate the probability of each data instance belonging to each Gaussian distribution and assign the instance to the distribution with the highest probability.

[0121] Noise sample screening:

[0122] Data instances assigned to the third Gaussian distribution (i.e., the distribution with the highest mean) are considered potential noise data. We collect these instances into the noise sample set.

[0123] Assume that there are 200 images assigned to the third Gaussian distribution, then these 200 images constitute our noise sample set.

[0124] Step 3: Noise Level Assessment

[0125] Proportional calculation:

[0126] We calculate the proportion of noise samples in the entire dataset to assess the noise level. In this example, the noise level is 200 / 1000 = 20%.

[0127] Quality Assessment:

[0128] By evaluating the noise level, we can understand the quality and reliability of the dataset. In this example, due to the high noise level (20%), we may need further data cleaning or label correction to improve the quality of the dataset.

[0129] S206 . Determine the aerial images that ultimately need to be relabeled based on a preset trust threshold Trust and a cleanup threshold c, and update the labels of the aerial images that need to be relabeled to corresponding verification labels.

[0130] The data instances that ultimately need to be relabeled are determined based on the pre-set trust threshold Trust and cleaning threshold c. Finally, the discriminator E is used to relabel these data instances, adjusting the labels in the noisy dataset based on the discriminator's output. The underlying assumption of this process is that the AC-GAN discriminator, having been trained on high-quality data, can effectively identify and correct label noise. The relabeling process is as follows:

[0131] ={( , ), = 𝑎𝑟𝑔𝑚𝑎igh(𝐸( )), ∀ ∈( [1 ∶ 𝑠𝑖𝑧𝑒])}.

[0132] In a specific example, the label update process is as follows:

[0133] Trust threshold: We set a trust threshold to determine whether the label of a data instance is sufficiently reliable. For example, we can set the trust threshold to 0.8, which means that only when the probability predicted by the discriminator E for a data instance exceeds 0.8, we consider the label to be reliable.

[0134] Cleaning threshold (c): We set a cleaning threshold to determine the number of data instances that need to be relabeled. For example, we can set the cleaning threshold to 0.2, indicating that we plan to relabel 20% of the samples in the dataset.

[0135] Data screening: First, we screen out all data instances with label reliability lower than 0.8 based on the confidence threshold. These instances will be considered as potential samples that need to be relabeled.

[0136] Discriminator evaluation: Next, we use the discriminator E to evaluate these potential relabeled samples. The discriminator E outputs the probability distribution of each sample belonging to different categories.

[0137] Label adjustment: Based on the output of the discriminator E, we adjust the labels in the noisy dataset. Specifically, we select the category with the highest probability as the new label and update it to the corresponding data instance.

[0138] Performance evaluation: During the relabeling process, we use methods such as cross-validation to evaluate the performance of the discriminator E. This ensures that the discriminator we use in the relabeling process is accurate and reliable.

[0139] Data Update: Finally, we update the relabeled labels to the corresponding aerial images. This ensures the accuracy and consistency of the dataset, providing strong support for subsequent analysis and applications.

[0140] In an embodiment of the present invention, based on step S4, a possible embodiment will be given below to illustrate its specific implementation scheme in a non-limiting manner.

[0141] S401. Generate multiple image copies

[0142] First, we need to generate multiple copies of the aerial images for the categories whose labels we want to augment. This can be achieved through a variety of methods, including but not limited to:

[0143] Image rotation: Rotate the original image at a certain angle (such as 90 degrees, 180 degrees, 270 degrees, etc.) to generate a new image copy.

[0144] Image scaling: Resizing the original image to generate copies of different sizes helps the model learn features of images at different sizes.

[0145] Image cropping: Cropping different regions or image blocks of different sizes from the original image as new image copies. This helps the model learn local features of the image.

[0146] In addition, other image enhancement techniques, such as color adjustment, contrast adjustment, etc., can be combined to generate more diverse image copies.

[0147] S402. Image Style Transfer

[0148] After generating multiple copies of an image, we need to perform style transfer on them. This includes rotation (already mentioned when generating copies, but emphasized again here as part of style transfer), scaling (also mentioned when generating copies), and various other processing methods such as blurring, brightness adjustment, and fusion with a template image.

[0149] Blur processing: Blurs an image to simulate effects such as different focal lengths or motion blur, and generate copies of the image with different degrees of blur.

[0150] Brightness adjustment: Change the brightness of an image to generate copies of the image at different brightness levels. This helps the model learn the characteristics of images under different lighting conditions.

[0151] Fusion with template images: The generated image copy is fused with a specific template image. These template images can include rain template images, snow template images, etc., to simulate different weather or environmental effects.

[0152] Raining template image: The generated image copy is merged with the raining template image to simulate the image effect in rainy weather.

[0153] Snowing template image: Similarly, the generated image copy is fused with the snowing template image to simulate the image effect under snowy weather.

[0154] This approach allows us to generate a large number of copies of images with different styles and features, significantly enriching the image dataset. This not only helps improve the generalization ability of the model, but also enables the model to better adapt to image recognition tasks in different scenarios and conditions.

[0155] Please refer to Figure 2 To facilitate understanding of the present invention, the following further describes the drone aerial image processing method provided by the present invention based on the principle of the drone aerial image processing method of the present invention and the process of processing the drone aerial image in the embodiment.

[0156] Step 1: Data collection and annotation. The server obtains drone data through multiple channels. These channels can include data collected during actual flight missions, data generated in simulation environments, and publicly available datasets (for example, from scientific research institutions, government open resources, or other legal channels).

[0157] The collected data should cover various scenarios that the drone may encounter when performing missions, including different weather conditions (sunny, rainy, foggy, etc.), various terrain features (plains, mountains, urban environments, etc.), different lighting conditions during the day and at night, and multiple target object types (people, vehicles, buildings, vegetation, etc.).

[0158] In order to ensure data quality, the data needs to be initially screened to exclude those images that are blurry, severely blocked or incomplete.

[0159] Step 2: Data preprocessing. The purpose of data preprocessing is to ensure that the collected raw data is suitable for AC-GAN model training. Through a series of preprocessing steps, the quality and consistency of the data are improved, laying the foundation for subsequent model training and evaluation. The preprocessing process mainly includes the following:

[0160] Deduplication and noise removal: Identify and remove duplicate data from a dataset and use filtering techniques (such as median filtering) to remove noise from images.

[0161] Format standardization: Ensure that all image data uses a unified file format, such as JPEG or PNG;

[0162] Dataset integrity verification: Check the amount of data in each category to ensure it meets the non-independent and identically distributed standard.

[0163] Step 3: Dataset Partitioning: Divide the preprocessed dataset into training, validation, and test sets, ensuring that each subset fully represents the characteristics of the entire dataset and provides a suitable benchmark for model training and evaluation.

[0164] Step 4: Initialize the AC-GAN model on the server side and train it using the training set data in the standard dataset. At the same time, use the validation set to regularly evaluate the model performance to ensure that the model does not overfit.

[0165] Step 5: Model Deployment. After training the AC-GAN model, the next step is to deploy it on the drone so that the model can run smoothly on the drone. Considering the limited computing resources of the drone, it is necessary to properly allocate CPU and memory resources to the AC-GAN model.

[0166] Step 6: UAV data collection. UAV data collection is an important part of the entire workflow, which involves multiple aspects such as UAV flight mission planning, data collection, and storage.

[0167] Determine the collection target: Identify the type of data required, such as terrain mapping, crop monitoring, or urban planning. Determine the specific area and scope of data collection. The collection target is closely related to the federated learning task;

[0168] Flight path design: Design the flight route to cover the required collection area, plan the starting point, end point, and flight altitude, speed and other parameters;

[0169] Safety considerations: Confirm whether there are no-fly zones or restricted flight zones within the flight area, plan emergency landing points, and develop emergency evacuation plans;

[0170] Data logging: Use appropriate storage media (such as SD cards) to ensure secure storage of collected data.

[0171] Step 7: Data label verification and correction. The quality of data labels directly affects the performance of machine learning models. Using the AC-GAN model can help us filter and correct incorrect labels in private datasets collected by drones, thereby improving the quality of the overall dataset. The following is the main process of this step:

[0172] Predict labels: Use pre-trained classifiers to predict labels in each drone’s private dataset;

[0173] Compare to true labels: Compare the predicted labels with the true labels. Identify data samples where the predicted labels are inconsistent with the true labels.

[0174] Anomaly detection: Pay extra attention to samples with low predicted probability or low confidence, which may contain incorrect labels or difficult-to-classify border cases;

[0175] Correct labels: Correct labels that are confirmed to be incorrect (the actual label that is inconsistent with the predicted label and whose predicted probability is higher than the preset threshold).

[0176] In specific implementation, after the drone receives the AC-GAN model, for each local data instance, the discriminant component in AC-GAN is used to calculate its loss value relative to the actual label:

[0177]

[0178] here, represents all data instances of the k-th drone, And represents the drone group;

[0179] The loss values ​​of all instances are passed as input features to a Gaussian Mixture Model (GMM). The goal of this stage is to identify the underlying distribution of loss values. Each distribution may represent a group or category in the dataset, reflecting different noise levels and data quality. The GMM soft-classifies the data instances based on the loss values ​​and generates soft classification labels as follows:

[0180] = {𝑝𝑟𝑒d 𝑐𝑡(𝐺𝑀𝑀( , 𝑐𝑜𝑚𝑝𝑜𝑛𝑒𝑛gna𝑠= 2)}, ∀ ∈ , 𝑘∈𝒦}

[0181] Among them, components represents the number of Gaussian distributions, and components=2 is intended to capture the loss value distribution of noisy samples and non-noise samples. The number corresponds to the value of components;

[0182] Next, the mean of each Gaussian distribution is calculated, and the distribution with the highest mean is selected to label the noise data. This establishes a noise sample set, and the proportion of noise samples in the entire dataset, i.e., the noise level, is determined accordingly. The data instances that ultimately need to be relabeled are then determined based on the pre-set trust threshold Trust and cleaning threshold c. Finally, the discriminator E is used to relabel these data instances, adjusting the labels in the noisy dataset based on the discriminator's output. The underlying assumption of this process is that the AC-GAN discriminator, having been trained on high-quality data, can effectively identify and correct label noise. The relabeling process is as follows:

[0183] ={( , ), = 𝑎𝑟𝑔𝑚𝑎igh(𝐸( )), ∀ ∈( [1 ∶ 𝑠𝑖𝑧𝑒])}

[0184] Step 8: Data augmentation decision. Data augmentation is a common technique used to increase the diversity and quantity of training data, thereby improving the generalization ability of machine learning models. After completing data label verification and correction, the next step is to evaluate whether the corrected dataset has class imbalance or insufficient sample size, and accordingly determine whether data augmentation is necessary. The following is the main process of this step:

[0185] Dataset evaluation

[0186] Statistical analysis: Count the number of samples in each category, calculate the mean and standard deviation of the category distribution, and whether it is the minority class or the majority class;

[0187] Visualization: Use histograms, pie charts, and other charts to display the number distribution of samples in each category and observe whether there is significant category imbalance.

[0188] Decision Making

[0189] Evaluation requirements: If the number of samples for some classes is much lower than that for other classes, there may be a class imbalance problem.

[0190] Formulate a strategy: If there is a class imbalance problem, use the AC-GAN generator to expand the samples of the minority class to achieve the purpose of generating more training samples.

[0191] Step 9: Federated learning initialization and local model training. Federated learning is a distributed machine learning framework that allows multiple clients (such as mobile devices, edge computing nodes, etc.) to collaboratively train models without sharing original data. In this scenario, we will explore how to implement federated learning in a network that includes multiple drones as clients. Figure 3 As shown, the following are the detailed steps and processes:

[0192] Server-side initialization

[0193] Depending on the type of task to be solved, the server initializes the model parameters and prepares to distribute these parameters to each drone; the server assigns IDs or identifiers to the drones participating in the training and selects an appropriate scheduling strategy.

[0194] Send the initial model to each drone

[0195] The server sends the initialized model parameters to each participating drone through a secure channel. Each drone receives the same version of the initial model.

[0196] Drones conduct local training

[0197] Each drone uses the received model parameters to train the model on the local dataset. After training is complete, the updated model parameters are obtained, packaged, and sent back to the server.

[0198] Step 10: Global model aggregation and iteration. The server aggregates the model parameters of all drones to form a global model, and then trains through multiple rounds of iterations to form the final global model. The following are the detailed steps and process:

[0199] Parameter aggregation

[0200] The server receives updated parameters from all participating drones and executes an aggregation algorithm (such as FedAvg) to update the global model parameters in an average or weighted average manner.

[0201] Multiple rounds of iteration

[0202] The server sends the aggregated global model parameters to the drone again, starting a new round of local training. This process continues for multiple rounds until certain convergence conditions are met or the predetermined number of iterations is reached.

[0203] End Condition

[0204] Training stops when the model reaches a predetermined convergence criterion or the communication reaches a specified number of rounds.

[0205] In some embodiments, the UAV aerial image processing system may include multiple functional modules composed of computer program segments. The computer program of each program segment in the UAV aerial image processing system may be stored in a memory of a computer device and executed by at least one processor to perform (see Figure 1 Description) Functions for processing drone aerial images.

[0206] In this embodiment, the UAV aerial image processing system can be divided into multiple functional modules according to the functions it performs, such as Figure 4 As shown. The functional modules of system 400 may include: image acquisition module 410, label correction module 420, quantity statistics module 430, and sample expansion module 440. A module as referred to in the present invention refers to a series of computer program segments that can be executed by at least one processor and can perform fixed functions, and is stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0207] An image acquisition module is used to acquire aerial images, generate labels for the aerial images, and save the labeled aerial images into a sample set;

[0208] A label correction module is used to generate verification labels for aerial images in the sample set using the generator model, generate loss values ​​for the verification labels using the discriminator model, and correct the original labels of the aerial images based on the verification labels and the loss values;

[0209] A quantity statistics module is used to count the number of various labels based on the corrected aerial image labels, and mark the label categories whose number does not reach a preset quantity threshold as the label categories to be expanded;

[0210] The sample expansion module is used to generate image copies for aerial images of the to-be-expanded label category, set the labels of the original aerial images as the labels of the corresponding image copies, and convert the image styles of the image copies.

[0211] Optionally, as an embodiment of the present invention, acquiring an aerial image, generating a label for the aerial image, and then saving the labeled aerial image to a sample set includes:

[0212] monitoring an update status of a storage file of the aerial image, and extracting an updated aerial image from the storage file based on the update status;

[0213] Use annotation tools to generate labels for the extracted aerial images and save the labeled aerial images to the sample collection.

[0214] Optionally, as an embodiment of the present invention, a generator model is used to generate verification labels for aerial images in a sample set, and a discriminator model is used to generate loss values ​​for the verification labels. Based on the verification labels and the loss values, the original labels of the aerial images are corrected, including:

[0215] Generating verification labels for the aerial images using a generator model comprising a graph neural network and a classifier;

[0216] The discriminator model generates a loss value of the verification label relative to the label of the aerial image;

[0217] The label loss values ​​of all aerial images are passed as input features to the Gaussian mixture model to identify the potential distribution model of the loss values, where each distribution represents a data class.

[0218] Soft-classify the labels according to the loss value to generate soft classification labels;

[0219] Calculate the average value of each Gaussian distribution and select the distribution with the highest average value to mark the noise data, thereby establishing a noise sample set and determining the proportion of noise samples in the entire data set;

[0220] The aerial images that need to be relabeled are determined based on the pre-set trust threshold Trust and cleaning threshold c, and the labels of the aerial images that need to be relabeled are updated to the corresponding verification labels.

[0221] Optionally, as an embodiment of the present invention, the discriminator model generates a loss value of the verification label relative to the label of the aerial image, including:

[0222] Convert labels into semantic vectors;

[0223] The loss value of the verification label relative to the label of the aerial image is calculated based on a preset loss function, and the loss function is:

[0224] ;

[0225] in, represents all data instances of the k-th drone, And represents the drone group;

[0226] ;

[0227] Among them, the vector A=(A1,A2,…,An) , is the semantic vector of the verification label of the i-th aerial image; vector B= (B1, B2, …, Bn) , is the semantic vector of the label of the i-th aerial image.

[0228] Optionally, as an embodiment of the present invention, generating an image copy for an aerial image of a to-be-expanded label category, setting the label of the original aerial image as the label of the corresponding image copy, and converting the image style of the image copy includes:

[0229] generating a plurality of image copies for the aerial image;

[0230] Image style conversion is performed on multiple image copies respectively, and the image style conversion includes rotation, scaling, blurring, brightness adjustment, and fusion with template images. The template images include rain template images and snow template images.

[0231] Optionally, as an embodiment of the present invention, the method further includes:

[0232] Deploy the model to be trained on multiple drones;

[0233] Under the federated learning framework, the models to be trained are deployed on multiple drones based on a pre-prepared dataset.

[0234] Optionally, as an embodiment of the present invention, training the to-be-trained model deployment of multiple drones based on a pre-prepared dataset under a federated learning framework includes:

[0235] The server initializes the model parameters according to the type of task to be solved and distributes the initialized model parameters to each drone;

[0236] The server assigns an ID or identifier to the drones participating in the training;

[0237] The drone uses the received model parameters to train the model on the local dataset. After the training is completed, it obtains the updated model parameters, packages these updated parameters and sends them back to the server.

[0238] The server receives updated parameters from all participating drones and executes an aggregation algorithm to update the global model parameters in an average or weighted average manner;

[0239] The aggregated global model parameters are sent to the UAV again to start a new round of local training, which is repeated multiple times until the set convergence conditions are met or the number of iterations reaches the predetermined number of iterations.

[0240] Figure 5The drone aerial image processing method provided for the embodiment of the present application can be applied to a device. Those skilled in the art will understand that the device structure involved in the embodiment of the present invention does not constitute a limitation on the device, and the device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. In the embodiment of the present invention, the device includes but is not limited to a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present application described and / or required herein.

[0241] The device 500 may include a processor 510, a memory 520, and a communication unit 530. These components communicate via one or more buses. Those skilled in the art will appreciate that the server structure shown in the figure does not limit the present invention. The server structure may be a bus structure or a star structure, and may include more or fewer components than shown, or may combine certain components or arrange the components differently.

[0242] The memory 520 can be used to store execution instructions of the processor 510. The memory 520 can be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk. When the execution instructions in the memory 520 are executed by the processor 510, the device 500 can perform some or all of the steps in the following method embodiments.

[0243] The processor 510 is the control center of the storage device, which uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and / or processes data by running or executing software programs and / or modules stored in the memory 520, and calling data stored in the memory. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of multiple packaged ICs with the same or different functions. For example, the processor 510 can only include a central processing unit (CPU). In an embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.

[0244] The communication unit 530 is configured to establish a communication channel so that the storage device can communicate with other devices, receive user data sent by other devices, or send user data to other devices.

[0245] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program that, when executed, may include some or all of the steps of each embodiment provided herein. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0246] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software and a necessary general-purpose hardware platform. Based on this understanding, the technical solutions in the embodiments of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, among other media capable of storing program code, and includes instructions for causing a computer device (which can be a personal computer, a server, or a second device, a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention.

[0247] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment.

[0248] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or modules, and can be electrical, mechanical or other forms.

[0249] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected to achieve the purpose of the present embodiment according to actual needs.

[0250] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0251] Although the present invention has been described in detail with reference to the accompanying drawings and in conjunction with preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, persons of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and such modifications or substitutions shall be within the scope of the present invention. Any changes or substitutions that can be easily conceived by persons skilled in the art within the technical scope disclosed in the present invention shall be within the scope of protection of the present invention.

Claims

1. A method for processing aerial images taken by a drone, characterized in that: include: Acquire an aerial image, generate a label for the aerial image, and save the labeled aerial image into a sample set; Generating verification labels for the aerial images in the sample set using the generator model, generating loss values ​​for the verification labels using the discriminator model, and correcting the original labels of the aerial images based on the verification labels and the loss values; Based on the corrected aerial image labels, the number of various labels is counted, and label categories whose number does not reach a preset threshold are marked as label categories to be expanded; Generating an image copy for the aerial image of the to-be-expanded label category, setting the label of the original aerial image as the label of the corresponding image copy, and converting the image style of the image copy; Generating verification labels for aerial images in the sample set using the generator model, generating loss values ​​for the verification labels using the discriminator model, and correcting the original labels of the aerial images based on the verification labels and the loss values, including: Generating verification labels for the aerial images using a generator model comprising a graph neural network and a classifier; The discriminator model generates a loss value of the verification label relative to the label of the aerial image; The label loss values ​​of all aerial images are passed as input features to the Gaussian mixture model to identify the potential distribution model of the loss values, where each distribution represents a data class. Soft-classify the labels according to the loss value to generate soft classification labels; Calculate the average value of each Gaussian distribution and select the distribution with the highest average value to mark the noise data, thereby establishing a noise sample set and determining the proportion of noise samples in the entire data set; The aerial images that need to be relabeled are determined based on the pre-set trust threshold Trust and cleaning threshold c, and the labels of the aerial images that need to be relabeled are updated to the corresponding verification labels.

2. The method according to claim 1, characterized in that Acquiring an aerial image, generating a label for the aerial image, and saving the labeled aerial image to a sample set includes: monitoring an update status of a storage file of the aerial image, and extracting an updated aerial image from the storage file based on the update status; Use annotation tools to generate labels for the extracted aerial images and save the labeled aerial images to the sample collection.

3. The method according to claim 1, characterized in that The discriminator model generates a loss value for the verification label relative to the aerial image's own label, including: Convert labels into semantic vectors; The loss value of the verification label relative to the label of the aerial image is calculated based on a preset loss function, and the loss function is: ； in, represents all data instances of the k-th drone, And represents the drone group; ； Among them, the vector A=(A 1 ,A 2 ,…，A n ） , is the semantic vector of the verification label of the i-th aerial image; vector B=(B1, B2,…,Bn) , is the semantic vector of the label of the i-th aerial image.

4. The method according to claim 1, wherein Generate an image copy for the aerial image of the to-be-expanded label category, set the label of the original aerial image as the label of the corresponding image copy, and convert the image style of the image copy, including: generating a plurality of image copies for the aerial image; Image style conversion is performed on multiple image copies respectively, and the image style conversion includes rotation, scaling, blurring, brightness adjustment, and fusion with template images. The template images include rain template images and snow template images.

5. The method according to claim 1, wherein The method further comprises: Deploy the model to be trained on multiple drones; Under the federated learning framework, the models to be trained are deployed on multiple drones based on a pre-prepared dataset.

6. The method according to claim 5, characterized in that In the federated learning framework, the training model deployment of multiple drones is trained based on a pre-prepared dataset, including: The server initializes the model parameters according to the type of task to be solved and distributes the initialized model parameters to each drone; The server assigns an ID or identifier to the drones participating in the training; The drone uses the received model parameters to train the model on the local dataset. After the training is completed, it obtains the updated model parameters, packages these updated parameters and sends them back to the server. The server receives updated parameters from all participating drones and executes an aggregation algorithm to update the global model parameters in an average or weighted average manner; The aggregated global model parameters are sent to the UAV again to start a new round of local training, which is repeated multiple times until the set convergence conditions are met or the number of iterations reaches the predetermined number of iterations.

7. A UAV aerial image processing system, characterized in that: include: An image acquisition module is used to acquire aerial images, generate labels for the aerial images, and save the labeled aerial images into a sample set; A label correction module is used to generate verification labels for aerial images in the sample set using the generator model, generate loss values ​​for the verification labels using the discriminator model, and correct the original labels of the aerial images based on the verification labels and the loss values; A quantity statistics module is used to count the number of various labels based on the corrected aerial image labels, and mark the label categories whose number does not reach a preset quantity threshold as the label categories to be expanded; A sample expansion module is used to generate image copies for aerial images of the to-be-expanded label category, set the labels of the original aerial images as the labels of the corresponding image copies, and convert the image styles of the image copies; Generating verification labels for aerial images in the sample set using the generator model, generating loss values ​​for the verification labels using the discriminator model, and correcting the original labels of the aerial images based on the verification labels and the loss values, including: Generating verification labels for the aerial images using a generator model comprising a graph neural network and a classifier; The discriminator model generates a loss value of the verification label relative to the label of the aerial image; The label loss values ​​of all aerial images are passed as input features to the Gaussian mixture model to identify the potential distribution model of the loss values, where each distribution represents a data class. Soft-classify the labels according to the loss value to generate soft classification labels; Calculate the average value of each Gaussian distribution and select the distribution with the highest average value to mark the noise data, thereby establishing a noise sample set and determining the proportion of noise samples in the entire data set; The aerial images that need to be relabeled are determined based on the pre-set trust threshold Trust and cleaning threshold c, and the labels of the aerial images that need to be relabeled are updated to the corresponding verification labels.

8. A device, characterized in that include: A memory for storing a UAV aerial image processing program; A processor is used to implement the steps of the drone aerial image processing method according to any one of claims 1 to 6 when executing the drone aerial image processing program.

9. A computer-readable storage medium storing a computer program, characterized in that: The readable storage medium stores a drone aerial image processing program, and when the drone aerial image processing program is executed by the processor, the steps of the drone aerial image processing method according to any one of claims 1 to 6 are implemented.

Images