A hybrid image-based real-time ground object segmentation method for unmanned aerial vehicles

Abstract

The application discloses a kind of unmanned vehicle real-time ground object segmentation methods based on mixed image, the method includes: collecting the remote sensing image of target area and carrying out partial annotation, making dataset;Ground object segmentation network is constructed and is trained;Real-time ground object segmentation is carried out using ground object segmentation network, the label obtained is visualized and then output to user equipment;The label obtained is carried out confidence calculation, and the label of confidence greater than or equal to threshold value is carried out image mixing processing, generates mixed image, and is input to ground object segmentation network and continues to be trained.The application adopts lightweight neural network based on consistency regularization semi-supervised method, and the delay on mobile device is small, only a small amount of label data is needed, that is, it can be realized, and classification precision is high.In addition, the application adopts image mixing, matching strategy, maximizes the use of data, increases the feature extraction capability of model to low proportion ground object class, improves model segmentation precision, reduces the risk of model overfitting.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of remote sensing image ground object segmentation, and particularly relates to a UAV real-time ground object segmentation method based on a mixed image. BACKGROUND

[0002] Real-time remote sensing ground object segmentation can accurately identify and classify each ground object category in a remote sensing image in real time, and provides key information for dynamic monitoring and decision support. Current real-time ground object segmentation mainly relies on full-supervised ground object segmentation methods, which train deep learning models through a large amount of labeled data to achieve accurate classification of each pixel in the image. UNet++ is a simple encoder-decoder structure, and through the establishment of multi-level skip connections between the encoder and the decoder, the transmission and fusion of features are enhanced. This design allows the network to better capture features at different levels and improve the accuracy of segmentation. Secondly, UNet++ has a certain flexibility, which allows users to adjust the depth and width of the network according to needs to adapt to different application scenarios and computing resource limitations. Although using this model to realize real-time ground object segmentation can guarantee high accuracy, a large amount of time is needed to train the model to obtain accurate segmentation results. Each time the region is changed, data needs to be collected and labeled again, and the model needs to be trained for the characteristics of the new region, which not only consumes time, but also poses a great challenge to real-time performance, increasing the complexity and cost of application. Due to the multi-level skip connection structure of UNet++, a lot of time is also needed to obtain the result during prediction, which greatly reduces the timeliness of real-time ground object segmentation.

[0003] For such models with complex network structure, some researchers have begun to design lightweight networks. Lightweight networks can reduce the time required for model prediction, solving the problem of high delay in traditional model implementation for real-time feature segmentation. The MobileNet series of networks is a group of lightweight deep learning models designed for mobile and embedded vision applications. The design goal of these models is to significantly reduce the demand for computing resources while maintaining reasonable accuracy, thereby achieving faster inference speed and lower power consumption. MobileNetV2 reduces unnecessary DW convolution weights and improves feature representation capabilities by introducing inverted residuals and linear bottlenecks. MobileNetV3 uses neural architecture search (NAS) to find the best balance between computational efficiency and model performance. In addition, MobileNetV3 also introduces the Hard Swish activation function, which is a simplified version of the Swish function that maintains the advantages of nonlinear activation while reducing computational complexity. Although lightweight networks can significantly reduce the time required for prediction, which ensures the timeliness of real-time remote sensing feature segmentation. However, during model training, a large number of accurate labels are still required, and this method requires a large amount of manpower and time cost for one-to-one annotation of remote sensing images.

[0004] With the rapid development of deep learning, some researchers have tried to use partially labeled remote sensing images and a large number of unlabeled remote sensing images to train models. Currently, semi-supervised algorithms in remote sensing feature segmentation research mainly have the following three directions:

[0005] One is based on pseudo-labeling, this method trains an initial model on a small amount of labeled data, then uses this model to predict unlabeled data, and uses the prediction results as "pseudo-labels" to further train the model. The core of this method is how to effectively generate high-quality pseudo-labels to reduce the impact of error propagation. ST++ adopts a picture-level screening and progressive utilization strategy, selects and sorts reliable pseudo-labels for re-training to reduce the impact of false pseudo-labels.

[0006] Second, the method based on consistency regularization, which ensures the consistency of the model's prediction under data augmentation transformation by introducing a consistency regularization term. Even with slight changes in input data, the model should give similar prediction results. Current semi-supervised remote sensing semantic algorithms are mainly based on consistency regularization. CPS method designs a very simple and good performance semi-supervised framework. CPS method uses two networks with the same structure but different initialization, adds constraints to make the output of the two networks similar for the same sample. Specifically, the pseudolabel generated by the current network will be used as the target of the prediction of another network, and this process can be supervised by cross entropy loss.

[0007] Third, the method based on generative adversarial network (GAN), which uses generative adversarial network to generate synthetic data to expand the training set. GAN includes generator and discriminator. The generator tries to generate synthetic samples consistent with the real data distribution, while the discriminator tries to distinguish real data and synthetic data. AdvSemiSeg is a method that uses the idea of generative adversarial network (GAN) to enhance the semi-supervised learning framework. AdvSemiSeg uses both labeled images and unlabeled images during training. When using labeled images, the segmentation network is subject to both standard cross-entropy loss based on ground truth label and adversarial loss based on discriminant network. When using unlabeled images, the preliminary segmentation result is obtained by the segmentation network, and then the preliminary segmentation result is sent to the discriminant network to obtain the confidence map. The confidence map is used as a supervision signal to mask the preliminary segmentation prediction result and use self-learning method to train the segmentation network. Benefiting from the discriminant network, AdvSemiSeg achieves good segmentation results on multiple datasets such as Pascal VOC and cityspace.

[0008] The above semi-supervised algorithm for remote sensing feature segmentation has the following shortcomings:

[0009] (1) When UNet++ and other models are used to realize real-time feature segmentation, the prediction speed is slow, resulting in certain delay and poor timeliness. This is because the network complexity of UNet++ and other models is high, resulting in high model calculation complexity and long processing time for each frame of image, poor real-time performance.

[0010] (2) The segmentation accuracy of lightweight networks such as MobileNet is lower than that of higher complexity models such as UNet++. When facing complex or variable data, the generalization ability of lightweight networks may not be as good as that of higher complexity models. This is because lightweight networks use techniques such as depthwise separable convolution to reduce computational complexity, but this also limits the ability of the model to capture complex features. When processing variable data, lightweight networks may not be able to capture all class features due to limited feature expression ability, affecting the generalization ability.

[0011] (3) Pseudo-label-based semi-supervised methods rely on high-quality initial models. The proportion of remote sensing features on the image is usually unbalanced. If the number of samples of a certain class is small, the pseudo-label method may cause that class to be ignored or misclassified. This is because a high-quality initial model is a prerequisite for generating reliable pseudo-labels. If the initial model is not of high quality, the generated pseudo-labels may contain more errors. When the initial model is of high quality, it is necessary to have higher network complexity, which will also increase the time it takes for the model to generate pseudo-labels, thereby reducing real-time performance.

[0012] (4) Semi-supervised feature segmentation methods based on generative adversarial networks require a large amount of computing resources, which is not conducive to real-time remote sensing feature segmentation. The masks generated by GAN between different domains (such as remote sensing images in different regions or different seasons) may not be consistent, affecting the robustness of the model. This is because the GAN method needs to train the generator and the discriminator, which has high computational complexity and is not suitable for real-time applications.

[0013] (5) Semi-supervised remote sensing feature segmentation methods based on consistency regularization have long model training times and are very sensitive to the selection of hyperparameters. Inappropriate hyperparameter settings may lead to unstable training or poor results. This is because models using consistency regularization methods require more time to train and consistency regularization methods may encounter challenges when dealing with cross-domain problems, especially when the real-time segmentation area exceeds the area where the training data is located. SUMMARY

[0014] The main purpose of the present application is to overcome the shortcomings and deficiencies of the prior art and provide a UAV real-time feature segmentation method based on mixed images. This method combines the advantages of lightweight networks (such as MobileOne), maintains prediction consistency under data augmentation conditions, fully utilizes unlabeled data, improves the robustness and generalization ability of the model, and can realize fast response to environmental monitoring, disaster assessment, urban management and other application scenarios.

[0015] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0016] In one aspect of the present application, a real-time unmanned aerial vehicle (UAV) ground object segmentation method based on mixed images is provided, comprising the following steps:

[0017] Collecting remote sensing images of a target area and performing partial labeling to produce a remote sensing data set of the target area;

[0018] Constructing a ground object segmentation network based on MobileOne and training the network using the remote sensing data set of the target area;

[0019] Using the trained ground object segmentation network to perform ground object segmentation on remote sensing images collected by the UAV, visualizing the obtained labels, and outputting the labels to a user device;

[0020] Performing confidence calculation on the obtained labels, performing image mixing processing on labels with a confidence greater than or equal to a threshold value, generating mixed images, and inputting the mixed images to the ground object segmentation network for further training.

[0021] As a preferred technical solution, the remote sensing images of the target area are collected and partially labeled, and a remote sensing data set of the target area is produced, specifically as follows:

[0022] Using a UAV to capture remote sensing images of the target area;

[0023] Setting the types of ground objects in the target area according to needs;

[0024] Classifying and labeling remote sensing images in a set proportion;

[0025] Dividing the labeled remote sensing image data into a training set, a validation set, and a test set;

[0026] Adding unlabeled data to the training set, reconfiguring the training set, and finally obtaining a remote sensing data set of the target area.

[0027] As a preferred technical solution, the ground object segmentation network based on MobileOne is specifically constructed as follows:

[0028] Based on the MobileOne lightweight neural network, a global depth separable convolution module is used to replace the depth convolution module in the MobileOne lightweight neural network, and an SENet attention module is introduced after the global depth separable convolution module.

[0029] As a preferred technical solution, the ground object segmentation network is trained using the remote sensing data set of the target area, specifically as follows:

[0030] Inputting labeled remote sensing images to a ground object segmentation network for training; simultaneously inputting unlabeled remote sensing images to the ground object segmentation network and another ground object segmentation network with the same initial parameters and gradient parameters frozen for training;

[0031] The cross-entropy is used to calculate the difference between the prediction result of the ground object segmentation network and the real label of the labeled remote sensing image; the MSE Loss is used to calculate the difference between the prediction results of the ground object segmentation network and the gradient parameter frozen ground object segmentation network of the same remote sensing image; and the two kinds of differences are used to update the gradient parameters of the ground object segmentation network.

[0032] The ground object segmentation network updates the parameters of the gradient parameter frozen ground object segmentation network by the exponential moving average method.

[0033] As a preferred technical solution, a breakpoint is set in the training of the ground object segmentation network, and the training is stopped when the proportion of the labeled remote sensing images in the entire training set is lower than a certain proportion; when new labeled remote sensing images are added into the data set, and the proportion of the labeled remote sensing images in the entire training set is higher than a certain proportion, the training is continued.

[0034] As a preferred technical solution, the pseudo label with the confidence greater than or equal to the threshold value is output to the mobile device or computer of the user as a real label, and specifically:

[0035] The ground object segmentation network is deployed on the unmanned aerial vehicle, real-time prediction is performed on the compressed remote sensing image collected by the unmanned aerial vehicle, and the prediction result is visualized and transmitted to the mobile device of the user;

[0036] The ground object segmentation network is deployed in the background system, receives the complete remote sensing image collected by the unmanned aerial vehicle and performs prediction, and the prediction result is visualized and transmitted to the computer of the user.

[0037] As a preferred technical solution, the pseudo label with the confidence greater than or equal to the threshold value is subjected to image mixing processing to generate a mixed image, and is input to the ground object segmentation network for continuous training, and specifically:

[0038] According to two images, two data with pseudo labels are generated, and it is judged whether the ground object categories of the two pseudo labels in the same region are the same or not, pixel by pixel; when the ground object categories are the same, True is marked, otherwise False is marked, and a marking graph containing only True and False is generated;

[0039] According to the region marked as True in the marking graph, the region of the remote sensing image corresponding to one of the pseudo labels is used to replace the same region of the remote sensing image corresponding to the other pseudo label, and a new mixed image is generated;

[0040] The newly generated mixed image is input to the ground object segmentation network for continuous training; when training, the prediction difference between the prediction result of the mixed image and the original label of the image mixing strategy is calculated, and is used to update the gradient parameters of the ground object segmentation network.

[0041] As a preferred technical solution, before judging whether the object categories of the two pseudo labels in the same region are the same or not, the two pseudo labels are subjected to image matching, specifically:

[0042] One of the remote sensing images is removed in a certain direction as a whole through a translation transformation, and then moved in from the opposite direction.

[0043] Another aspect of the application also provides an unmanned aerial vehicle real-time ground object segmentation system based on mixed images, which is applied to the unmanned aerial vehicle real-time ground object segmentation method based on the semi-supervised mixed image algorithm, and includes a training system and a real-time segmentation system.

[0044] The real-time segmentation system includes an embedded system deployed on an unmanned aerial vehicle and a background system;

[0045] The embedded system is deployed with a pre-trained ground object segmentation network, which is used to make predictions according to the compressed remote sensing images obtained by the unmanned aerial vehicle in real time, and the obtained results are visualized and then transmitted to the mobile device of the user;

[0046] The background system is deployed with a pre-trained ground object segmentation network, which receives the complete remote sensing images collected by the unmanned aerial vehicle and inputs them into the segmentation network for prediction, and then transmits the obtained results to the computer of the user after visualization;

[0047] The training system is used to pre-train the ground object segmentation network deployed in the embedded system and the background system according to the remote sensing images collected by the unmanned aerial vehicle, and to calculate the confidence of the labels obtained by the ground object segmentation network, and to perform image mixing processing on the labels with a confidence greater than or equal to a threshold value, to generate mixed images, and to input them into the ground object segmentation network for further training.

[0048] Another aspect of the application also provides a storage medium storing a program, which, when executed by a processor, implements the unmanned aerial vehicle real-time ground object segmentation method based on mixed images.

[0049] Compared with the prior art, the application has the following advantages and beneficial effects:

[0050] (1) The application uses a lightweight network combined with a semi-supervised method based on consistency regularization, which has a small delay on a mobile device, requires only a small amount of labeled data, and can achieve high classification accuracy, not only meeting the real-time requirements of remote sensing ground object segmentation, but also reducing the time and labor costs required for labeling.

[0051] (2) The application uses an image mixing strategy combined with an optimized image mixing matching strategy, which maximizes the use of labeled data and real-time acquired remote sensing data, while also increasing the feature extraction capability of the model for low-proportion ground object categories, improving the segmentation accuracy of the model, and reducing the risk of model overfitting.

[0052] (3) The unmanned aerial vehicle-mounted ground object segmentation network of the present application can compress and transmit the real-time segmentation results to the mobile device of the user, which is simple to operate and has high real-time performance. In addition, the background system can receive the remote sensing influence transmitted by the unmanned aerial vehicle and make predictions to provide the user with more accurate visual results.

[0053] (4) The present application can be applied to real-time environmental monitoring, disaster assessment and urban management and other application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is a flowchart of a real-time ground object segmentation method based on mixed images of an unmanned aerial vehicle according to an embodiment of the present application;

[0055] Figure 2 is a schematic diagram of remote sensing images taken by an unmanned aerial vehicle and manually annotated masks according to an embodiment of the present application;

[0056] Figure 3 is a flowchart of a remote sensing image dataset production process according to an embodiment of the present application;

[0057] Figure 4 is a network architecture of a real-time ground object segmentation method based on mixed images of an unmanned aerial vehicle according to an embodiment of the present application;

[0058] Figure 5 is a training flowchart of a real-time ground object segmentation method based on mixed images of an unmanned aerial vehicle according to an embodiment of the present application;

[0059] Figure 6 is a processing process diagram of an image mixing strategy according to an embodiment of the present application;

[0060] Figure 7 is a schematic diagram of an optimized image matching strategy according to an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to enable personnel in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0062] Embodiment one:

[0063] As shown in Figure 1 , the present embodiment provides a real-time ground object segmentation method based on mixed images of an unmanned aerial vehicle, comprising the following steps:

[0064] S1, collect remote sensing images of the target area and perform partial annotation, as follows:

[0065] Remote sensing images of the target area are photographed by a UAV, and the remote sensing images of the target area all belong to RGB images;

[0066] The types of ground objects in the target area are set according to needs;

[0067] The remote sensing images in a set proportion (in this embodiment, set to 10%) are classified and annotated; the annotation is performed by manual or software (in this embodiment, argis tool is used), and the collected UAV images and the annotation results are as shown in Figure 2

[0068] The annotated mask is presented with a visual effect corresponding to the types of ground objects. For example, the data set is designed to have 6 types of classification, namely, impervious surface, building, low vegetation, tree, car and background.

[0069] S2, making a remote sensing data set of the target area.

[0070] As shown in Figure 3 , according to the annotated data, the training set, the validation set and the test set are divided in proportion (in this embodiment, 7:2:1), and then the unlabeled data is added to the training set to restructure the training set, and finally the remote sensing data set of the target area is obtained.

[0071] S3, constructing a ground object segmentation network based on MobileOne.

[0072] In this embodiment, based on the MobileOne lightweight neural network, a global depthwise convolution module (Global Depthwise Convolution, hereinafter referred to as GDC module) is used instead of the depthwise convolution module in the MobileOne lightweight neural network, and an SENet attention module is introduced after the GDC module.

[0073] ​Specifically, the global depth separable convolution is a special convolution operation that combines the idea of depth separable convolution and performs global processing in the spatial dimension. It can be regarded as a form of applying depth convolution in the spatial dimension of the entire input feature map, which means that for each input channel, a convolution kernel covering the entire input feature map size (i.e. width and height equal to the input feature map size) is used for operation. This is actually equivalent to performing a global average pooling or maximum pooling operation on each channel, but the difference is that the "pooling" kernel here is a learnable parameter rather than a fixed statistical quantity (such as the average or maximum value). This setting allows the model to learn more complex cross-channel relationships while still maintaining a relatively low computational cost. It can efficiently utilize the learning mechanism of parameters and is particularly suitable for task scenarios that require information extraction from the entire input region, such as the feature aggregation stage in image classification. In this way, global context information can be effectively captured while maintaining a relatively lightweight design.

[0074] Specifically, SENet (Squeeze-and-Excitation Networks) is a convolutional neural network architecture that introduces an attention mechanism. The core idea is to enhance useful features and suppress less important features through automatic learning, thereby improving the performance of the model. The key component in SENet is the "squeeze-excitation" (Squeeze-and-Excitation, SE) module, which is used to recalibrate the importance of each channel in the feature map. SENet improves the model's ability to capture subtle differences by introducing a new attention mechanism, while maintaining the simplicity and efficiency of the model structure.

[0075] S4, training the land cover segmentation network using the target area remote sensing dataset, specifically:

[0076] As shown in Figure 4 , a land cover segmentation network (named MobileOne-student model) is constructed, and an initial parameter identical but gradient parameter frozen land cover segmentation network (named MobileOne-teacher model) is copied;

[0077] The labeled remote sensing images in the target area remote sensing dataset are input into the MobileOne-student model, and the unlabeled remote sensing images are simultaneously input into the MobileOne-student model and the MobileOne-teacher model for training;

[0078] As shown in Figure 5As shown, in the training process, the MobileOne-student model uses cross-entropy as the loss function of the labeled remote sensing image to calculate the difference between the prediction result and the true label; uses MSE loss as the loss function of the unlabeled data to calculate the difference between the prediction result of the MobileOne-student model and the prediction result of the MobileOne-student model. At the same time, with the help of the image mixing strategy, the remote sensing image is obtained in real time, after the mixed image is generated by the image mixing strategy, the prediction result of the mixed image by the MobileOne-student model is obtained, and the prediction difference between the prediction result and the original label of the image mixing strategy is calculated, further improving the utilization rate of the remote sensing image label.

[0079] During training, the MobileOne-student model updates the parameters of the MobileOne-teacher model through the EMA (Exponential Moving Average) method.

[0080] The MobileOne-student model with the best effect on the validation set is deployed / updated to the unmanned aerial vehicle or background system for prediction.

[0081] S5, image mixing strategy.

[0082] Only the ground object segmentation network is prone to overfitting, so the embodiment proposes an image mixing strategy to effectively reduce the risk of overfitting and reduce the number of manually labeled remote sensing images, as shown in Figure 6 As shown, it specifically includes the following steps:

[0083] First, calculate the confidence of the real-time ground object segmentation model for the corresponding prediction maps generated by Fig. 1 and Fig. 2, when the confidence of the prediction map is greater than or equal to the set threshold, use argmax to change the two prediction maps into pseudo labels, and consider that the pseudo label can be used as the true label of Fig. 1 and Fig. 2.

[0084] According to the pseudo labels of Fig. 1 and Fig. 2, judge whether the categories of the two pseudo labels are the same pixel by pixel, mark the areas with the same ground object category, mark True if the same, otherwise mark False, and finally generate a marking map (the marking map only contains True and False).

[0085] According to the marking map, a new mixed image is generated, and the label does not need to be changed, specifically: use the area of Fig. 2 to replace the same area of Fig. 1 in the marking map True area to generate Fig. 1 – new.

[0086] In the newly generated mixed image (Fig. 1 - new) is input to the trained ground object segmentation network, reinforcement training.

[0087] Among them, the image mixing strategy image comes from the unlabeled image and the real-time acquisition of remote sensing image, which can maximize the rich data set and improve the segmentation ability of the model.

[0088] S5, optimize image matching strategy.

[0089] Because of the imbalance of the proportion of ground object categories in remote sensing images, some ground objects are difficult to appear in the same area of two remote sensing images at the same time. At the same time, such ground objects usually account for a small proportion, which is easy to increase the risk of model overfitting. Therefore, the embodiment proposes an optimized image matching strategy, which moves the image as a whole to a certain direction by translation transformation, and then moves it from the opposite direction, increasing the probability of successful image mixing. The optimized image matching strategy can increase the diversity of low-proportion ground object categories, thereby reducing the risk of model overfitting. The schematic diagram of the optimized image matching strategy is shown in Figure 7 , where the numbers represent ground object categories, and True and False represent the areas to be replaced and not to be replaced, respectively.

[0090] S6, using the trained ground object segmentation network to perform ground object segmentation on the remote sensing image collected by the unmanned aerial vehicle, and visualizing the prediction result and outputting it to the user's mobile device or computer.

[0091] Further, the ground object segmentation network is carried on the unmanned aerial vehicle to perform real-time prediction on the compressed remote sensing image collected by the unmanned aerial vehicle, and the prediction result is visualized and transmitted to the user's mobile device;

[0092] Further, the ground object segmentation network is carried on the computer background system, receives the complete remote sensing image collected by the unmanned aerial vehicle and performs prediction, and transmits the prediction result to the user's computer after visualization.

[0093] Further, in the training of the ground object segmentation network, set a breakpoint, stop training when the number of labeled remote sensing images in the entire training set is less than a certain proportion (set to 1% in this embodiment); when new labeled remote sensing images are added to the data set, and the number of labeled remote sensing images in the entire training set is higher than a certain proportion, continue training.

[0094] Further, when the segmentation degree of the ground object segmentation network deployed in the background system is much higher than that of the ground object segmentation network deployed on the unmanned aerial vehicle, update the ground object segmentation network deployed on the unmanned aerial vehicle.

[0095] Embodiment two:

[0096] This embodiment builds an intelligent image processing system based on DJI Mavic 3 aircraft and Jetson Nano development board, uses the E-Port interface and E-Port developer kit of Mavic 3 to realize hardware connection and data transmission. The system deploys an efficient image segmentation algorithm, and realizes remote communication combined with a 4G module, which can send the segmentation results to mobile devices in real time, making it convenient for users to view and analyze.

[0097] The hardware devices used this time include DJI Mavic 3, E-Port developer kit and Jetson Nano developer kit. The E-Port provides power supply for Jetson, and provides data through USB line. The E-Port adapter board plays a role in connecting the DJI Mavic 3 host and the Jetson nano development board. The E-Port web axis is used to connect the host of DJI Mavic 3 and the E-Port connection board. The XT30 DC5.5 power line is used to connect the Jetson nano development board and the E-Port connection board, realizing the power supply of the Jetson development board. The USB line and the USB-COTG adapter line are used to realize the data interaction between Jetson and Mavic 3 host. The pictures taken by Mavic host are transmitted to Jetson.

[0098] The segmentation algorithm is deployed on the Jetson nano development board. In order to speed up the inference, the model is converted to onnx to speed up the inference. After receiving the pictures sent by DJI, Jetson calls the segmentation algorithm and gets the result. Jetson is installed with a 4G module, which can send the result to mobile devices through http protocol. The image segmentation result can be viewed on mobile devices.

[0099] Embodiment three:

[0100] In this embodiment, a real-time ground object segmentation system for unmanned aerial vehicles based on a semi-supervised hybrid image algorithm is provided, which includes a training system and a real-time segmentation system;

[0101] The real-time segmentation system includes an embedded unmanned aerial vehicle remote sensing image processing platform system and a background remote sensing image processing Web platform system;

[0102] The embedded unmanned aerial vehicle remote sensing image processing platform system is deployed with a ground object segmentation network trained by using a large number of unlabeled remote sensing images and a part of labeled remote sensing images collected in the early stage. When the unmanned aerial vehicle acquires remote sensing images in real time, it can predict in real time and present the compressed visual results on mobile devices.

[0103] The background remote sensing image processing Web platform system is used for continuously acquiring more remote sensing images taken by the unmanned aerial vehicle, feeding back to the training system through image mixing and other strategies, updating the data set of the unlabeled part in the training system, and updating the model with the best deployment effect to the background remote sensing image processing Web platform system to provide more accurate and high-definition visual results for the user to refer to.

[0104] The compressed visual results can ensure high segmentation accuracy and real-time performance at the same time, and the high-definition visual results can provide more accurate reference results for the user.

[0105] The training system is used for pre-training the embedded unmanned aerial vehicle remote sensing image processing platform system and the ground object segmentation network deployed in the background remote sensing image processing Web platform system according to the remote sensing images collected by the unmanned aerial vehicle, performing confidence calculation on the labels predicted by the ground object segmentation network, performing image mixing processing on the labels with a confidence greater than or equal to a threshold value, generating mixed images, and inputting the mixed images to the ground object segmentation network for continuous training.

[0106] It should be noted that the system provided in the above embodiments is only exemplified by the division of the above functional modules, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure is divided into different functional modules to complete all or part of the functions described above. The system can be applied to the unmanned aerial vehicle real-time ground object segmentation method based on the semi-supervised mixed image algorithm of the above embodiments.

[0107] In another embodiment of the present application, a storage medium storing a program is also provided, and the program is executed by a processor to implement the unmanned aerial vehicle real-time ground object segmentation method based on the semi-supervised mixed image algorithm, specifically:

[0108] S1, collecting remote sensing images of a target area and performing partial labeling, and making a target area remote sensing data set;

[0109] S2, constructing a ground object segmentation network based on MobileOne, and training the ground object segmentation network by using the target area remote sensing data set;

[0110] S3, using the trained ground object segmentation network to perform ground object segmentation on remote sensing images collected by the unmanned aerial vehicle, visualizing the obtained labels, and outputting the labels to a user device;

[0111] S4, performing confidence calculation on the obtained labels, performing image mixing processing on the labels with a confidence greater than or equal to a threshold value, generating mixed images, and inputting the mixed images to the ground object segmentation network for continuous training.

[0112] It should be understood that portions of the present application can be implemented in hardware, software, firmware, or combinations thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, implementation can be with any or a combination of the following technologies, which are all well known in the art: a discrete logic circuit having logic gates for implementing logic functions upon an application of data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0113] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications, etc. made without departing from the spirit and principles of the present application are equivalent replacement modes and are included in the protection scope of the present application.

Claims

1. A method for real-time ground object segmentation of a UAV based on mixed imagery, the method comprising: The method comprises the following steps: Collect remote sensing images of a target area and perform partial labeling to produce a remote sensing dataset of the target area; Construct a MobileOne-based feature segmentation network, and train the network using the remote sensing dataset of the target area, specifically as follows: Input labeled remote sensing images into a feature segmentation network for training; input unlabeled remote sensing images into the feature segmentation network and another feature segmentation network with the same initial parameters and gradient parameters frozen for training; Calculate the difference between the predicted results of the feature segmentation network and the true labels for the labeled remote sensing images using cross-entropy; calculate the difference between the predicted results of the feature segmentation network and the gradient parameter frozen feature segmentation network for the same remote sensing image using MSE Loss; and update the gradient parameters of the feature segmentation network using the two differences; The feature segmentation network updates the parameters of the gradient parameter frozen feature segmentation network through an exponential moving average method; Use the trained feature segmentation network to perform feature segmentation on remote sensing images collected by a UAV, visualize the obtained labels, and output the labels to a user device; Calculate the confidence of the obtained labels, perform image mixing processing on labels with a confidence greater than or equal to a threshold, generate mixed images, and input the mixed images into the feature segmentation network for further training, specifically as follows: According to two images, generate two pseudo-labeled data, and determine whether the feature categories of the two pseudo-labeled data in the same region are the same pixel by pixel; if the feature categories are the same, mark True, otherwise mark False, to generate a label map containing only True and False; According to the regions marked as True in the label map, replace the same regions of the remote sensing image corresponding to the other pseudo-labeled data with the regions of the remote sensing image corresponding to one of the pseudo-labeled data to generate a new mixed image; Input the newly generated mixed image into the feature segmentation network for further training; Calculate the prediction difference between the prediction results of the mixed image and the original labels of the image mixing strategy during training to update the gradient parameters of the feature segmentation network. 2.The hybrid image based real-time ground object segmentation method for UAVs according to claim 1, wherein, The method of collecting remote sensing images of a target area and performing partial labeling to produce a remote sensing dataset of the target area specifically comprises: Use a UAV to capture remote sensing images of the target area; Set the feature categories of the target area according to needs; Classify and label remote sensing images in a set proportion; Divide the labeled remote sensing image data into a training set, a validation set, and a test set; Add unlabeled data to the training set, restructure the training set, and finally obtain a remote sensing dataset of the target area. 3.The hybrid image based real-time ground object segmentation method for UAVs according to claim 1, wherein, The method of constructing a MobileOne-based feature segmentation network specifically comprises: Based on the MobileOne lightweight neural network, use a global depth separable convolution module to replace the depth convolution module in the MobileOne lightweight neural network, and introduce an SE-Net attention module after the global depth separable convolution module. 4.The method of claim 1, wherein, Setting a breakpoint in the training of the feature segmentation network, stopping training when the proportion of labeled remote sensing images in the entire training set is less than a certain proportion; when new labeled remote sensing images are added to the dataset, and the proportion of labeled remote sensing images in the entire training set is higher than a certain proportion, then continue training.

5. The method of claim 1, wherein the method is based on a hybrid image. The obtained label is visualized and output to the user equipment, specifically: Deploying the feature segmentation network on the unmanned aerial vehicle, real-time prediction of the compressed remote sensing image collected by the unmanned aerial vehicle, and visualizing the prediction result and transmitting it to the user's mobile device; Deploying the feature segmentation network in the background system, receiving the complete remote sensing image collected by the unmanned aerial vehicle and performing prediction, and transmitting the prediction result to the user's computer after visualizing it. 6.The method of claim 1, wherein, Before judging whether the feature categories of the two pseudo labels in the same region are the same, the two pseudo labels are matched, specifically: Translate one of the remote sensing images to move it in a certain direction, and then move it back in the opposite direction. 7.A hybrid image based real-time UAV ground object segmentation system, characterized in that, A kind of unmanned aerial vehicle real-time feature segmentation method based on mixed image applied to any one of claims 1-6, comprising a training system and a real-time segmentation system; The real-time segmentation system includes an embedded system deployed on an unmanned aerial vehicle and a background system; The embedded system is deployed with a pre-trained feature segmentation network, which predicts according to the compressed remote sensing image obtained by the unmanned aerial vehicle in real time, and transmits the obtained result to the user's mobile device after visualization; The background system is deployed with a pre-trained feature segmentation network, which receives the complete remote sensing image collected by the unmanned aerial vehicle and inputs it into the feature segmentation network for prediction, and transmits the obtained result to the user's computer after visualization; The training system is used to pre-train the feature segmentation network deployed in the embedded system and the background system according to the remote sensing image collected by the unmanned aerial vehicle, and to calculate the confidence of the label obtained by the feature segmentation network, and to perform image mixing processing on the label with confidence greater than or equal to the threshold value, to generate mixed images, and to input them into the feature segmentation network for further training.

8. A storage medium storing a program, characterized by comprising: The program is executed by the processor to realize the unmanned aerial vehicle real-time feature segmentation method based on mixed image of any one of claims 1-6.

Images