Unmanned aerial vehicle real-time ground object segmentation method based on mixed images

Abstract

The invention discloses an unmanned aerial vehicle real-time ground object segmentation method based on a mixed image, and the method comprises the steps: collecting a remote sensing image of a target region, carrying out the partial labeling, and making a data set; constructing a ground feature segmentation network and performing training; performing real-time surface feature segmentation by using a surface feature segmentation network, visualizing the obtained label and outputting the visualized label to user equipment; and performing confidence calculation on the obtained labels, performing image mixing processing on the labels of which the confidence is greater than or equal to a threshold value, generating a mixed image, and inputting the mixed image into the ground feature segmentation network for continuous training. According to the invention, the lightweight neural network based on consistency regularization semi-supervised method is adopted, the delay on the mobile device is small, the method can be realized only by a small amount of label data, and the classification precision is high. In addition, an image mixing and matching strategy is adopted, data is utilized to the maximum extent, the feature extraction capacity of the model for low-proportion surface feature categories is improved, the model segmentation precision is improved, and the risk of model over-fitting is reduced.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing image ground object segmentation, and in particular relates to a real-time ground object segmentation method for an unmanned aerial vehicle based on mixed images. Background Art

[0002] Real-time remote sensing object segmentation can instantly and accurately identify and classify various object categories in remote sensing images, providing key information for dynamic monitoring and decision support. Current real-time object segmentation mainly relies on fully supervised object segmentation methods, which use a large amount of labeled data to train deep learning models to achieve accurate classification of each pixel in the image. UNet++ is a simple encoder-decoder structure, and by establishing multi-level jump connections between the encoder and decoder, the feature transfer and fusion are enhanced. This design allows the network to better capture features at different levels and improve the accuracy of segmentation. Secondly, UNet++ has a certain degree of flexibility, which allows users to adjust the depth and width of the network as needed to adapt to different application scenarios and computing resource constraints. Although the use of this model to achieve real-time object segmentation can guarantee a high degree of accuracy, it takes a lot of time to train the model to obtain accurate segmentation results. Whenever the region is changed, it is necessary to collect and annotate the data again, and train the model according to the characteristics of the new region, which is not only time-consuming, but also poses a great challenge to real-time performance, increasing the complexity and cost of the application. Since the UNet++ network structure contains multi-level jump connections, it takes a long time to obtain the results during prediction, which greatly reduces the timeliness of real-time object segmentation.

[0003] For this type of model with a complex network structure, some researchers have begun to design lightweight networks. Lightweight networks can reduce the time required for model prediction and solve the problem of high latency in traditional models for real-time object segmentation. The MobileNet series of networks is a set of lightweight deep learning models designed for mobile and embedded vision applications. The design goal of these models is to significantly reduce the demand for computing resources while maintaining reasonable accuracy, thereby achieving faster inference speed and lower power consumption. MobileNetV2 reduces unnecessary DW convolution weights and improves the ability of feature representation by introducing inverted residuals and linear bottlenecks. MobileNetV3 uses neural architecture search (NAS) to find an architecture that strikes the best balance between computational efficiency and model performance. In addition, MobileNetV3 also introduces the hard Swish activation function, which is a simplified version of the approximate Swish function that maintains the advantages of nonlinear activation and reduces computational complexity. Although lightweight networks can significantly reduce the time required for prediction, this ensures the timeliness of real-time remote sensing object segmentation. However, a large number of accurate labels are still required when training the model. This type of method requires a lot of manpower and time costs to perform one-to-one labeling of remote sensing images.

[0004] With the rapid development of deep learning, some researchers have tried to use some labeled remote sensing images and a large number of unlabeled remote sensing images to train models. At present, the research on semi-supervised algorithms in remote sensing object segmentation mainly focuses on the following three directions: The first is based on pseudo-labeling. This method trains an initial model on a small amount of labeled data, then uses this model to predict unlabeled data, and uses the prediction results as "pseudo-labels" to further train the model. The core of this method is how to effectively generate high-quality pseudo-labels to reduce the impact of error propagation. ST++ adopts a strategy of image-level screening and progressive utilization. Through selection and sorting, reliable pseudo-labels are selected for re-training to reduce the impact of erroneous pseudo-labels.

[0005] The second is a method based on consistency regularization. This method introduces a consistency regularization term to ensure the prediction consistency of the model under data augmentation transformation. Even when the input data changes slightly, the model should give similar prediction results. The current semi-supervised remote sensing semantic algorithm is based on consistency regularization. The CPS method designs a very simple and high-performance semi-supervised framework. The CPS method uses two networks with the same structure but different initializations, and adds constraints so that the outputs of the two networks for the same sample are similar. Specifically, the pseudolabel generated by the current network will be used as the target of the prediction of the other network. This process can be supervised by cross entropy loss.

[0006] The third is a method based on generative adversarial networks (GANs), which uses generative adversarial networks to generate synthetic data to expand the training set. GANs consist of two parts: a generator and a discriminator. The generator attempts to generate synthetic samples that are consistent with the distribution of real data, while the discriminator attempts to distinguish between real data and synthetic data. AdvSemiSeg is a method that uses the idea of ​​generative adversarial networks (GANs) to enhance the semi-supervised learning framework. AdvSemiSeg uses both labeled and unlabeled images during training. When using labeled images, the segmentation network is subject to both the standard cross entropy loss based on the ground truth label and the adversarial loss based on the discriminant network. When using unlabeled images, the segmentation network is used to obtain a preliminary segmentation result, and then the preliminary segmentation result is sent to the discriminant network to obtain a confidence map. The confidence map is used as a supervisory signal, and the preliminary segmentation prediction result is masked as a label. The segmentation network is trained using a self-learning method. Benefiting from the discriminant network, AdvSemiSeg has achieved good segmentation results on multiple datasets such as Pascal VOC and cityspace.

[0007] The above-mentioned semi-supervised algorithm for remote sensing object segmentation has the following disadvantages: (1) When models such as UNet++ achieve real-time object segmentation, their prediction speed is slow, resulting in a certain delay and poor timeliness. This is because the network complexity of models such as UNet++ is high, which makes the model calculation complexity high, the time to process each frame of the image is long, and the real-time performance is poor.

[0008] (2) The segmentation accuracy of lightweight networks such as MobileNet is lower than that of more complex models such as UNet++. When faced with complex or variable data, their generalization ability may not be as good as that of more complex models. This is because lightweight networks use techniques such as deep separable convolution to reduce the amount of computation, but this also limits the model's ability to capture complex features. When dealing with variable data, lightweight networks may not be able to capture all categories of features well due to their limited feature expression capabilities, which affects generalization ability.

[0009] (3) The pseudo-label-based semi-supervised method relies on a high-quality initial model. The proportion of remote sensing objects in images is usually unbalanced. If there are fewer samples in a certain category, the pseudo-label method may cause that category to be ignored or misclassified. This is because a high-quality initial model is a prerequisite for generating reliable pseudo-labels. If the initial model quality is not high, the generated pseudo-labels may contain more errors. When the initial model quality is high, higher network complexity is inevitably required, which will also increase the time it takes to generate pseudo-labels, thereby reducing real-time performance.

[0010] (4) The semi-supervised object segmentation method based on generative adversarial networks requires a lot of computing resources, which is not conducive to real-time remote sensing object segmentation. The masks generated by GAN may be inconsistent between different domains (such as remote sensing images in different regions and seasons), affecting the robustness of the model. This is because the GAN method requires training the generator and discriminator, which has high computational complexity and is not suitable for real-time applications.

[0011] (5) The model training time of the semi-supervised remote sensing object segmentation method based on consistency regularization is long and it is very sensitive to the choice of hyperparameters. Improper hyperparameter settings may lead to unstable training or poor results. This is because the model training using the consistency regularization method requires a long time and the consistency regularization method may encounter challenges when dealing with cross-domain problems, especially when the real-time segmentation area exceeds the area where the training data is located. Summary of the invention

[0012] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide a real-time object segmentation method for UAVs based on hybrid images. The method combines the advantages of lightweight networks (such as MobileOne), maintains prediction consistency under data enhancement conditions, makes full use of unlabeled data, improves the robustness and generalization ability of the model, and can achieve rapid response in environmental monitoring, disaster assessment, urban management and other application scenarios.

[0013] In order to achieve the above object, the present invention adopts the following technical solutions: One aspect of the present invention provides a method for real-time object segmentation of an unmanned aerial vehicle based on mixed images, comprising the following steps: Collect remote sensing images of the target area and mark them partially to create a remote sensing dataset of the target area; Build a MobileOne-based object segmentation network and use the target area remote sensing dataset for training; Use the trained object segmentation network to segment the remote sensing images collected by the drone, visualize the obtained labels and output them to the user's device; The confidence of the obtained labels is calculated, and the labels with confidence greater than or equal to the threshold are subjected to image mixing processing to generate mixed images, which are then input into the ground object segmentation network for further training.

[0014] As a preferred technical solution, the remote sensing images of the target area are collected and partially annotated, and a remote sensing data set of the target area is prepared, specifically: Use drones to capture remote sensing images of target areas; Set the type of features in the target area as needed; Classify and label remote sensing images of a set scale; Divide the training set, validation set and test set according to the labeled remote sensing image data; The unlabeled data is added to the training set, the training set is reconstructed, and finally the remote sensing dataset of the target area is obtained.

[0015] As a preferred technical solution, the construction of the MobileOne-based feature segmentation network is specifically as follows: Based on the MobileOne lightweight neural network, the global depthwise separable convolution module is used to replace the deep convolution module in the MobileOne lightweight neural network, and the SENet attention module is introduced after the global depthwise separable convolution module.

[0016] As a preferred technical solution, the object segmentation network is trained using the remote sensing data set of the target area, specifically: Inputting labeled remote sensing images into a land object segmentation network for training; inputting unlabeled remote sensing images into the land object segmentation network and another land object segmentation network with the same initial parameters and frozen gradient parameters for training; For labeled remote sensing images, the cross entropy is used to calculate the gap between the predicted results of the object segmentation network and the actual label; for the same remote sensing image, the MSE Loss is used to calculate the gap between the predicted results of the object segmentation network and the object segmentation network with frozen gradient parameters; the gradient parameters of the object segmentation network are updated using the two gaps; The object segmentation network updates the parameters of the object segmentation network whose gradient parameters are frozen by an exponential moving average method.

[0017] As an optimal technical solution, breakpoints are set in the training of the object segmentation network. When the number of labeled remote sensing images in the entire training set is lower than a certain proportion, training is stopped; when new labeled remote sensing images are subsequently added to the data set, and the number of labeled remote sensing images in the entire training set is higher than a certain proportion, training is continued.

[0018] As a preferred technical solution, the pseudo-label with a confidence level greater than or equal to a threshold is output as a true label to the user's mobile device or computer, specifically: Deploy a ground segmentation network on the drone to perform real-time prediction on the compressed remote sensing images collected by the drone, and visualize the prediction results and transmit them to the user's mobile device; The ground segmentation network is deployed in the background system to receive the complete remote sensing images collected by the drone and make predictions. The prediction results are visualized and transmitted to the user's computer.

[0019] As a preferred technical solution, the pseudo labels with confidence greater than or equal to the threshold are subjected to image mixing processing to generate mixed images, and the mixed images are input into the ground object segmentation network for further training, specifically: Based on the two images, two pseudo-labeled data are generated. The two pseudo-labeled data are judged pixel by pixel whether the object categories in the same area are the same. If the object categories are the same, they are marked as True, otherwise they are marked as False, and a labeled map containing only True and False is generated. According to the area marked as True in the labeling map, the area of ​​the remote sensing image corresponding to one of the pseudo labels is used to replace the same area of ​​the remote sensing image corresponding to the other pseudo label to generate a new mixed image; The newly generated mixed image is input into the object segmentation network and training is continued. During training, the prediction gap between the prediction result of the mixed image and the original label of the image mixing strategy is calculated to update the gradient parameters of the object segmentation network.

[0020] As a preferred technical solution, before determining pixel by pixel whether the two pseudo labels have the same ground object category in the same area, image matching is performed on the two pseudo labels, specifically: Through translation transformation, one of the remote sensing images is removed as a whole in a certain direction and then moved in from the opposite direction.

[0021] Another aspect of the present invention further provides a UAV real-time object segmentation system based on hybrid images, which is applied to the above-mentioned UAV real-time object segmentation method based on semi-supervised hybrid image algorithm, including a training system and a real-time segmentation system; The real-time segmentation system includes an embedded system and a background system deployed on the drone; The embedded system deploys a pre-trained object segmentation network, performs prediction based on the compressed remote sensing images acquired by the drone in real time, and transmits the obtained results to the user's mobile device after visualization; The backend system is deployed with a pre-trained ground object segmentation network, which receives the complete remote sensing image collected by the drone and inputs it into the segmentation network for prediction, and then visualizes the obtained result and transmits it to the user's computer; The training system is used to pre-train the object segmentation network deployed in the embedded system and the background system according to the remote sensing images collected by the drone, and to calculate the confidence of the labels predicted by the object segmentation network, perform image mixing processing on the labels whose confidence is greater than or equal to the threshold, generate a mixed image, and input it into the object segmentation network for further training.

[0022] Another aspect of the present invention further provides a storage medium storing a program, which, when executed by a processor, implements the above-mentioned method for real-time object segmentation of a drone based on mixed images.

[0023] Compared with the prior art, the present invention has the following advantages and beneficial effects: (1) This application uses a lightweight network combined with a semi-supervised method based on consistency regularization. It has low latency on mobile devices and only requires a small amount of labeled data to be implemented. It also has high classification accuracy. It can not only meet the real-time requirements of remote sensing object segmentation, but also reduce the time and labor costs required for labeling.

[0024] (2) This application uses an image mixing strategy in combination with an optimized image mixing matching strategy to maximize the use of labeled data and remote sensing data acquired in real time. At the same time, it can also increase the model's feature extraction capabilities for low-proportion land object categories, improve the model's segmentation accuracy, and reduce the risk of model overfitting.

[0025] (3) The present application can utilize drones to carry ground segmentation networks and compress and transmit the real-time segmentation results to the user's mobile device. The operation is simple and the real-time performance is high. In addition, the background system can be used to receive the remote sensing impact transmitted by the drone and make predictions, providing users with high-definition, complete and more accurate visualization results.

[0026] (4) This application can be applied to a variety of application scenarios such as real-time environmental monitoring, disaster assessment and urban management. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart of a method for real-time object segmentation of a UAV based on mixed images according to an embodiment of the present invention; Figure 2 is a schematic diagram of a remote sensing image taken by a drone and a manually annotated mask according to an embodiment of the present invention; Figure 3 is a flowchart of remote sensing image data set production according to an embodiment of the present invention; Figure 4 It is a network architecture of a mixed image-based real-time object segmentation method for unmanned aerial vehicles according to an embodiment of the present invention; Figure 5 It is a training flow chart of a method for real-time object segmentation of a drone based on mixed images according to an embodiment of the present invention; Figure 6 is a processing diagram of the image mixing strategy of an embodiment of the present invention; Figure 7 Schematic diagram of an optimized image matching strategy according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0029] Embodiment 1: like Figure 1 As shown, this embodiment provides a method for real-time object segmentation of a drone based on mixed images, comprising the following steps: S1. Collect remote sensing images of the target area and mark them partially, as follows: Using a drone to take remote sensing images of a target area, wherein the remote sensing images of the target area are all RGB images; Set the type of features in the target area as needed; The remote sensing images of a set ratio (in this embodiment, set to 10%) are classified and labeled; the labeling is performed manually or by software (in this embodiment, the argis tool is used). The collected drone images and the labeling results are as follows: Figure 2 As shown; The labeled masks are visualized to show the corresponding types of objects. For example, the dataset is designed to have 6 types of objects, namely: impervious surface, building, low vegetation, tree, car and background.

[0030] S2. Produce remote sensing datasets for the target area.

[0031] like Figure 3 As shown, the training set, validation set, and test set are divided according to the ratio (7:2:1 in this embodiment) of the labeled data, and then the unlabeled data are added to the training set, and the training set is reconstructed to finally obtain the remote sensing data set of the target area.

[0032] S3. Build a terrain segmentation network based on MobileOne.

[0033] In this embodiment, based on the MobileOne lightweight neural network, a global depthwise separable convolution module (Global Depthwise Convolution, hereinafter referred to as GDC module) is used to replace the deep convolution module in the MobileOne lightweight neural network, and a SENet attention module is introduced after the GDC module.

[0034] Specifically, global depthwise separable convolution is a special convolution operation that combines the idea of ​​depthwise separable convolution and performs global processing in the spatial dimension. It can be seen as a form of applying depthwise convolution in the spatial dimension of the entire input feature map, which means that for each input channel, a convolution kernel covering the entire input feature map size (i.e., width and height equal to the input feature map size) is used for operation. This is actually equivalent to performing a global average pooling or maximum pooling operation on each channel, but the difference is that the "pooling" kernel here is a learnable parameter rather than a fixed statistic (such as the mean or maximum value). This setting allows the model to learn more complex cross-channel relationships while still maintaining a low computational cost. It can efficiently utilize the learning mechanism of parameters and is particularly suitable for task scenarios that require information to be extracted from the entire input area, such as the feature aggregation stage in image classification. In this way, global context information can be effectively captured while maintaining a relatively lightweight design.

[0035] Specifically, SENet (Squeeze-and-Exitation Networks) is a convolutional neural network architecture that introduces an attention mechanism. The core idea is to improve the performance of the model by enhancing useful features and suppressing less important features through automatic learning. The key component in SENet is the Squeeze-and-Excitation (SE) module, which is used to recalibrate the importance of each channel in the feature map. SENet improves the model's ability to capture subtle differences by introducing a new attention mechanism, while maintaining the simplicity and efficiency of the model structure.

[0036] S4. Use the target area remote sensing data set to train the ground object segmentation network, specifically: like Figure 4 As shown, a feature segmentation network (named MobileOne-student model) is constructed, and a feature segmentation network (named MobileOne-teacher model) with the same initial parameters but frozen gradient parameters is copied; Input the labeled remote sensing images in the target area remote sensing dataset into the MobileOne-student model, and input the unlabeled remote sensing images into both the MobileOne-student model and the MobileOne-teacher model for training; like Figure 5 As shown in the figure, during the training process, the MobileOne-student model uses cross entropy as the loss function of labeled remote sensing images to calculate the gap between the predicted results and the true labels; and uses MSE loss as the loss function of unlabeled data to calculate the gap between the MobileOne-student model and the MobileOne-student model's predicted results. At the same time, with the help of the image mixing strategy, the remote sensing images are acquired in real time. After the image mixing strategy generates the mixed image, the prediction results of the MobileOne-student model for the mixed image are obtained, and then the prediction gap between the predicted results and the original labels of the image mixing strategy is calculated, further improving the utilization rate of remote sensing image labels.

[0037] During training, the MobileOne-student model updates the parameters of the MobileOne-teacher model through the EMA (exponential moving average) method.

[0038] The MobileOne-student model with the best performance on the validation set is deployed / updated to the drone or backend system for prediction.

[0039] S5. Image mixing strategy.

[0040] Relying solely on the object segmentation network is prone to overfitting. Therefore, this embodiment proposes an image mixing strategy to effectively reduce the risk of overfitting and reduce the number of manually annotated remote sensing images. Figure 6 As shown, the specific steps include: First, the confidence of the corresponding predicted images generated by the real-time object segmentation model for Fig.1 and Fig.2 is calculated. When the confidence of the predicted image is greater than or equal to the set threshold, argmax is used to convert the two predicted images into pseudo labels, and it is considered that the pseudo labels can be used as the real labels of Fig.1 and Fig.2.

[0041] According to the pseudo labels in Fig.1 and Fig.2, we determine whether the two pseudo labels are the same pixel by pixel. We mark the areas with the same ground object category. If they are the same, we mark them as True, otherwise they are False. Finally, we generate a label map (the label map only contains True and False).

[0042] A new mixed image is generated according to the label map, and the label does not need to be changed. Specifically, the area marked as True in the label map is replaced by the area in Fig.2 to generate Fig.1 – new.

[0043] The newly generated mixed image (Fig.1 – new) is input into the trained object segmentation network to strengthen the training.

[0044] The images of the image mixing strategy come from unlabeled images and remote sensing images acquired in real time, which can maximize the enrichment of the data set and improve the segmentation ability of the model.

[0045] S5. Optimize image matching strategy.

[0046] Since remote sensing images have an unbalanced proportion of object categories, it is difficult for some objects to appear in the same area of ​​two remote sensing images at the same time. At the same time, such objects usually account for a small proportion, which easily increases the risk of model overfitting. Therefore, this embodiment proposes an optimized image matching strategy, which removes the entire image in a certain direction through a translation transformation, and then moves it in from the opposite direction, thereby increasing the probability of successful image mixing. This optimized image matching strategy can increase the diversity of low-proportion object categories, thereby reducing the risk of model overfitting. The schematic diagram of the optimized image matching strategy is shown in the figure. Figure 7 As shown, the numbers represent the feature categories, and True and False indicate the areas to be replaced and not to be replaced, respectively.

[0047] S6. Use the trained object segmentation network to segment the remote sensing images collected by the drone, visualize the prediction results and output them to the user's mobile device or computer.

[0048] Furthermore, a ground segmentation network is mounted on the drone to perform real-time prediction on the compressed remote sensing images collected by the drone, and the prediction results are visualized and transmitted to the user's mobile device; Furthermore, a ground segmentation network is installed in the computer background system to receive the complete remote sensing images collected by the drone and make predictions, and the prediction results are visualized and transmitted to the user's computer.

[0049] Furthermore, breakpoints are set in the training of the object segmentation network, and training is stopped when the number of labeled remote sensing images in the entire training set is lower than a certain proportion (set to 1% in this embodiment); when new labeled remote sensing images are subsequently added to the data set, and the number of labeled remote sensing images in the entire training set is higher than a certain proportion, training is continued.

[0050] Furthermore, when the segmentation degree of the feature segmentation network deployed by the background system is much higher than that of the feature segmentation network deployed on the UAV, the feature segmentation network deployed on the UAV is updated.

[0051] Embodiment 2: This example builds an intelligent image processing system based on the DJI Mavic 3 aircraft and the Jetson Nano development board, using the Mavic 3's E-Port interface and the E-Port developer kit to achieve hardware connection and data transmission. The system deploys an efficient image segmentation algorithm and combines it with a 4G module to achieve remote communication. It can send segmentation results to mobile devices in real time, making it convenient for users to view and analyze.

[0052] The hardware equipment used this time includes DJI Mavic 3, E-Port developer kit and Jetson Nano developer kit. Jetson is powered by E-Port and data is provided by USB cable. E-Port adapter board is used to connect DJI Mavic 3 host and Jetson nano development board. Use E-Port network axis to connect DJI Mavic 3 host and E-Port connection board. Use XT30 to DC5.5 power cable to connect Jetson nano development board and E-Port connection board to realize power supply for Jetson development board. Use USB cable and USB-COTG adapter cable to realize data exchange between Jetson and Mavic 3 host, and the pictures taken by Mavic host are transmitted to Jetson.

[0053] The segmentation algorithm is deployed on the Jetson nano development board. In order to speed up the inference, the model is converted to onnx to speed up the inference. After receiving the picture from DJI, Jetson calls the segmentation algorithm and gets the result. Jetson is equipped with a 4G module, which can send the result to the mobile device through the http protocol. The image segmentation results can be viewed on the mobile device.

[0054] Embodiment three: In this embodiment, a UAV real-time object segmentation system based on a semi-supervised hybrid image algorithm is provided, the system comprising a training system and a real-time segmentation system; The real-time segmentation system includes an embedded UAV remote sensing image processing platform system and a background remote sensing image processing Web platform system; The embedded UAV remote sensing image processing platform system is deployed with a ground object segmentation network trained using remote sensing images collected and partially annotated in the early stage and a large number of unannotated remote sensing images. When the UAV acquires remote sensing images in real time, it predicts in real time and presents the compressed visualization results on the mobile device.

[0055] The background remote sensing image processing Web platform system is used to continuously obtain more remote sensing images taken by drones, feed them back to the training system through strategies such as image blending, update the data set of the unlabeled part in the training system, and update the model with the best deployment effect to the background remote sensing image processing Web platform system to provide more accurate and high-definition visualization results for user reference.

[0056] The compressed visualization results can ensure both high segmentation accuracy and real-time performance, while high-definition visualization results can provide users with more accurate reference results.

[0057] The training system is used to pre-train the object segmentation network deployed in the embedded UAV remote sensing image processing platform system and the background remote sensing image processing Web platform system according to the remote sensing images collected by the UAV, and calculate the confidence of the labels predicted by the object segmentation network, perform image mixing processing on the labels whose confidence is greater than or equal to the threshold, generate a mixed image, and input it into the object segmentation network for further training.

[0058] It should be noted here that the system provided in the above embodiment is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure is divided into different functional modules to complete all or part of the functions described above. The system can be applied to the real-time object segmentation method of drone based on semi-supervised hybrid image algorithm in the above embodiment.

[0059] In another embodiment of the present application, a storage medium is provided, which stores a program. When the program is executed by a processor, a method for real-time object segmentation of a drone based on a semi-supervised hybrid image algorithm is implemented, specifically: S1. Collect remote sensing images of the target area, annotate them partially, and create a remote sensing dataset of the target area; S2, build a ground segmentation network based on MobileOne and use the remote sensing dataset of the target area for training; S3, using the trained object segmentation network to segment the remote sensing images collected by the drone, and visualize the obtained labels and output them to the user device; S4. Calculate the confidence of the obtained labels, perform image mixing processing on the labels whose confidence is greater than or equal to the threshold, generate a mixed image, and input it into the ground object segmentation network to continue training.

[0060] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0061] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A real-time object segmentation method for UAV based on mixed images, characterized in that: The steps include: Collect remote sensing images of the target area and mark them partially to create a remote sensing dataset of the target area; Build a MobileOne-based object segmentation network and use the target area remote sensing dataset for training; Use the trained object segmentation network to segment the remote sensing images collected by the drone, visualize the obtained labels and output them to the user's device; The confidence of the obtained labels is calculated, and the labels with confidence greater than or equal to the threshold are subjected to image mixing processing to generate mixed images, which are then input into the ground object segmentation network for further training.

2. The method for real-time object segmentation of a drone based on mixed images according to claim 1 is characterized in that: The remote sensing images of the target area are collected and partially annotated, and a remote sensing data set of the target area is prepared, specifically: Use drones to capture remote sensing images of target areas; Set the type of features in the target area as needed; Classify and label remote sensing images of a set scale; Divide the training set, validation set and test set according to the labeled remote sensing image data; The unlabeled data is added to the training set, the training set is reconstructed, and finally the remote sensing dataset of the target area is obtained.

3. The method for real-time object segmentation of a drone based on mixed images according to claim 1 is characterized in that: The construction of the MobileOne-based object segmentation network is specifically as follows: Based on the MobileOne lightweight neural network, the global depthwise separable convolution module is used to replace the deep convolution module in the MobileOne lightweight neural network, and the SENet attention module is introduced after the global depthwise separable convolution module.

4. The method for real-time object segmentation of a drone based on mixed images according to claim 1 is characterized in that: The object segmentation network is trained using the target area remote sensing dataset, specifically: Inputting labeled remote sensing images into a land object segmentation network for training; inputting unlabeled remote sensing images into the land object segmentation network and another land object segmentation network with the same initial parameters and frozen gradient parameters for training; For labeled remote sensing images, the cross entropy is used to calculate the gap between the predicted results of the object segmentation network and the actual label; for the same remote sensing image, the MSE Loss is used to calculate the gap between the predicted results of the object segmentation network and the object segmentation network with frozen gradient parameters; the gradient parameters of the object segmentation network are updated using the two gaps; The object segmentation network updates the parameters of the object segmentation network whose gradient parameters are frozen by an exponential moving average method.

5. The method for real-time object segmentation of a drone based on mixed images according to claim 1, characterized in that: Breakpoints are set in the training of the object segmentation network. When the number of labeled remote sensing images in the entire training set is lower than a certain proportion, training is stopped; when new labeled remote sensing images are added to the data set, and the number of labeled remote sensing images in the entire training set is higher than a certain proportion, training continues.

6. The method for real-time object segmentation of a drone based on mixed images according to claim 1, characterized in that: The method of outputting the pseudo-label with a confidence level greater than or equal to a threshold as a true label to the user's mobile device or computer is specifically as follows: Deploy a ground segmentation network on the drone to perform real-time prediction on the compressed remote sensing images collected by the drone, and visualize the prediction results and transmit them to the user's mobile device; The ground segmentation network is deployed in the background system to receive the complete remote sensing images collected by the drone and make predictions. The prediction results are visualized and transmitted to the user's computer.

7. The method for real-time object segmentation of a drone based on mixed images according to claim 1, characterized in that: The pseudo labels with confidence greater than or equal to the threshold are subjected to image mixing processing to generate mixed images, and are input into the ground object segmentation network for further training, specifically: Based on the two images, two pseudo-labeled data are generated. The two pseudo-labeled data are judged pixel by pixel whether the object categories in the same area are the same. If the object categories are the same, they are marked as True, otherwise they are marked as False, and a labeled map containing only True and False is generated. According to the area marked as True in the labeling map, the area of ​​the remote sensing image corresponding to one of the pseudo labels is used to replace the same area of ​​the remote sensing image corresponding to the other pseudo label to generate a new mixed image; After the newly generated mixed image is input into the object segmentation network, the training continues; During training, the prediction gap between the prediction result of the mixed image and the original label of the image mixing strategy is calculated to update the gradient parameters of the object segmentation network.

8. The method for real-time object segmentation of a drone based on mixed images according to claim 1, characterized in that: Before judging pixel by pixel whether the two pseudo labels have the same object category in the same area, the two pseudo labels are image matched, specifically: Through translation transformation, one of the remote sensing images is removed as a whole in a certain direction and then moved in from the opposite direction.

9. A real-time object segmentation system for UAV based on mixed images, characterized in that: A method for real-time object segmentation of a UAV based on a semi-supervised hybrid image algorithm applied to any one of claims 1 to 8, comprising a training system and a real-time segmentation system; The real-time segmentation system includes an embedded system and a background system deployed on the drone; The embedded system deploys a pre-trained object segmentation network, performs prediction based on the compressed remote sensing images acquired by the drone in real time, and transmits the obtained results to the user's mobile device after visualization; The backend system is deployed with a pre-trained ground object segmentation network, which receives the complete remote sensing image collected by the drone and inputs it into the segmentation network for prediction, and then visualizes the obtained result and transmits it to the user's computer; The training system is used to pre-train the object segmentation network deployed in the embedded system and the background system according to the remote sensing images collected by the drone, and to calculate the confidence of the labels predicted by the object segmentation network, perform image mixing processing on the labels whose confidence is greater than or equal to the threshold, generate a mixed image, and input it into the object segmentation network for further training.

10. A storage medium storing a program, characterized in that: When the program is executed by the processor, the method for real-time object segmentation of a drone based on mixed images as described in any one of claims 1 to 8 is implemented.

Images