A method for UAV visual scene understanding based on multi-task learning networks

Abstract

This invention discloses a method for UAV visual scene understanding based on a multi-task learning network. The method first selects the efficient classification network VoVNet as a feature-sharing network to obtain multi-scale encoded features. Second, based on a single-stage anchor-free target detection network, an attention-based feature selection supplementary unit is designed to enhance the detection capability of potential targets. Then, for semantic segmentation and depth estimation tasks, a cascaded CRP module is used as a parameter-sharing decoder for semantic segmentation and depth estimation, ensuring running speed through the parameter-sharing decoder structure. Finally, to improve the model's generalization ability, a general-specific paired dataset is constructed for network training. This invention can simultaneously achieve target detection, semantic segmentation, and depth estimation in UAV flight scenarios, completing potential target discovery, obstacle detection, and distance information acquisition, effectively realizing visual scene understanding and further enhancing the autonomous reconnaissance capability of UAVs.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically a method for UAV visual scene understanding based on multi-task learning networks. Technical Background

[0002] UAV visual scene understanding is the process of perceiving, analyzing, and interpreting observed 3D dynamic scenes using visual sensors. This involves labeling the categories of all targets in the scene and determining their locations, requiring good real-time performance. Traditional UAV visual scene understanding methods are mostly based on SLAM (Simultaneous Localization and Mapping), but these often involve expensive equipment and complex structures, making them difficult to implement on UAVs. Monocular vision, due to its lower cost and simpler structure, is widely used in UAVs. Deep learning-based visual scene understanding methods use neural networks with specific structures to simultaneously perform visual tasks such as target detection and semantic segmentation, providing environmental information for UAV flight.

[0003] In recent years, scene understanding using visual images has focused primarily on object detection and semantic segmentation, failing to provide richer scene information for drone flight. With the continuous development of deep learning, multi-task learning networks based on deep learning have attracted researchers' attention, aiming to achieve multiple visual prediction tasks through a single, appropriate network model. MaskR-CNN, through joint learning, has achieved detection, classification, and segmentation tasks in general scenarios. BlitzNet, based on an encoder-decoder structure, has achieved object detection and semantic segmentation for unmanned vehicles in general scenarios. MultiNet, employing shared encoded features, has been designed to complete vehicle detection, road segmentation, and classification tasks. Nekrasov et al. improved the RefineNet semantic segmentation network using an asymmetric dataset, designing a multi-task learning network that includes real-time semantic segmentation and depth estimation. However, these methods are mostly geared towards ground scenarios, and their performance is not ideal for drone reconnaissance and autonomous decision-making.

[0004] To address the shortcomings of existing visual scene understanding algorithms, such as insufficient information provision and inadequate real-time performance to meet the reconnaissance and autonomous decision-making requirements of UAVs in complex environments, this invention proposes a multi-task learning network based on an encoder-decoder structure for UAV visual scene understanding. Building upon an efficient classification network as the encoder, a feature filtering supplementary unit is added to the anchor-free target detection algorithm to enhance potential target detection capabilities. Simultaneously, a parameter-shared decoder for semantic segmentation and depth estimation is constructed to accelerate model processing. In experiments, the multi-task learning network for scene understanding is trained using a general-to-specific dataset. During actual UAV operation, it provides depth information on targets, obstacles, and the flight environment to meet the decision-making information needs of UAVs in real-time reconnaissance. Summary of the Invention

[0005] The purpose of this invention is to provide a method for UAV visual scene understanding based on a multi-task learning network. This method constructs a multi-task learning network based on an encoder-decoder structure. First, a classification network is used as an encoder to obtain multi-scale encoded features, which are then fed into the decoder network. Second, the object detection branch constructs a feature pyramid from the encoded features, while the semantic segmentation and depth estimation branches obtain decoded features through a decoder with shared parameters. These decoded features are then fed into their respective independent prediction heads and the object detection feature pyramid to obtain fused features. Finally, the visual scene understanding results are obtained using the respective prediction networks for object detection, semantic segmentation, and depth estimation.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] Methods for UAV visual scene understanding based on multi-task learning networks include:

[0008] Step 1: Construct a scene understanding multi-task learning network model, which consists of three parts: object detection, semantic segmentation, and depth estimation. It includes a VoVNet feature encoding network, an object detection network combined with the feature selection supplementary unit FSS, a semantic segmentation network and a depth estimation network that use cascaded CRP decoder parameter sharing.

[0009] Step 2: Establish a general-specific dataset for the visual scene understanding multi-task learning network, and simultaneously construct the loss function L for the multi-task learning network. total ;

[0010] Step 3: Train the scene understanding multi-task learning network model using the dataset. Specifically:

[0011] (3.1) Input the dataset images into VoVNet for feature extraction to obtain multi-scale encoded features {F}. encoded};

[0012] (3.2) Encode the multi-scale feature {Fe ncoded The data is fed into a parameter-sharing cascaded CRP decoder to generate joint decoding features for semantic segmentation and depth estimation. joint_decoded};

[0013] (3.3) Jointly decode the feature {f joint_decoded The input feature filtering supplementary unit (FSS) obtains supplementary features for object detection {f}. obj_sup};

[0014] (3.4) Supplement the target detection with features {f obj_sup} and multi-scale coding features {F encoded After fusion, the data is fed into the Feature Pyramid Network (FPN) to obtain the target detection decoding features {f}. obj_decoded};

[0015] (3.5) Decode the target detection features {f} respectively obj_decoded} and semantic segmentation and depth estimation jointly decode features {f joint_decoded The data is fed into the corresponding network prediction section, using the loss function L. total The prediction results and the real labels in the dataset are processed until the network iterative training is completed;

[0016] Step four involves inputting the images obtained by the drone camera during actual operation into the trained scene understanding multi-task learning network. Following the process in step three, target detection, semantic segmentation, and depth estimation are obtained, which are used for discovering potential targets, avoiding obstacles, and acquiring depth information of the flight environment during drone operation, respectively, to achieve scene understanding during drone flight.

[0017] The present invention has the following advantages:

[0018] 1. This invention addresses the problem of missed detections in target detection from the perspective of UAVs by designing a feature selection and supplement unit based on an attention mechanism. This unit fuses multi-scale encoded features with joint decoded features as decoded features for target detection, thereby improving the potential target detection capability of the target detection branch.

[0019] 2. This invention addresses the problems of insufficient information and difficulty in meeting the requirements of existing visual scene understanding methods. It adopts a multi-task learning network based on an encoder-decoder structure, which completes three visual prediction tasks—target detection, semantic segmentation, and depth estimation—while reducing the number of parameters. This enables potential target discovery, obstacle detection, and acquisition of flight environment depth information, providing sufficient environmental information for UAV flight. Attached Figure Description

[0020] Figure 1 This is a flowchart of the present invention.

[0021] Figure 2 This is an overall block diagram of the present invention.

[0022] Figure 3 This is a comparison of the target detection performance before and after adding the Feature Filtering Supplement (FSS) unit to the network in the method of this invention.

[0023] Figure 4 This is a partial test result diagram of the method of the present invention on a dataset. Detailed Implementation

[0024] The technical solution of the present invention will be described in detail with reference to the accompanying drawings and tables.

[0025] like Figure 1As shown, the UAV visual scene understanding method based on a multi-task learning network of the present invention specifically includes the following steps:

[0026] Step 1 constructs a scene-understanding multi-task learning network model, consisting of three parts: object detection, semantic segmentation, and depth estimation, as follows: Figure 2 As shown, it includes a VoVNet feature encoding network, an object detection network combined with a feature selection supplementary unit (FSS), a semantic segmentation network using parameter sharing of a cascaded CRP decoder, and a depth estimation network, specifically:

[0027] (1.1) Based on the encoder-decoder structure, a feature encoding network based on VoVNet is designed. The network structure is divided into 5 groups of convolutional blocks. The first group of convolutional blocks is the Stem module, which includes 3 groups of 3×3 convolutional layers. The second to fifth groups of convolutional blocks are OSA modules. Each OSA module consists of 3×3 convolutional layers and 1×1 convolutional layers. The encoded features are composed of the outputs of the four groups of OSA modules.

[0028] (1.2) The target detection network combined with the feature selection supplementary unit FSS includes the FSS unit, FPN, and prediction structure. The FSS unit adopts a residual structure, consisting of two 1×1 convolutional layers and one 3×3 convolutional layer. The FPN uses a traditional structure for multi-scale feature processing. The prediction structure uses three sets of two 3×3 convolutional layers corresponding to classification, regression, and center position prediction, respectively.

[0029] (1.3) The semantic segmentation network and depth estimation network with decoder parameter sharing are based on the CRP structure, which consists of two 5×5 max pooling layers and 1×1 convolutional layers. The decoded features are output as the input of the prediction structure in a cascaded manner, and the prediction structures for semantic segmentation and depth estimation are composed of two 3×3 convolutional layers respectively.

[0030] Step 2 establishes a general-purpose / dedicated dataset for a multi-task learning network for visual scene understanding, specifically as follows:

[0031] Combining the knowledge distillation concept, we designed and constructed a general-to-specific dataset for COCO general scene and VisDrone2019 UAV view target detection; a general-to-specific dataset for COCO general scene and AreoScapes UAV flight semantic segmentation; and a complete depth estimation dataset combining NYUDv2 indoor scene and KITTI outdoor scene.

[0032] Step 3: Construct a visual scene understanding multi-task learning network with loss function L. total , by target detection L obj Semantic segmentation L seg and depth estimation L depthIt consists of three parts, which alleviate overfitting caused by single-task-dominated training through weight adjustment, specifically:

[0033]

[0034] Among them, w i (i = 1, 2, 3) are the task balancing adjustment weights, satisfying... Loss function L for object detection obj Including classification cls Regression reg and center position l cent Three types of losses. y and I represent the predicted values ​​and ground truth labels for object detection, semantic segmentation, and depth estimation, respectively, and θ... o θ s θ d These represent the object detection parameters, semantic segmentation parameters, and depth estimation parameters, respectively.

[0035] Step 4 involves training the scene understanding multi-task learning network model using the dataset, specifically as follows:

[0036] (4.1) Input the real-time images from the UAV camera into VoVNet to generate multi-scale encoded features {F encoded};

[0037] (4.2) Encode the multi-scale features {F} encoded The data is fed into a parameter-sharing cascaded CRP decoder to generate joint decoding features for semantic segmentation and depth estimation. joint_decoded}; where the CRP module consists of a series of alternating 5×5 max pooling layers and 1×1 convolutional layers, encoding multiple features {F encoded After fusion by this module, scale-uniform joint decoding features {f} are obtained. joint_decoded};

[0038] (4.3) Jointly decode the feature {f joint_decoded The target detection supplementary features {f} are obtained through the Feature Selection Supplementary Unit (FSS). obj_sup Specifically:

[0039] For the input joint decoding features {f joint_decoded}, define the joint decoding feature of the i-th layer as Feature selection attention mask is f m After passing through the Feature Filtering and Supplementing Unit (FSS), supplementary features for target detection {f} are formed. obj_sup Specifically:

[0040]

[0041] In the formula, This indicates that two feature maps are multiplied element-wise. This indicates that the features are sequentially processed through a 3×3 convolutional layer, a batch normalization (BN) layer, and a ReLU non-linear activation function for feature fusion after multiplication, with the corresponding mask feature f. m This is learned through attention mechanisms, specifically:

[0042]

[0043] In the formula, This indicates that the feature transformation is performed sequentially through a 1×1 convolutional layer, a BN layer, and a ReLU non-linear activation function. i The function used to learn the attention mask mainly includes 1×1 convolution, batch normalization (BN), and a sigmoid activation function, such that the ultimately learned mask f... m ∈[0,1].

[0044] (4.4) Supplement the target detection with features {f obj_sup} and multi-scale coding features {F encoded After fusion, the data is fed into the Feature Pyramid Network (FPN) to obtain the target detection decoding features {f}. obj_decoded};

[0045] (4.5) Decode the target detection features {f} respectively obj_decoded} and semantic segmentation and depth estimation jointly decode features {f joint_decoded The data is fed into the corresponding network prediction section, using the loss function L. total The prediction results and the real labels in the dataset are processed until the network iterative training is completed.

[0046] Step 5 replaces the dataset images with images obtained by the UAV camera during actual operation and inputs them into the trained scene understanding multi-task learning network. Following steps (4.1) to (4.5), the final results of target detection, semantic segmentation, and depth estimation prediction are obtained and used for the discovery of potential targets, obstacle avoidance, and distance information acquisition during UAV operation, respectively, to achieve scene understanding during UAV flight.

[0047] To verify the feasibility and effectiveness of the present invention, the present invention will be further described in detail below with reference to examples.

[0048] This validation was conducted on a desktop PC with an Intel(R) Core(TM) i5-2450M CPU@2.50GHz, 4GB of RAM, and an NVIDIA GeForce GTX 1660 Ti graphics card, implemented using PyCharm, CUDA 9.0, and CUDNN 7.0. The experiments first analyzed object detection results on the VisDrone2019 dataset from a UAV perspective, comparing the object detection performance before and after adding the specially selected supplementary unit FSS to the network. Next, semantic segmentation results were compared on the AeroScapes dataset for UAV flight scenes. Then, the performance of depth estimation was validated on the NYUDv2-KITTI complete dataset. Finally, the results of the scene understanding multi-task learning network were analyzed using the test dataset.

[0049] Table 1 shows the results of the method of the present invention after being trained on the COCO general dataset - UAV perspective VisDrone dedicated dataset, on the VisDrone2019 validation set, compared with the one-stage object detection algorithms Faster R-CNN, Mask R-CNN and the two-stage object detection algorithms YOLOv3, M2Det.

[0050] Table 1 Comparison of results from this invention and related algorithms on the VisDrone test set from the perspective of unmanned aerial vehicles.

[0051]

[0052] As shown in the table, the object detection branch in this invention achieves a mean precision (AP) comparable to M2DNet, but lower than the overall best method, CenterNet. However, its average recall (AR) for multi-object detection is 32.7% higher than the best one-stage object detection algorithm, Mask R-CNN, and 4.9% higher than the best two-stage object detection algorithm, CenterNet. Furthermore, its running speed is comparable to the real-time object detection algorithms of YOLOv3. In summary, the object detection algorithm in this invention significantly improves multi-object detection capabilities while maintaining real-time running speed.

[0053] Figure 3 This image shows a comparison of target detection performance before and after adding the Feature Filtering Supplement (FSS) unit to the network in the method of this invention. As can be seen from the image, in... Figure 3 In the first and fourth rows of highway scenarios, targets affected by occlusion and shadows are undetectable in the baseline network, but can be detected and marked after adding the FSS module; Figure 3In the second and third rows of small object detection, although there are still undetected human targets compared to the real annotations, the detection capability of the target detection branch has been significantly improved compared to the baseline network, indicating that the feature selection supplement module has enhanced the target detection branch's ability to discover potential targets.

[0054] Table 2 shows the performance comparison of the method of the present invention with related algorithms on the AeroScapes test set after being trained on the COCO general dataset - AeroScapes semantic segmentation dataset for UAV flight scenes.

[0055] Table 2. AeroScapes Test Set Results and Algorithm Comparison

[0056]

[0057] As can be seen from the table, the semantic segmentation performance of the method in this invention is second only to PSPNet and BiseNet in Mean IoU, but it is the best in pixel accuracy (PA). Furthermore, its real-time performance fully meets the requirements for real-time semantic segmentation.

[0058] Table 3 shows the depth estimation results of the method of this invention on the complete NYUDv2-KITTI dataset. The root mean square error (RMS) and log-RMS evaluation criteria are used to compare and verify the method with related algorithms, and the real-time performance is also compared.

[0059] Table 3 Depth estimation results for the NYUDv2-KITTI dataset

[0060]

[0061] As can be seen from the table, the depth estimation network in the method of this invention, due to the adoption of a shared parameter decoding structure and an independent prediction module, is comparable to the Garg algorithm in terms of RMSE, slightly lower than the Goddard algorithm, but far surpasses other algorithms in terms of real-time performance, with the highest improvement being 73.6%, which can fully meet the real-time requirements for depth estimation of UAVs.

[0062] Figure 4The images show partial test results of the method of this invention on a dataset. The images depict operating scenes captured by a drone over a common highway. Potential targets include pedestrians and vehicles, and obstacles include streetlights, public transportation equipment, and buildings. The first column shows the original scene images, the second column shows the target detection results, demonstrating that pedestrians and vehicles, appearing as small targets from the drone's perspective, can still be detected even with overlap and partial occlusion. The third column shows the semantic segmentation results, showing good segmentation performance for obstacles such as buildings and public transportation equipment, but slightly insufficient segmentation for public transportation equipment. The last column shows the depth estimation results, providing distance estimations for different objects and offering some distance information for actual drone flight.

Claims

1. A method for UAV visual scene understanding based on multi-task learning networks, characterized in that, The method includes the following steps: Step 1: Construct a scene understanding multi-task learning network model, which consists of three parts: object detection, semantic segmentation, and depth estimation. It includes a VoVNet feature encoding network, an object detection network combined with the feature selection supplementary unit FSS, a semantic segmentation network and a depth estimation network that use cascaded CRP decoder parameter sharing. Step 2: Establish a general-specific dataset for the visual scene understanding multi-task learning network, and simultaneously construct the loss function L for the multi-task learning network. total ; Step 3: Train the scene understanding multi-task learning network model using the dataset. Specifically: (3.1) Input the dataset images into VoVNet for feature extraction to obtain multi-scale encoded features {F}. encoded }; (3.2) Encode the multi-scale features {F} encoded The data is fed into a parameter-sharing cascaded CRP decoder to generate joint decoding features for semantic segmentation and depth estimation. joint_decoded }; (3.3) Jointly decode the feature {f joint_decoded The input feature filtering supplementary unit (FSS) obtains supplementary features for object detection {f}. obj_sup }; (3.4) Supplement the target detection with features {f obj_sup } and multi-scale coding features {F encoded After fusion, the data is fed into the Feature Pyramid Network (FPN) to obtain the target detection decoding features {f}. obj_decoded }; (3.5) Decode the target detection features {f} respectively obj_decoded } and semantic segmentation and depth estimation jointly decode features {f joint_decoded The data is fed into the corresponding network prediction section, using the loss function L. total The prediction results and the real labels in the dataset are processed until the network iterative training is completed; Step four involves inputting the images obtained by the drone camera during actual operation into the trained scene understanding multi-task learning network. Following the process in step three, target detection, semantic segmentation, and depth estimation are obtained, which are used for discovering potential targets, avoiding obstacles, and acquiring depth information of the flight environment during drone operation, respectively, to achieve scene understanding during drone flight.

2. The UAV visual scene understanding method based on a multi-task learning network as described in claim 1, characterized in that, The scene understanding multi-task learning network model constructed in step one consists of three parts: object detection, semantic segmentation, and depth estimation. It includes a VoVNet feature encoding network, an object detection network combined with a feature selection supplementary unit (FSS), a semantic segmentation network using a cascaded CRP decoder with parameter sharing, and a depth estimation network. Specifically: (2.1) The feature encoding network with VoVNet as the core has a network structure divided into 5 groups of convolutional blocks. The first group of convolutional blocks is the Stem module, which includes 3 groups of 3×3 convolutional layers. The second to fifth groups of convolutional blocks are OSA modules. Each OSA module consists of 3×3 convolutional layers and 1×1 convolutional layers. The encoded features are composed of the outputs of the four groups of OSA modules. (2.2) The target detection network combined with the feature selection supplement unit FSS consists of FSS unit, FPN and prediction structure. The FSS unit adopts residual structure, which consists of two 1×1 convolutional layers and one 3×3 convolutional layer. The FPN uses traditional structure for multi-scale feature processing. The prediction structure uses three sets of two 3×3 convolutional layers to correspond to classification, regression and center position prediction respectively. (2.3) The core of the cascaded CRP decoder parameter-sharing semantic segmentation network and depth estimation network is the CRP structure, which consists of two layers of 5×5 max pooling layers and 1×1 convolutional layers alternately. The output decoding features are used as the input of the prediction structure in a cascaded manner, and two layers of 3×3 convolutional layers are used to form the prediction structure for semantic segmentation and depth estimation respectively.

3. The UAV visual scene understanding method based on a multi-task learning network as described in claim 1, characterized in that, In step two, a general-specific dataset for the visual scene understanding multi-task learning network is established, and the training loss function L for the multi-task learning network is constructed. total Specifically: (3.1) Establish a general-specific dataset for COCO general scene and VisDrone2019 UAV view target detection for object detection; a general-specific dataset for Microsoft COCO general scene and AreoScapes UAV flight semantic segmentation for semantic segmentation; and a complete depth estimation dataset combining NYUDv2 indoor scene and KITTI outdoor scene for depth estimation. (3.2) Define the total loss function of the visual scene understanding multi-task learning network as L total , by target detection L obj Semantic segmentation L seg and depth estimation L depth It consists of three parts, which alleviate overfitting caused by single-task-dominated training through weight adjustment, specifically: Among them, w i i = 1, 2, 3 are the task balancing adjustment weights, satisfying Loss function L for object detection obj Including classification cls Regression reg and center position l cent Three types of loss; y and I represent the predicted values ​​and ground truth annotations for object detection, semantic segmentation, and depth estimation, respectively, and θ o θ s θ d These represent the object detection parameters, semantic segmentation parameters, and depth estimation parameters, respectively.

4. The UAV visual scene understanding method based on a multi-task learning network as described in claim 1, characterized in that, In step three (3.3), the joint decoding feature {f joint_decoded The input feature filtering supplementary unit (FSS) obtains supplementary features for object detection {f}. obj_sup Specifically: Define the joint decoding feature of the i-th layer as Feature selection attention mask feature is f m After passing through the Feature Filtering and Supplementing Unit (FSS), supplementary features for target detection {f} are formed. obj_sup Specifically: In the formula, This indicates that two feature maps are multiplied element-wise. This indicates that the feature is passed through a 3×3 convolutional layer, a batch normalization layer, and a ReLU non-linear activation function in sequence, which is used for feature fusion after multiplication. The corresponding feature selection attention mask feature f is used. m This is learned through attention mechanisms, specifically: In the formula, This indicates that the feature transformation is performed sequentially through a 1×1 convolutional layer, a BN layer, and a ReLU non-linear activation function. i The function for learning the attention mask includes 1×1 convolution, batch normalization (BN), and a sigmoid activation function, such that the ultimately learned mask f... m ∈[0,1].

Images