Low-altitude relative depth of field estimation and target segmentation fusion method and system

Abstract

The application discloses a low-altitude relative depth-of-field estimation and target segmentation fusion method and system, and relates to the technical field of unmanned system environment perception and airworthiness guarantee. The method obtains continuous frames of low-altitude airspace video collected by a monocular camera carried by a UAV, carries out joint self-supervised training through three networks of depth estimation, semantic segmentation and pose estimation after camera calibration, constructs an overall training loss function of weighted fusion of a re-projection loss function, an edge smoothing loss function and a multi-scale loss function, and completes model optimization. The UAV is equipped with a low-altitude relative depth-of-field estimation and target segmentation fusion system which is adapted to the method. The application realizes depth optimization and interference suppression of a key obstacle area, can output a semantic-enhanced relative depth-of-field map in real time under monocular vision conditions, improves the depth prediction stability and boundary consistency of slender structures and weak texture areas, and enhances the structure understanding and obstacle identification capability of an unmanned system on a scene.

Description

Technical Field

[0001] This invention relates to the field of environmental perception and airworthiness assurance technology for unmanned systems, and in particular to a method and system for low-altitude relative depth estimation and target segmentation fusion. Background Technology

[0002] Currently, with the rapid development of the "low-altitude economy," drones are increasingly widely used in fields such as power line inspection and urban security. The low-altitude airspace environment is complex, and obstacles with "weak texture and fine structure," such as high-voltage power lines and communication cables, can easily cause drone collisions, seriously threatening flight safety. Existing perception technologies have significant limitations: lidar is expensive and prone to "leakage" in thin power lines; millimeter-wave radar has low resolution, making it difficult to meet the needs of precise obstacle avoidance; binocular vision is limited by baseline length, resulting in insufficient accuracy for long-distance depth perception. Monocular vision, with its advantages of low cost and simple structure, combined with the development of deep learning, has made monocular depth estimation a research hotspot. Monocular depth estimation is divided into supervised and self-supervised methods. Supervised learning requires a massive number of "image-depth ground truth" pairs, which is extremely costly to obtain; self-supervised learning utilizes inter-frame geometric constraints and can be trained without depth labels, gradually becoming the mainstream. However, existing self-supervised methods still suffer from drawbacks in low-altitude applications, such as scale ambiguity, loss of slender targets, failure in weakly textured regions, and semantic deficiencies. Furthermore, current fusion schemes for semantic segmentation and depth estimation are mostly designed for autonomous driving, lacking optimization for the perspective of low-altitude aircraft, and are particularly inadequate for the network structure and fusion strategies for extremely fine targets. Therefore, there is an urgent need for a low-altitude depth estimation system that can effectively capture the features of slender structures and combine semantic information for depth optimization.

[0003] Furthermore, existing monocular depth estimation methods still have limitations in their training paradigms: some methods rely on stereo image pairs or known camera poses as supervision signals, making them difficult to apply directly in actual low-altitude flight scenarios where only monocular video is available; depth estimation networks and pose estimation networks are mostly trained independently or optimized only in one direction, failing to achieve end-to-end joint self-supervised training, requiring pose information to be provided by external sensors, increasing system complexity and deployment costs; moving objects in dynamic scenes can violate the inter-frame geometric consistency assumption, leading to depth estimation failure. These problems are particularly prominent in complex low-altitude environments, restricting the practical application of monocular vision in autonomous obstacle avoidance for UAVs. To address these challenges, this invention proposes a method and system for fusing low-altitude relative depth estimation and target segmentation. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for fusing low-altitude relative depth estimation and target segmentation. This involves joint self-supervised training using three networks: depth estimation, semantic segmentation, and pose estimation. A weighted fusion of reprojection loss function, edge smoothing loss function, and multi-scale loss function is constructed to optimize the model. A low-altitude relative depth estimation and target segmentation fusion system adapted to this method is then implemented on a UAV. This addresses the problems of existing monocular depth estimation methods, such as weak perception of slender obstacles, unstable prediction of weakly textured regions, and lack of semantic understanding in complex low-altitude environments.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] A method for fusing low-altitude relative depth estimation and target segmentation, the method comprising the following steps:

[0007] Step S1: Acquire continuous video frames of the low-altitude airspace captured by the UAV's onboard monocular camera;

[0008] Step S2: After the camera calibrates the continuous video frames, joint self-supervised training is performed. The corresponding outputs are obtained through the depth estimation network, semantic segmentation network, and pose estimation network, including the original relative depth map, semantic mask, and six-degree-of-freedom pose transformation parameters. The overall training loss function is constructed and the model is optimized to train the optimal model. The current target frame of the continuous video frames is input to the semantic segmentation network, and the semantic mask is output in real time.

[0009] Step S3: Use the semantic segmentation network trained by joint self-supervised training to reason about video frames and generate pixel-level semantic masks;

[0010] Step S4: Perform semantic and depth fusion. Based on the semantic mask, perform sky region suppression, elongated obstacle enhancement, and dynamic object masking operations to optimize the original relative depth map and generate a semantically enhanced relative depth map.

[0011] Furthermore, in step S2, the camera calibration video continuous frames include extracting frames from the continuous video sequence in time order, dividing two adjacent frames into training sample groups, with the previous frame as the target frame and the next frame as the source frame, and inputting them into the network after normalization and size unification preprocessing.

[0012] Furthermore, the joint self-supervised training in step S2 further includes the following steps:

[0013] Step S21: Input the current target frame of the continuous video frames into the depth estimation network to obtain the original relative depth map;

[0014] Step S22: Input the adjacent frames of the continuous video frames into the pose estimation network to predict the six-degree-of-freedom pose transformation parameters between adjacent frames;

[0015] Step S23: The depth estimation network and pose estimation network are jointly trained again under self-supervised supervision to construct the reprojection loss function, edge smoothing loss function and multi-scale loss function, and the model training is completed by optimizing the overall training loss function.

[0016] Furthermore, in step S21, the depth estimation network connects the encoder and decoder feature maps through nested convolutional nodes. Each convolutional node fuses features from the same layer, upsampled features from the next layer, and features from skip connections. The ECANet channel attention mechanism module is set after feature fusion and before upsampling operation in the decoder to adaptively calculate channel weights through one-dimensional convolution, thereby enhancing the response to the feature channels of slender targets.

[0017] Furthermore, the joint self-supervised training in step S23 further includes:

[0018] Based on the relative depth map and six-DOF pose transformation parameters, a reconstructed image is generated by reprojecting the source frame onto the target frame's viewpoint using a camera imaging geometric model. The reprojection loss function, edge smoothing loss function, and multi-scale loss function are weighted and fused together as the overall training loss function. The overall training loss function is defined as follows:

[0019]

[0020] in, Let the reprojection loss function be... The edge smoothing loss function is... For multi-scale loss functions, These are the weight coefficients of the corresponding loss function; the overall training loss is backpropagated to update the trainable parameters of the two networks; the validation set loss value is calculated after each training round, and training is stopped when the loss value tends to stabilize for several consecutive rounds and is less than the preset threshold, or when the preset maximum number of iterations is reached, and the optimal model parameters are saved.

[0021] Furthermore, in step S23, the edge smoothing loss function uses the image gradient as weight to constrain the smoothing of flat areas of the depth map and allows abrupt changes in depth in edge areas; the multi-scale loss function calculates the loss at different resolution levels of the decoder to avoid the network getting trapped in local optima; the reprojection loss function combines the L1 norm and SSIM structural similarity to calculate the photometric error and uses the minimum reprojection error strategy to solve the occlusion problem.

[0022] Furthermore, in step S3, the semantic segmentation network utilizes the hollow spatial pyramid pooling module to extract multi-scale semantic features, classifies the input image pixels into predefined categories such as sky and slender obstacles, and outputs a semantic mask in real time.

[0023] Furthermore, the semantic and depth fusion in step S4 includes depth suppression of the sky region based on a semantic mask, and depth boundary sharpening of the elongated obstacle region; the sky region suppression is to set the depth value of sky-type pixels to a preset maximum depth value; the elongated obstacle enhancement is to use semantic boundaries as constraints to perform local sharpening and guided filtering on the original relative depth map, so that the depth edge is aligned with the semantic edge; the dynamic object masking is to use a semantic mask to generate a dynamic object mask, and reset the weight of pixels belonging to the dynamic category in the reprojection loss calculation to zero, so as to eliminate the interference of dynamic objects on the deep learning of static scenes.

[0024] A low-altitude relative depth estimation and target segmentation fusion system, the system comprising a data acquisition module, a depth perception module, a pose estimation module, a semantic understanding module, and a fusion processing module:

[0025] The data acquisition module is used to acquire video streams captured by the drone's monocular camera.

[0026] The depth perception module includes a depth estimation network, which is used to receive a single frame of RGB image and output a relative depth map in real time;

[0027] The pose estimation module is only enabled during the training phase and is used to assist in localization. It includes a pose estimation network based on a fully convolutional network. The input is adjacent frame images stitched together along the channel dimension, and the output is the six-degree-of-freedom motion parameters of the camera between adjacent frames.

[0028] The semantic understanding module includes a semantic segmentation network based on DeepLabv3. The semantic segmentation network adopts the DeepLabv3 architecture and includes a hollow space pyramid pooling module.

[0029] The fusion processing module is used to receive the original relative depth map and semantic mask, execute a depth optimization algorithm based on semantic rules, and output the optimized depth map;

[0030] The aforementioned low-altitude relative depth estimation and target segmentation fusion system is carried by an unmanned aerial vehicle (UAV). The UAV's onboard monocular camera collects data, and its flight control system is configured with the semantically enhanced relative depth map to perform obstacle detection and autonomous obstacle avoidance path planning.

[0031] Furthermore, the depth estimation network adopts an encoder-decoder structure. The encoder uses a ResNet-18 residual network as the backbone to extract features, and the decoder is designed based on the UNet++ architecture, integrating an ECANet channel attention mechanism module to perform multi-scale feature fusion using dense skip connections. The ECANet channel attention mechanism module recalibrates the weights of feature channels by adaptively learning the interaction between channels through one-dimensional convolution. The dense skip connections consist of a series of convolutional layers, bridging the semantic gap between the feature maps of the encoder and decoder and reducing information loss during the upsampling process.

[0032] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0033] (1) The present invention provides a method and system for low-altitude relative depth estimation and target segmentation fusion, which integrates shallow geometric details and deep semantic features through the dense skip connection structure of UNet++, and enhances the response of slender features with the ECANet channel attention module, thus significantly improving the ability to capture small targets.

[0034] (2) The present invention provides a method and system for fusing low-altitude relative depth estimation and target segmentation, which introduces DeepLabv3 semantic segmentation and uses semantic prior knowledge to solve the problem of depth estimation failure in weak texture regions, effectively eliminates false obstacles in the sky region, and reduces the probability of UAVs avoiding obstacles.

[0035] (3) The present invention provides a method and system for low-altitude relative depth estimation and target segmentation fusion, which corrects the depth map edge through semantic boundary correction to achieve accurate positioning of obstacle boundary, and adapts to the needs of UAVs to pass through narrow spaces.

[0036] (4) The present invention provides a method and system for low-altitude relative depth estimation and target segmentation fusion. The core network adopts a pure self-supervised learning paradigm. It only uses continuous video sequence frames collected by the UAV's onboard monocular camera and constructs supervision signals through inter-frame geometric constraints. It can complete adaptive fine-tuning for specific scenes without stereo image pairs, known camera poses, or expensive LiDAR depth ground truth. The depth estimation network and pose estimation network are jointly optimized end-to-end through the reprojection loss function. The loss function simultaneously backpropagates to update the trainable parameters of the two networks, so that the two promote each other and converge together during training. Combined with the design of ResNet-18 and lightweight attention module, it can realize real-time inference on the onboard embedded computing platform and meet the real-time requirements of low-altitude flight.

[0037] (5) The present invention provides a method and system for low-altitude relative depth estimation and target segmentation fusion. In the self-supervised training stage, a semantic segmentation network is introduced to assist training. A dynamic object mask is generated using a semantic mask. The weight of pixels belonging to the dynamic category in the reprojection loss calculation is reset to zero, which effectively eliminates the interference of moving objects on the deep learning of static scenes and solves the problem of depth estimation failure of traditional self-supervised methods in dynamic scenes. Attached Figure Description

[0038] Figure 1 This is an overall architecture diagram of a low-altitude relative depth estimation and target segmentation fusion method and system according to the present invention;

[0039] Figure 2 This is a diagram of the depth estimation network structure based on UNet++ and ECANet improvements of the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] A method for fusing low-altitude relative depth estimation and target segmentation, the method comprising the following steps:

[0042] Step S1: Acquire continuous video frames of the low-altitude airspace captured by the UAV's onboard monocular camera;

[0043] Step S2: After the camera calibrates the continuous video frames, joint self-supervised training is performed. The corresponding outputs are obtained through the depth estimation network, semantic segmentation network, and pose estimation network, including the original relative depth map, semantic mask, and six-degree-of-freedom pose transformation parameters. The overall training loss function is constructed and the model is optimized to train the optimal model. The current target frame of the continuous video frames is input to the semantic segmentation network, and the semantic mask is output in real time.

[0044] Step S3: Use the semantic segmentation network trained by joint self-supervised training to reason about video frames and generate pixel-level semantic masks;

[0045] Step S4: Perform semantic and depth fusion. Based on the semantic mask, perform sky region suppression, elongated obstacle enhancement, and dynamic object masking operations to optimize the original relative depth map and generate a semantically enhanced relative depth map.

[0046] Furthermore, in step S2, the camera calibration video continuous frames include extracting frames from the continuous video sequence in time order, dividing two adjacent frames into training sample groups, with the previous frame as the target frame and the next frame as the source frame, and inputting them into the network after normalization and size unification preprocessing.

[0047] Furthermore, the joint self-supervised training in step S2 further includes the following steps:

[0048] Step S21: Input the current target frame of the continuous video frames into the depth estimation network to obtain the original relative depth map;

[0049] Step S22: Input the adjacent frames of the continuous video frames into the pose estimation network to predict the six-degree-of-freedom pose transformation parameters between adjacent frames;

[0050] Step S23: The depth estimation network and pose estimation network are jointly trained again under self-supervised supervision to construct the reprojection loss function, edge smoothing loss function and multi-scale loss function, and the model training is completed by optimizing the overall training loss function.

[0051] Furthermore, in step S21, the depth estimation network connects the encoder and decoder feature maps through nested convolutional nodes. Each convolutional node fuses features from the same layer, upsampled features from the next layer, and features from skip connections. The ECANet channel attention mechanism module is set after feature fusion and before upsampling operation in the decoder to adaptively calculate channel weights through one-dimensional convolution, thereby enhancing the response to the feature channels of slender targets.

[0052] Furthermore, the joint self-supervised training in step S23 further includes:

[0053] Based on the relative depth map and six-DOF pose transformation parameters, a reconstructed image is generated by reprojecting the source frame onto the target frame's viewpoint using a camera imaging geometric model. The reprojection loss function, edge smoothing loss function, and multi-scale loss function are weighted and fused together as the overall training loss function. The overall training loss function is defined as follows:

[0054]

[0055] in, Let the reprojection loss function be... The edge smoothing loss function is... For multi-scale loss functions, These are the weight coefficients of the corresponding loss function; the overall training loss is backpropagated to update the trainable parameters of the two networks; the validation set loss value is calculated after each training round, and training is stopped when the loss value tends to stabilize for several consecutive rounds and is less than the preset threshold, or when the preset maximum number of iterations is reached, and the optimal model parameters are saved.

[0056] Furthermore, in step S23, the edge smoothing loss function uses the image gradient as weight to constrain the smoothing of flat areas of the depth map and allows abrupt changes in depth in edge areas; the multi-scale loss function calculates the loss at different resolution levels of the decoder to avoid the network getting trapped in local optima; the reprojection loss function combines the L1 norm and SSIM structural similarity to calculate the photometric error and uses the minimum reprojection error strategy to solve the occlusion problem.

[0057] Furthermore, in step S3, the semantic segmentation network utilizes the hollow spatial pyramid pooling module to extract multi-scale semantic features, classifies the input image pixels into predefined categories such as sky and slender obstacles, and outputs a semantic mask in real time.

[0058] Furthermore, the semantic and depth fusion in step S4 includes depth suppression of the sky region based on a semantic mask, and depth boundary sharpening of the elongated obstacle region; the sky region suppression is to set the depth value of sky-type pixels to a preset maximum depth value; the elongated obstacle enhancement is to use semantic boundaries as constraints to perform local sharpening and guided filtering on the original relative depth map, so that the depth edge is aligned with the semantic edge; the dynamic object masking is to use a semantic mask to generate a dynamic object mask, and reset the weight of pixels belonging to the dynamic category in the reprojection loss calculation to zero, so as to eliminate the interference of dynamic objects on the deep learning of static scenes.

[0059] A low-altitude relative depth estimation and target segmentation fusion system, the system comprising a data acquisition module, a depth perception module, a pose estimation module, a semantic understanding module, and a fusion processing module:

[0060] The data acquisition module is used to acquire video streams captured by the drone's monocular camera.

[0061] The depth perception module includes a depth estimation network, which is used to receive a single frame of RGB image and output a relative depth map in real time;

[0062] The pose estimation module is only enabled during the training phase and is used to assist in localization. It includes a pose estimation network based on a fully convolutional network. The input is adjacent frame images stitched together along the channel dimension, and the output is the six-degree-of-freedom motion parameters of the camera between adjacent frames.

[0063] The semantic understanding module includes a semantic segmentation network based on DeepLabv3. The semantic segmentation network adopts the DeepLabv3 architecture and includes a hollow space pyramid pooling module.

[0064] The fusion processing module is used to receive the original relative depth map and semantic mask, execute a depth optimization algorithm based on semantic rules, and output the optimized depth map;

[0065] The aforementioned low-altitude relative depth estimation and target segmentation fusion system is carried by an unmanned aerial vehicle (UAV). The UAV's onboard monocular camera collects data, and its flight control system is configured with the semantically enhanced relative depth map to perform obstacle detection and autonomous obstacle avoidance path planning.

[0066] Furthermore, the depth estimation network adopts an encoder-decoder structure. The encoder uses a ResNet-18 residual network as the backbone to extract features, and the decoder is designed based on the UNet++ architecture, integrating an ECANet channel attention mechanism module to perform multi-scale feature fusion using dense skip connections. The ECANet channel attention mechanism module recalibrates the weights of feature channels by adaptively learning the interaction between channels through one-dimensional convolution. The dense skip connections consist of a series of convolutional layers, bridging the semantic gap between the feature maps of the encoder and decoder and reducing information loss during the upsampling process.

[0067] To better illustrate the specific implementation methods, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0068] Example 1

[0069] This embodiment provides a vision-based depth estimation and semantic fusion system for low-altitude scenes on unmanned aerial vehicles (UAVs). The system runs on the UAV's onboard computer, taking as input an RGB video stream captured by a monocular camera and outputting an optimized dense depth map with semantic tags for use by the downstream path planning module.

[0070] like Figure 1 As shown, the system is logically divided into three core neural network modules and one post-processing fusion module, specifically including the following modules:

[0071] The dataset acquisition module is used to acquire RGB video stream datasets and divide the datasets into training and testing sets. The datasets are derived from low-altitude flight videos collected by a monocular camera mounted on a drone, including typical low-altitude scenes such as power line inspection and urban low-altitude flight. This module converts the videos into continuous image sequences at a fixed frame rate and divides them into training and testing sets proportionally.

[0072] The model building module is used to construct a depth estimation network, a pose estimation network, and a semantic segmentation network. The depth estimation network is used to infer the geometric structure of the scene and includes an encoding-side feature extraction module, an intermediate feature aggregation module, and a decoding-side feature fusion module. The decoding-side feature fusion module includes a channel attention module connected to the output end, which is used to allocate weights according to the different importance of feature channels, thereby increasing the influence of important channels of the neural network on the output results. The pose estimation network is used to assist self-supervised training and adopts a fully convolutional network structure. The semantic segmentation network is used to perform pixel-level classification of the input image and adopts the DeepLabv3 architecture.

[0073] The training module is used to train the depth estimation network, pose estimation network, and semantic segmentation network using the training set. The training processes for the depth estimation network and pose estimation network are self-supervised learning processes, while the semantic segmentation network employs supervised pre-training. The structure of the depth estimation network is as follows: Figure 2 As shown.

[0074] The testing module is used to test the trained depth estimation network and semantic segmentation network using the test set. The testing phase includes the depth estimation network and the semantic segmentation network, which process the input single-frame RGB image in parallel, outputting the original depth map and semantic mask, respectively. The pose estimation network is removed during the inference phase and does not participate in the forward computation.

[0075] The deep fusion module is used to fuse the original depth map with a semantic mask using a semantic-depth fusion algorithm to generate an optimized depth map.

[0076] The output module is used to output the optimized depth map to the downstream path planning module for the UAV's autonomous navigation and obstacle avoidance.

[0077] The training module specifically includes:

[0078] The first input unit is used to input the previous frame image in the training set into the depth estimation network to obtain the depth value corresponding to each pixel in the previous frame image.

[0079] The second input unit is used to input the current frame image and the previous frame image in the training set into the pose estimation network to obtain the pose transformation matrix between corresponding pixels in the current frame image and the previous frame image.

[0080] The reconstruction unit is used to reconstruct the current frame image using a reprojection formula based on the depth value and the pose transformation matrix.

[0081] The error calculation unit is used to calculate the error between the reconstructed image and the current frame image, and to perform weighted calculation in conjunction with the dynamic object mask provided by the semantic segmentation network.

[0082] The parameter optimization unit is used to back-optimize the parameters of the depth estimation network and the pose estimation network based on the error, so that the error is continuously reduced to a threshold range, thereby completing the training.

[0083] The deep fusion module specifically includes:

[0084] The sky suppression unit is used to traverse the semantic mask and set the depth value corresponding to the sky region to infinity to eliminate depth estimation artifacts in the sky region.

[0085] The elongated obstacle sharpening unit is used to extract the connected regions of elongated obstacles, perform depth correction, and apply bilateral filtering to align the depth edges with the semantic edges, thereby improving the depth perception accuracy of elongated obstacles.

[0086] Example 2

[0087] like Figure 1 As shown, the vision-based depth estimation and semantic fusion method for low-altitude UAV scenes provided in this embodiment mainly runs on the UAV's onboard computer (such as NVIDIA Jetson AGX Orin). The system input is an RGB video stream acquired by a monocular camera, and the output is an optimized dense depth map with semantic tags, which is used by the downstream path planning module. Logically, the system consists of three core neural network modules and one post-processing fusion module, specifically including the following steps:

[0088] Step A1: Obtain the RGB video stream dataset and divide the dataset into a training set and a test set.

[0089] The dataset is derived from low-altitude flight videos collected by a monocular camera mounted on a drone. It includes typical low-altitude scenes such as power line inspection and urban low-altitude flight. The videos are made into continuous image sequences at a fixed frame rate and divided into training and test sets according to the proportion. Since a self-supervised learning method is used, there is no need for manual annotation of depth labels.

[0090] Step A2: Construct a depth estimation network, a pose estimation network, and a semantic segmentation network.

[0091] The depth estimation network is used to infer the geometric structure of the scene and includes an encoding-end feature extraction module, an intermediate feature aggregation module, and a decoding-end feature fusion module. The decoding-end feature fusion module includes a channel attention module connected to the output end. The pose estimation network is used to assist self-supervised training and adopts a fully convolutional network. The semantic segmentation network is used to perform pixel-level classification of the input image and provide a semantic mask for post-processing fusion.

[0092] The construction process of the depth estimation network is as follows:

[0093] Constructing an encoding-side feature extraction module: The encoding-side feature extraction module consists of a ResNet-18 network, containing four residual blocks, each outputting a feature at a scale equal to the original. Figure 1 Feature maps of 1 / 2, 1 / 4, 1 / 8, and 1 / 16. Compared to VGG or ResNet-50, ResNet-18 significantly reduces the number of parameters and inference latency while maintaining feature extraction capabilities, making it suitable for real-time computation on UAVs. This embodiment initializes the features using transfer learning based on ImageNet pre-trained weights.

[0094] Constructing the intermediate feature aggregation module: This embodiment uses the UNet++ architecture to construct the intermediate feature aggregation module. Let... Indicates the first Layer downsampling layer, the first The output of each convolutional block, where Indicates the downsampling layer index. Indicates the index of a convolutional block at the same level. Node The input comes from the previous node in the same layer. The output, and the next layer The upsampling feature. Its mathematical expression is:

[0095]

[0096] in This indicates that the convolution operation is followed by an activation function. Indicates an upsampling operation. This indicates channel splicing. This dense connection method acts as a semantic filter, gradually fusing deep and shallow features, enabling the decoder to more smoothly recover the spatial resolution of small objects such as wires, and reducing the checkerboard effect and detail blurring during the upsampling process.

[0097] Constructing a decoding-end feature fusion module: The decoding-end feature fusion module stacks the decoding-end feature vector of the lower layer dimension, the intermediate aggregated feature vector of the same layer dimension, and the encoding-end feature vector of the same layer dimension in the channel dimension, and uses them as the input of the attention module. At the same time, a channel attention module is embedded in the decoding-end feature fusion module to allocate weights to different channels according to the different importance of feature channels, thereby increasing the influence of important channels of the neural network on the output results.

[0098] The channel attention module is ECANet (Efficient Channel Attention). Compared to SENet, ECANet abandons dimensionality reduction operations and uses one-dimensional convolution to capture cross-channel interactions, thus avoiding the negative impact of dimensionality reduction on channel attention prediction.

[0099] The specific operations of the ECANet module include:

[0100] Global average pooling: converts the input feature map Compressed into channel descriptor :

[0101]

[0102] Adaptive one-dimensional convolution: using a kernel size of One-dimensional convolution processing . Size depends on the number of channels Adaptive decision, The adaptive calculation formula is:

[0103]

[0104] in Indicates taking away The most recent odd number, in this embodiment, is set , This ensures that the more feature channels there are, the greater the range of interaction.

[0105] Sigmoid function activation: generating channel weights .

[0106] Reweighting: Multiply the weights back into the original feature map. .

[0107] In low-altitude scenes, power lines and towers often exhibit specific high-frequency channel responses in feature maps. ECANet can automatically learn and amplify the weights of these channels, suppressing redundant channels in large background areas (such as grass and roads), thereby highlighting obstacles in the depth map.

[0108] The construction process of the pose estimation network is as follows:

[0109] The pose estimation network is an improved version of ResNet-18, employing a fully convolutional network structure. Its input consists of two adjacent frames (the target frame). and source frame (Concatenate along the channel dimension; the input shape is...) The output is a 6-dimensional vector containing 3 translation parameters. and 3 rotation parameters (Euler angles) In post-processing, the 6-dimensional vector is converted to... Transformation matrix This is used for subsequent geometric projection calculations.

[0110] The decoding part of the pose estimation network includes a series of connected... Convolutional layer, 2 Convolutional layer and a The convolutional layer ultimately outputs six degrees of freedom information.

[0111] The construction process of the semantic segmentation network is as follows:

[0112] In this embodiment, DeepLabv3 is selected as the semantic segmentation module. Its core component is Spatial Pyramid Pooling with Hollows (ASPP). At low altitudes, the scale of objects varies greatly. Nearby tree branches occupy half the screen, while distant power towers appear as tiny dots. By paralleling multiple dilated convolutions with different dilation rates (such as 6, 12, and 18), ASPP can expand the receptive field without reducing resolution and capture multi-scale contextual information.

[0113] Based on low-altitude airworthiness requirements, this embodiment divides pixels into key categories: Sky, Terrain, Building, Wire / Pole, and Dynamic.

[0114] Step A3: Train the depth estimation network, pose estimation network, and semantic segmentation network using the training set; wherein the training process of the depth estimation network and pose estimation network is a self-supervised learning process, and the semantic segmentation network adopts supervised pre-training.

[0115] The core of this self-supervised learning lies in utilizing the geometric consistency of video sequences. For the target frame Each pixel in Deep networks predict their depth According to camera internal parameters Relative pose predicted by pose network It is possible to calculate the pixel in the source frame. Projected coordinates in :

[0116]

[0117] Using a bilinear interpolation sampling mechanism, based on coordinates From the source image Mid-sampling yields synthetic images If the depth and pose estimations are accurate, Should be consistent with real images Highly similar.

[0118] In this embodiment, the input consists of two consecutive frames from a continuous video. The depth estimation network receives the previous frame as input, while the pose estimation network receives both frames. The depth estimation network performs pixel-level depth estimation on the input images, extracting the depth value corresponding to each pixel in the previous frame. The pose estimation network extracts the pose transformation matrix between corresponding pixels in two consecutive frames. .

[0119] Output depth value With pose transformation matrix In the image reconstruction section, the current frame is reconstructed based on the reprojection formula, and the error between the reconstructed current frame and the original image is calculated.

[0120] The reprojection formula is as follows:

[0121]

[0122] in, This represents the depth value corresponding to each pixel in the current frame image. This represents the depth value corresponding to each pixel in the previous frame image. Represents the pose transformation matrix. Represents the camera intrinsic parameter matrix. This indicates the position of a pixel in the previous frame of the image. It indicates the position of a pixel in the current frame of the image.

[0123] In this embodiment, it is necessary to calculate the reprojection loss pixel by pixel to construct self-supervised information, back-optimize the relevant learning parameters in the depth estimation network and pose estimation network, output more accurate depth values ​​and pose transformation matrices to the image reconstruction part, and iterate this process repeatedly to reduce reprojection error, and finally continuously optimize the depth estimation network and pose estimation network models.

[0124] The loss function is designed as follows:

[0125] Total loss function Due to light loss and smoothing loss composition.

[0126] Minimum reprojection loss: To handle occlusion and field-of-view shift issues, instead of calculating the average error across all source frames, we take the error from all source frames (such as the previous frame). and the next frame The one with the smallest reprojection error:

[0127]

[0128] in SIM stands for Structural Similarity Index. It is typically set to 0.85. This strategy significantly reduces erroneous depth gradients caused by occlusion.

[0129] Edge-aware smoothing loss: To prevent noise in textured areas of the depth map while allowing abrupt depth changes at object edges, the following smoothing loss is used:

[0130]

[0131] in It is the normalized inverse depth. This formula means that when the image gradient is large (i.e., object edges), the smoothing constraint weights are reduced, allowing abrupt depth changes; when the image gradient is small (flat regions), depth smoothing is forced.

[0132] Multi-scale loss: Depth maps are output at different levels of the UNet++ decoder (e.g., 1 / 2, 1 / 4, 1 / 8 scales), and the above loss is calculated and then weighted and summed. This helps the network learn the scene structure progressively from coarse to fine.

[0133] During the training phase, the static scene assumption is violated if there are moving objects in the scene (such as cars driving on the ground). In this case, a binary weight map is generated using the semantic mask output by the semantic segmentation network. For pixels belonging to the categories of "vehicles" and "pedestrians", their weight in the photometric loss is reset to 0. This prevents the network from learning incorrect depth information and improves the accuracy of reconstructing static backgrounds (road surfaces, buildings).

[0134] Step A4: Test the trained depth estimation network and semantic segmentation network using the test set.

[0135] The testing phase includes a depth estimation network and a semantic segmentation network, which process the input single-frame RGB image in parallel. The depth estimation network outputs the original depth map. The semantic segmentation network outputs a semantic mask. The pose estimation network is removed during the inference phase and does not participate in the forward computation.

[0136] Step A5: The original depth map is fused with the semantic mask using a semantic-depth fusion algorithm to generate an optimized depth map.

[0137] The semantic-deep fusion algorithm specifically includes the following strategies:

[0138] Fusion Strategy 1: Sky Region Suppression. Self-supervised depth estimation often fails in the sky region because the sky lacks texture, and any depth value can satisfy luminosity consistency, causing the network to often predict the sky as a nearby object (infinite depth artifact). To address this scenario, the proposed fusion strategy will traverse the semantic mask. ,like Then set (Set to the sensor's maximum range or infinity marker). This eliminates the most troublesome skywall false alarm problem in low-altitude flight.

[0139] Fusion Strategy 2: Sharpening of Slender Obstacles. For slender objects such as power lines and towers, the edges of the depth map often exhibit dilation or erosion. To address this scenario, the algorithm's fusion strategy extracts connected regions belonging to the slender obstacle category from the semantic mask. For pixels within this region, its depth distribution histogram is calculated. Since power lines are usually suspended and closer to the camera than the background, local minima filtering or percentile-based depth correction is used to uniformly correct the depth value of this region to the foreground depth estimate of the object region. Bilateral filtering is applied at the mask boundaries, using semantic boundaries as guides to force the depth map edges to align with the semantic edges.

[0140] Through the above fusion strategy, the final output is an optimized dense depth map with semantic tags. .

[0141] Step A6: Output the optimized depth map to the downstream path planning module for the UAV's autonomous navigation and obstacle avoidance.

[0142] This embodiment improves the structure of the depth estimation network model and incorporates semantic information for post-processing fusion, significantly enhancing the monocular depth estimation performance in low-altitude scenes. Specifically, the improvements are as follows:

[0143] Improvement 1: The UNet++ architecture is adopted as the intermediate feature aggregation module. The skip connection method is modified and many intermediate feature aggregation nodes are added to the network to realize the capture and reuse of features at different levels, which enhances the ability to extract information on thin obstacles such as wires.

[0144] Improvement 2: The channel attention module ECANet was added to the decoder of the depth estimation network, which abandoned the dimensionality reduction operation, effectively captured the interaction relationship between channels, and improved the network's response to key channels;

[0145] Improvement 3: A semantic-depth fusion algorithm was designed to eliminate sky artifacts and sharpen obstacle edges using semantic information, which significantly improves the quality and practicality of the depth map.

[0146] Those skilled in the art should understand that, unless otherwise specified, the meanings of the technical and scientific terms used herein are consistent with the general understanding of the relevant technical field. Furthermore, terms defined in general dictionaries should be understood in the context of the technical background in this field and should not be interpreted in an overly idealized or formalistic manner divorced from practical application scenarios.

[0147] The above embodiments have described in detail the main concept, technical solution, and technical effects of the present invention. It should be noted that the above content is merely illustrative and not intended to limit the scope of protection of the present invention. Any equivalent modifications, substitutions, or optimizations based on the present invention without departing from its core principles are within the scope of the present invention.

Claims

1. A method for fusing low-altitude relative depth estimation and target segmentation, characterized in that, The method includes the following steps: Step S1: Acquire continuous video frames of the low-altitude airspace captured by the UAV's onboard monocular camera; Step S2: After the camera calibrates the continuous video frames, joint self-supervised training is performed. The corresponding outputs are obtained through the depth estimation network, semantic segmentation network, and pose estimation network, including the original relative depth map, semantic mask, and six-degree-of-freedom pose transformation parameters. The overall training loss function is constructed and the model is optimized to train the optimal model. The current target frame of the continuous video frames is input to the semantic segmentation network, and the semantic mask is output in real time. Step S3: Use the semantic segmentation network trained by joint self-supervised training to reason about video frames and generate pixel-level semantic masks; Step S4: Perform semantic and depth fusion. Based on the semantic mask, perform sky region suppression, elongated obstacle enhancement, and dynamic object masking operations to optimize the original relative depth map and generate a semantically enhanced relative depth map.

2. The low-altitude relative depth estimation and target segmentation fusion method according to claim 1, characterized in that, In step S2, the camera calibration video continuous frames include extracting frames from the continuous video sequence in time order, dividing two adjacent frames into training sample groups, with the previous frame as the target frame and the next frame as the source frame, and inputting them into the network after normalization and size unification preprocessing.

3. The low-altitude relative depth estimation and target segmentation fusion method according to claim 2, characterized in that, The joint self-supervised training in step S2 further includes the following steps: Step S21: Input the current target frame of the continuous video frames into the depth estimation network to obtain the original relative depth map; Step S22: Input the adjacent frames of the continuous video frames into the pose estimation network to predict the six-degree-of-freedom pose transformation parameters between adjacent frames; Step S23: The depth estimation network and pose estimation network are jointly trained again under self-supervised supervision to construct the reprojection loss function, edge smoothing loss function and multi-scale loss function, and the model training is completed by optimizing the overall training loss function.

4. The low-altitude relative depth estimation and target segmentation fusion method according to claim 3, characterized in that, In step S21, the depth estimation network connects the encoder and decoder feature maps through nested convolutional nodes. Each convolutional node fuses features from the same layer, upsampled features from the next layer, and features from skip connections. The ECANet channel attention mechanism module is set after feature fusion and before upsampling in the decoder to adaptively calculate channel weights through one-dimensional convolution, thereby enhancing the response to the feature channels of slender targets.

5. The low-altitude relative depth estimation and target segmentation fusion method according to claim 4, characterized in that, The joint self-supervised training in step S23 further includes: Based on the relative depth map and six-DOF pose transformation parameters, a reconstructed image is generated by reprojecting the source frame onto the target frame's viewpoint using a camera imaging geometric model. The reprojection loss function, edge smoothing loss function, and multi-scale loss function are weighted and fused together as the overall training loss function. The overall training loss function is defined as follows: in, Let the reprojection loss function be... The edge smoothing loss function is... For multi-scale loss functions, These are the weight coefficients of the corresponding loss function; the overall training loss is backpropagated to update the trainable parameters of the two networks; the validation set loss value is calculated after each training round, and training is stopped when the loss value tends to stabilize for several consecutive rounds and is less than the preset threshold, or when the preset maximum number of iterations is reached, and the optimal model parameters are saved.

6. The low-altitude relative depth estimation and target segmentation fusion method according to claim 5, characterized in that, In step S23, the edge smoothing loss function uses the image gradient as weight to constrain the smoothing of flat areas of the depth map and allows abrupt changes in depth in edge areas; the multi-scale loss function calculates the loss at different resolution levels of the decoder to avoid the network getting trapped in local optima; the reprojection loss function combines the L1 norm and SSIM structural similarity to calculate the photometric error and uses the minimum reprojection error strategy to solve the occlusion problem.

7. The low-altitude relative depth estimation and target segmentation fusion method according to claim 6, characterized in that, In step S3, the semantic segmentation network uses the hollow spatial pyramid pooling module to extract multi-scale semantic features, classifies the input image pixels into predefined categories such as sky and slender obstacles, and outputs a semantic mask in real time.

8. The low-altitude relative depth estimation and target segmentation fusion method according to claim 7, characterized in that, The fusion of semantics and depth in step S4 includes depth suppression of the sky region based on a semantic mask and depth boundary sharpening of the slender obstacle region; the sky region suppression is to set the depth value of the sky-like pixels to a preset maximum depth value; the slender obstacle enhancement is to use the semantic boundary as a constraint to perform local sharpening and guided filtering on the original relative depth map so that the depth edge is aligned with the semantic edge. The dynamic object masking method utilizes semantic masks to generate dynamic object masks, and resets the weights of pixels belonging to the dynamic category in the reprojection loss calculation to zero, thereby eliminating the interference of dynamic objects on the deep learning of static scenes.

9. A low-altitude relative depth estimation and target segmentation fusion system, characterized in that, The system applies a low-altitude relative depth estimation and target segmentation fusion method as described in any one of claims 1-8. The system includes a data acquisition module, a depth perception module, a pose estimation module, a semantic understanding module, and a fusion processing module. The data acquisition module is used to acquire video streams captured by the drone's monocular camera. The depth perception module includes a depth estimation network, which is used to receive a single frame of RGB image and output a relative depth map in real time; The pose estimation module is only enabled during the training phase and is used to assist in localization. It includes a pose estimation network based on a fully convolutional network. The input is adjacent frame images stitched together along the channel dimension, and the output is the six-degree-of-freedom motion parameters of the camera between adjacent frames. The semantic understanding module includes a semantic segmentation network based on DeepLabv3. The semantic segmentation network adopts the DeepLabv3 architecture and includes a hollow space pyramid pooling module. The fusion processing module is used to receive the original relative depth map and semantic mask, execute a depth optimization algorithm based on semantic rules, and output the optimized depth map; The aforementioned low-altitude relative depth estimation and target segmentation fusion system is carried by an unmanned aerial vehicle (UAV). The UAV's onboard monocular camera collects data, and its flight control system is configured with the semantically enhanced relative depth map to perform obstacle detection and autonomous obstacle avoidance path planning.

10. A low-altitude relative depth estimation and target segmentation fusion system according to claim 9, characterized in that, The depth estimation network adopts an encoder-decoder structure. The encoder uses a ResNet-18 residual network as the backbone network to extract features, and the decoder is designed based on the UNet++ architecture. The decoder integrates an ECANet channel attention mechanism module and uses dense skip connections to perform multi-scale feature fusion. The ECANet channel attention mechanism module recalibrates the weights of feature channels by learning the interaction between channels through one-dimensional convolution adaptive learning. The dense skip connections consist of a series of convolutional layers, bridging the semantic gap between the encoder and decoder feature maps and reducing information loss during the upsampling process.

Images