Unmanned aerial vehicle safe landing area identification method based on dual-task lightweight model

Abstract

The application provides a UAV safe landing area identification method based on a dual-task lightweight model, which uses synthetic aerial 3D data generated by three-dimensional reconstruction to construct a training set, simultaneously learns ground surface semantic categories and pixel-level depth information through a unified network, and obtains scene representation with semantic structure and spatial geometric features; in the model application stage, the RGB image collected by the UAV in real time is taken as input, the dual-task lightweight model synchronously outputs semantic segmentation results and depth estimation results, and safe landing area screening is completed based on depth gradient and local inclination; the application realizes a continuous processing flow of semantic segmentation, depth estimation and landing area determination through a single model, avoids calculation redundancy and structural complexity caused by multiple models in series, has the characteristics of unity, real-time performance and easy deployment, and can efficiently support autonomous safe landing of the UAV in a complex low-altitude environment on an embedded platform.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle vision processing, and particularly relates to a method for identifying a safe landing area of an unmanned aerial vehicle based on a dual-task lightweight model. BACKGROUND

[0002] At present, when an unmanned aerial vehicle (UAV) performs a task (such as inspection, surveying and mapping, emergency delivery and safe landing) in a city low-altitude scene, it usually relies on multi-sensor fusion (such as a laser radar, an ultrasonic wave and a depth camera) to complete environment perception and depth estimation. However, the above-mentioned sensors are high in price, high in power consumption and heavy in weight, and are not suitable for lightweight platforms.

[0003] In addition, a traditional vision algorithm usually models semantic segmentation (used for identifying a ground surface area) and depth estimation (used for judging a spatial geometric relationship) separately, needs to train a network and perform post-fusion respectively, and causes large calculation overhead, poor real-time performance and difficulty in embedding real-time operation of an on-board end of the unmanned aerial vehicle. SUMMARY

[0004] The embodiments of the present application provide a method for identifying a safe landing area of an unmanned aerial vehicle based on a dual-task lightweight model, solve the problems of task fragmentation, calculation redundancy, insufficient real-time performance and poor model generalization ability in a model processing process in the prior art, realize synchronous inference of semantic segmentation and depth estimation on an image collected by an on-board camera of the unmanned aerial vehicle, and thus complete environment structure understanding and safe landing area detection in a single model.

[0005] To achieve the above-mentioned purpose, the embodiments of the present application provide a method for identifying a safe landing area of an unmanned aerial vehicle based on a dual-task lightweight model, which comprises the following steps:

[0006] An RGB image is collected in real time by an on-board camera of the unmanned aerial vehicle, and illumination enhancement and color normalization processing are performed on the RGB image; the illumination enhancement and color normalization processing comprises one or more of Retinex illumination enhancement, adaptive histogram equalization and color consistency normalization, so as to reduce the influence of illumination change on the stability of the dual-task lightweight model;

[0007] input the processed RGB image into a dual-task lightweight model to obtain a semantic segmentation map and a depth map output by the dual-task lightweight model; the dual-task lightweight model is composed of an encoder, a center bottleneck module and a decoder; the dual-task lightweight model is obtained by pre-training in the following manner: a synthetic aerial 3D data set is constructed based on real geographical scene three-dimensional reconstruction data; the three-dimensional reconstruction data is obtained by using a Google Earth application and is used to generate a virtual data set containing an RGB image, a depth map and a semantic inclination label; the dual-task lightweight model is trained based on the synthetic aerial 3D data set and a joint loss function;

[0008] a pixel region with a class of horizontal in the semantic segmentation map is screened out, and a local inclination θ of the pixel region is determined by combining a height gradient extracted from the depth map and through a determination formula of the local inclination θ; the determination formula of the local inclination θ is: ; wherein, is a height gradient, is a pixel interval in a direction, is a pixel interval in a direction;

[0009] in a case where the local inclination θ is less than a preset angle threshold and an area of the pixel region is greater than a preset area threshold, the pixel region is taken as a safe landing region;

[0010] a connected region analysis and a boundary smoothing algorithm are used to process the safe landing region to output a landing area mask map; and the landing area mask map is superimposed on the RGB image to obtain a display result;

[0011] the display result is transmitted to a flight control system of the unmanned aerial vehicle for visual display.

[0012] In a possible implementation, the encoder is configured to perform multi-scale convolution feature extraction on an input image; the encoder includes a plurality of down-sampling convolution blocks and an ECA attention module, each down-sampling convolution block is composed of two 3x3 convolution layers, a batch normalization layer and a ReLU activation function; the down-sampling convolution block introduces a depth separable convolution to improve feature expression capability, and the ECA attention module is configured to dynamically adjust channel weights;

[0013] the center bottleneck module includes a multi-scale hollow convolution and a context concatenation mechanism, and is configured to realize fusion of multi-scale semantic and geometric features; the center bottleneck module further includes a cross-space attention module and a lightweight Transformer encoding unit, and is configured to realize cross-region dependency modeling and remote geometric relationship capturing;

[0014] The decoding block of each layer in the decoder includes a transpose convolution, a channel splicing, a standard convolution, and a normalization layer, and the image spatial resolution is recovered step by step through deconvolution and a jump connection; a bidirectional feature fusion strategy is introduced in the decoding stage, the jump connection is used to recover the detail information, and the context attention fusion is introduced to guide the decoding features to focus on the key ground surface area.

[0015] In a possible implementation, the decoding end of the dual-task lightweight model is divided into two output branches; the first branch generates a class probability distribution of each pixel through a 1*1 convolution layer, and outputs a ground surface class graph; the ground surface class includes horizontal, vertical or other; a Softmax layer is used for class normalization; the second branch uses a 1*1 convolution layer to output a continuous depth value of each pixel, and uses a scale-invariant depth loss and an inverse depth normalization mechanism to obtain relative real distance information.

[0016] In a possible implementation, the illumination enhancement and color normalization processing includes one or more of Retinex illumination enhancement, adaptive histogram equalization, and color consistency normalization, to reduce the influence of illumination changes on the stability of the dual-task lightweight model.

[0017] In a possible implementation, the joint loss function is:

[0018] ; wherein, L total is a joint loss function, L seg is a semantic segmentation loss, L depth is a depth estimation loss, L consistency is a semantic-geometric consistency constraint loss, and λ1, λ2, λ3 are weight coefficients.

[0019] In a possible implementation, the weight coefficients λ1, λ2, λ3 are dynamically adjusted based on a task adaptive weight adjustment mechanism by monitoring the gradient norm.

[0020] In a possible implementation, the RGB image is processed through a size normalization and a dynamic exposure compensation algorithm to automatically adjust the brightness in strong light / low light environments; the RGB image is based on the camera intrinsic parameters to perform real-time fisheye correction and perspective transformation.

[0021] In a possible implementation, the method further includes:

[0022] storing the display result for flight task backtracking of the unmanned aerial vehicle; and

[0023] retraining the dual-task lightweight model based on the display result.

[0024] In a possible implementation, the method further includes:

[0025] based on the display result, controlling the unmanned aerial vehicle to land in the safe landing area; or

[0026] controlling the unmanned aerial vehicle to adjust the path to land in the safe landing area.

[0027] The technical scheme provided in the embodiments of the present application has at least the following technical effects or advantages:

[0028] (1) The present application integrates semantic segmentation and depth estimation two visual tasks in a unified lightweight neural network framework, realizing the synchronous perception of structural semantic understanding and spatial geometric reconstruction of the unmanned aerial vehicle on-board monocular image, overcoming the problems of task fragmentation, calculation redundancy and insufficient real-time in the prior art; and combining area, continuity and confidence index for landing area determination, improving the accuracy of unmanned aerial vehicle landing area identification;

[0029] (2) The dual-task lightweight model of the present application adopts a shared encoder and a multi-scale bottleneck fusion structure, simultaneously modeling semantic and geometric information in a unified feature space, enabling collaborative learning and complementary enhancement of the two types of features, thereby significantly improving the representation ability and perception consistency of the model; and by introducing a joint loss optimization mechanism, the semantic segmentation task provides accurate area boundary constraints at the pixel level, and the depth estimation task provides structural priors of continuous space at the geometric level, forming a complementary optimization relationship between the two in the training process, effectively reducing the false detection and missed detection rate of the model in complex scenes;

[0030] (3) The synthetic aerial 3D data generation mechanism based on the three-dimensional reconstruction data construction of real geographic scenes proposed by the present application generates training samples by using three-dimensional reconstruction models derived from real ground structures to construct a virtual training domain covering multiple types of terrain and multiple scales of perspective; this mechanism can provide complete RGB-depth-semantic inclination labels without real depth annotation samples, breaking through the bottleneck of the lack of depth annotation in unmanned aerial vehicle aerial scenes, and significantly improving the robustness and generalization performance of the model under conditions of illumination variation, terrain difference, texture complexity and structural noise;

[0031] (4) The present application can be directly deployed on an embedded GPU platform (such as NVIDIA Jetson TX2, Orin NX, etc.), through channel sharing and lightweight operator design, realizing real-time inference performance of hundreds of frames (≥130FPS), significantly reducing the calculation energy consumption and delay while ensuring high-precision output. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application or the prior art. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0033] Figure 1 A flow chart of a method for identifying a safe landing area of a UAV based on a dual-task lightweight model according to an embodiment of the present application is shown in FIG. 4.

[0034] Figure 2 A flow chart of another method for identifying a safe landing area of a UAV based on a dual-task lightweight model according to an embodiment of the present application is shown in FIG. 5.

[0035] Figure 3 A block diagram of a system for identifying a safe landing area of a UAV based on a dual-task lightweight model according to an embodiment of the present application is shown in FIG. 6. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort fall within the scope of the present application.

[0037] In the description of the embodiments of the present application, it should be noted that the terms “center”, “upper”, “lower”, “left”, “right”, “vertical”, “horizontal”, “inner”, “outer” and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only used to facilitate the description of the embodiments of the present application and simplify the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. The terms “first”, “second”, “third” are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance. In addition, the terms “mounting”, “connecting”, “connecting” should be understood broadly, for example, it can be fixedly connected, or detachably connected, or integrally connected; it can be mechanically connected, or electrically connected; it can be directly connected, or indirectly connected through an intermediate medium, or the communication between two elements inside. For those skilled in the art, the specific meaning of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0038] Firstly, the application scenario of the present application is introduced, and the present application is applied in the terrain recognition scene in the landing process of the unmanned aerial vehicle. It is proposed in related researches to calculate the ground safety area (such as FreeLSD, Slope-based Terrain Reconstruction, etc.) by using a three-dimensional reconstruction algorithm or a multi-view geometric method, and the idea is to reconstruct a local terrain model through stereo vision, and then estimate the plane inclination according to the normal vector to judge the landing. However, such methods have a very large amount of calculation, and usually take several seconds to process small size images (about 300x300 pixels) on a desktop CPU, which cannot meet the real-time processing requirements of several tens of frames per second in the flight of the unmanned aerial vehicle.

[0039] It can be seen that there are problems of task fragmentation and calculation redundancy in the existing algorithms: the depth estimation and semantic segmentation network are usually trained separately, and the complementary information between the two is not fully utilized, resulting in a large network size, repeated calculation, and slow inference speed.

[0040] In addition, the depth annotation data in the field of unmanned aerial vehicle aerial photography is extremely scarce, resulting in insufficient accuracy and stability of the model in real scenes; most methods can only run on desktop-level GPUs, making it difficult to realize real-time inference on embedded platforms such as Jetson TX2; at the same time, the existing models have poor universality, and the integrated visual perception ability of "depth estimation based on monocular image and safety area recognition" has not been realized under a unified system.

[0041] To solve the above technical problems, the application provides a UAV safe landing area identification method based on a dual-task lightweight model, which comprises the following steps: collecting an RGB image in real time through a UAV on-board camera, and performing illumination enhancement and color normalization processing on the RGB image; the illumination enhancement and color normalization processing comprises one or more of Retinex illumination enhancement, adaptive histogram equalization and color consistency normalization, so as to reduce the influence of illumination changes on the stability of the dual-task lightweight model; inputting the processed RGB image into the dual-task lightweight model to obtain a semantic segmentation map and a depth map output by the dual-task lightweight model; the dual-task lightweight model is composed of an encoder, a center bottleneck module and a decoder; the dual-task lightweight model is obtained by pre-training in the following manner: constructing a synthetic aerial 3D data set based on real geographical scene three-dimensional reconstruction data; the three-dimensional reconstruction data is obtained through Google Earth application and used to generate a virtual data set containing RGB images, depth maps and semantic inclination labels; the preset network model is trained based on the synthetic aerial 3D data set and a joint loss function, and is used as the dual-task lightweight model; filtering out a pixel region with a horizontal class in the semantic segmentation map, combining the height gradient extracted from the depth map, and determining the local inclination θ of the pixel region through a local inclination θ determination formula; in the case that the local inclination θ is less than a preset angle threshold and the area of the pixel region is greater than a preset area threshold, the pixel region is regarded as a safe landing area; processing the safe landing area by using a connected region analysis and boundary smoothing algorithm to output a landing area mask map; and superimposing the landing area mask map on the RGB image to obtain a display result; and transmitting the display result to the flight control system of the UAV for visual display.

[0042] According to the above technical scheme, by integrating semantic segmentation and depth estimation two visual tasks in a unified lightweight neural network framework, synchronous perception of structural semantic understanding and spatial geometric reconstruction of the UAV on-board monocular image is realized, and the problems of task fragmentation, calculation redundancy and insufficient real-time performance in the prior art are overcome; the network adopts a shared encoder and a multi-scale bottleneck fusion structure to model semantic and geometric information in a unified feature space, so that the two types of features realize collaborative learning and complementary enhancement, thereby significantly improving the representation ability and perception consistency of the model; and by introducing a joint loss optimization mechanism, the semantic segmentation task provides accurate regional boundary constraints at the pixel level, and the depth estimation task provides structural priors of continuous space at the geometric level, and the two form a complementary optimization relationship in the training process, effectively reducing the false detection and missed detection rate of the model in complex scenes.

[0043] Figure 1 A flowchart of a UAV safe landing area identification method based on a dual-task lightweight model provided by the embodiment of the application is shown in FIG. 1. Figure 1As shown, the method can include the following steps.

[0044] S101, real-time collection of an RGB image by a UAV on-board camera, and performing illumination enhancement and color normalization processing on the RGB image.

[0045] In one possible implementation, the illumination enhancement and color normalization processing includes one or more of Retinex illumination enhancement, adaptive histogram equalization, and color consistency normalization, to reduce the impact of illumination changes on the stability of the dual-task lightweight model.

[0046] In an example, the illumination enhancement and color normalization processing can be performed on the RGB image in low-light conditions, thereby improving the robustness of the dual-task lightweight model in low-light environments and enhancing the model's robust perception ability under different time, weather, and lighting conditions.

[0047] S102, inputting the processed RGB image into a dual-task lightweight model to obtain a semantic segmentation map and a depth map output by the dual-task lightweight model; the dual-task lightweight model is composed of an encoder, a center bottleneck module, and a decoder.

[0048] The dual-task lightweight model is pre-trained in the following manner: a synthetic aerial 3D data set is constructed based on real geographical scene three-dimensional reconstruction data; the three-dimensional reconstruction data is obtained through a Google Earth application and used to generate a virtual data set containing RGB images, depth maps, and semantic inclination labels; and the dual-task lightweight model is trained based on the synthetic aerial 3D data set and a joint loss function.

[0049] S103, filtering out a pixel region of a class of horizontal in the semantic segmentation map, and combining a height gradient extracted from the depth map to determine a local inclination θ of the pixel region through a determination formula of the local inclination θ.

[0050] The determination formula of the local inclination θ is: ; wherein, is a height gradient, is a pixel interval in the direction, is a pixel interval in the direction. is a pixel interval in the direction.

[0051] S104, in a case where the local inclination θ is less than a preset angle threshold and a region area of the pixel region is greater than a preset area threshold, the pixel region is taken as a safe fall region.

[0052] In an example, the region area of the pixel region can be determined through the semantic segmentation map.

[0053] ​For example, the preset angle threshold can be within the range of 10°-20°, such as 10°, 15°, or 20°, etc., and is not limited here. The preset area threshold can be within 0.1m. 2 -0.6m 2 Within a range, for example, 0.1m 2 0.3m 2 0.5m 2 or 0.6m 2 etc., no specific limit is specified here.

[0054] S105. The safe landing area is processed using connected component analysis and boundary smoothing algorithms to output a landing area mask; and the landing area mask is superimposed on the RGB image to obtain the display result.

[0055] S106. Transmit the display results to the flight control system of the UAV for visualization.

[0056] This application integrates two visual tasks, semantic segmentation and depth estimation, into a unified lightweight neural network framework, achieving simultaneous perception of structural semantic understanding and spatial geometric reconstruction of UAV-borne monocular images. This overcomes the problems of task fragmentation, computational redundancy, and insufficient real-time performance in existing technologies. Furthermore, by combining area, continuity, and confidence indices for landing area determination, the accuracy of UAV landing area identification is improved.

[0057] In one possible implementation, the encoder is used to extract multi-scale convolutional features from the input image. The encoder includes multiple downsampled convolutional blocks and an ECA attention module. Each downsampled convolutional block consists of two 3×3 convolutional layers, a batch normalization layer, and a ReLU activation function. The downsampled convolutional blocks introduce depthwise separable convolution to improve feature representation capabilities, and the ECA attention module dynamically adjusts channel weights. The central bottleneck module includes multi-scale dilated convolution and a context concatenation mechanism to achieve the fusion of multi-scale semantic and geometric features. The central bottleneck module also includes a cross-spatial attention module and a lightweight Transformer encoding unit to achieve cross-region dependency modeling and long-range geometric relationship capture. Each layer of the decoder's decoding block contains transposed convolution, channel concatenation, standard convolution, and a normalization layer, progressively restoring the image's spatial resolution through deconvolution and skip connections. A bidirectional feature fusion strategy is introduced during the decoding stage, restoring detailed information through skip connections and introducing contextual attention fusion to guide the decoded features to focus on key surface regions.

[0058] The encoder extracts multi-scale features from local texture to global structure through layer-by-layer downsampling, and preserves shallow spatial details through residual connection and channel concatenation structure to provide high-fidelity geometric information for subsequent segmentation and depth prediction. The cross-spatial attention block and mobile ViT are further embedded in the center bottleneck module to realize cross-region dependency modeling and remote geometric relationship capture. This design can balance local texture and scene geometric depth features on the premise of not significantly increasing the number of parameters, thereby enhancing the understanding of environmental structure.

[0059] The encoder and decoder of the dual-task lightweight model can adopt various lightweight or multi-scale structures, such as U-Net, SegNeXt, PIDNet, BiSeNetV2 convolutional model, or new Transformer-based structure (such as SegFormer, ConvNeXt or MobileViT) to enhance remote dependency modeling capability.

[0060] In one possible implementation, the decoding end of the dual-task lightweight model is divided into two output branches; the first branch generates the class probability distribution of each pixel through a 1x1 convolution layer, and outputs the ground surface class map; the ground surface class includes horizontal, vertical or other; the Softmax layer is used for class normalization; the second branch uses a 1x1 convolution layer to output the continuous depth value of each pixel, and uses scale-invariant depth loss and inverse depth normalization mechanism to obtain relative true distance information. Through the two output branches, a dual-task structure is formed, and a shared encoder and bottleneck layer design are used to output high-precision semantic and geometric results simultaneously under the premise of only about 1M parameters, realizing end-to-end real-time inference.

[0061] The channel sharing and feature concatenation strategy can be used to replace the traditional multi-branch repeated convolution, which reduces the number of parameters by about 30 times and reduces the computational complexity by more than 50% while maintaining the model accuracy, significantly saving memory and power compared to DeepLabV3+ or U-Net structure, solving the problem of limited computing power on embedded platforms and providing structural support for embedded real-time deployment.

[0062] It should be noted that the dual-task lightweight model can be deployed on an NVIDIA Jetson TX2 platform, a Jetson Orin NX, a Rockchip RK3588, an Ascend 310, or an FPGA accelerator, and the dual-task lightweight model can also be compiled and quantization accelerated by an inference framework such as TensorRT, ONNX Runtime, OpenVINO, or TVM to obtain the same real-time inference performance and energy efficiency optimization effect.

[0063] In a possible implementation, the dataset of the dual-task lightweight model can be obtained by generating large-scale synthetic samples based on a Google Earth three-dimensional reconstruction technology, and the data simultaneously contains RGB images, depth maps, and inclination labels (divided into three categories of horizontal, vertical, and other), and then using the synthetic aerial 3D dataset generated by the three-dimensional reconstruction model as the training input sample of the dual-task lightweight model, a virtual training domain covering various urban and suburban terrains can be constructed, so that the dual-task lightweight model obtains stable semantic structure feature and spatial geometric feature representation in the training stage, thereby improving the robustness and generalization ability of the model under complex lighting, texture change, and terrain difference conditions.

[0064] For example, the input sample can also be a multi-view sample generated based on a virtual simulation platform (such as a Microsoft AirSim, CARLA, or Unreal Engine environment) for model pre-training and domain adaptation learning. By adding simulation enhancement strategies such as wind field disturbance, lighting change, and terrain randomization, the cross-domain generalization performance of the model can be further improved. For actual deployment scenarios, small-scale real scene samples can be collected in the target area for fine-tuning to achieve the same adaptability target.

[0065] For example, the two output branches in the dual-task network model are trained in two stages through a multi-stage collaborative optimization mechanism to fully utilize the complementarity between the semantic and geometric tasks. In the first stage, the depth estimation task is dominant, and the three-dimensional structure perception of the network is improved through geometric reconstruction constraints. In the second stage, the semantic segmentation task is introduced and optimized jointly to utilize the semantic boundary information to refine the depth edges and ground surface determination. This collaborative training method of “geometric leading and semantic refinement” effectively avoids the problem of single-task overfitting and significantly improves the transferability of the model in unknown scenes.

[0066] In a possible implementation, the joint loss function is: ; wherein L total is the joint loss function, L seg is the semantic segmentation loss, L depth is the depth estimation loss, and L consistencyFor the semantic-geometric consistency constraint loss, λ1, λ2, and λ3 are weight coefficients.

[0067] For example, the semantic segmentation loss can be a cross-entropy loss function and an OHEM (Online Hard Example Mining) hard example mining mechanism, or a Focal Loss, a Dice Loss, etc., to optimize the sample imbalance problem or improve the discrimination ability for small obstacle regions. The depth estimation loss can be an L1 loss, an L2 loss, a Scale-Invariant loss, a depth error, or a BerHu loss, to adapt to different depth distribution characteristics. The semantic-geometric consistency constraint loss can use a gradient similarity or edge alignment based method to ensure the consistency of the segmentation boundary and the depth gradient.

[0068] In a possible implementation, the weight coefficients λ1, λ2, and λ3 are dynamically adjusted based on a task adaptive weight adjustment mechanism by monitoring the gradient norm. In this way, the training stability and task trade-off can be improved, and the two tasks can be promoted in the training by dynamically adjusting the weight coefficients based on the gradient norm.

[0069] In a possible implementation, the RGB image is processed by a size normalization and a dynamic exposure compensation algorithm to automatically adjust the brightness in strong light / low light environments, and the RGB image is real-time processed by a fisheye correction and a perspective transformation based on the camera intrinsic parameters. For example, the size normalization can adapt the image size to the model input size, which can be 640x480 pixels or 512x512 pixels, and the dynamic exposure compensation algorithm can suppress the influence of illumination changes.

[0070] In a possible implementation, the method further includes: storing the display result for flight task backtracking of the UAV; and retraining the dual-task lightweight model based on the display result.

[0071] In a possible implementation, the method further includes: controlling the UAV to land in the safe landing area based on the display result; or controlling the UAV to adjust the path to land in the safe landing area.

[0072] Figure 2 Another flowchart of a method for identifying a safe landing area of a UAV based on a dual-task lightweight model is provided in the embodiments of the present application. As shown in Figure 2 the method can include the following steps.

[0073] S201, a 3D synthetic dataset.

[0074] The training sample is read from a synthetic aerial 3D data set constructed based on real geographic scene three-dimensional reconstruction data, and the training sample includes one-to-one corresponding RGB images, pixel-by-pixel depth maps (Depth), and semantic tilt angle labels (horizontal H, vertical V, and other O). Among them, the RGB images in the synthetic aerial 3D data set can be used as the input of the dual-task lightweight model, and the depth value and the semantic tilt angle label can be used as a supervision signal for training the dual-task lightweight model.

[0075] S202, image input and preprocessing.

[0076] Before the dual-task lightweight model is trained by the synthetic aerial 3D data set, size normalization and brightness normalization processing (using CLAHE) can be performed on it to adapt the model input size (640x480 or 512x512 pixels) and improve the uneven lighting problem caused by the three-dimensional reconstruction texture difference. According to the needs, the Retinex lighting enhancement algorithm and color consistency normalization processing can also be performed on the RGB images in the synthetic aerial 3D data set to improve the robustness and multi-scene learning ability of the model under different terrain, lighting and texture conditions.

[0077] S203, feature extraction and coding.

[0078] The input image is subjected to multi-scale convolution feature extraction by the encoder module of the dual-task lightweight model. The encoder includes multiple down-sampling convolution blocks, each block consisting of two 3x3 convolution layers, batch normalization layers and ReLU activation functions. In order to improve the feature expression ability while maintaining high efficiency, the encoder introduces depthwise separable convolution and ECA attention module (Efficient Channel Attention) to dynamically adjust the channel weight. Through layer-by-layer down-sampling, the model extracts multi-scale features from local texture to global structure, and retains the spatial details of the shallow layer through residual connection and channel concatenation (Concatenation) structure, providing high-fidelity geometric information for subsequent segmentation and depth prediction.

[0079] S204, feature fusion bottleneck layer.

[0080] The encoded features enter a central bottleneck module of the dual-task lightweight model, which adopts multi-scale dilated convolution (Dilation Rates = 1, 2, 4, 8, 16, 32) and a context concatenation mechanism to realize the fusion of multi-scale semantic and geometric features. Unlike the single-scale bottleneck layer of traditional U-Net or DeepLabV3+, the bottleneck layer of the present application further embeds a cross-space attention module (Cross-Spatial Attention Block) and a lightweight Transformer encoding unit (MobileViT) to realize cross-region dependency modeling and remote geometric relationship capture. This design can simultaneously consider the local texture and scene geometric depth features of the ground surface without significantly increasing the parameter amount, thereby enhancing the environmental structure understanding ability.

[0081] S205, decoding and up-sampling.

[0082] The decoder module of the dual-task lightweight model restores the image spatial resolution step by step through deconvolution and skip connection. Each decoding block contains a transpose convolution, channel concatenation, standard convolution and normalization layer. To strengthen the feature reconstruction quality, a bi-directional feature fusion (Bi-Directional Feature Fusion) strategy is introduced in the decoding stage. On the one hand, the skip connection is used to restore the detail information, and on the other hand, the context-aware fusion (Context-Aware Fusion) is introduced to guide the decoding features to focus on the key ground area. The final output decoding features are consistent with the input image in spatial size, providing a unified feature base for dual-task prediction.

[0083] S206, dual-task prediction output.

[0084] The dual-task lightweight model is divided into two output branches at the decoding end. Branch 1 (semantic segmentation head) generates the class probability distribution of each pixel through a 1x1 convolution layer, outputs the ground surface class map (including horizontal / vertical / other three categories), and the Softmax layer is used for class normalization. Branch 2 (depth estimation head) uses a 1x1 convolution layer to output the continuous depth value of each pixel, and uses a scale-invariant depth loss (Scale-Invariant Depth Loss) and an inverse depth normalization mechanism to obtain relative true distance information. The dual-task structure of the dual-task lightweight model adopts a shared encoder and bottleneck layer design, which can output high-precision semantic and geometric results while the parameter amount is only about 1M, realizing end-to-end real-time inference.

[0085] S207, joint loss training.

[0086] The dual-task lightweight model adopts a joint loss function in the network training process: ; wherein, L total is a joint loss function, L seg is a semantic segmentation loss (cross-entropy and OHEM hard example mining mechanism), L depth is a depth estimation loss (L2 or scale invariant depth error), L consistency is a semantic-geometric consistency constraint loss for ensuring consistency between the segmentation boundary and the depth gradient, and λ1, λ2, λ3 are weight coefficients.

[0087] To improve the training stability and task weighting, a task adaptive weight adjustment mechanism (Dynamic Weight Averaging, DWA) can be introduced to dynamically adjust the coefficient by monitoring the gradient norm, so that the two tasks promote each other in training.

[0088] S208, reasoning and safety zone determination.

[0089] The dual-task lightweight model in the reasoning stage takes a single-frame RGB image as input and simultaneously outputs a semantic segmentation map and a depth map. By screening the pixel region with a class of "horizontal" in the semantic segmentation map and combining the depth map to calculate the local tilt angle θ: ; wherein, is the height gradient, is the pixel interval in the horizontal direction, is the pixel interval in the vertical direction. In the case where the local tilt angle θ < 15° and the area is > 0.3m 2 , the region is determined as a safe landing zone.

[0090] Meanwhile, to improve robustness, connected component analysis (Connected Component Analysis) and edge refinement with CRF (Edge Refinement with CRF) are used for post-processing of the results, and the final safe landing zone mask map is output, which can be superimposed on the original RGB image for real-time visualization.

[0091] It should be noted that the above S201 can be a dataset construction stage, and the above S202 to S207 can be a dual-task network training stage, and the above S208 can be a UAV image reasoning stage. The dataset construction stage and the dual-task network training stage can be pre-processed on a computer or a server to obtain a trained dual-task lightweight model. Then, the trained dual-task lightweight model can be deployed on a UAV, so that the UAV can collect images in real time during work and perform synchronous reasoning of semantic segmentation and depth estimation through the dual-task lightweight model, thereby completing environment structure understanding and safe landing zone detection in a single model.

[0092] According to the technical solution, the application has at least the following technical effects:

[0093] (1) The application proposes a single-network dual-task collaborative architecture, which can simultaneously realize the structure of semantic segmentation and depth estimation under a unified network framework, adopts a shared encoder and a dual-output decoder design, realizes the complementary features of semantic and geometric information, significantly reduces the calculation redundancy and improves the inference speed, and ensures real-time perception on an embedded platform;

[0094] (2) The application proposes a multi-scale feature fusion module based on a hollow convolution, which can introduce a multi-expansion rate hollow convolution and a feature splicing mechanism at the bottleneck layer, can simultaneously capture the detailed texture and global spatial structure information of the ground surface at different scales, improve the recognition ability of the model for complex terrain and obstacle areas, and enhance the comprehensiveness and robustness of perception;

[0095] (3) The application proposes a semantic-geometric joint loss optimization mechanism, which constructs a joint optimization framework of semantic segmentation cross-entropy loss and depth estimation L2 loss, so that the two tasks constrain and promote each other in the training process, the semantic segmentation task provides boundary and class priori, and the depth estimation task provides geometric consistency information, thereby significantly improving the accuracy and generalization performance of the model in complex scenes.

[0096] Figure 3 A block diagram of an unmanned aerial vehicle safe landing area recognition system based on a dual-task lightweight model is provided for the embodiments of the application. As shown in Figure 3 The system can include the following modules: perception input module 301, feature calculation module 302, feature fusion module 303, dual-task collaborative module 304, perception result cache module 305, inference and landing area determination module 306, and flight control interface and display module 307.

[0097] (1) The perception input module 301 is used to collect the RGB image of the onboard camera of the unmanned aerial vehicle, and performs brightness equalization, distortion correction and normalization. The perception input module 301 is located at the front end of the system data stream, and is responsible for providing a stable and uniform input source for the neural network, which significantly improves the robustness of subsequent feature extraction.

[0098] The perception input module 301 can include: an imaging unit: using a low-delay CMOS sensor, connected with the host system through a MIPI or USB interface; an illumination adaptive unit: using adaptive histogram equalization (CLAHE) and dynamic exposure compensation algorithm to automatically adjust brightness in strong light / low light environment; distortion and geometric correction unit: based on camera internal parameters for real-time fisheye correction and perspective transformation; cache and prefetch queue: constructing a ring buffer (Ring Buffer) in the host memory to ensure continuous input of image stream and timestamp alignment; interface output: transmitting the preprocessed image to the feature calculation module in Tensor format.

[0099] (2) The feature calculation module 302 is connected with the perception input module 301, and is used for performing convolution extraction, channel attention weighting and multi-scale downsampling of image features. The multi-scale feature maps output by the feature calculation module 302 are directly sent to the feature fusion module 303.

[0100] The feature calculation module 302 is composed of multi-level convolution calculation units and attention mechanism units, and has the following functions: a lightweight convolution structure (such as Depthwise Separable Conv) is used to reduce the calculation overhead; an efficient channel attention (ECA) and a batch normalization (BN) unit are embedded to improve the feature expression quality; multi-scale down-sampling (×2, ×4, ×8 times) feature layers are realized to facilitate subsequent multi-task feature fusion; parallel multi-thread CUDA kernel calls are supported to utilize GPU shared memory to optimize bandwidth.

[0101] (3) The feature fusion module 303 is used for performing dilated convolution and Transformer global modeling on multi-scale features. The feature fusion module 303 serves as the core input layer of the double-task collaborative module 304, and provides high-dimensional fused semantic and geometric feature representation.

[0102] The feature fusion module 303 internally integrates a multi-branch fusion structure, including: a dilated convolution branch: realizing multi-scale context perception through dilated convolution with different expansion rates (1, 2, 4, 8, 16, 32); a global modeling branch: using a lightweight Transformer encoding unit (MobileViT) to capture long-distance dependency relationships; a feature fusion unit: integrating local texture and global geometric features through splicing and weighted fusion mechanisms; a dimension reduction unit: using 1×1 convolution to compress the channel number to form a shared feature tensor.

[0103] (4) Dual-task coordination module 304, connected with the feature fusion module 303, is used for simultaneously performing semantic segmentation and depth estimation in a unified feature space. The dual-task coordination module 304 realizes mutual compensation of semantic and geometric information through shared bottleneck features and consistency constraint mechanism, thereby improving the overall robustness and accuracy of the system.

[0104] The dual-task coordination module 304 is composed of two cooperative sub-modules (a semantic segmentation sub-module and a depth estimation sub-module), a task coordination control unit, and an asynchronous inference interface. The semantic segmentation sub-module is used to output a pixel-level ground surface class probability map based on a convolution-deconvolution structure; the depth estimation sub-module is used to predict a pixel-level relative depth value under the same feature input; the task coordination control unit is used to be responsible for feature sharing and inference scheduling between the two tasks, so as to ensure that resource competition does not occur during parallel calculation; and the asynchronous inference interface is used to allow the system to alternately update segmentation and depth results between different frames, thereby realizing stream output.

[0105] (5) Perception result cache module 305, connected with the dual-task coordination module 304, is used for storing inference results and ensuring data consistency. The perception result cache module 305 realizes decoupling of data input and inference output through a “producer-consumer” parallel model, thereby improving system stability and concurrent performance.

[0106] The perception result cache module 305 includes: a result buffer area for storing semantic probability maps, depth maps, and time stamps thereof; a synchronization management unit for realizing cross-task result alignment through frame identification and time stamp mechanism; data integrity detection for monitoring the legitimacy of result data and discarding abnormal frames in real time; and an interface service for providing a unified access channel for the subsequent inference and landing area determination module 306.

[0107] (6) Inference and landing area determination module 306, connected with the perception result cache module 305, is used for fusing semantic and depth results, calculating terrain inclination, and identifying a landing area. The inference and landing area determination module 306 is the decision core of the system, and realizes mapping from semantic-geometric joint inference to flight control.

[0108] The main functions of the inference and landing area determination module 306 include: a geometric analysis unit for extracting height gradient information from a depth map and calculating a local inclination; a region screening unit for selecting a “horizontal” region according to a semantic result; a region verification unit for determining a landing area in combination with area, continuity, and confidence indicators; a post-processing unit for performing boundary smoothing (CRF) and connected region labeling; and an output interface for sending a safe landing area mask and position information to a flight control interface and a display module 307.

[0109] (7) Flight control interface and display module 307, connected with the inference and landing area determination module 306, is used for transmitting landing area results to a flight control system and performing visual display.

[0110] The flight control interface and display module 307 includes: a flight control communication unit that interacts with the flight control mainboard through UART, CAN or MAVLink protocol; a display output unit that superimposes semantic segmentation map, depth map and landing area boundary in real time; a log recording unit that stores historical prediction results for flight task backtracking and model retraining; task instruction feedback: the landing area result can be used for triggering and path adjustment of the flight control automatic landing logic.

[0111] The above-mentioned system forms a hierarchical structure of "sensing input-feature extraction-feature fusion-task coordination-result caching-landing area determination-flight control feedback", and uses shared memory (Shared Memory) and asynchronous pipeline mechanism (Asynchronous Pipeline) inside the system to realize parallel operation of image acquisition, neural network inference and landing area determination. During operation, the system can dynamically adjust the inference frame rate and task priority according to the flight task, support single task degradation mode (such as only segmentation or only depth estimation), and ensure stable operation on different computing platforms. The various modules in the system communicate through a high-bandwidth data bus, and run in an asynchronous pipeline parallel manner on an embedded GPU platform, thereby realizing high-precision, low-power visual perception and landing area decision while ensuring real-time performance.

[0112] In summary, the unmanned aerial vehicle safe landing area recognition method and system based on the dual-task lightweight model of the present application have at least the following improvements:

[0113] (1) Task structure: the traditional scheme models depth estimation and semantic segmentation separately, and the features cannot be shared. The present application realizes complementary perception of semantic and geometric features by sharing the encoder and bottleneck layer, and improves the task coordination efficiency;

[0114] (2) Computing efficiency: existing methods usually require two independent forward inferences, while the present application realizes dual-task parallel inference in a single network, reducing the overall inference overhead by more than 50%;

[0115] (3) Real-time performance: the traditional network can only achieve a speed of less than 10 frames per second on a desktop GPU, while the present application can stably run at about 130 frames per second on an embedded Jetson TX2 platform, realizing real-time safe landing area recognition;

[0116] (4) Data dependency: existing models are highly dependent on real depth annotation data, which is subject to problems such as high acquisition cost and narrow coverage. The hybrid training mechanism of the present application effectively alleviates the annotation bottleneck and improves the model generalization performance;

[0117] (5) Deployability: The traditional model has a large number of parameters and a complex structure, which is difficult to directly embed in the flight control system, while the model of the application has only about 1M parameters, and has strong lightweight and portability;

[0118] (6) Reliability: The traditional model is prone to failure in low light, complex terrain and other environments, while the application significantly improves the model robustness and environmental adaptability through semantic-geometric joint constraints.

[0119] Each embodiment in the specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other, and each embodiment mainly describes the difference from other embodiments.

[0120] The above examples are only used to illustrate the technical solutions of the application, and are not limited to the application; although the application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the application.

Claims

1. A method for identifying a safe landing area of a UAV based on a dual-task lightweight model, characterized in that, The method comprises: real-time collection of RGB images by a UAV on-board camera, performing illumination enhancement and color normalization on the RGB images; inputting the processed RGB images into a dual-task lightweight model to obtain a semantic segmentation map and a depth map output by the dual-task lightweight model; the dual-task lightweight model is composed of an encoder, a center bottleneck module and a decoder; the dual-task lightweight model is pre-trained in the following manner: a synthetic aerial 3D dataset is constructed based on real geographical scene 3D reconstruction data; the 3D reconstruction data is obtained by using a Google Earth application and is used to generate a virtual dataset containing RGB images, depth maps and semantic inclination labels; the dual-task lightweight model is trained based on the synthetic aerial 3D dataset and a joint loss function; Screening the pixel region of the semantic segmentation map with the class of horizontal, and combining the height gradient extracted from the depth map, the local inclination θ of the pixel region is determined by a determination formula of the local inclination θ. ; wherein, is the height gradient, is the pixel interval of the direction, is the pixel interval of the direction; in a case where the local inclination θ is less than a preset angle threshold and the area of the pixel region is greater than a preset area threshold, the pixel region is taken as a safe landing region; processing the safe landing region by using a connected region analysis and a boundary smoothing algorithm to output a landing area mask map; and superimposing the landing area mask map on the RGB images to obtain a display result; transmitting the display result to a flight control system of the UAV for visual display; the encoder is used for multi-scale convolution feature extraction of an input image; the encoder comprises a plurality of down-sampling convolution blocks and an ECA attention module; each down-sampling convolution block is composed of two 3×3 convolution layers, a batch normalization layer and a ReLU activation function; the down-sampling convolution block introduces a depth separable convolution to improve feature expression capability; and the ECA attention module is used for dynamically adjusting channel weights; the center bottleneck module comprises a multi-scale hollow convolution and a context concatenation mechanism, and is used for realizing fusion of multi-scale semantic and geometric features; the center bottleneck module further comprises a cross-space attention module and a lightweight Transformer coding unit, and is used for realizing cross-region dependency modeling and remote geometric relationship capturing; each decoding block in the decoder comprises a transposed convolution, a channel concatenation, a standard convolution and a normalization layer, and gradually recovers the image spatial resolution through inverse convolution and skip connection; a bidirectional feature fusion strategy is introduced in the decoding stage to recover detailed information through skip connection, and a context attention fusion is introduced to guide the decoding features to focus on key ground surface regions; the decoding end of the dual-task lightweight model is divided into two output branches; a first branch generates a class probability distribution of each pixel through a 1×1 convolution layer to output a ground surface class map; the ground surface class includes horizontal, vertical or other; a Softmax layer is used for class normalization; and a second branch outputs a continuous depth value of each pixel by using a 1×1 convolution layer, and uses a scale-invariant depth loss and an inverse depth normalization mechanism to obtain relative real distance information; the joint loss function is: ; wherein L total is a joint loss function, L seg is a semantic segmentation loss, L depth is a depth estimation loss, L consistency is a semantic-geometric consistency constraint loss, and λ1, λ2, λ3 are weight coefficients.

2. The method of claim 1, wherein, the illumination enhancement and color normalization processing includes one or more of Retinex illumination enhancement, adaptive histogram equalization and color consistency normalization, to reduce the influence of illumination changes on the stability of the dual-task lightweight model.

3. The method of claim 2, wherein, The weight coefficients λ1, λ2 and λ3 are dynamically adjusted by monitoring the gradient norm based on a task adaptive weight adjustment mechanism.

4. The method of claim 1, wherein, The RGB image is processed by a size normalization and dynamic exposure compensation algorithm to automatically adjust the brightness in strong light / low light environment; and the RGB image is corrected by fisheye and perspective transformation based on the real-time camera internal parameters of the on-board camera.

5. The method of claim 1, wherein, The method further comprises: storing the display result for flight task backtracking of the unmanned aerial vehicle; and retraining the dual-task lightweight model based on the display result.

6. The method of claim 1, wherein, The method further comprises: based on the display result, controlling the unmanned aerial vehicle to land in the safe landing area; or controlling the unmanned aerial vehicle to adjust the path to land in the safe landing area.

Images