A method for obstacle detection and segmentation in UAV aerial photography based on bi-branch contrastive learning

Abstract

This invention discloses a method for obstacle detection and segmentation in UAV aerial photography based on bi-branch contrastive learning, belonging to the field of computer vision and UAV application technology. The invention constructs a bi-branch instance segmentation network comprising a teacher model and a student model. Clean images and dynamically augmented images are input into the teacher and student models respectively. By calculating the alignment loss of the feature layer, detection layer, and segmentation layer, consistency constraints are applied between the two branches to update the student model weights, and an exponential moving average strategy is used to softly update the teacher model weights. During the inference phase, the teacher model is discarded, and only the student model is retained for prediction. This invention, through dynamic data augmentation and multi-level contrastive learning, effectively improves the model's feature extraction capability under complex lighting and occlusion conditions, achieving high-precision obstacle detection and instance segmentation in UAV aerial photography without increasing the computational overhead during actual deployment and inference.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and unmanned aerial vehicle (UAV) application technology, and more specifically, to a UAV aerial obstacle detection and segmentation method based on bi-branch contrastive learning. Background Technology

[0002] In recent years, unmanned aerial vehicles (UAVs) have been widely used in fields such as power line inspection, agricultural plant protection, urban security, and disaster relief. In these application scenarios, autonomous obstacle avoidance is a key technology to ensure the safe flight of UAVs, and accurately and in real time detecting and segmenting various obstacles (such as power towers, buildings, tree canopies, birds, and suspended cables) from aerial images is a prerequisite for achieving autonomous obstacle avoidance by UAVs.

[0003] Existing methods for visual obstacle extraction in drones largely rely on traditional image processing techniques or conventional deep learning object detection and segmentation networks (such as Mask R-CNN and YOLO series). However, drone aerial photography environments are highly complex and uncontrollable. Conventional algorithms exhibit the following significant shortcomings in practical applications:

[0004] 1. Insufficient robustness in complex environments: Aerial images are susceptible to changes in lighting, complex weather conditions, and occlusion. Conventional models rely solely on clean data for supervised learning, and existing dynamic data augmentation strategies are not integrated with the model training mechanism, failing to achieve progressive learning of difficult samples. This results in a significant increase in the false negative rate of the model in extreme scenarios such as fog and occlusion.

[0005] 2. Low pixel-level segmentation accuracy: Existing segmentation networks typically only calculate global cross-entropy loss, lacking explicit constraints on pixel-level feature consistency; moreover, existing bi-branch contrastive learning models are mostly applied to image classification tasks, focusing only on the output consistency of the feature layer and the detection layer for instance segmentation scenarios, failing to simultaneously achieve overall segmentation of artificially attached holes and the main building body, as well as accurate differentiation of natural hollow paths in buildings, easily leading to one-size-fits-all missegmentation problems.

[0006] 3. Difficulty in balancing accuracy and inference computing power: Conventional methods to improve model accuracy include increasing the number of network layers or introducing complex attention mechanisms, which inevitably leads to a significant increase in computing power overhead during the inference stage, making it unsuitable for the computing power and power consumption limitations of UAV onboard edge devices.

[0007] Therefore, there is an urgent need for a new method that can effectively enhance the model's feature extraction capability in complex environments and achieve high-precision pixel-level obstacle segmentation without increasing the computational overhead of the inference stage. Summary of the Invention

[0008] This invention proposes a method for obstacle detection and segmentation in UAV aerial photography based on bi-branch contrastive learning.

[0009] The technical solution to achieve the purpose of this invention is: a method for obstacle detection and segmentation in UAV aerial photography based on bi-branch contrastive learning, comprising:

[0010] A dual-branch teacher-student model is constructed, which includes a teacher model and a student model. The teacher model and the student model adopt the same instance segmentation network architecture. The teacher model is initialized with pre-trained weights, while the student model is initialized with weights independently using a random initialization method. The weights are not shared during the training process.

[0011] Using a drone aerial obstacle training dataset, the dual-branch teacher-student model is trained through comparative learning. During the training process, the original clean image is input into the teacher model, and the enhanced image obtained by data augmentation of the original clean image is input into the student model. By constraining the consistency of the outputs of the dual branches in the feature layer, detection layer and segmentation layer, the weights of the student model are updated, and the weights of the teacher model are softly updated using an exponential moving average strategy.

[0012] After training is completed, the teacher model is discarded, and only the final student model is retained as the inference model. The inference model is then used to perform obstacle detection and instance segmentation on the input drone aerial images.

[0013] Compared with the prior art, the significant advantages of this invention are:

[0014] 1. Significantly improves detection robustness in complex environments: This invention innovatively designs a dynamic data augmentation strategy (local occlusion and lighting transformation) with monotonically increasing intensity as the number of training rounds increases. By inputting the original image into the teacher model and the highly challenging augmented image into the student model for consistency learning, the network is forced to actively mine the essential structural features of obstacles under occlusion and poor lighting conditions, significantly reducing missed detections and missegmentation in actual flight.

[0015] 2. Enhanced pixel-level feature constraints for improved segmentation accuracy: This invention overcomes the limitations of traditional single cross-entropy loss by constructing a multi-level contrastive learning mechanism consisting of a feature layer (BYOL), a detection layer (InfoNCE), and a segmentation layer (double-weighted InfoNCE). In particular, the proposed double-weighted alignment strategy between obstacle regions and global pixels for the segmentation layer maximizes the statistical correlation of similar pixel features, resulting in sharper obstacle boundary segmentation and effectively suppressing misclassification of small internal regions.

[0016] 3. Balancing high accuracy and real-time requirements: This invention applies the complex contrastive learning and dual-branch architecture only to the offline training phase. In actual deployment and inference applications, the teacher branch is directly removed and discarded, and only the well-trained student model is used for forward propagation, i.e., it is directly deployed to the UAV-borne edge computing device, which has extremely strong practical application value.

[0017] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0018] Figure 1 The above is a flowchart of the method provided in the embodiments of the present invention.

[0019] Figure 2 The dual-branch teacher-student instance segmentation network architecture constructed for embodiments of the present invention.

[0020] Figure 3 This is a schematic diagram comparing the expected effect of obstacle segmentation in complex UAV aerial photography environments before and after introducing dual-branch comparative learning in this invention. Detailed Implementation

[0021] A novel obstacle detection and segmentation method for UAV aerial photography based on bi-branch contrastive learning is proposed. Its core concept involves constructing an isomorphic bi-branch teacher-student instance segmentation network. This network combines dynamic data augmentation, whose intensity monotonically increases with training epochs, with multi-level contrastive learning constraints. The bi-branch contrastive learning mechanism is introduced only during the training phase, while the single-branch student model is retained during the inference phase. This enhances the model's ability to learn the essential features of obstacles without increasing inference computational overhead. Simultaneously, a dual-layer weighted contrastive loss in the segmentation layer achieves joint optimization of the overall semantics and local details of the target, resolving the technical contradiction between "overall merging" and "precise differentiation." By introducing a teacher-student architecture and dynamic data augmentation mechanism, combined with multi-level contrastive learning constraints, this method strengthens the model's ability to represent robust obstacle features, significantly improving obstacle segmentation accuracy for UAVs under complex occlusion and lighting conditions.

[0022] A method for obstacle detection and segmentation in UAV aerial photography based on bi-branch contrastive learning includes the following steps:

[0023] Step 1: Construct a dual-branch teacher-student model, which includes a teacher model and a student model, both of which use the same YOLACT++ instance segmentation network architecture; wherein, the teacher model is initialized with pre-trained weights, and the student model is initialized independently with weights using a random initialization method, and the weights are not shared during training;

[0024] Furthermore, the YOLACT++ instance segmentation network architecture described in step 1 includes a backbone feature extraction network, a feature pyramid network (FPN) module, a prototype generation branch, and a prediction head branch connected in sequence; the backbone feature extraction network uses ResNet50 or ResNet101; the prototype generation branch has 32 prototype mask dimensions; the prediction head branch includes a parallel classification branch, a bounding box regression branch, and a mask coefficient branch; the teacher model loads YOLACT++ weights pre-trained on a large segmentation dataset, and the student model uses He initialization or Xavier initialization for random weight initialization.

[0025] Step 2: Using the drone aerial obstacle training dataset, the dual-branch teacher-student model is trained through comparative learning. During training, the original clean image is input into the teacher model, and the enhanced image obtained by data augmentation of the original image is input into the student model. By constraining the consistency of the outputs of the dual branches in the feature layer, detection layer, and segmentation layer, the weights of the student model are updated, and the weights of the teacher model are softly updated using an exponential moving average strategy. The data augmentation includes local occlusion and lighting transformation, and the intensity of the data augmentation increases monotonically with the number of training rounds.

[0026] Further, the specific steps for dataset preprocessing and annotation in step 2 are as follows: Collect original clean aerial images taken by drones at low altitudes, including urban roads, rural fields, mountainous forests, and industrial parks. These original clean images refer to unprocessed aerial images without any data augmentation, artificial occlusion, or lighting changes. All original clean images are uniformly resolved to 640×640 pixels. The LabelMe tool is used to annotate obstacle targets in the images, generating target detection bounding boxes and instance segmentation ground truth masks (MASK_GT). The dataset is randomly divided into training, validation, and test sets in a 7:2:1 ratio. Pixel values ​​are normalized and mean-variance standardized for all images, mapping pixel values ​​to the 0-1 range to match the input distribution of the pre-trained model.

[0027] The specific generation rules and intensity increment implementation method for enhanced images are as follows:

[0028] 1) Local Occlusion Transformation: Randomly select rectangular regions with a predetermined area ratio from the original image and fill them with neutral gray pixels for occlusion. The location of the occluded region is random. Let the current training epoch be t, and the total number of training epochs be T. The occlusion area ratio p(t) gradually increases from its initial value to its final value with each training epoch, using an exponential growth function:

[0029]

[0030] in, For growth rate control parameters, This represents the initial occlusion area ratio. For the final occlusion area ratio and ;

[0031] 2) Lighting Transformation: Randomly adjusts the image's brightness, contrast, and saturation coefficients. The brightness coefficient adjustment range starts from the initial range. Extend to the final interval according to the same exponential pattern. ,in Contrast coefficient adjustment range from Expand to ,in The saturation coefficient adjustment range is from Expand to ,in In the formula, These are the initial adjustment increments for brightness, contrast, and saturation, respectively. These represent the final adjustment ranges for brightness, contrast, and saturation, respectively.

[0032] 3) All transformations apply only to the input image of the student model; the original image of the teacher model remains unchanged.

[0033] In a further embodiment, the specific process of using drone aerial photography to train the obstacle dataset is as follows:

[0034] Prepare a training dataset for drone aerial obstacle photography, complete dataset preprocessing and obstacle annotation, and set a complete set of hyperparameters for model training, including the maximum number of training epochs N, loss convergence threshold ε, weight coefficients of each loss term, training batch size, exponential moving average update coefficient, optimizer hyperparameters, and learning rate strategy; start an outer loop for each epoch to traverse the entire training dataset, with one complete traversal of the entire dataset constituting one training epoch; simultaneously, start an inner loop within each batch to extract training samples and corresponding ground truth labels from the training dataset without replacement for the current batch;

[0035] Parallel dual-path processing is performed on the current batch of homogeneous training samples to generate the corresponding input data for the dual-branch model: the input to the teacher model is the original clean aerial image X. clean Without making any additional modifications; the input to the student model is the enhanced image X obtained by locally occluding and transforming the original clean image from the same source. aug Furthermore, the strength of data augmentation increases monotonically with the number of training epochs.

[0036] The two-branch model performs forward propagation synchronously. The teacher model extracts features from the input clean image and outputs the corresponding teacher features. Target detection results Instance segmentation results The student model performs synchronous feature extraction and inference on the input augmented image, and outputs the corresponding student features. Target detection results Instance segmentation results Based on the corresponding outputs of the same input, three sets of one-to-one positive sample pairs are constructed, which are the feature layer positive sample pairs. Detection layer positive sample pair positive sample pairs of segmentation layers ;

[0037] Specifically, the construction rules for the three positive sample pairs are as follows: the positive sample pairs of the feature layer are constructed using the same scale deep feature map output by the dual-branch FPN module; the positive sample pairs of the detection layer are constructed using the same dimension classification results and bounding box regression results output by the dual-branch prediction head; and the positive sample pairs of the segmentation layer are constructed using the instance segmentation mask results with the same size as the original image output by the dual-branch module. All positive sample pairs are dimensionally perfectly matched.

[0038] Calculate the three types of constraint loss terms during model training: For positive sample pairs in the feature layer, use the BYOL no-negative-sample contrastive loss framework to calculate the feature alignment loss. The feature distribution of the two branches is forced to be consistent; for positive sample pairs in the detection layer, the InfoNCE contrastive loss is used to calculate the detection alignment loss. For positive sample pairs in the segmentation layer, an InfoNCE-based contrastive loss is adopted, and a two-layer weighting strategy is introduced to calculate the segmentation alignment loss. ;

[0039] The total training loss is calculated by weighting and fusing the three types of constraint loss terms. The weights of the two-branch model are updated based on backpropagation and the total loss: first, the gradient is calculated based on the total loss, and the full weights of the student model are updated through the gradient descent optimizer; then, the teacher model weights are softly updated using the exponential moving average strategy.

[0040] Feature alignment loss The specific calculation steps are as follows:

[0041] 1) Student features in positive sample pairs of the feature layer and teacher characteristics Mapping is performed using a multilayer perceptron (MLP) projection head with shared parameters;

[0042] 2) The student projection features are transformed using a multilayer perceptron projection head to obtain the transformed student features. The transformed teacher projection features are obtained by stopping gradient propagation of the teacher projection features. ;

[0043] 3) Calculate the cosine similarity loss between the transformed student features and the teacher's projected features as the feature alignment loss:

[0044]

[0045] Where · represents the vector dot product, express Norm.

[0046] Furthermore, detect alignment loss. The contrastive loss is calculated using noise-contrast estimation: For the classification branch and the bounding box regression branch, the output of the student model is used as the query q, the output of the teacher model corresponding to the same sample is used as the positive sample k⁺ (the teacher model output stops gradient propagation), and the teacher model outputs of other samples in the current batch are used as the negative sample set N. First, the single-sample contrastive loss for each sample in the current batch is calculated. After summing the single-sample contrastive losses for all samples, the losses of the classification branch and the bounding box regression branch are added together to obtain the final detection alignment loss. The formula for calculating the single-sample InfoNCE contrastive loss is:

[0047]

[0048] Where τ is a preset temperature coefficient.

[0049] Furthermore, the segment alignment loss A contrast loss based on noise contrast estimation is adopted, and a two-layer weighting strategy is introduced: global pixel-level alignment loss is calculated. Alignment loss with obstacle area The weighted sum of the two is calculated using the following formula:

[0050]

[0051] in, and The weight coefficients are such that the weight coefficients of the obstacle region alignment loss are greater than the weight coefficients of the global alignment loss. ).

[0052] Both the global pixel-level alignment loss and the obstacle region alignment loss adopt the aforementioned noise contrast estimation contrast loss calculation method. The feature vector of each pixel position in the segmentation mask feature map is used as the query unit q, and the teacher outputs of all other pixel positions in the current batch (including other pixels in the same image and all pixels in other images in the batch) are used as the negative sample set.

[0053] The global pixel-level alignment loss is calculated for all pixels on the feature map, and its formula is as follows:

[0054] Where H and W are the height and width of the segmentation feature map, respectively, (x,y) are the pixel coordinates, and q(x,y) is the output feature of the student model at that pixel location. This refers to the output features of the teacher model at the same source and location.

[0055] The obstacle region alignment loss utilizes a truth mask. Pixels within the obstacle region are extracted, and the noise contrast estimation contrast loss is calculated only for these pixels. The formula is as follows:

[0056]

[0057] in, For truth mask The total number of pixels covering the obstacle area.

[0058] Furthermore, total training loss The specific calculation formula is as follows:

[0059]

[0060] In the formula, These are feature alignment loss, detection alignment loss, and segmentation alignment loss, respectively. These are the non-negative weight coefficients of the feature alignment loss, detection alignment loss, and segmentation alignment loss, respectively.

[0061] Furthermore, the optimizer for updating the student model weights employs the AdamW algorithm, with the optimizer hyperparameters set as follows: momentum parameter... The weight decay coefficient is 1e -5 Initial learning rate 1e -4 The training process employs a cosine annealing learning rate decay strategy, where the learning rate dynamically decreases with the number of training rounds.

[0062] Furthermore, the specific formula for calculating the teacher model weights using the EMA exponential moving average strategy is as follows:

[0063] in, For teacher model weights, The updated student model weights are α, which is the preset exponential moving average update coefficient.

[0064] Step 3: After updating the weights for the current batch, determine if all batches in the current round have been traversed. If not, continue the batch loop; if all batches are traversed, determine if the current training meets the termination condition. After training terminates, discard the teacher model and retain only the final student model as the inference model, used for obstacle detection and instance segmentation of the input UAV aerial images.

[0065] Furthermore, the specific rules for determining the training termination condition are as follows: First, determine whether the current round has reached the preset maximum number of training rounds N. If it has, training is terminated directly. If it has not, determine whether the decrease in the total loss of the validation set for five consecutive rounds is less than the preset loss convergence threshold. If it is satisfied, training is terminated early to avoid overfitting.

[0066] Furthermore, the final student model is retained as the inference model. The steps for obstacle detection and instance segmentation using the inference model on the input drone aerial images are as follows:

[0067] 1. Input the drone aerial image to be detected, and perform resolution adjustment and normalization preprocessing consistent with the training set;

[0068] 2. Input the preprocessed image into the student model. After forward propagation of the model, the target detection bounding box, class confidence and instance segmentation mask of the obstacles in the image are directly output.

[0069] 3. Non-maximum suppression (NMS) is used to remove redundant detection boxes, and the detection and segmentation results with confidence scores higher than a preset threshold are retained as the final obstacle detection and segmentation output.

[0070] Furthermore, the method is applied to scenarios such as obstacle avoidance during low-altitude flight of UAVs, power line inspection, oil and gas pipeline inspection, obstacle identification in agricultural and forestry plant protection, and abnormal target detection in urban security.

[0071] Example

[0072] This embodiment addresses the practical application needs of low-altitude inspection and autonomous obstacle avoidance by UAVs. With obstacle detection and segmentation in low-altitude aerial photography as its core task, it fully implements the technical solution described in this invention. The specific operation steps are as follows:

[0073] 1. Collect 8000 drone aerial images across four scenarios: low-altitude urban road inspection, rural field plant protection, mountain forest fire prevention inspection, and industrial park equipment inspection. Flight altitudes ranged from 10m to 50m, and flight speeds from 3m / s to 8m / s. All images were uniformly scaled to 640×640 pixels, ensuring that at least 40% of the samples contained complex scenes such as partial occlusion, backlighting, strong light, weak light, and rain / fog blur. LabelMe was used for pixel-level annotation of the images, labeling obstacles into 12 categories: pedestrians, motor vehicles, non-motor vehicles, power poles, transmission cables, communication towers, building components, trees, piles of stones, slopes, and fences. For each target, both axis-aligned detection bounding boxes and instance segmentation polygon masks were simultaneously annotated. The bounding box format was... The mask is a binary mask with the same size as the original image, with a pixel value of 1 for the target region and 0 for the background region. The LabelMe annotation format is converted to the standard JSON format of the COCO instance segmentation dataset. The annotated dataset is randomly partitioned into training, validation, and test sets in a 7:2:1 ratio. The training set contains 5600 images, the validation set 1600 images, and the test set 800 images, ensuring that complex scene samples account for at least 50% of the test set, and that no subsets contain duplicate samples. Pixel values ​​are normalized for all images, linearly mapping the RGB three-channel pixel values ​​from 0-255 to the 0-1 range. The RGB channel mean of the ImageNet dataset is used. ,variance =[0.229,0.224,0.225], and perform standardization processing on the normalized image. The calculation formula is as follows: Filter out invalid samples whose bounding boxes exceed the image boundaries, whose mask area is less than 10 pixels, or whose labeling category is incorrect.

[0074] A dual-parallel branch architecture is constructed for the teacher and student models. Both branches employ a completely isomorphic YOLACT++ instance segmentation network architecture, ensuring identical network layers, kernel size, number of channels, and output dimensions. The weights of the two branches are initialized asymmetrically and independently: the teacher model is initialized with weights from a pre-trained full YOLACT++ model on a large segmentation dataset; the student model is initialized independently using a random initialization method. Its backbone network, feature pyramid, prototype generation branch, and prediction head branch are all initialized using either He (Kaiming) or Xavier initialization. The weights of the two branches are not shared during training.

[0075] 1) The backbone feature extraction network adopts the ResNet50 residual network, which outputs three levels of feature maps: C3, C4, and C5, corresponding to downsampling ratios of 8, 16, and 32, and output channels of 512, 1024, and 2048, respectively.

[0076] 2) Construct the FPN feature pyramid module to perform multi-scale fusion of C3, C4, and C5 feature maps, and output fused feature maps of three scales: P3, P4, and P5. The number of output channels is uniformly 256.

[0077] 3) Build a prototype generation branch, take the P3 feature map as input, and generate a 138×138×32 prototype feature map through 4 layers of 3×3 convolutional layers and 1 layer of 1×1 convolutional layer;

[0078] 4) Construct a parallel prediction head branch, including a classification branch, a bounding box regression branch, and a mask coefficient branch, sharing weights across feature maps of all scales; the classification branch outputs the confidence score of the class corresponding to each anchor point, with an output dimension of num_classes×num_anchors; the bounding box regression branch outputs the coordinate offset of each anchor box, with an output dimension of 4×num_anchors; the mask coefficient branch outputs the 32-dimensional mask coefficients corresponding to each target, with an output dimension of 32×num_anchors; where num_classes is set to 12 and num_anchors is set to 3.

[0079] 3. In training mode, both the teacher model and the student model are used simultaneously to process the input image, calculate the loss, and update the weights in parallel. In inference mode, the teacher model is discarded, and only the trained student model is retained. The input aerial image directly outputs the detection box and segmentation mask.

[0080] 4. The parameter settings used during training are shown in Table 1.

[0081] Table 1 Experimental Environment and Model Parameters

[0082] 5. Begin model training.

[0083] 1) Load the preprocessed training and validation sets, initialize the dual-branch teacher-student model, and independently load or randomly initialize the pre-trained weights for each branch. Configure the AdamW optimizer and cosine annealing learning rate strategy, initialize the loss function and training log recording tool, and set the model to training mode.

[0084] 2) Execute the outer loop for each round. Initialize the current round number t to 1, start the outer loop, and complete a full traversal of the training set data once in each round. After the traversal is completed, verify the model performance on the validation set.

[0085] 3) Perform batch inner loop. Randomly select 16 samples without replacement from the training set for the current batch. Each sample contains the original source image, ground truth bounding box, and ground truth segmentation mask. Perform dual-path processing on the source samples: the teacher model directly inputs the original clean image, which remains unchanged; the student model performs data augmentation on the original source image. First, based on the current training round number t and the total number of rounds T, calculate the current occlusion area ratio using an exponential growth function. And the corresponding photometric transformation interval. Then, an enhanced image is generated and input only into the student model. Simultaneously, forward propagation is performed to construct three sets of positive sample pairs: feature layer positive sample pairs, detection layer positive sample pairs, and segmentation layer positive sample pairs. The multi-dimensional consistency constraint loss is calculated:

[0086] a) Feature alignment loss calculation: Based on the BYOL framework, the two-branch features are mapped using a multilayer perceptron (MLP) projection head with shared parameters. The Student projection feature is transformed into PS through the MLP prediction head, and the Teacher projection feature ZT stops gradient propagation. Cosine similarity loss is calculated:

[0087]

[0088] b) Calculation of Alignment Loss: Based on the InfoNCE framework. For both the classification branch and the bounding box regression branch, the output of the student model is used as the query q, the output of the teacher model corresponding to the same sample is used as the positive sample k+, and the teacher outputs of other samples in the current batch are used as the negative sample set N. The calculation formula is as follows:

[0089]

[0090] Where Lhead is the sum of the LNCE losses of the classification branch and the regression branch;

[0091] c) Two-layer weighted segmentation alignment loss calculation: Also based on the InfoNCE loss calculation method described above, the feature vector of each pixel position in the segmentation mask feature map is used as the query unit. The global pixel-level alignment loss Lseg_global (negative samples include the teacher's output of all other pixels in the batch) is calculated; simultaneously, obstacle regions are extracted using the ground truth mask, and the alignment loss Lseg_GT is calculated only for pixels within the obstacle regions. The weighted fusion yields:

[0092]

[0093] d) Calculation of total loss for weighted fusion:

[0094]

[0095] 4) Backpropagation and weight update: The student model weight gradient is calculated based on the total loss, and only the student model weight is updated through the AdamW optimizer; the teacher model weight is softly updated through the EMA exponential moving average strategy, and the teacher model does not participate in gradient backpropagation.

[0096] 5) Determine whether all batches in the current round have been traversed. If not, return to step 1 to continue execution. If completed, proceed to the verification process.

[0097] 6. Set the model to evaluation mode and perform forward inference on the validation set; calculate and record the total validation set loss, detection AP, and segmentation mIoU metrics, and update the learning rate.

[0098] 7. Determine if the termination condition is met: The number of rounds reaches T=100, or the loss on the validation set decreases by less than 1e-5 for 5 consecutive rounds. If the condition is met, terminate training; otherwise, increment the number of rounds by 1 and return to the outer loop to continue execution.

[0099] 8. After training terminates, the teacher model is discarded, and only the final student model is retained as the inference model. To verify the performance advantages of the model of this invention, a test set is used to verify the model's performance. The original YOLACT++ algorithm is set as the baseline for comparison. The comparison method uses the same dataset, preprocessing procedure, training hyperparameters, and hardware environment as this invention for training and testing. The specific performance verification scheme is as follows:

[0100] 1) Comprehensive performance test: Test the core metrics of the present invention model and the baseline on the full test set, namely average precision (AP) and mean intersection over union (mIoU).

[0101] 2) Scene robustness test: Test the detection AP of the model in clean and interference-free scenes and complex interference scenes respectively, calculate the index decrease in occluded scenes and lighting change scenes, and verify the model's anti-interference ability.

[0102] 3) Inference efficiency test: Under the NVIDIA RTX 3090 hardware environment, the average inference time of the test model in processing a single 640×640 pixel image was measured to verify the deployment adaptability of the model.

[0103] Based on the technical solution of this invention, it is reasonably expected that compared with the original YOLACT++ baseline, the detection and segmentation accuracy of the model of this invention will be significantly improved in complex interference scenarios, while the inference time will not increase significantly, and real-time, high-precision obstacle detection and segmentation can be achieved.

[0104] To intuitively illustrate the technical principles and expected effects of this invention in typical complex scenarios, the following three types of core pain points in drone obstacle avoidance are selected for a comparative demonstration of the principles and effects:

[0105] A. Foggy Road Vehicle Scenario. The fog obscuring and blurred target boundaries in this scenario perfectly simulate the core scenarios of "light transformation and progressive occlusion enhancement" in student branch training. This invention, through feature layer contrastive loss, forces the model to ignore the apparent interference of fog and learn the essential global features of vehicles. Based on the technical solution of this invention, it is expected that: the original YOLACT++ can only identify two high-confidence targets nearby, resulting in serious missed detections and incomplete segmentation; the area of ​​the bus obscured by fog is not completely segmented, and multiple distant vehicles are completely missed; while the model of this invention, under foggy interference, can completely detect all road vehicles, accurately reconstruct the complete outline of each vehicle, with no missed detections or incomplete segments. Even distant vehicles with low visibility can be accurately segmented, fully demonstrating the strong robustness and anti-interference capability brought by the training strategy.

[0106] B. Architectural Scenes with U-Shaped Hollow Passages. Addressing the core requirement of "distinguishing between false holes and real passageways" in this scenario, this invention specifically supplements the training dataset for this type of scene. Through the joint loss of the detection and segmentation layers, the model can learn the semantic differences between "natural architectural passageways" and "artificially attached holes," while retaining the ability to identify passable areas. Based on the technical solution of this invention, it is expected that: the original YOLACT++ has poor ability to distinguish between natural passageways and artificial holes, easily segmenting the U-shaped passageway and the main building as a whole, with masks covering the passageway area, leading to misclassification of passable areas as obstacles; while the model of this invention possesses scene semantic discrimination capabilities, accurately distinguishing between natural architectural hollow passageways and artificially attached holes, generating masks only for the main building, leaving the U-shaped passageway area unmasked, achieving accurate segmentation of obstacles and passable areas, simultaneously meeting the two core requirements, and avoiding the problem of "one-size-fits-all" misclassification.

[0107] C. Building-attached billboard / notice scenario. The local holes formed by the attached objects in this scenario are the core simulation scenario of "local occlusion enhancement" in student branch training. This invention, through the global pixel alignment loss of the segmentation layer, allows the model to learn the overall outline and contextual semantic information of the building, and will not be misled by the visual differences of local attachments. Based on the technical solution of this invention, it is expected that: the original YOLACT++ will separate the main building and the attached billboard / notice into two independent entities, forming a hollow segment in the building area, which is easy to cause the area to be misjudged as an unobstructed path; while the model of this invention can combine the building context information to determine the attached billboard and the main building as the same whole, generating a continuous and complete building mask without local hollow segmentation, effectively avoiding the safety risk of drone collisions, and realizing the core requirement of "hole and overall building segmentation".

[0108] A schematic diagram comparing the underlying principles and effects of the three scenarios described above is attached to the instruction manual. Figure 3 The three scenarios, from top to bottom, are: comparison of expected effects for vehicles on foggy roads, comparison of expected effects for buildings with U-shaped hollow passageways, and comparison of expected effects for buildings with attached billboards.

[0109] The PyTorch Student inference model was converted to the ONNX general format, and operator fusion and constant folding optimization were performed. The ONNX model was converted to the NVIDIA TensorRT .engine format and the FP16 half-precision inference mode was adopted.

[0110] The optimized TensorRT model was deployed on the NVIDIA Jetson Xavier NX airborne edge computing device of the UAV, and connected to the real-time aerial video stream from the UAV's onboard camera. Preprocessing, model inference, and post-processing were performed on the video frames sequentially, and obstacle location information was sent to the UAV's flight control system. Actual flight tests were conducted in urban roads and mountainous forest areas to verify the model's real-time performance, detection accuracy, and robustness in real-world operating environments.

Claims

1. A method for detecting and segmenting obstacles in aerial images based on dual-branch contrastive learning, characterized in that, include: A dual-branch teacher-student model is constructed, which includes a teacher model and a student model. The teacher model and the student model adopt the same instance segmentation network architecture. The teacher model is initialized with pre-trained weights, while the student model is initialized with weights independently using a random initialization method. The weights are not shared during the training process. Using a drone aerial obstacle training dataset, the dual-branch teacher-student model is trained through comparative learning. During training, the original clean image is input into the teacher model, and the enhanced image obtained by data augmentation of the original clean image is input into the student model. By constraining the consistency of the outputs of the two branches in the feature layer, detection layer, and segmentation layer, the weights of the student model are updated, and the weights of the teacher model are softly updated using an exponential moving average strategy. The total training loss is as follows: ； In the formula, These are feature alignment loss, detection alignment loss, and segmentation alignment loss, respectively. These are the non-negative weight coefficients of the feature alignment loss, detection alignment loss, and segmentation alignment loss, respectively. ; Segment alignment loss Specifically: in, and These are the weighting coefficients, and ; The local pixel-level alignment loss is as follows: ； Where H and W are the height and width of the segmentation feature map, respectively. For pixel coordinates, The output features of the student model at pixel locations. For the teacher model's output features at the same source and location, The single-sample InfoNCE contrastive loss uses the current pixel as the query and the teacher's outputs of all other pixels on the segmentation feature map as negative samples. The obstacle region alignment loss is as follows: in, For truth mask The total number of pixels covering the obstacle area. The single-sample InfoNCE contrastive loss is calculated only for pixels within the ground truth mask, with the current pixel as the query and the teacher's output for all other pixels on the segmentation feature map as the negative sample. After training is completed, the teacher model is discarded, and only the final student model is retained as the inference model. The inference model is then used to perform obstacle detection and instance segmentation on the input drone aerial images.

2. The UAV aerial obstacle detection and segmentation method based on bi-branch contrastive learning according to claim 1, characterized in that, The instance segmentation network architecture includes a backbone feature extraction network, a feature pyramid network module, a prototype generation branch, and a prediction head branch connected in sequence; the backbone feature extraction network adopts ResNet50 or ResNet101; the prototype generation branch is set with 32 prototype mask dimensions; the prediction head branch includes a parallel classification branch, a bounding box regression branch, and a mask coefficient branch.

3. The UAV aerial obstacle detection and segmentation method based on bi-branch contrastive learning according to claim 1, characterized in that, Data augmentation of original clean images from the same source includes local occlusion and lighting transformation, and the intensity of data augmentation increases monotonically with the number of training epochs. The specific process is as follows: A rectangular region with a predetermined area ratio is randomly selected from the original image and filled with neutral gray pixels for occlusion. The location of the occluded region is random. Let the current training round number be t, the total number of training rounds be T, and the occlusion area ratio p(t) gradually increases from the initial value to the final value with each training round, using an exponential growth function. ; in, For growth rate control parameters, This represents the initial occlusion area ratio. For the final occlusion area ratio and ; The brightness, contrast, and saturation coefficients of the image are randomly adjusted to complete the light transformation.

4. The UAV aerial obstacle detection and segmentation method based on bi-branch contrastive learning according to claim 1, characterized in that, Using a training dataset of drone aerial obstacle photography, the dual-branch teacher-student model is trained through comparative learning, including: The outer loop of each round is started to traverse the entire training dataset. One complete traversal of the entire dataset is one training round. At the same time, the inner loop of each batch is started within each round to extract the training samples and corresponding labeled ground truth values ​​of the current batch from the training dataset without replacement. Parallel dual-path processing is performed on the same training samples in the current batch. The input to the teacher model is the original clean image; the input to the student model is the enhanced image obtained after local occlusion and lighting transformation of the original clean image. The two-branch model performs forward propagation synchronously. The teacher model extracts features from the input clean image and outputs the corresponding teacher features. Target detection results Instance segmentation results The student model performs synchronous feature extraction and inference on the input augmented image, and outputs the corresponding student features. Target detection results Instance segmentation results Based on the corresponding outputs of the same input, three sets of one-to-one positive sample pairs are constructed, which are the feature layer positive sample pairs. Detection layer positive sample pair positive sample pairs of segmentation layers ; Based on positive sample pairs, the total loss of the two-branch teacher-student model is calculated. Backpropagation and weight updates of the two-branch model are performed based on the total loss, including calculating the gradient based on the total loss and updating the full weights of the student model through the gradient descent optimizer. Then, the teacher model weights are softly updated using an exponential moving average strategy.

5. The UAV aerial obstacle detection and segmentation method based on bi-branch contrastive learning according to claim 1, characterized in that, The specific process for calculating the feature alignment loss is as follows: Student features in positive sample pairs of the feature layer and teacher characteristics Mapping is performed using a multilayer sensor projection head with shared parameters; The student's projected features are transformed using a multilayer perceptron projector to obtain the transformed student features. The transformed teacher projection features are obtained by stopping gradient propagation of the teacher projection features. ; The cosine similarity loss between the transformed student features and the teacher's projected features is calculated as the feature alignment loss: ; Where · represents the vector dot product, express Norm.

6. The UAV aerial obstacle detection and segmentation method based on bi-branch contrastive learning according to claim 1, characterized in that, The detection alignment loss is the sum of the noise contrast estimation contrast losses of all samples in the current batch. The contrast loss for each batch is calculated as follows: Perform the following operations on the classification branch and the bounding box regression branch respectively: use the output of the student model as the query vector q, the output of the teacher model corresponding to the same sample as the positive sample vector k⁺, and the output of the teacher model of all other samples in the current batch as the negative sample set N; For each sample in the current batch, calculate its single-sample InfoNCE contrast loss; The average of the individual sample losses for all samples in the current batch is used to obtain the batch loss for that branch. The batch loss of the classification branch and the bounding box regression branch are added together to obtain the final batch detection alignment loss; The formula for calculating the single-sample InfoNCE contrastive loss is as follows: ； In the formula, τ is the preset temperature coefficient, and k⁻ is the negative sample vector in the negative sample set.

7. The UAV aerial obstacle detection and segmentation method based on bi-branch contrastive learning according to claim 1, characterized in that, The specific method for softly updating the teacher model weights using the exponential moving average strategy is as follows: in, For teacher model weights, For the updated student model weights, The preset exponential moving average update coefficients.

Images