Method and system for real-time detection and determination of blockage and damage of unmanned aerial vehicle side slope side ditch

Abstract

This invention discloses a method and system for real-time detection and judgment of blockage and damage in slopes and ditches using unmanned aerial vehicles (UAVs). The method includes: collecting real-time image data of the target area using a UAV and preprocessing it; constructing and training a deep learning-based multi-task network model, which collaboratively performs target detection, attitude estimation, and multi-task feature association through a shared feature extraction layer and task-specific branches; inputting the preprocessed image into the trained network model, outputting detection results and spatial attitude parameters; automatically generating an inspection route adapted to the target structural features based on the attitude parameters; and determining anomalies based on the detection results and generating output information. This invention achieves parallel processing of target detection and attitude estimation through a multi-task network model, combined with attitude parameter-driven dynamic route planning, realizing fully automated inspection from data acquisition to result output, effectively improving the intelligence level and inspection efficiency of infrastructure surface damage detection.

Description

Technical Field

[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) intelligent inspection technology, specifically relating to a method and system for detecting surface damage and monitoring health of infrastructure based on UAVs in multiple scenarios. Background Technology

[0002] Unmanned aerial vehicle (UAV) inspection technology has been widely used in the field of surface damage detection for infrastructure such as bridges, power transmission lines, and wind turbines due to its advantages of high efficiency, safety, and low cost. However, existing technologies still have the following shortcomings:

[0003] First, traditional drone inspections typically rely on manual remote control or preset fixed routes, making it impossible to dynamically adjust the inspection path based on the target's current real-time attitude. For example, in wind turbine inspections, the blade rotation angle and nacelle orientation directly affect the shooting angle and coverage integrity. Existing methods struggle to adaptively plan the optimal route, easily leading to missed shots of critical areas or poor image quality.

[0004] Secondly, existing deep learning-based detection methods are mostly single-task models, capable of performing only object detection or classification tasks, and struggling to simultaneously perform damage detection and target pose estimation. While related technologies have attempted multi-task learning, their feature interaction and collaborative optimization capabilities between tasks are insufficient, making it difficult to fully leverage the advantages of multi-task parallel processing.

[0005] Furthermore, existing methods typically only consider the geometric consistency of the bounding box in the design of the loss function, ignoring the impact of sample quality distribution and predicted probability distribution on model training. This leads to the model overfitting to low-quality samples and underfitting to high-quality samples, thus limiting detection accuracy and generalization ability.

[0006] To address the aforementioned issues, this invention proposes a method and system for multi-scenario infrastructure surface damage detection and health monitoring based on unmanned aerial vehicles (UAVs). By collaboratively executing target detection and attitude estimation through a multi-task network model and combining attitude parameter-driven dynamic flight path planning, the entire process of automated inspection from data acquisition to result output is achieved. Summary of the Invention

[0007] This invention aims to solve the problems in existing UAV inspection technologies, such as flight path planning relying on preset paths and being unable to adaptively adjust, detection models being mostly single-task and difficult to process in parallel, loss function design being simplistic and ignoring sample quality distribution, and insufficient collaborative optimization capabilities between tasks.

[0008] In a first aspect, embodiments of this application provide a method for real-time detection and determination of blockage and damage in slopes and ditches using unmanned aerial vehicles (UAVs), the method comprising:

[0009] The drone is used to collect real-time image data of the target area, and the collected images are preprocessed.

[0010] Construct and train a deep learning-based multi-task network model. The multi-task network model performs target detection, pose estimation and multi-task feature association by sharing a feature extraction layer and task-specific branches. The multi-task network model includes a target pose and structure perception module, a hybrid feature extraction and enhancement module and a multi-task correlation module.

[0011] The training employs a loss function that comprehensively constrains multiple dimensions, including geometric consistency, sample quality distribution, and prediction probability distribution. It also adopts a phased training strategy that sequentially completes backbone network pre-training, relevance module training, and joint optimization to enhance multi-task collaborative learning capabilities.

[0012] The preprocessed image is input into the trained network model, which outputs the detection results of target damage or defects and the spatial pose parameters of the target.

[0013] Based on the output spatial attitude parameters, the UAV inspection route is automatically generated to adapt to the current target structural characteristics, realizing dynamic planning and adaptive adjustment of the inspection path;

[0014] Based on the detection results, abnormal situations are identified and corresponding output information is generated, completing the entire process of automated inspection from data collection to result output.

[0015] Secondly, embodiments of this application provide a real-time detection and judgment system for blockage and damage in slopes and ditches using unmanned aerial vehicles (UAVs), applied to the method described in the first aspect, the system comprising:

[0016] The data acquisition module is used to collect real-time image data of the target area using a drone and to preprocess the acquired images.

[0017] The model building and training module is used to build and train a deep learning-based multi-task network model. The multi-task network model performs target detection, pose estimation and multi-task feature association by sharing a feature extraction layer and task-specific branches. The training adopts a multi-dimensional joint optimization loss function and a phased training strategy.

[0018] The detection and estimation module is used to input the preprocessed image into the trained network model and output the detection results of target damage or defects and the spatial pose parameters of the target.

[0019] The route planning module is used to automatically generate UAV inspection routes that adapt to the current target structure characteristics based on the output spatial attitude parameters, so as to realize dynamic planning and adaptive adjustment of the inspection path.

[0020] The output module is used to determine abnormal situations based on the detection results and generate corresponding output information, completing the entire process of automated inspection from data acquisition to result output.

[0021] Thirdly, embodiments of this application provide an electronic device, including:

[0022] processor;

[0023] Memory used to store processor-executable instructions;

[0024] The processor is configured to implement the real-time detection and judgment method for blockage and damage of slope and ditch by UAV as described in the first aspect when executing the instructions.

[0025] Fourthly, embodiments of this application provide a computer-readable storage medium storing a program that instructs a device to execute the real-time detection and determination method for blockage and damage in slopes and ditches as described in the first aspect.

[0026] Compared with existing technologies, this invention has the following advantages: By collaboratively executing target detection and attitude estimation through a multi-task network model, parallel processing of damage detection and attitude parameter extraction is achieved, significantly improving the execution efficiency and information acquisition capability of inspection tasks. A multi-dimensional joint optimization loss function is adopted, comprehensively constraining the model from multiple dimensions such as geometric consistency, sample quality distribution, and predicted probability distribution, effectively improving the model's detection accuracy and generalization ability in complex scenes. A phased training strategy is employed, sequentially completing backbone network pre-training, relevance module training, and joint optimization, fully promoting information interaction and collaborative learning among task branches, further improving the accuracy of multi-task parallel processing. Based on the output spatial attitude parameters, a dynamic inspection route adapted to the target structural features is automatically generated, realizing adaptive planning of the inspection path, effectively avoiding the problem of missed images of key parts caused by traditional fixed routes, and ensuring the integrity of inspection coverage and image acquisition quality. A fully automated inspection closed loop is constructed, from data acquisition, model training, real-time detection, route planning to anomaly output, significantly reducing the need for manual intervention and improving the intelligence level and operation and maintenance efficiency of infrastructure surface damage detection. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of a method for real-time detection and judgment of blockage and damage in slopes and ditches provided in an embodiment of this application.

[0028] Figure 2 This is a diagram illustrating the architecture of a real-time detection and judgment system for blockage and damage in slopes and ditches provided in an embodiment of this application.

[0029] Figure 3 A schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0031] It should be noted that in the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0032] Based on the embodiments described in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] Example 1

[0034] Figure 1 This is a schematic flowchart illustrating a method for real-time detection and judgment of blockage and damage in slopes and ditches using a drone, provided as an embodiment of this application. Figure 1 As shown, a method for real-time detection and judgment of blockage and damage in slopes and ditches using unmanned aerial vehicles (UAVs) includes:

[0035] S1. Use a drone to collect real-time image data of the target area and preprocess the collected images. This step is responsible for controlling the drone to fly near the target to be detected, acquiring on-site images through the onboard camera, and performing noise reduction, correction, scaling, and other processing on the raw images to make them meet the input requirements of the subsequent algorithm model.

[0036] Specifically, in this embodiment, the step of using a drone to collect real-time image data of the target area and preprocessing the collected images includes:

[0037] S1.1 Control the drone to fly to the target area and collect continuous video stream data of the target area through the camera equipment on board the drone. In this embodiment, the ground control station sends flight commands to the drone, which autonomously navigates to the vicinity of the slope, ditch, or trench structure to be detected using its onboard Global Positioning System (GPS) and visual positioning system. After reaching the target area, the drone hovers and activates its onboard high-definition camera to continuously capture dynamic images of the target area from a first-person perspective, forming continuous video stream data. The continuous video stream data here refers to a series of image frames arranged in chronological order, recorded in real time by the drone camera during flight, rather than a single still photograph, in order to provide sufficient raw material for subsequent extraction of keyframes.

[0038] S1.2. Extract keyframe images from the continuous video stream data at a preset frame rate to obtain the original image dataset. After the UAV transmits or stores the acquired continuous video stream to the local computing platform in real time, the system performs uniform or keypoint sampling on the video stream according to a preset frame rate parameter (e.g., extracting 2 or 5 frames per second). The preset frame rate here refers to the image extraction frequency preset according to the UAV's flight speed and detection accuracy requirements. Its purpose is to effectively control the total number of images to be processed while ensuring that important image details are not missed, thus avoiding waste of computing resources. Through this step, the system selects a series of representative single-frame images from the lengthy video to form the original image dataset.

[0039] S1.3. The original image dataset is sequentially processed by image denoising, distortion correction, and resolution normalization to generate preprocessed standard image data. For the original images obtained in the previous step, the system first uses a filtering algorithm to remove noise caused by insufficient light or sensor interference (i.e., randomly occurring pixel brightness anomalies in the image); secondly, it uses camera calibration parameters to perform reverse compensation for the bending distortion at the image edges caused by the wide-angle characteristics of the lens, restoring straight lines in the image to their normal shape—this operation is called distortion correction; finally, all images are uniformly scaled to a fixed size (e.g., 640×640 pixels) required by the deep learning network model, ensuring that different images have the same data dimension. The image output after the above three sequential processing steps is the standard image data that can be directly input into the subsequent model.

[0040] S2. Construct and train a deep learning-based multi-task network model. The multi-task network model performs target detection, pose estimation, and multi-task feature association collaboratively through a shared feature extraction layer and task-specific branches. The multi-task network model includes a target pose and structure perception module, a hybrid feature extraction and enhancement module, and a multi-task correlation module. The training adopts a loss function that comprehensively constrains multiple dimensions such as geometric consistency, sample quality distribution, and prediction probability distribution. It also adopts a phased training strategy that sequentially completes the pre-training of the backbone network, the training of the correlation module, and the joint optimization to improve the multi-task collaborative learning capability.

[0041] This step is used to build a deep learning network that can perform multiple tasks simultaneously. This network can identify damaged areas in an image and determine the current spatial orientation and rotation angle of the target. At the same time, this step also designs a comprehensive optimization objective and a phased learning strategy to enable the model to learn the correlation features between multiple tasks more effectively from the data.

[0042] Specifically, in this embodiment, the target pose and structure perception module includes a multi-scale adaptive channel attention submodule, an adaptive attention deformable fusion submodule, and a keypoint heatmap regression submodule. The multi-scale adaptive channel attention submodule extracts input features in parallel using local and global feature paths and combines them with a hybrid convolutional attention feature extractor for multi-scale adaptive weighted fusion. The adaptive attention deformable fusion submodule uses an attention feature fusion mechanism to weightedly fuse shallow features and deep keypoint features, and introduces deformable convolution to dynamically adjust the sampling point positions according to the target's geometric deformation. The keypoint heatmap regression submodule uses an encoder-decoder structure to extract spatial features of key target parts and generates a two-dimensional Gaussian distribution heatmap to represent the keypoint positions for end-to-end target pose estimation.

[0043] The core function of the multi-scale adaptive channel attention submodule is to enhance the neural network's ability to capture features at different scales in the target image. In actual processing, the input feature map is simultaneously fed into two parallel processing paths: one is called the local feature path, which is responsible for extracting subtle details such as edges and textures in the image; the other is called the global feature path, which is responsible for extracting macroscopic structural information such as the overall contour and orientation of the target.

[0044] The outputs of the two paths are adaptively weighted and fused through a hybrid convolutional attention feature extractor. This extractor contains both traditional convolutional layers and attention layers, automatically determining whether detailed information or overall information is more important in the current scene, and dynamically adjusting the fusion ratio accordingly. Specifically, the extractor consists of parallel convolutional and attention branches. The convolutional branches extract local texture and edge features using standard convolutional kernels, while the attention branches capture global dependencies through a self-attention mechanism. The outputs of the two branches are weighted and summed using learnable fusion weights, which are dynamically generated by a small fully connected network based on statistical information from the input features.

[0045] The multi-scale adaptive weighted fusion here refers to the network's ability to flexibly adjust the weights of local details and global structure in feature representation based on the actual situation of different input images, thereby more accurately understanding the spatial composition of the target.

[0046] To quantitatively describe the multi-scale adaptive weighted fusion process of input features by this module, the present invention defines the following feature fusion mechanism:

[0047] Let the input feature map be ,in For the number of channels, and These represent the height and width of the feature map, respectively. Local feature path output. With global feature path output The calculation method is as follows:

[0048] , ,

[0049] in, Detailed features extracted for local feature paths The overall structural features extracted from the global feature path; This represents the feature transformation function of the hybrid convolutional attention feature extractor. The global feature representation of the input feature map after adaptive average pooling is in matrix form, with its only non-zero dimension being the channel dimension. The calculation method is as follows:

[0050] ,

[0051] Fusion weights Generate in the following way:

[0052] ,

[0053] ,

[0054] Final weighted fusion output feature map for: ;in, Input feature map (matrix form); This refers to the row index (vertical direction) in the feature map. Column indexes (horizontal direction) in the feature map. For the input feature map, the first Channel, First line, number The element values ​​of the column; The feature map is obtained after adaptive average pooling; The first feature map after pooling The unique element value of the channel. For the first The fusion weight (scalar) of each channel. `\` represents the Sigmoid activation function, and `\` represents the Hadamard product operation with element-wise multiplication. (Row index) Corresponding to the height direction of the feature map (from top to bottom), column index Corresponding to the width direction of the feature map (from left to right), after pooling, the height dimension of the feature map is 1, and the width dimension is also 1. Therefore, its element index is fixed. .

[0055] The Adaptive Attention-Based Deformable Fusion Submodule addresses the issue of shape variations in targets under different viewpoints or poses. In images captured by drones, the same target (such as a wind turbine blade or a bridge pillar) may exhibit geometric deformations such as stretching or rotation due to different shooting angles. This submodule first employs an attention-based feature fusion mechanism, weightedly fusing the detailed features extracted by the shallow network with the semantic features extracted by the deep network. Subsequently, a deformable convolutional layer is introduced. Unlike traditional convolutions, its sampling points are not fixed grid points but dynamically shifted according to the actual shape of the target. Specifically, an independent convolutional layer is set up, taking the input feature map as input. Through convolution operations, it predicts a two-dimensional spatial offset for each sampling point. These offsets are then applied to a standard sampling grid to obtain the offset sampling position. Feature values ​​are then obtained at this sampling position through bilinear interpolation for further convolution operations. Here, deformable convolution refers to the fact that the sampling position of the convolutional kernel can adaptively adjust according to changes in the target's geometry during feature extraction, thereby more accurately matching the actual contour of the target and avoiding the extraction of background noise.

[0056] The keypoint heatmap regression submodule is responsible for accurately locating several key parts of the target from the image and inferring the overall pose of the target based on this. This submodule adopts an encoder-decoder network structure: the encoder part progressively compresses the image size through multiple convolutions to extract high-level semantic features; the decoder part progressively restores the image resolution through deconvolutions to generate a heatmap corresponding to the original image size. Here, the heatmap refers to a two-dimensional grayscale image, where the brightness of each pixel represents the probability that the location is a keypoint of the target; the higher the brightness, the more likely the location is to be a keypoint. To simulate the uncertainty of keypoint location, each keypoint is represented as a two-dimensional Gaussian distribution, that is, a bright spot that gradually decays outward from the keypoint.

[0057] By detecting peak positions in the heatmap, the system can accurately obtain the coordinates of key target points, and then calculate the target's orientation angle and component rotation angle, achieving end-to-end attitude estimation. The specific method for peak positioning is as follows: For each key point's corresponding heatmap channel, first, find the pixel position with the largest response value in the heatmap as the coarse coordinate. Then, within the local region formed by this pixel and its eight adjacent pixels, use a Gaussian fitting method to refine the coordinates to sub-pixel precision. That is, using this coarse coordinate as the initial value, solve for the sub-pixel extreme points of the heatmap response through a second-order Taylor expansion.

[0058] To accurately regress the location of target key points, this invention defines a key point heatmap loss function. as follows:

[0059] Let the predicted heat map be... The actual heat map is ,in H represents the number of keypoints, and H and W represent the height and width of the heatmap, respectively. The actual heatmap is generated using a two-dimensional Gaussian distribution.

[0060] ,

[0061] in, Let σ be the true coordinates of the k-th key point, and σ be the standard deviation of the Gaussian kernel.

[0062] The loss in the key point heatmap is the mean square error:

[0063] ,

[0064] in, Let be the true coordinates of the k-th key point.

[0065] Specifically, in this embodiment, the hybrid feature extraction and enhancement module includes a hybrid convolutional attention feature extractor and a feature pre-fusion path aggregation network. The hybrid convolutional attention feature extractor includes both convolutional layers and attention layers. The convolutional layers extract local edge and texture features, while the attention layers capture global dependencies. The two are weighted and fused using adaptive weights. The feature pre-fusion path aggregation network embeds a feature pre-fusion module before the top-down and bottom-up fusion paths of the feature pyramid network. It employs a hierarchical fusion strategy to interactively fuse the features of the current layer with the features of the next two layers, and combines a focused linear attention mechanism to establish potential associations between global and local features. This reduces the loss of detailed information during feature fusion and enhances the detection capability for small targets.

[0066] The hybrid convolutional attention feature extractor is a core component of the hybrid feature extraction and enhancement module, used to simultaneously extract complementary information from the input image, including both local details and global semantics. In practice, the input feature map is fed into two parallel processing branches: the first branch uses a traditional convolutional layer to extract local micro-features such as edges, corners, and textures through a sliding window; the second branch uses an attention layer to capture the dependencies between distant pixels by calculating the correlation scores between different locations in the feature map, thereby understanding the overall structure and context of the target. The adaptive weights here refer to the network automatically learning a fusion coefficient based on the characteristics of the current input image, dynamically adjusting the proportion of convolutional and attention features in the final output. When the image is rich in detail, the network automatically increases the weight of the convolutional features; when the background is complex or the target is partially occluded, the network increases the weight of the attention features, thus achieving adaptive responses to different scenes.

[0067] To quantify the adaptive fusion process of convolutional features and attention features, this invention defines the following feature extraction and fusion formulas:

[0068] The Feature Pre-fusion Path Aggregation Network is an improvement on the traditional Feature Pyramid Network, primarily addressing the issue of lost detail information during feature propagation for small-sized targets. In conventional Feature Pyramid Networks, higher-level features propagate semantic information to lower levels via a top-down path, while lower-level features propagate positional information to higher levels via a bottom-up path. However, this direct cross-level fusion can easily dilute the detail information in intermediate levels. To address this problem, this network embeds a pre-fusion module before the traditional two fusion paths. Pre-fusion here refers to allowing small-scale interactions between adjacent or nearby feature layers before performing large-scale cross-level fusion: specifically, features from the current layer are locally fused with features from the next two layers, limiting the span of feature propagation and thus preserving more original details.

[0069] Building upon this foundation, the network also introduces a focused linear attention mechanism. This mechanism establishes potential correlations between global and local features with low computational cost through a linearized attention calculation method. This allows the network to pay attention to the subtle appearance of small targets while understanding their relative position within the entire scene, thus significantly improving the detection capability for small, damaged targets. The specific computational method of this focused linear attention mechanism is as follows: First, an element-wise exponentiation operation is performed on the query vector and key vector using a focusing function, giving higher attention weights to similar query-key pairs, thereby simulating the sharpening distribution characteristics of the Softmax function. Then, the query vector and key vector are projected onto a high-dimensional feature space using kernel functions, and similarity is calculated using a linear kernel function in the high-dimensional space. This avoids explicitly calculating the attention matrix, reducing the computational complexity from the square of the sequence length to linear.

[0070] Specifically, in this embodiment, the multi-task correlation module uses a probability mapping method to generate inter-task correlation labels, and uses an adaptive average pooling layer, a fully connected layer, and a normalized output probability distribution vector to establish information interaction and collaborative learning between the target detection task and the pose estimation task, so as to promote knowledge sharing between tasks and improve the accuracy of multi-task parallel processing.

[0071] The core function of the multi-task correlation module is to break down the information barriers between the target detection task and the attitude estimation task, enabling the two tasks to learn from and promote each other during the learning process. In practical applications, certain types of damage are often strongly correlated with specific spatial locations. For example, lightning strike damage to wind turbine blades usually occurs at the blade tip, while corrosion mostly occurs at the blade root or middle section.

[0072] Based on this physical prior, the module first employs a probabilistic mapping method to transform the keypoint location information output by the attitude estimation task into inter-task correlation labels in the form of a probability distribution. This probabilistic mapping method refers to converting deterministic position coordinates into a set of probability values, each representing the likelihood that the current input image belongs to a specific attitude category (such as blade tip, blade root, nacelle, etc.). The specific mapping rules are as follows: multiple attitude categories are predefined based on the structural characteristics of the target, with each attitude category corresponding to a spatial region; for the input image, based on which attitude category the detected keypoint coordinates fall into, the correlation label is set to a one-hot distribution with a probability of 1 for that category and 0 for other categories; alternatively, a soft labeling method is used, calculating the probability value of each category based on the distance between the keypoint and the center of each attitude category region using a Gaussian kernel function, with closer distances resulting in higher probabilities. Subsequently, the module uses a small network consisting of an adaptive average pooling layer, a fully connected layer, and a normalization layer connected in series to perform dimensionality reduction, mapping, and probabilistic processing on the high-dimensional features. The adaptive average pooling layer compresses input features of arbitrary size into a fixed-size feature vector, ensuring a uniform output dimension regardless of the input feature map size. The fully connected layer further maps this feature vector to a low-dimensional space equal to the number of pose categories. The normalization layer uses a softmax function to compress all output values ​​to between 0 and 1, ensuring the sum of all output values ​​is 1, thus creating a standard probability distribution vector. This final probability distribution vector serves as a supplementary output to the pose estimation results and is also fed back to the target detection branch, enabling it to determine the current pose category of the target and adjust the expected probability for different damage types based on prior pose information. For example, when the correlation module outputs a high probability indicating that the current target is a blade tip, the target detection branch automatically increases its sensitivity to lightning strike damage. Through this two-way information exchange mechanism, the two tasks achieve collaborative learning and knowledge sharing, significantly improving the accuracy of multi-task parallel processing.

[0073] Specifically, in this embodiment, the multi-dimensional joint optimization loss function used in the training includes one or more of geometric optimization loss, sample quality weighted loss, distribution distance loss, and distribution focusing loss.

[0074] The geometric optimization loss comprehensively considers the overlap area between the predicted and ground truth bounding boxes, the center point distance, the aspect ratio, angular cost, distance cost, and shape cost. This loss function measures the degree of matching between the model's predicted target bounding box and the ground truth bounding box, guiding the model to learn how to more accurately locate damaged areas. Traditional bounding box loss only considers the overlap area between the predicted and ground truth bounding boxes; when there is no overlap, the loss is zero, and the model cannot obtain an effective optimization direction. To address this issue, the geometric optimization loss comprehensively evaluates the predicted box from five dimensions: first, the overlap area, i.e., the size of the intersection of the predicted and ground truth bounding boxes; second, the center point distance, i.e., the spatial distance between the centers of the two boxes; third, the aspect ratio, i.e., whether the ratio of the width to the height of the predicted box is close to the ratio of the ground truth bounding box; fourth, the angular cost, i.e., the rotational deviation of the predicted box relative to the ground truth bounding box, used to handle targets with strong directionality (such as diagonal cracks); and fifth, the overall similarity of the two boxes in terms of their outlines. This loss function weights and combines these five indicators into a comprehensive score; the higher the score, the closer the predicted box is to the ground truth bounding box. In actual training, when the predicted bounding box partially overlaps with the ground truth bounding box or even does not overlap at all, the loss function can still provide effective gradient signals through other dimensions such as center point distance and angle cost, guiding the predicted bounding box to gradually move closer to the ground truth bounding box, thereby significantly improving the model's localization accuracy of the damaged area.

[0075] To accurately measure the degree of matching between the predicted bounding box and the ground truth bounding box, this invention defines a geometric optimization loss function. As follows: Let the prediction box be... The real frame is ,in Let w be the coordinates of the bounding box center point, and h be the width and height, respectively. The overlap area loss term is:

[0076] ,

[0077] The center point distance loss term is:

[0078] ,

[0079] in, Let c be the Euclidean distance function, and c be the length of the diagonal of the minimum bounding rectangle between the predicted box and the ground truth box.

[0080] Angular cost Λ is defined as:

[0081] ,

[0082] Where α is the angle between the line connecting the center of the predicted box and the center of the true box and the horizontal direction, and σ is the Euclidean distance between the two centers.

[0083] Shape cost Ω is defined as:

[0084] ,

[0085] in, , θ is the shape cost control factor.

[0086] Comprehensive geometric optimization loss for:

[0087] ,

[0088] in, The bounding box predicted by the model; The bounding box is the actual labeled bounding box; The coordinates of the center point of the prediction box; The width and height of the predicted bounding box; The coordinates of the center point of the true bounding box; The width and height of the actual bounding box; For intersection-union ratio, the value range is [0,1]. The loss term is the distance from the center point; Normalize the differences in width and height.

[0089] The sample quality-weighted loss introduces a weighting factor to enhance the gradient contribution of high-quality samples and suppress the negative impact of low-quality samples. This loss function addresses the imbalance between high-quality and low-quality samples in the training data, making the model focus more on samples that are easy to learn and accurately labeled. In real-world datasets, the shooting conditions of different images vary greatly: some images have ample lighting, clear targets, and minimal occlusion, and are considered high-quality samples; others are blurry, have small targets, cluttered backgrounds, or are partially occluded, and are considered low-quality samples. If both types of samples are treated equally during training, the model may over-adapt to noise and anomalies in low-quality samples, leading to a decline in generalization ability. This loss function introduces a weighting factor that automatically calculates the weight of each sample based on the degree of overlap between the predicted and ground truth bounding boxes: samples with high overlap (i.e., high-quality samples) are assigned higher weights and have a larger gradient contribution; samples with low overlap (i.e., low-quality samples) are assigned lower weights and their gradient contribution is suppressed. The weighting factor here is a coefficient between 0 and 1. When the model updates parameters during backpropagation, the loss value of each sample is multiplied by this coefficient before participating in the gradient calculation. In this way, the model focuses more on learning effective features from high-quality samples during training, while reducing the interference of noise information from low-quality samples, thereby improving the stability and detection accuracy of the model.

[0090] To balance the contributions of high-quality and low-quality samples during training, this invention defines a sample quality-weighted loss function. as follows:

[0091] Let the intersection-union ratio (IU) of the predicted bounding box and the ground truth bounding box of sample i be . Then the weighting factor for:

[0092] ,

[0093] Where τ is the temperature coefficient used to adjust the smoothness of the weight distribution, and N is the total number of samples in the current batch.

[0094] The sample quality-weighted loss is:

[0095] ,

[0096] in, This represents the confidence score of the model's prediction for the i-th sample. Let be the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box of the i-th sample. τ is the quality weighting factor for the i-th sample; τ is the temperature coefficient, ranging from (0,1], with smaller values ​​indicating more concentrated high-weighted samples; N is the total number of samples in the current training batch.

[0097] The distribution distance loss combines JS divergence and Hellinger distance to measure the difference between the predicted probability distribution and the true label probability distribution. This loss function measures the difference between the probability distribution predicted by the model and the probability distribution of the true labels, and is mainly used for the optimization of multi-task relevance modules. In object detection and attitude estimation tasks, the model output is often not a single deterministic value, but a probability distribution vector representing the confidence level of each possible class. For example, the attitude estimation task might output a distribution with a probability of 0.7 for the blade tip, 0.2 for the blade root, and 0.1 for the nacelle. To determine whether the model's output distribution is close to the reality, this loss function uses a combination of two distance metrics: the first is JS divergence, which measures the similarity between two probability distributions, with values ​​ranging from 0 to 1; the closer the value is to 0, the more similar the two distributions are. The second is the squared Hellinger distance, which also measures the difference between two distributions. Its advantage is that even if the two distributions do not overlap at all, it can still provide non-zero gradient information, avoiding model training stagnation. This loss function adds the square of the JS divergence to the square of the Hellinger distance, combining the advantages of both: it has the ability of the JS divergence to sensitively capture subtle differences when the distributions overlap, and the gradient preservation ability of the Hellinger distance when the distributions do not overlap at all. This ensures that the correlation module can obtain an effective optimization direction in all situations and accurately learn the correlation between tasks.

[0098] To measure the difference between the predicted probability distribution and the true label probability distribution, this invention defines a distribution distance loss function. as follows:

[0099] Let the predicted probability distribution be... The true label probability distribution is Where K is the total number of categories, satisfying and .

[0100] JS divergence is defined as:

[0101] ,

[0102] The square of the Hellinger distance is defined as:

[0103] ,

[0104] Distance loss for:

[0105] ,

[0106] Where λ is a balance factor used to adjust the contribution ratio of JS divergence and Hellinger distance to the total loss. This is the probability distribution vector predicted by the model; This is the probability distribution vector of the true labels; This represents the probability value of the k-th class predicted by the model. This represents the probability value of the k-th category of the true label; The Jensen-Shannon divergence function has a value range of [0,1]. It is the square of the Hellinger distance, with a value in the range [0,1].

[0107] The distribution-focusing loss maximizes the probabilities of the two boundaries closest to the true label, allowing the model to focus on regions near the true label. This loss function improves the model's prediction accuracy of target boundary locations, making it particularly suitable for detection tasks requiring precise localization. In object detection, the bounding box output by the model is determined by four boundary values ​​(left, right, top, and bottom). Traditional regression loss directly predicts a specific numerical value; the loss is large when the predicted value deviates significantly from the true value, and small when the deviation is small. However, this approach ignores the uncertainty of boundary locations: within a certain interval near the true boundary, the confidence level varies at different locations. The distribution-focusing loss addresses this problem by having the model predict the probability distribution of boundary locations rather than a single numerical value. Specifically, the model outputs a set of discrete probability values, representing the likelihood of the true boundary falling at each candidate location. The core operation of this loss function is focusing: it does not require the model to predict all candidate locations accurately, but rather focuses on the two candidate locations closest to the true label. By maximizing the sum of the probabilities at these two locations, the model's predicted distribution quickly converges to the vicinity of the true boundary. Here, the two nearest boundary probabilities refer to the probability values ​​corresponding to the two discrete points closest to the true boundary value in the probability distribution output by the model. By continuously maximizing these two probability values, the model can quickly focus its attention on a small area around the true boundary, thereby achieving accurate localization of the target boundary and significantly improving the regression accuracy of the detection box.

[0108] To enable the model to focus on the prediction region near the true boundary, this invention defines a distributed focusing loss function. as follows:

[0109] Let the true boundary location be y, and the discrete probability distribution predicted by the model be... Where M is the number of discrete positions. This represents the probability that the model predicts the true boundary to be located at the i-th discrete position.

[0110] set up and To the true boundary The two most recent discrete locations satisfy The focus loss is:

[0111] .

[0112] Combining the above loss components, this invention defines a multi-dimensional joint optimization total loss function. as follows:

[0113] ,

[0114] in, These are the weighting coefficients for each loss component.

[0115] Specifically, in this embodiment, the training employs a phased training strategy to optimize the multi-task network model: the phased training strategy sequentially includes a backbone network pre-training phase, a correlation module training phase, and a joint optimization phase. The switching timing of each phase is determined by a preset fixed number of rounds: the backbone network pre-training phase consists of 200 training rounds, during which the correlation module parameters are fixed, and only the basic feature extraction capabilities of the object detection branch and pose estimation branch are optimized; the correlation module training phase consists of 10 training rounds, during which the backbone network weights are fixed, and only the parameters of the multi-task correlation module are optimized; the joint optimization phase consists of 90 training rounds, during which all network parameters are unfrozen, and both the backbone network and the correlation module are optimized simultaneously. The total number of rounds for the three phases is 300 rounds, and training ends after all rounds are completed.

[0116] In the pre-training phase of the backbone network, the parameters of the correlation module are fixed, and only the basic feature extraction capabilities of the object detection and pose estimation branches are optimized. This phase is the starting point of the entire training process, aiming to enable the model to master the basic processing capabilities of a single task, laying a good foundation for subsequent multi-task collaborative learning. During this phase, the object detection and pose estimation branches in the model are activated and participate in training, while the parameters of the multi-task correlation module are temporarily fixed and do not participate in parameter updates. "Fixed" here means that the gradient of the module's parameters is not calculated during backpropagation, and no adjustments are made to their values, keeping them in their initial state. The two task branches share the same backbone feature extraction network, which is responsible for extracting common visual features from the input image. During training, the model simultaneously optimizes the recognition ability of the detection branch and the estimation ability of the pose branch, enabling the backbone network to learn to extract shared features that are beneficial for both damage detection and pose prediction. This phase does not involve inter-task information interaction; the two branches learn independently, aiming to allow the model to first establish a basic understanding of different tasks, avoiding convergence difficulties caused by simultaneous optimization of multiple tasks.

[0117] During the training phase of the correlation module, the backbone network weights are fixed, and only the parameters of the multi-task correlation module are optimized to establish inter-task correlations. This phase focuses on establishing and strengthening the correlation between the target detection task and the pose estimation task. In this phase, the backbone network weights trained in the previous phase are completely fixed; that is, the backbone network no longer updates its parameters, maintaining its basic feature extraction capabilities learned in the previous phase. Simultaneously, the parameters of the target detection branch and the pose estimation branch are also fixed. The only module in training is the multi-task correlation module, which is responsible for learning the mapping relationship between the two tasks. Specifically, it determines the pose category of the current target based on the features output by the pose estimation branch and passes this information to the detection branch. Here, inter-task correlation refers to the correspondence between the output features of one task and the features required by another task, such as the statistical correlation between the blade tip position and the probability of lightning damage. By fixing all other modules and only allowing the correlation module to participate in training, the model can concentrate all its learning capabilities on exploring the intrinsic connections between tasks, avoiding interference from changes in other module parameters on correlation learning, thereby establishing accurate inter-task mapping relationships more efficiently.

[0118] In the joint optimization phase, all network parameters are unfrozen, and the backbone network and related modules are optimized simultaneously to achieve inter-task collaborative learning. This phase is the final step in the entire training process, aiming to achieve deep collaborative learning between the two tasks. In this phase, all parameters fixed in the first two phases are unfrozen; that is, the parameters of the backbone network, object detection branch, pose estimation branch, and multi-task related modules can all participate in the update. The unfrozen model, based on the basic feature extraction and inter-task correlation modeling capabilities already established in the first two phases, simultaneously performs joint optimization on all modules. During this process, the gradient information generated by each task branch is not only used to update its own parameters but also transmitted to another task branch through the shared backbone network and related modules, forming a mutually reinforcing virtuous cycle. For example, the keypoint location information learned by the pose estimation branch can help the detection branch more accurately determine the type of damage (e.g., blade tips are more susceptible to lightning strike damage), while the damage location identified by the detection branch can, in turn, help the pose estimation branch more accurately locate keypoints. Through this bidirectional information flow, the model can fully utilize the complementarity between the two tasks, achieving higher overall performance than training each task individually. Training continues in this phase until the model's loss value converges to a stable level, at which point the entire training process of the multi-task network model is complete.

[0119] S3. Input the preprocessed image into the trained network model, and output the detection results of target damage or defects and the spatial pose parameters of the target. This step feeds the preprocessed image into the trained model, and through one forward computation, obtains two aspects of information: first, whether there is damage in the image and its location and type; second, the target's current pose data, such as the orientation angle and blade rotation angle.

[0120] Specifically, in this embodiment, after the preprocessed image is input into the trained network model, the damage or defect detection results of the target surface and the spatial attitude parameters of the target are output in parallel through forward inference; the damage or defect detection results include damage type, location and confidence information; the spatial attitude parameters include the target's orientation angle and component rotation angle, which are used to provide input basis for the dynamic planning of subsequent inspection routes.

[0121] When the preprocessed standard image data is fed into a multi-task network model that has completed phased training, the model outputs two types of information in parallel through a single forward inference computation: damage or defect detection results and spatial pose parameters. Forward inference here refers to the process of passing the input data from the network's input layer to the output layer for computation. This process does not require backpropagation or parameter updates, thus it is fast and suitable for real-time applications.

[0122] Regarding damage or defect detection results, the model outputs information in three dimensions. First is the damage category, which is the specific type of damage identified by the model. For example, in a wind turbine inspection scenario, it can be categorized as lightning damage, cracks, rust spots, spalling, and corrosion; in a bridge inspection scenario, it can be categorized as rebar corrosion, bridge body cracks, and large-area spalling. Second is location information, which marks the specific coordinate range of the damaged area in the original image in the form of a bounding box, including the pixel coordinates of the upper left and lower right corners of the bounding box, for subsequent visualization and marking of the damage location in images or videos. Third is confidence information, which is a score on the reliability of the model's judgment result, with a value between 0 and 1. The closer the value is to 1, the more confident the model is in the detection result. This confidence information can be used to set alarm thresholds: when the confidence score output by the model is lower than a preset threshold (e.g., 0.6), the system can determine it as a suspected anomaly, prompting manual review to avoid false alarms or missed alarms.

[0123] Regarding spatial attitude parameters, the model outputs two key parameters: the target's orientation angle and the component rotation angle. The orientation angle refers to the overall pointing angle of the target in the horizontal direction, such as the orientation of a wind turbine nacelle, ranging from 0 to 360 degrees. The component rotation angle refers to the rotation angle of a movable component within the target relative to a reference position, such as the current rotation phase of a wind turbine blade, also ranging from 0 to 360 degrees. These two attitude parameters form the core input basis for subsequent UAV inspection flight path planning: the orientation angle determines the direction from which the UAV should approach the target to obtain the best shooting angle, while the component rotation angle determines the spatial direction along which the UAV's hovering shooting points should be arranged. For example, when the blade rotation angle is 90 degrees, the flight path planning system needs to set waypoints in the vertical direction; when the blade rotation angle is 0 degrees, waypoints need to be set in the horizontal direction. By outputting the detection results and attitude parameters in parallel, the model can identify where damage is located while also informing the UAV of the target's current attitude, providing complete information support for the next step of adaptive flight path planning.

[0124] S4. Based on the output spatial attitude parameters, automatically generate a UAV inspection route that adapts to the current target's structural characteristics, realizing dynamic planning and adaptive adjustment of the inspection path. This step uses the target attitude parameters obtained in the previous step, combined with the UAV's camera parameters and preset safety distance, to automatically calculate an optimal flight path that can cover all key parts of the target, and dynamically adjusts the shooting point position according to the target's actual orientation and shape to avoid missed or repeated shooting.

[0125] Specifically, in this embodiment, the automatic generation of UAV inspection routes adapted to the current target structural features based on the output spatial attitude parameters, realizing dynamic planning and adaptive adjustment of the inspection path, specifically includes:

[0126] S4.1. Based on the target orientation angle and component rotation angle, a three-dimensional spatial model of the target structure is constructed using a rotation matrix, and the starting waypoint for the inspection is calculated. This step utilizes the attitude parameters output by the model to construct a three-dimensional spatial model of the target in the computer and determine the starting position of the UAV inspection. Specifically, the system first obtains the target orientation angle (e.g., the direction the wind turbine nacelle points) and component rotation angle (e.g., the current rotation phase of the wind turbine blades), and then transforms the target's geometry from a preset reference attitude to the current actual attitude through rotation matrix operations. Here, the rotation matrix is ​​a mathematical transformation tool used to rotate a point in three-dimensional space around a coordinate axis by a specified angle, thereby achieving accurate simulation of the target orientation and blade angles in the computer. After completing the spatial modeling, the system sets the starting waypoint for the inspection at a position equal to a preset safe distance from the hub center on the reverse extension line of the nacelle vector. This starting waypoint is located directly in front of the target, ensuring that the UAV has a wide field of vision when taking off and reserving sufficient maneuvering space for subsequent flight along the blade direction.

[0127] S4.2 Calculate the hovering shooting interval based on the camera's field of view and the ideal distance between the UAV and the target. This step determines the distance between two adjacent hovering shooting points during the UAV's flight along the leaf. The system first obtains two key parameters: one is the field of view of the camera on the UAV, i.e., the angle of the field of view that the camera lens can cover; the other is the ideal distance between the UAV and the target, i.e., the flight distance preset according to camera resolution and safety requirements, at which clear and detailed images can be captured while ensuring sufficient safety margin between the UAV and the target. Based on these two parameters, the system calculates the actual physical range length that the camera can cover during each hovering shooting using geometric relationships. To ensure sufficient overlap between two adjacent images and avoid missed shots, the system multiplies this coverage length by an overlap coefficient (usually three-quarters) to obtain the final hovering shooting interval. Here, the hovering shooting interval refers to the spatial distance between two adjacent hovering shooting points when the UAV flies along the inspection route. This interval ensures the continuity of shooting coverage while avoiding inefficiency caused by overly dense shooting points.

[0128] S4.3 Generate hovering shooting waypoints on the windward and leeward sides along the direction of the target structure, and set flight paths to avoid obstacles, so as to realize dynamic planning and adaptive adjustment of the inspection path.

[0129] This step, based on the calculations in the previous two steps, generates a complete inspection route. The system first determines the actual orientation of the three blades in three-dimensional space based on the target's facing angle and the component's rotation angle. Then, it sequentially deploys hovering waypoints along the direction of each blade from root to tip. To achieve omnidirectional inspection of the target surface, the system generates two inspection paths for each blade: one on the windward side of the blade, starting from the initial waypoint and flying along the side of the blade facing the airflow from root to tip, with a path length slightly longer than the actual blade length; the other on the leeward side of the blade, starting from the back of the blade near the hub and flying along the side of the blade away from the airflow from root to tip. Furthermore, in the transition area where the blade root connects to the nacelle, the system generates an obstacle avoidance route that bypasses the nacelle. This route follows the outer contour of the nacelle, ensuring that the UAV maintains a safe distance from the nacelle when turning from the windward side to the leeward side or from one blade to another, thus avoiding collisions. The inspection route generated by the above method can automatically adjust the waypoint position and flight direction according to the real-time attitude of the target, and achieve full coverage and multi-angle shooting of key parts of the target surface while ensuring safety.

[0130] S5. Based on the detection results, determine the abnormal situation and generate corresponding output information to complete the fully automated inspection process from data acquisition to result output. This step determines whether there are abnormal conditions on the target surface based on the damage information output by the detection model, and integrates all detection results, attitude information, and flight data into a readable report or alarm, which is provided to maintenance personnel, thereby completing the closed loop of the entire inspection task.

[0131] This step is the final link in the entire inspection method, responsible for transforming the damage or defect information detected by the model into decision-making basis that can be directly used by maintenance personnel, forming a complete automated closed loop from image acquisition to result output. The system first receives the damage category, bounding box location, and confidence level information output by the detection and estimation module, comparing each detection result with a preset confidence level threshold. This confidence level threshold is a pre-set value, typically set to 0.6 or 0.7, used to filter reliable detection results: when the confidence level of the model output is higher than this threshold, the system classifies the detection result as a valid anomaly; when the confidence level is lower than this threshold, the system marks it as a suspected anomaly, prompting manual review to avoid unnecessary maintenance dispatch due to false alarms. In addition, the system also sets a high-risk threshold to trigger immediate alarms. This high-risk threshold is typically set to 0.9, higher than the regular confidence level threshold, used to filter serious anomalies requiring immediate attention. When the detected anomaly category is a damage type that directly affects safety, such as structural cracks or large-area spalling, and the confidence level is higher than both the regular confidence level threshold and the high-risk threshold, the system automatically triggers an immediate alarm.

[0132] After completing the anomaly detection, the system categorizes and summarizes all valid detection results according to damage type, and calculates the actual size and area of ​​each damage based on the bounding box coordinates. The system also correlates the detection results with the location information (including GPS coordinates, flight altitude, gimbal angle, etc.) in the UAV flight log, thereby accurately locating the actual geographical location of each damage, providing precise navigation for subsequent on-site troubleshooting by maintenance personnel.

[0133] Finally, the system generates a complete inspection report and visualizes the relevant information. The inspection report includes, but is not limited to: identification information of the inspected target (such as wind turbine number or bridge pile number), inspection time, flight path data, a list of all detected anomalies (including damage category, location coordinates, confidence score, damage area, etc.), and the original image with labeled boxes. For visualization, the system marks the damage location on the original image with colored rectangles, with different colors representing different damage categories, and labels the category name and confidence score next to the box. For suspected anomalies with low confidence scores, the system distinguishes them with dashed boxes or special colors.

[0134] Furthermore, the system supports outputting inspection reports in multiple formats, including PDF documents for archiving, JSON data for integration with other management systems, and real-time alarm push notifications to mobile terminals or monitoring center dashboards. When a serious anomaly (such as structural cracks or large-area corrosion) is detected and the confidence level exceeds the high-risk threshold, the system can automatically trigger an immediate alarm, notifying maintenance personnel to handle the situation as soon as possible via SMS, application push notifications, or audible and visual alerts. Thus, from drone takeoff and data collection, model training and optimization, real-time detection and attitude estimation, dynamic flight path planning to anomaly detection and result output, the entire inspection task forms a complete automated closed loop, effectively reducing the need for manual intervention and improving the intelligence level and operational efficiency of infrastructure surface health monitoring.

[0135] Example 2

[0136] like Figure 2 As shown in the diagram, this application provides an architecture diagram of a real-time detection and judgment system for blockage and damage in slopes and ditches using a drone, which is applied to the real-time detection and judgment method for blockage and damage in slopes and ditches using a drone as described in Embodiment 1, including:

[0137] The data acquisition module 210 is used to collect real-time image data of the target area using a UAV and preprocess the acquired images. The model building and training module 220 is used to build and train a deep learning-based multi-task network model. This module is executed only once during system initialization. After training, the model parameters are fixed and passed to the detection and estimation module 230. The detection and estimation module 230 receives the preprocessed image output by the data acquisition module 210, inputs the image into the trained network model, outputs the detection results of target damage or defects and the target's spatial attitude parameters, and then sends the detection results to the output module 250 and the attitude parameters to the flight path planning module 240. The flight path planning module 240 automatically generates a UAV inspection flight path adapted to the current target structural characteristics based on the received spatial attitude parameters, realizing dynamic planning and adaptive adjustment of the inspection path, and sends the generated flight path instructions to the UAV control system. The output module 250 determines abnormal situations based on the received detection results and generates corresponding output information, completing the fully automated inspection process from data acquisition to result output. The above modules are connected in series to form a complete processing chain of data acquisition, model inference, route planning, and anomaly output.

[0138] Figure 3 This is an electronic device provided in one embodiment of this application. For example... Figure 3 As shown, the electronic device includes at least the following components: processor 301 and memory 300, communication interface 303, and bus 302.

[0139] In this embodiment of the application, memory 300 is used to store executable instructions of processor 301, which, when configured to execute instructions, implements the method as described in the first aspect.

[0140] In embodiments of this application, a computer-readable storage medium includes instructions that instruct a device to perform the method as described in the first aspect. For example, the instructions instruct the device to perform... Figure 1 The method is shown in the process steps.

[0141] In one embodiment of this application, the program operating in the electronic device may be a program that controls a central processing unit (CPU) or similar device to achieve the functions of the above-described embodiments of the present invention (a program that enables the computer to function). Information processed by these systems is then temporarily stored in random access memory (RAM) during processing, and subsequently stored in various ROMs such as read-only memory (FlashROM) and hard disk drives (HDDs), and read, corrected, and written by the CPU as needed.

[0142] It should be noted that a portion of the electronic device described above can also be implemented using a computer. In this case, the program for implementing the control function can be recorded on a computer-readable recording medium, and the program recorded on the recording medium can be read into the computer and executed.

[0143] It should be noted that the computer mentioned here refers to a computer built into an electronic device, employing hardware including an operating system and peripheral devices. Furthermore, computer-readable recording media refers to removable media such as floppy disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage systems such as hard drives built into the computer.

[0144] Furthermore, computer-readable recording media can include: media that dynamically stores programs for short periods of time, such as communication lines used when transmitting programs via networks like the Internet or communication lines like telephone lines; and media that store programs for fixed periods of time, such as volatile memory inside a computer that serves as a server or client in this case. In addition, the aforementioned program can be a program used to implement the above-mentioned functions, or it can be a program that can implement the above-mentioned functions by combining them with programs already recorded in the computer.

[0145] Furthermore, the electronic device in the above embodiments can also be implemented as an assembly (system group) composed of multiple systems. Each system constituting the system group can possess some or all of the functions or functional blocks of the electronic device in the above embodiments. As a system group, it is sufficient to have all the functions or functional blocks of the electronic device.

[0146] Those skilled in the art should recognize that the above embodiments are only used to illustrate this application and are not intended to limit this application. Any appropriate changes and variations made to the above embodiments within the essential spirit and scope of this application fall within the scope of protection claimed in this application.

Claims

1. A method for real-time detection and judgment of blockage and damage in slopes and ditches using unmanned aerial vehicles (UAVs), characterized in that, Includes the following steps: The drone is used to collect real-time image data of the target area, and the collected images are preprocessed. Construct and train a deep learning-based multi-task network model. The multi-task network model performs target detection, pose estimation and multi-task feature association by sharing a feature extraction layer and task-specific branches. The multi-task network model includes a target pose and structure perception module, a hybrid feature extraction and enhancement module and a multi-task correlation module. The training employs a loss function that comprehensively constrains multiple dimensions, including geometric consistency, sample quality distribution, and prediction probability distribution. It also adopts a phased training strategy that sequentially completes backbone network pre-training, relevance module training, and joint optimization to enhance multi-task collaborative learning capabilities. The preprocessed image is input into the trained network model, which outputs the detection results of target damage or defects and the spatial pose parameters of the target. Based on the output spatial attitude parameters, the UAV inspection route is automatically generated to adapt to the current target structural characteristics, realizing dynamic planning and adaptive adjustment of the inspection path; Based on the detection results, abnormal situations are identified and corresponding output information is generated, completing the entire process of automated inspection from data collection to result output.

2. The method according to claim 1, characterized in that, The process of using a drone to collect real-time image data of the target area and preprocessing the collected images includes: Control the drone to fly to the target area and collect continuous video stream data of the target area through the camera equipment on the drone; Keyframe images are extracted from the continuous video stream data at a preset frame rate to obtain the original image dataset; The original image dataset is sequentially subjected to image denoising, distortion correction, and resolution normalization to generate preprocessed standard image data.

3. The method according to claim 1, characterized in that, The target attitude and structure perception module includes a multi-scale adaptive channel attention submodule, an adaptive attention deformable fusion submodule, and a key point heatmap regression submodule. The multi-scale adaptive channel attention submodule extracts local and global feature paths in parallel and combines them with a hybrid convolutional attention feature extractor to perform multi-scale adaptive weighted fusion of input features. The adaptive attention-deformable fusion submodule uses an attention feature fusion mechanism to weight and fuse shallow features and deep key point features, and introduces deformable convolution to dynamically adjust the sampling point position according to the geometric deformation of the target. The key point heatmap regression submodule uses an encoder-decoder structure to extract spatial features of key parts of the target and generates a two-dimensional Gaussian distribution heatmap to represent the key point location for end-to-end target attitude estimation.

4. The method according to claim 1, characterized in that, The hybrid feature extraction and enhancement module includes a hybrid convolutional attention feature extractor and a feature pre-fusion path aggregation network; The hybrid convolutional attention feature extractor includes both convolutional layers and attention layers. The convolutional layers are used to extract local edge and texture features, while the attention layers are used to capture global dependencies. The two are weighted and fused using adaptive weights. The feature pre-fusion path aggregation network embeds a feature pre-fusion module before the top-down and bottom-up fusion paths of the feature pyramid network. It adopts a hierarchical fusion strategy to interactively fuse the features of the current layer with the features of the next two layers, and combines a focused linear attention mechanism to establish the potential correlation between global and local features. This is used to reduce the loss of detailed information during the feature fusion process and enhance the detection capability of small targets.

5. The method according to claim 1, characterized in that, The multi-task correlation module uses a probability mapping method to generate inter-task correlation labels, and uses an adaptive average pooling layer, a fully connected layer, and a normalized output probability distribution vector to establish information interaction and collaborative learning between the target detection task and the pose estimation task, so as to promote knowledge sharing between tasks and improve the accuracy of multi-task parallel processing.

6. The method according to claim 1, characterized in that, The multi-dimensional joint optimization loss function used in the training includes one or more of the following: geometric optimization loss, sample quality weighted loss, distribution distance loss, and distribution focusing loss. The geometric optimization loss comprehensively considers the overlap area between the predicted box and the ground truth box, the distance between the center points, the aspect ratio, the angle cost, the distance cost, and the shape cost. The sample quality weighted loss introduces a weighting factor to enhance the gradient contribution of high-quality samples and suppress the negative impact of low-quality samples. The distribution distance loss combines JS divergence and Hellinger distance to measure the difference between the predicted probability distribution and the true label probability distribution; The distribution focusing loss focuses the model on the region close to the true label by maximizing the probabilities of the two boundary points closest to the true label.

7. The method according to claim 1, characterized in that, The training employs a phased training strategy to optimize the multi-task network model: The phased training strategy includes, in sequence, the backbone network pre-training phase, the relevant module training phase, and the joint optimization phase; During the pre-training phase of the backbone network, the parameters of the correlation module are fixed and only the basic feature extraction capabilities of the target detection branch and the pose estimation branch are optimized. During the training phase of the correlation module, the weights of the backbone network are fixed and only the parameters of the multi-task correlation module are optimized to establish the correlation between tasks; In the joint optimization phase, all network parameters are unfrozen and the backbone network and related modules are optimized simultaneously to achieve collaborative learning between tasks.

8. The method according to claim 1, characterized in that, After the preprocessed image is input into the trained network model, the damage or defect detection results of the target surface and the spatial pose parameters of the target are output in parallel through forward inference. The damage or defect detection results include damage type, location, and confidence level information; The spatial attitude parameters include the target's orientation angle and component rotation angle, which are used to provide input for the dynamic planning of subsequent inspection routes.

9. The method according to claim 1, characterized in that, The process of automatically generating UAV inspection routes based on the output spatial attitude parameters, adapting to the current target structural characteristics, and realizing dynamic planning and adaptive adjustment of the inspection path, specifically includes: Based on the target orientation angle and component rotation angle, a three-dimensional spatial model of the target structure is created using a rotation matrix, and the starting waypoint for the inspection is calculated. The hovering shooting interval is calculated based on the camera's field of view and the ideal distance between the drone and the target; Hovering waypoints for shooting on the windward and leeward sides are generated along the direction of the target structure, and flight paths to avoid obstacles are set to achieve dynamic planning and adaptive adjustment of the inspection path.

10. A real-time detection and judgment system for blockage and damage in slopes and ditches by unmanned aerial vehicles (UAVs), applied to the method described in any one of claims 1 to 9, characterized in that, The system includes: The data acquisition module is used to collect real-time image data of the target area using a drone and to preprocess the acquired images. The model building and training module is used to build and train a deep learning-based multi-task network model. The multi-task network model performs target detection, pose estimation and multi-task feature association by sharing a feature extraction layer and task-specific branches. The training adopts a multi-dimensional joint optimization loss function and a phased training strategy. The detection and estimation module is used to input the preprocessed image into the trained network model and output the detection results of target damage or defects and the spatial pose parameters of the target. The route planning module is used to automatically generate UAV inspection routes that adapt to the current target structure characteristics based on the output spatial attitude parameters, so as to realize dynamic planning and adaptive adjustment of the inspection path. The output module is used to determine abnormal situations based on the detection results and generate corresponding output information, completing the entire process of automated inspection from data acquisition to result output.

Images