Road disease detection method and device under unmanned aerial vehicle perspective, electronic equipment and program product

Abstract

The application discloses a road disease detection method and device under the perspective of a UAV, an electronic device and a program product. The method is realized based on a trained target detection model. When the target detection model is trained, the initial query is dynamically adjusted through a self-adaptive noise query generation mechanism, so that the generated noise query is more consistent with the target distribution characteristics, thereby improving the query expression stability, model convergence efficiency and detection accuracy. After training, the model removes the noise query generation process without increasing the inference complexity. In addition, during training, a Hungarian matching cost function and a consistency loss function are designed in combination with scale perception, shape consistency and direction consistency, which can improve the matching reliability of small targets and complex disease targets. Moreover, whether the model is in the training process or the application process, the backbone network is provided with a direction perception strip feature enhancement module, which can enhance the representation ability of the spatial continuity features of cracks, joints and strip-shaped diseases.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to a method for detecting road defects from the perspective of an unmanned aerial vehicle (UAV), a device for detecting road defects from the perspective of an UAV, electronic equipment, and computer program products. Background Technology

[0002] With the widespread application of drone platforms in urban inspection and road maintenance, automatic road defect detection based on drone aerial images is gradually becoming an important technical means in smart transportation and refined urban management. Compared with traditional vehicle-mounted inspection methods, drones have advantages such as flexible viewing angle, wide coverage, and low deployment cost, enabling rapid inspection of urban roads, park roads, and highways without affecting traffic flow.

[0003] However, road defects in drone aerial images typically exhibit large scale variations, discrete distribution, and a high proportion of small targets, making it difficult for existing detection models to consistently and accurately detect different defects. Especially in query-based target detection frameworks, noise queries are often generated by randomly adding noise to the real target bounding boxes to aid model convergence. However, in complex scenarios, this type of denoising training method offers limited improvement to model convergence efficiency and detection accuracy. Summary of the Invention

[0004] This application provides a road defect detection method, a road defect detection device, electronic equipment, and computer program product from the perspective of an unmanned aerial vehicle (UAV). Through an adaptive noise query generation mechanism, the initial query is dynamically adjusted to make the generated noise query more consistent with the target distribution characteristics, thereby improving the stability of the query expression and assisting the model to converge quickly.

[0005] Firstly, this application provides a method for detecting road defects from the perspective of an unmanned aerial vehicle (UAV), including:

[0006] The acquired image to be detected is input into a pre-trained target detection model to obtain the road defect detection result corresponding to the image to be detected; the image to be detected is a road image from the perspective of an UAV.

[0007] The target detection model is trained from the initial detection model, which includes: a backbone network for extracting features from the input image to obtain image features; a Transformer encoding network for globally modeling the image features to obtain enhanced image features; and a Transformer decoding network for outputting the detection result corresponding to the input image based on the enhanced image features.

[0008] During the training of the initial detection model, the Transformer encoding network includes an ADQG module. The ADQG module is used to generate a noise query related to the content of the training samples based on the real target boxes of the training samples and the image features corresponding to the training samples. The Transformer decoding network is also used to perform a denoising task based on the noise query to obtain the real target boxes corresponding to the noise query, thereby assisting the initial detection model in convergence. The target detection model is obtained from the converged initial detection model with the ADQG module removed.

[0009] Optionally, the ADQG module is specifically used for:

[0010] For each training sample:

[0011] The encoded image features corresponding to the current training sample are split into multi-scale two-dimensional features;

[0012] Determine the local features of the target content corresponding to the true target box of the current training sample in the two-dimensional features at each scale;

[0013] The target content features are obtained by stitching together the local features of the target content at all scales;

[0014] Noise queries are generated based on the target content features and the true target bounding boxes to correspond to the current training samples.

[0015] Optionally, a noisy query corresponding to the current training sample is generated based on the target content features and the true target bounding box, including:

[0016] Generating perturbation information based on target content characteristics;

[0017] Determine the noisy reference box based on the perturbation information and the scale information corresponding to the real target box;

[0018] Noise queries are generated based on target content features and noise reference boxes to correspond to the current training samples.

[0019] Optionally, the loss function of the initial detection model includes the Hungarian matching cost function and the matching consistency loss function. The Hungarian matching cost function introduces scale constraints to characterize target scale differences, morphological constraints to characterize target shape differences, and directional constraints to characterize target orientation differences, based on the classification cost, location cost, and overlap cost. The matching consistency loss function is used to constrain the consistency between prediction results at different decoding stages.

[0020] The acquired image containing road cracks is converted into a block-level representation, and location information is introduced to obtain a feature sequence.

[0021] State modeling is performed on the feature sequence to obtain global semantic features;

[0022] Extract local detail features from the image to be segmented;

[0023] The global semantic features and the local detail features are fused to obtain the target fusion features;

[0024] Based on the target fusion features, the crack segmentation result corresponding to the image to be segmented is output.

[0025] Optionally, the Hungarian matching cost function is defined as:

[0026]

[0027] ,

[0028] ,

[0029] , ,

[0030] ,

[0031] The matching consistency loss function is:

[0032] .

[0033] Indicates the first The first real frame and the first The overall matching cost between prediction boxes; Indicates the actual target index; Indicates the target index for prediction; Indicates the first One real frame; Indicates the first One prediction box; , These represent the x and y coordinates of the center point of the actual bounding box, respectively. , These represent the width and height of the actual bounding box, respectively. , These represent the x and y coordinates of the center point of the prediction box, respectively. , These represent the width and height of the prediction box, respectively; Represents the weighting coefficient of the classification cost item; Indicates the first The first real goal and the first The classification cost among the prediction targets; Indicates the scale weighting factor; Indicates the first The area of ​​a real frame; This represents a very small constant, used to prevent the denominator from being zero; This represents the weight coefficient of the L1 position regression term; The L1 distance represents the positional parameter between the ground truth bounding box and the predicted bounding box; Represents the weighting coefficient of the IoU cost term; This represents the intersection-over-union ratio between the ground truth bounding boxes and the predicted bounding boxes; The weighting coefficient of the morphological consistency cost term; This represents the cost of morphological consistency matching between the ground truth bounding box and the predicted bounding box. Indicates the first Anisotropy coefficients of a real bounding box; Indicates the first Anisotropy coefficients of each prediction box; Indicates the first The second-order morphological matrix corresponding to each real frame; Indicates the first The second-order morphological matrix corresponding to each prediction box; Represents determinant operations; Represents trace operation; The weighting coefficients of the directional consistency cost term; This represents the cost of directional consistency matching between the ground truth bounding box and the predicted bounding box. Indicates the first The main orientation angle of a real bounding box; Indicates the first The main orientation angle of each prediction box; This represents the consistency loss function. Indicates the total number of decoding layers; Indicates the decoding layer index; Indicates the first The prediction box is in the... The output of the layer decoder; Indicates the first The prediction box is in the... The output of the layer decoder; This represents the L2 norm distance between the decoding outputs of two adjacent layers.

[0034] Optionally, the backbone network is equipped with a DSFEM module, which is used to extract orientation-aware features along the two directions of the feature map to model the spatial continuity features of strip-shaped disease targets.

[0035] Optionally, the DSFEM module is used to extract direction-aware features from the two directions of the input features to model the spatial continuity features of strip-shaped disease targets, including:

[0036] The input features are transformed to obtain the basic features;

[0037] Strip spatial attention operations are performed on the basic features in two directions to obtain the corresponding directional strip enhanced features;

[0038] Statistical processing is performed on the enhancement features of each direction strip to obtain the corresponding direction weights;

[0039] The corresponding directional strip enhancement features are weighted based on the directional weight, and the weighted directional strip enhancement features are fused to obtain the directional fusion features;

[0040] Based on preset learnable parameters, the directional fusion features and basic features are fused to obtain target enhancement features;

[0041] The target enhancement features are fused with the input features to obtain the output features corresponding to the input features.

[0042] Optionally, performing strip spatial attention operations on the basic features along two directions yields corresponding directional strip enhanced features, including:

[0043] In each direction:

[0044] Perform global average pooling on the basic features to obtain the pooling result;

[0045] The pooling results are sequentially subjected to convolution, attention, and reshaping operations to obtain the strip weights.

[0046] The basic features are expanded along the current direction using a sliding window to obtain multiple local stripes in the current direction;

[0047] Based on the strip weights, the local strips in the current direction after reshaping are weighted separately to obtain the local strip enhancement features in the current direction;

[0048] Summing all local strip enhancement features and reshaping the summation result yields the strip enhancement features in the current direction.

[0049] Secondly, this application provides a road defect detection device from the perspective of an unmanned aerial vehicle (UAV), comprising:

[0050] The detection module is used to input the acquired image to be detected into a pre-trained target detection model to obtain the road defect detection result corresponding to the image to be detected; the image to be detected is a road image from the perspective of an UAV.

[0051] The target detection model is trained from the initial detection model, which includes: a backbone network for extracting features from the input image to obtain image features; a Transformer encoding network for globally modeling the image features to obtain enhanced image features; and a Transformer decoding network for outputting the detection result corresponding to the input image based on the enhanced image features.

[0052] During the training of the initial detection model, the Transformer encoding network includes an ADQG module. The ADQG module is used to generate a noise query related to the content of the training samples based on the real target boxes of the training samples and the image features corresponding to the training samples. The Transformer decoding network is also used to perform a denoising task based on the noise query to obtain the real target boxes corresponding to the noise query, thereby assisting the initial detection model in convergence. The target detection model is obtained from the converged initial detection model with the ADQG module removed.

[0053] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described in the first aspect.

[0054] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in the first aspect above.

[0055] Fifthly, this application provides a computer program product comprising a computer program that, when executed by one or more processors, implements the steps of the method described in the first aspect.

[0056] The first aspect of this application offers a beneficial effect compared to existing technologies: It uses a trained target detection model to detect road defects in an image and obtains corresponding detection results. This target detection model is trained from an initial detection model, which includes: a backbone network for feature extraction from the input image; a Transformer encoding network for global modeling of the image features to obtain enhanced image features; and a Transformer decoding network for outputting the detection results corresponding to the input image based on the enhanced image features. During the training of the initial detection model, an Adaptive Denoising Query Generator (ADQG) module is introduced into the Transformer encoding network to generate noise queries related to the training sample content, based on the real target boxes of the training samples and the corresponding image features. These noise queries are input into the Transformer decoding network as auxiliary supervisory information, enabling the network to perform denoising on the noise queries, reconstruct the corresponding real target boxes, and learn to recover the offset positions and category biases of the real targets from the disturbed targets during training. This allows for faster establishment of the matching relationship between the query and the real target, improving the model's convergence speed and detection accuracy. After the initial detection model is trained and converged, it is pruned to remove the ADQG module, thus obtaining the target detection model. This avoids the execution of the noise query generation process during the inference stage, reducing the computational load and inference complexity of the model while ensuring detection performance.

[0057] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a flowchart illustrating the road defect detection method from the perspective of an unmanned aerial vehicle (UAV) provided in an embodiment of this application.

[0060] Figure 2 This is a schematic diagram of the network structure of the ADQG module provided in the embodiments of this application;

[0061] Figure 3This is a schematic diagram of the network structure of the DSFEM module provided in the embodiments of this application;

[0062] Figure 4 This is a schematic diagram of the network structure of the road defect detection model provided in the embodiments of this application;

[0063] Figure 5 This is a schematic diagram of the road defect detection device from the perspective of an unmanned aerial vehicle (UAV) provided in an embodiment of this application.

[0064] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0065] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0066] In query-based object detection frameworks, noise queries are usually generated by randomly adding noise to the real object bounding box location to assist model convergence. However, in complex scenarios, this type of denoising training method has limited effect on improving model convergence efficiency and detection accuracy.

[0067] This study finds that most existing noise queries are constructed based on random noise or fixed embedding methods, lacking the utilization of image content, target scale, and multi-scale feature information, resulting in a deviation between the generated noise query and the real target distribution. In UAV road damage scenarios, due to the large differences in the scale of the damage targets, their discrete distribution, and the complex background, traditional noise queries struggle to stably adapt to different target features, easily affecting the matching effect between the query and the real target, thus leading to slow model convergence speed and insufficient detection accuracy.

[0068] Based on this discovery, this application proposes a road defect detection method from the perspective of unmanned aerial vehicles (UAVs). Through an adaptive noise query generation mechanism, image features corresponding to training samples and real target bounding box information are used to generate noise queries specifically related to the target content, making the generated noise queries more consistent with the target distribution characteristics. The generated noise queries are then input as auxiliary supervision information into the Transformer decoding network, enabling the Transformer decoding network to perform denoising on the noise queries and reconstruct the corresponding real target bounding boxes. Furthermore, during training, the network learns to recover the offset positions and category differences of real targets from disturbed targets, thereby establishing a faster matching relationship between the query and the real target, improving the stability of the query representation, and ultimately assisting the model in rapid convergence.

[0069] The urban management inspection method based on drones provided in this application can be applied to electronic devices such as drones, mobile phones, tablets, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and edge computing devices. This application does not impose any restrictions on the specific type of electronic device.

[0070] Considering the limited computing resources of UAVs, the deployment method of the target detection model can be flexibly configured according to actual application needs. In some embodiments, the trained target detection model can be directly deployed on the UAV to achieve real-time detection of road defects. In other embodiments, the UAV is used to collect image or video data to be detected and transmit the collection results to a server, edge computing device, or other computing terminal, where the corresponding device completes the training process of the target detection model, or completes the training process and the defect detection process based on the target detection model. Through these methods, the computing burden and energy consumption of the UAV can be reduced while ensuring detection efficiency.

[0071] To illustrate the technical solutions proposed in this application, the following description will use an electronic device as the execution subject to illustrate various embodiments.

[0072] Figure 1 A schematic flowchart of the road defect detection method from the perspective of an unmanned aerial vehicle (UAV) provided in this application is shown. The road defect detection method from the perspective of an UAV includes:

[0073] Step 110: The electronic device inputs the acquired image to be detected into the pre-trained target detection model to obtain the road defect detection result corresponding to the image to be detected.

[0074] The images to be detected are road images captured by drones, typically including urban roads, park roads, or highways, and may contain road defects such as cracks, potholes, joints, and repair marks. By inputting the images to be detected into a trained target detection model, the corresponding road defect detection results can be obtained. The detection results include information such as the location, category, and confidence level of the defect targets.

[0075] Electronic devices can use a processor to call the program code corresponding to the target detection model to realize the detection of road defects in the image to be detected; the program code can be stored in a computer-readable storage medium.

[0076] The target detection model can be obtained by pruning an initial detection model after training convergence. This initial detection model includes a backbone network, a Transformer encoder network, and a Transformer decoder network. The backbone network performs multi-level feature extraction on the input image to obtain image features containing texture, edge, shape, and semantic information. The Transformer encoder network performs global correlation modeling on the image features to enhance the contextual relationships between different regions, resulting in enhanced image features. The Transformer decoder network interacts with the enhanced image features through a preset query to progressively determine the location and category of the target, outputting the detection result corresponding to the input image.

[0077] For query-based object detection models, the essence is based on Hungarian matching, which uses the matching relationship between the query and image features to progressively predict the location and category of the target. Therefore, the quality of the query directly affects the accuracy of target matching and the convergence effect of the model during training.

[0078] Considering that road images captured by UAVs typically have high resolution, large field of view, complex backgrounds, small proportions of damaged targets, and significant scale variations, conventional queries may struggle to stably match different damaged targets, thus affecting model training performance. Therefore, during the training phase, an ADQG module is added to the Transformer encoding network. The ADQG module generates a noisy query related to the training sample content based on the ground truth bounding boxes and the corresponding image features. This noisy query is then input into the Transformer decoding network for denoising, recovering the corresponding ground truth bounding boxes from the perturbed queries. This approach allows the model to learn a more stable correspondence between preset queries and ground truth targets during training, thereby improving target matching ability, accelerating model convergence, and increasing detection accuracy for small targets, complex backgrounds, and targets with significant scale variations.

[0079] After the model training is completed and converged, the initial detection model can be pruned to remove the ADQG module, resulting in the final target detection model. This avoids the execution of the noise query generation process during the inference stage, reducing the computational load and inference complexity while maintaining detection performance.

[0080] In some embodiments, during training, in order for the ADQG module to generate high-quality noisy queries to assist model training, the electronic device may perform the following operations through the ADQG module for each training sample:

[0081] Step A1: Decompose the encoded image features corresponding to the current training sample into multi-scale two-dimensional features.

[0082] For the image features output by the backbone network, after being input into the Transformer encoding network, the Transformer encoding network performs global correlation modeling and serialization encoding on the image features to obtain a feature embedding sequence. This feature embedding sequence typically integrates semantic information from different spatial locations and scales, which helps to enhance the model's ability to represent the global context.

[0083] However, since feature embedding sequences are typically represented as one-dimensional sequences, they no longer retain the two-dimensional spatial structure corresponding to features at different scales. Therefore, it is difficult to directly locate the corresponding regions of the real target bounding boxes in features at each scale, and it is also not conducive to subsequent extraction of local visual features corresponding to the target region. Based on this, electronic devices can use the ADQG module to split the encoded feature embedding sequence according to the index position of each scale feature in the feature embedding sequence. For example, a splitting method can be used to separate the feature sequences corresponding to different scales and restore them to the form of two-dimensional feature maps of the corresponding scales, thereby obtaining two-dimensional feature representations corresponding to multiple scales. This preserves the global semantic information encoded by the Transformer and restores the spatial positional relationship of features at different scales, enabling subsequent extraction of multi-scale local features of the corresponding regions for the real target bounding boxes.

[0084] Step A2: Determine the local features of the target content corresponding to the true target box of the current training sample in the two-dimensional features of each scale.

[0085] For the ground truth bounding boxes in the current training samples, the electronic device can determine the normalized position of the ground truth bounding boxes in the original image through the ADQG module, and map the ground truth bounding boxes to the corresponding spatial regions in the feature maps at each scale based on the scale correspondence between the feature maps at each scale and the original image. Subsequently, a region alignment method can be used to extract local feature vectors for the corresponding spatial regions in the feature maps at each scale as local features of the target content, thereby obtaining a multi-scale feature representation corresponding to the current ground truth bounding box.

[0086] This approach not only allows for the acquisition of local semantic information of the target region under different receptive fields, but also preserves the detailed features of small targets and the overall structural features of large targets. This enhances the ability to represent disease targets with large scale variations, irregular shapes, and complex backgrounds, providing a more accurate feature basis for generating noise queries related to the target content.

[0087] Step A3: Segment the local features of the target content at all scales to obtain the target content features.

[0088] Step A4: Generate a noise query corresponding to the current training sample based on the target content features and the real target bounding box.

[0089] To obtain more complete target content information, electronic devices can stitch together and fuse local features of the target region extracted at different scales using the ADQG module, and map them to a unified dimension to obtain target content features. These target content features are used to characterize the comprehensive visual features of the current road surface defects under different receptive fields. Since features at different scales focus on the target's local details, edge texture, overall structure, and contextual semantic information, fusing multi-scale target features can yield a more comprehensive representation of the target content, avoiding the problems of incomplete target features or insufficient semantic expression caused by relying solely on single-scale features.

[0090] After obtaining the target content features, the electronic device can further combine them with the real target bounding box to generate a corresponding noise query. The target content features are used to determine the content vector of the noise query, while the real target bounding box is used to determine the location reference information of the noise query. Compared to noise queries generated through random noise, fixed embeddings, or single-category information, the noise queries generated in this way can simultaneously reflect the actual visual features, spatial location, and scale information of the target, thus better adapting to the distribution characteristics of different disease targets.

[0091] Furthermore, by introducing scale-aware noise modeling, the perturbation range can be adaptively adjusted according to the target scale, avoiding excessive positional shifts for small-scale disease targets and improving stability during small target training. Since the noise query originates from the real visual features of the target region, it can provide more discriminative query information to the decoder in the early stages of training, alleviating the problem of query initialization instability in large field-of-view and high-resolution scenes of UAVs, accelerating model convergence speed, and improving detection accuracy.

[0092] In some embodiments, generating a noisy query corresponding to the current training sample based on target content features and the true target bounding box includes:

[0093] Step A41: Generate perturbation information based on the characteristics of the target content.

[0094] Electronic devices can generate corresponding perturbation information based on the target feature vector through the ADQG module. For example, the perturbation information includes the offset of the target box center position and the scaling of the target box width and height. That is, this perturbation information can characterize the offset trend of the real target box in position, size or center point, so that the subsequent noise query no longer depends only on a fixed noise distribution, but can be adaptively adjusted in combination with the visual features of the current target.

[0095] Step A42: Determine the noisy reference box based on the perturbation information and the scale information corresponding to the real target box.

[0096] Then, the electronic device can combine the perturbation information with the scale information corresponding to the real target box through the ADQG module to determine the noise reference box. Since targets of different scales are less sensitive to positional offsets, the perturbation amplitude can be constrained based on the target scale, so that the generated noise reference box remains adjacent to the real target box but has a certain offset. This not only avoids introducing excessive perturbation to small-scale disease targets, but also enhances the perturbation diversity of large-scale targets, thereby improving the rationality of noise modeling.

[0097] Step A43: Generate a noise query corresponding to the current training sample based on the target content features and the noise reference box.

[0098] Finally, the electronic device can generate a content vector corresponding to the noisy query based on the target content features using the ADQG module, and generate positional reference information corresponding to the noisy query based on the noisy reference box, thereby constructing the noisy query corresponding to the current training sample. The generated noisy query can be input into the decoder along with the original target query to guide the decoder in learning to recover the position and category of the real target from perturbed targets during training, thereby enhancing the matching ability between the query and the real target, accelerating the model convergence speed, and improving the detection accuracy in complex scenes.

[0099] In this embodiment, the electronic device utilizes the ADQG module to generate a noisy query by combining target content features, the true target bounding box, and scale information. This allows the noisy query to simultaneously possess the visual semantics, spatial location, and scale features of the target, thus more closely approximating the distribution of real targets. Compared to noisy queries generated based on random noise or fixed embedding methods, this approach improves the matching ability between the query and the real target, enhances the stability of target localization and classification during the training phase, and effectively improves the model's convergence speed and detection accuracy in complex scenes.

[0100] For example, see Figure 2 , Figure 2 This illustrates one possible network structure for the ADQG module.

[0101] Assume that the encoded image features corresponding to the current training sample are F :

[0102]

[0103] in, , E For a unified embedding dimension, Corresponding to different scales. The result after stitching is:

[0104]

[0105] Will Femb Split by scale:

[0106]

[0107] Then, a dimensional transformation is performed to obtain multi-scale two-dimensional features:

[0108]

[0109] Let the first i The normalized image size of the annotation boxes for each pavement defect is:

[0110]

[0111] In the two-dimensional features at each scale, the regions corresponding to the ground truth bounding boxes are sampled and aligned using global pooling. This indicates Region Alignment (RoI Alignment), where h and w represent the scale by which each real target bounding box needs to be aligned.

[0112]

[0113] Disease features from different scales are spliced ​​together and mapped. Indicates a linear mapping. :

[0114]

[0115] Noise offset (i.e., disturbance information) prediction:

[0116]

[0117] Scale noise modeling:

[0118]

[0119] Construct a noise reference box. Noise adjustment factor:

[0120]

[0121] Obtain the noise content query:

[0122]

[0123] Obtain the noise location query (Position Query):

[0124]

[0125] Compared with existing methods that construct noise queries based on random noise or fixed embedding methods, Figure 2 The ADQG module shown can directly generate noise queries (including noise content queries and noise location queries) based on the target region features in the encoded features, making the noise queries closely related to the image content, target location and scale information, thereby improving the semantic consistency and spatial rationality of the noise queries.

[0126] In UAV-based road damage scenarios, different damage targets exhibit significant differences in scale, shape, and texture. For example, cracks typically present as elongated, small-scale features, while potholes and subsidence have a larger spatial range. Based on this, the ADQG module fuses multi-scale target features and combines them with target scale information to generate perturbation information. This allows the noise query to adaptively adjust the perturbation range according to the actual scale of different damage targets, avoiding excessive offsets for small-scale targets and improving the stability of training for small damage targets. Simultaneously, since the content vector of the noise query originates from the real visual features of the target region, rather than random initialization or fixed-class embedding, it can provide more discriminative query information to the decoder in the early stages of training. This alleviates the problem of unstable query initialization in UAV-based large field-of-view, high-resolution scenarios, accelerates model convergence, and improves detection accuracy. Furthermore, the ADQG module can be removed from the model after training, thus fully utilizing the performance improvement brought by the high-quality noise queries generated during training without increasing the computational complexity of the inference stage. Therefore, it is particularly suitable for application scenarios with large scale variations of road damage targets and complex backgrounds from a UAV top-down perspective.

[0127] This application also found that in road defect detection tasks from the perspective of UAVs, targets typically exhibit slender, irregular, highly directional, and significantly scale-varying structural features, such as longitudinal cracks, transverse cracks, network cracks, and strip-shaped repair areas. These types of targets are often difficult to stably perceive in traditional convolutional or standard self-attention mechanisms due to insufficient spatial relationship modeling, lack of directional sensitivity, and inadequate multi-scale structural coupling.

[0128] In some embodiments, to address the aforementioned issues, a direction-aware strip feature enhancement module can be introduced into both the initial detection model and the target detection model. The DSFEM module can extract direction-aware features along two directions of the feature map to explicitly model strip-shaped defects, such as the continuous spatial distribution of road cracks and / or joints. Compared to approaches that only focus on local receptive fields or global feature associations, the DSFEM module enhances the model's ability to represent slender structures, edge orientation, and directional continuity, making it easier for the model to distinguish crack orientation, strip boundaries, and complex texture backgrounds.

[0129] Furthermore, the DSFEM module can adaptively enhance directional features consistent with the pathology of road defects by combining feature responses from different directions and suppressing background interference from irrelevant directions, thereby improving the ability to perceive fine cracks, discontinuous cracks, and complex-shaped defects. For defects with large scale variations, the DSFEM module can also enhance long-distance directional dependencies while preserving local detail information, thus improving the model's detection stability and localization accuracy for different types of road defects.

[0130] In some embodiments, the DSFEM module is used to extract direction-aware features from two directions of the input features to model the spatial continuity features of strip-shaped defects such as road cracks and joints, including:

[0131] Step B1: Perform feature transformation on the input features to obtain the basic features.

[0132] Before modeling, in order to compress the input features, adjust the dimensions, or enhance the semantics, and to filter out some irrelevant noise, electronic devices can perform feature transformation on the input features through the DSFEM module. For example, the number of channels can be adjusted by 1×1 convolution, and local texture and edge information can be enhanced by 3×3 convolution and activation function to obtain basic features, thereby providing a more stable feature foundation for subsequent directional feature extraction.

[0133] Step B2: Perform strip spatial attention operations on the basic features along two directions to obtain the corresponding directional strip enhancement features.

[0134] After obtaining the basic features, to enhance the model's ability to perceive the spatial continuity of strip-shaped defects such as road cracks and joints, the electronic device can perform strip spatial attention operations on the basic features in two directions through the DSFEM module. For example, horizontal strip convolution, vertical strip convolution, pooling operations along the row direction, or pooling operations along the column direction can be used to model the continuous structure of the basic features in different directions, obtaining corresponding directional strip enhancement features. This enables the model to more accurately perceive the extension trend and structural features of defects in different directions.

[0135] Step B3: Perform statistical processing on the enhancement features of each direction strip to obtain the corresponding direction weights.

[0136] Step B4: Weight the corresponding directional strip enhancement features based on the directional weights of each feature, and fuse the weighted directional strip enhancement features to obtain the directional fusion features.

[0137] Furthermore, to enable the model to adaptively determine the importance of features in different directions, the electronic device can perform statistical processing on the strip enhancement features in each direction using the DSFEM module. For example, mean pooling, max pooling, or global statistical operations can be performed on the strip enhancement features in each direction to obtain the feature response intensity in different directions, and the corresponding directional weights can be determined based on the feature response intensity.

[0138] Based on this, electronic devices can weight the corresponding directional strip enhancement features according to the directional weights of each. For example, multiplicative weighting can be used to enhance the feature response in high-weight directions. Then, the weighted directional strip enhancement features can be fused through methods such as concatenation, addition, or convolutional fusion to obtain directional fusion features. In this way, the model can adaptively enhance directional information consistent with the direction of the disease target based on the directional features of the current target, while suppressing background interference in irrelevant directions, thereby enhancing the model's feature representation ability for slender and irregular disease targets.

[0139] Step B5: Based on preset learnable parameters, fuse the directional fusion features and the basic features to obtain the target enhancement features.

[0140] To balance the proportion of directional enhancement features and original base features, and to avoid the loss of local texture or semantic details due to excessive directional information, electronic devices can use the DSFEM module to perform weighted fusion of directional fusion features and base features based on preset learnable parameters. For example, the contribution levels of directional fusion features and base features can be adjusted separately using learnable coefficients, and then the target enhancement features can be obtained through methods such as addition, concatenation, or convolutional fusion. This allows the model to adaptively adjust the balance between directional information and original semantic information according to the characteristics of different disease targets.

[0141] Step B6: Fuse the target enhancement features with the input features to obtain the output features corresponding to the input features.

[0142] Finally, to preserve the original information in the input features while enhancing the directional continuity features, the electronic device can fuse the target enhancement features with the input features through the DSFEM module. For example, residual connections, feature concatenation, or element-wise addition can be used to obtain the output features corresponding to the input features. This approach not only enhances the model's ability to represent the directional features of the disease target but also avoids feature degradation or information loss, improving the stability of the model during training.

[0143] In this embodiment, the DSFEM module can explicitly enhance continuous structural features in different directions while preserving the original feature information, adaptively highlight directional information consistent with the direction of the disease, and suppress complex background interference, thereby improving the model's detection stability and positioning accuracy for slender and irregular disease targets such as cracks, joints, and strip-shaped repair areas.

[0144] In some embodiments, the electronic device may obtain the corresponding strip enhancement feature in each direction by performing the following operations via the DSFEM module:

[0145] Step B21: Perform global average pooling on the basic features to obtain the pooling result.

[0146] To obtain the overall response information of the basic features in the current direction with lower computational overhead, the electronic device can perform global average pooling on the basic features using the DSFEM module to obtain the pooling result. This not only compresses the spatial information of the basic features but also preserves the overall statistical characteristics of each channel in the current direction, thus providing a basis for subsequent generation of strip weights.

[0147] Step B22: Perform convolution, attention, and reshaping operations on the pooling results in sequence to obtain the strip weights.

[0148] To generate strip weights that reflect the importance of local regions in different directions based on the pooling results, the electronic device can sequentially perform convolution, attention, and reshaping operations on the pooling results using the DSFEM module. For example, convolution operations can be used to map channels and transform features in the pooling results; attention operations can enhance salient responses in key directions; and reshaping operations can adjust the results to a weight form corresponding to the local strips in the current direction, thus obtaining the strip weights. In this way, the model can adaptively determine the importance of different local strips based on the overall response characteristics in the current direction.

[0149] Step B23: Expand the basic features along the current direction using a sliding window to obtain multiple local stripes in the current direction.

[0150] To locally model the continuous structure of a basic feature in the current direction, the electronic device can use the DSFEM module to perform a sliding window expansion of the basic feature along the current direction, resulting in multiple local stripes in the current direction. For example, in the horizontal direction, the basic feature can be expanded along the row direction; in the vertical direction, it can be expanded along the column direction. That is, by splitting the basic feature into multiple local regions with directional continuity, a foundation can be laid for subsequently capturing the local structural features of cracks, joints, and strip-shaped defects in the current direction.

[0151] Step B24: Based on the strip weights, weight each reshaped local strip in the current direction to obtain the local strip enhancement features in the current direction.

[0152] Electronic devices can use the DSFEM module to weight local strips in each current direction based on strip weights, obtaining the enhanced features of each local strip in the current direction. This can enhance the features of local areas consistent with the target direction of the disease. Specifically, the weighting can be applied to the corresponding local strip features through element-wise multiplication, thereby enhancing the feature response of important strip areas while suppressing interference from irrelevant or background areas.

[0153] Step B25: Sum all the local strip enhancement features and reshape the summation result to obtain the strip enhancement features in the current direction.

[0154] Finally, the electronic device can sum all local strip enhancement features using the DSFEM module and perform a reshaping operation on the summation result. This allows for the reintegration of local strip information in the current direction, resulting in strip enhancement features for that direction. This process restores the overall spatial layout in the current direction while preserving local structural features, thus forming directional strip enhancement features that possess both directional continuity and global consistency.

[0155] In this embodiment, the DSFEM module first generates strip weights related to the current direction using global statistical information, and then models the local continuous structure in different directions using a sliding window expansion method, thereby enhancing the feature response of strip-shaped defects such as cracks and seams in the corresponding directions. Simultaneously, by weighting and reconstructing the local strips, the directional continuity, edge morphology, and spatial structure information of the target can be preserved while suppressing background noise and irrelevant texture interference. This allows the model to more stably perceive slender, irregular, and highly directional road defect targets, and improves detection accuracy and robustness in complex UAV scenarios.

[0156] In some embodiments, see Figure 3 , Figure 3 A schematic diagram of the network structure of a DSFEM module is shown. Based on this structure, the electronic device can construct a strip perception mechanism along the horizontal and vertical directions based on input features, explicitly modeling the spatial directional continuity features of strip-shaped defects such as road cracks and joints. The DSFEM module does not require explicit construction of a query-key-value attention structure; instead, it achieves efficient modeling of long-distance structural information based on strip unrolling and adaptive weight generation. This modeling is then embedded as residuals into the multi-scale feature layers of the backbone network, significantly enhancing directional discrimination capabilities with almost no increase in computational complexity.

[0157] In its implementation, the DSFEM module includes a horizontal strip enhancement branch and a vertical strip enhancement branch. Each branch first performs global aggregation on the input features to generate direction-dependent strip weights, and then extracts local continuous structural features in the corresponding direction through strip unrolling. By weighted fusion of the strip weights and the unrolled features, it enhances the response to typical linear and strip patterns in road defects. Furthermore, to avoid manually setting the importance of directions, the DSFEM module introduces an implicit direction gating mechanism, adaptively adjusting the fusion ratio of horizontal and vertical features based on the global response strength of the enhanced features in different directions. Finally, the enhanced features are superimposed on the original features in residual form, achieving stable training and allowing flexible application to multiple scale output layers of the backbone network (such as P3, P4, and P5 layers).

[0158] Let the input features be The input features can come from any scale of the backbone network or feature pyramid (such as P3 / P4 / P5). The forward propagation process of the DSFEM module can be represented as:

[0159] First, a convolutional layer is used to model the local context of the input features, where... The activation function is the sigmoid function, as described in the text.

[0160]

[0161] Basic features Perform global average pooling to obtain the pooling result of the channel-level description vector:

[0162]

[0163] pass Convolution maps pooling results to direction-dependent strip weights through convolution, attention, and reshaping operations:

[0164]

[0165] right Expanding the window horizontally is equivalent to doing so in each spatial location ( i, j Extract a local horizontal strip of length K at point )

[0166]

[0167] Strip enhancement output is defined as:

[0168]

[0169] Similarly, vertical strip enhancement features can be obtained in the vertical direction. .

[0170] To adaptively adjust the importance of different directions, a directional response intensity estimate is introduced through a mean operation to obtain the corresponding directional weights:

[0171]

[0172]

[0173] In each direction, the corresponding directional strip enhancement features are weighted based on the corresponding directional weights, and the weighted results from the two directions are fused to obtain directional fusion features. Finally, the directional fusion features and the basic features are fused based on preset learnable parameters to obtain the target enhancement features.

[0174]

[0175] After integration, the target enhancement features can be added element-wise with the input features to achieve residual fusion and obtain the corresponding output features.

[0176]

[0177]

[0178] In this embodiment, the DSFEM module can explicitly model the spatial directional continuity features of road defects, making it particularly suitable for strip-shaped anomalies such as longitudinal cracks and lateral damage commonly seen in UAV aerial photography. By constructing strip perception mechanisms in both the horizontal and vertical directions, the module can effectively capture long-distance structural correlation information while maintaining local detail perception capabilities, thereby significantly improving the detection and differentiation capabilities for slender, small-scale road defect targets. Furthermore, this module does not rely on query-key-value calculations and Softmax operations in self-attention structures; instead, it uses strip unrolling and channel-level weight mapping to achieve directional modeling, resulting in low computational complexity and low parameter overhead, making it suitable for multi-scale feature enhancement of high-resolution UAV images. Simultaneously, by introducing an implicit directional gating mechanism, the module can adaptively adjust directional feature weights according to different road scenarios without requiring manual rule setting, demonstrating good generalization ability and engineering deployment value.

[0179] In some embodiments, see Figure 4 , Figure 4 A schematic diagram of a possible network structure for the initial detection model is shown. The initial detection model can be divided into an image encoding part and an output decoding part.

[0180] The image encoding part is used for multi-level feature extraction from the input image. The input image first undergoes image patch embedding, dividing the original image into multiple patches and mapping them to initial features. Subsequently, DSFEM modules are introduced at different stages to enhance the directional continuity structure of the feature maps, thereby improving the model's directional perception ability for cracks, ruts, joints, and strip-shaped defects. The feature map resolution is gradually reduced and the number of feature channels is increased through patch merging operations between stages, enabling the network to gradually expand its receptive field and extract richer high-level semantic features.

[0181] In some stages, fully connected layers and reshaping operations can be used to uniformly map features at different scales, enabling subsequent multi-scale feature fusion. The fused features can then be further input into an image encoder, which may include multi-head attention mechanisms, residual connections and normalization structures, and feedforward neural networks to model long-distance dependencies between different spatial locations and enhance feature representation capabilities.

[0182] Furthermore, an ADQG module can be introduced during the image encoding stage. The ADQG module generates a noise query based on the target content features corresponding to the real target bounding box, including content encoding and location encoding. Content encoding is used to represent the texture, edge, and semantic features of the target region, while location encoding is used to represent the spatial location information of the target region. Content encoding and location encoding can be further processed through a fully connected mapping to jointly constitute the noise query input-output decoding part.

[0183] The output decoding section may include multiple stacked decoding layers, each of which may incorporate a multi-head attention mechanism and a feedforward neural network. The decoder receives global features from the image encoding section, as well as target and noise queries, to progressively classify and locate road damage targets. Finally, the detection head outputs the category and bounding box information of the road damage targets, thereby enabling the identification and localization of different types of road damage targets.

[0184] In some embodiments, during the training of the initial detection model, the electronic device first acquires training samples containing road defect annotation information and performs preprocessing operations such as image scaling, cropping, and flipping on the training samples. Subsequently, the processed training samples are input into the backbone network and feature fusion network to extract multi-layer features at different scales. Based on this, the DSFEM module can enhance the direction perception capability of slender cracks, joints, and strip-shaped defects, and the ADQG module can generate noise queries related to the image content based on the target content features corresponding to the real target boxes. Then, the target queries and noise queries are jointly input into the decoder to perform classification and localization prediction of the road defect targets. Based on the classification loss, bounding box regression loss, and overlap loss between the prediction results and the real annotations, the model parameters are backpropagated and iteratively updated. After multiple rounds of training, an initial detection model capable of adapting to the needs of detecting defects at multiple scales in complex backgrounds under UAV overhead views can be obtained.

[0185] In some embodiments, in order to address the highly uneven characteristics of road damage targets in terms of scale, morphology, and spatial distribution from the perspective of UAVs, the loss function of the initial detection model includes a Hungarian matching cost function and a matching consistency loss function. The Hungarian matching cost function, based on the classification cost, location cost, and overlap cost, introduces scale constraints to characterize target scale differences, morphological constraints to characterize target morphological differences, and directional constraints to characterize target directional differences. The matching consistency loss function is used to constrain the consistency between prediction results at different decoding stages.

[0186] In some embodiments, the Hungarian matching cost function and the matching consistency loss function can be constructed based on structure-aware weighted Hungarian matching (SA-WHM) and consistency optimization methods.

[0187] This method, while maintaining the original Hungarian matching framework, introduces scale-aware weights and structural consistency constraints, enabling the matching process to more accurately reflect the geometric characteristics of different types of road defects, thereby improving the model's matching stability and detection accuracy in complex road scenarios. Unlike traditional matching strategies that rely solely on classification cost and IoU cost, this method explicitly models the target scale and shape information during the matching stage. This allows small-scale, slender, or structurally fragmented defect targets to receive more reasonable optimized weights during the matching process, preventing them from being ignored or incorrectly assigned in the global matching.

[0188] Let the first i The labels for each real bounding box are:

[0189]

[0190] No. j The prediction boxes are:

[0191]

[0192] in, Indicates the actual target index; Indicates the target index for prediction; Indicates the first One real frame; Indicates the first One prediction box; , These represent the x and y coordinates of the center point of the actual bounding box, respectively. , These represent the width and height of the actual bounding box, respectively. , These represent the x and y coordinates of the center point of the prediction box, respectively. , These represent the width and height of the prediction box, respectively.

[0193] Traditional Hungarian matching can be represented as:

[0194]

[0195] in, Indicates the first The first real frame and the first The overall matching cost between prediction boxes; Represents the weighting coefficient of the classification cost item; Indicates the first The first real goal and the first The classification cost among the prediction targets; This represents the weight coefficient of the L1 position regression term; The L1 distance represents the positional parameter between the ground truth bounding box and the predicted bounding box; Represents the weighting coefficient of the IoU cost term; This represents the intersection-over-union ratio (IoU) between the ground truth bounding boxes and the predicted bounding boxes.

[0196] To reduce the imbalance effect of targets of different scales in the matching process, a scale weighting factor based on the actual target area is introduced to weight and modulate the location-related matching cost.

[0197] First, define the... Area of ​​each real bounding box:

[0198]

[0199] The scale weighting factor is defined by the following formula, where It is a minimal constant:

[0200]

[0201] By introducing scale-aware weights, this method explicitly distinguishes the importance of targets at different scales during the matching stage, enabling small-scale road defect targets to obtain greater geometric constraint weights in global matching, thereby effectively alleviating the problem that small targets are easily ignored from the perspective of UAVs.

[0202] Considering that road defects (such as cracks and spalling) have obvious shape and structural features, a second-order geometric morphology matrix is ​​introduced to model the target structure.

[0203] Define a second-order morphological matrix to describe the predicted bounding box and the ground truth bounding box; Indicates the first The second-order morphological matrix corresponding to each real bounding box:

[0204]

[0205] Define the morphological anisotropy coefficient, where Represents trace operation. Determinant operations:

[0206]

[0207] in, ,when hour, Isotropic, when or hour, This exhibits anisotropy. The cost of morphological consistency matching can be expressed as:

[0208]

[0209] The morphological consistency cost automatically characterizes the balance and anisotropy of the target in spatial expansion through matrix invariants. For road defects such as cracks and spalling that present slender structures, this metric can effectively distinguish between prediction results with similar and mismatched morphologies, making the matching process more consistent with the actual defect structure characteristics and improving the stability and accuracy of the matching.

[0210] Considering that some road defects have obvious main directional characteristics, we introduce directional consistency constraints to align the spatial orientation of the predicted box with the actual box.

[0211] Define the principal orientation angles of the ground truth bounding box and the predicted bounding box as follows:

[0212]

[0213] Indicates the first The main orientation angle of a real bounding box; Indicates the first The main orientation angle of each prediction box;

[0214] Based on the difference in principal orientation angles, the cost of orientation consistency matching between the ground truth bounding box and the predicted bounding box is defined as:

[0215]

[0216] The orientation consistency cost geometrically constrains the consistency between the predicted bounding box and the actual target in spatial orientation, which is particularly effective for disease targets with significant orientation, such as cracks. This cost naturally degenerates when the target is approximately square, and does not introduce additional constraints for disease types that are not sensitive to orientation, thus maintaining the adaptability of the matching process.

[0217] Considering the constraints of scale-aware weights, morphological consistency, and directional consistency, the final Hungarian matching cost function is defined as follows:

[0218]

[0219] After completing the Hungarian matching, to improve the continuity of prediction results during multi-layer decoding, a matching consistency loss is introduced:

[0220]

[0221] This represents the consistency loss function. Indicates the total number of decoding layers; Indicates the decoding layer index; Indicates the first The prediction box is in the... The output of the layer decoder; Indicates the first The prediction box is in the... The output of the layer decoder; This represents the L2 norm distance between the decoding outputs of two adjacent layers.

[0222] This loss is added directly to the existing regression and classification losses, which can constrain the model to refine the prediction results layer by layer during the decoding process and avoid prediction oscillations.

[0223] In this embodiment, a structure-aware weighted Hungarian matching and consistency loss method is used to jointly model road defect targets from three levels—scale, shape, and orientation—during the matching stage. This makes the matching results more consistent with the true geometric characteristics of road defect targets from the perspective of an UAV. This method effectively improves the matching reliability of small-scale defect targets without introducing additional annotation information, enhances the ability to distinguish slender and structurally complex defect targets, and improves the predictive stability of the decoding process through matching consistency loss. It has good versatility and engineering scalability.

[0224] In some embodiments, the PyTorch deep learning framework can be used to build and train an initial detection model. First, the electronic device acquires a road damage dataset and cleans, filters, and converts the original images and annotation information to meet the data format requirements for model training. Then, the dataset can be divided according to a preset ratio, such as an 8:2 ratio between the training and validation sets, to ensure that the model can not only learn features from known samples during training but also evaluate its generalization ability on unseen samples using the validation set.

[0225] To improve the model's adaptability to complex scenes, electronic devices can perform data augmentation operations on training images before inputting training samples into the network. For example, image sizes can be randomly adjusted, random noise introduced, brightness and contrast altered, or imaging effects under different weather conditions such as cloudy, rainy, or foggy days can be simulated. These methods expose the model to a richer distribution of samples, thereby enhancing its robustness under varying shooting heights, lighting conditions, and complex backgrounds.

[0226] During the training phase, the electronic device inputs enhanced training images into the initial detection model. Through forward propagation, it obtains the classification and localization results of road damage targets and calculates the loss function based on the difference between the predicted results and the ground truth annotations. Subsequently, stochastic gradient descent is used to backpropagate and iteratively update the network parameters, allowing the model to gradually converge to an optimal state. As the number of training epochs increases, the model can gradually learn the differential features of different types of road damage in terms of texture, edges, morphology, and spatial distribution, thereby improving its detection accuracy.

[0227] After training, the electronic device can save the final network structure parameters and weight parameters, and prune the ADQG module to form a directly deployable model file. In subsequent practical applications, this model file can be directly loaded to perform road defect detection tasks on road images captured by the UAV, thereby achieving rapid identification and localization of road defect targets.

[0228] In other words, this application proposes a road defect detection method from an UAV perspective, addressing the challenges of small scale, significant morphological differences, irregular spatial distribution, and severe background interference associated with road defect targets. This method focuses on key aspects such as feature enhancement, query generation, and matching optimization, enabling the detection network to better adapt to the imaging characteristics of road defect targets in UAV-view scenarios. This effectively improves detection accuracy, training stability, and generalization ability in complex scenarios.

[0229] First, this application introduces a direction-aware strip feature enhancement module in the feature extraction stage. This module extracts directional strip enhancement features along both horizontal and vertical directions to model directional defects such as cracks, ruts, joints, and strip-shaped repair areas. Since strip features in different directions can reflect the spatial continuity, extensibility, and structural orientation of defects, this module not only expands the network's effective receptive field for elongated targets but also enhances its ability to express target edges, textures, and local structural information. Simultaneously, by adaptively adjusting the feature responses in different directions through directional weights, interference from complex road textures, shadows, lane lines, and background noise can be effectively suppressed, thereby improving the model's ability to identify and locate slender, small-scale, and irregular defects.

[0230] Secondly, this application proposes an adaptive noise query generation mechanism. This mechanism extracts target content features corresponding to the real target bounding boxes from multi-scale feature maps and generates perturbation information related to the target scale based on the target content features, thereby constructing denoised reference boxes and corresponding noise queries. Since the content vector of the noise query comes from the visual features of the real target region, rather than being randomly initialized or fixedly embedded, the generated noise query can more accurately reflect the texture, shape, edge, and spatial distribution features of the target. At the same time, by combining the target bounding box scale information to adaptively adjust the perturbation range, excessive positional shifts can be avoided for small-scale disease targets, thereby improving the model's learning stability for fine-grained disease targets. This mechanism effectively alleviates the problem of fixed noise settings and decoupling from image content in traditional denoising strategies, enabling the decoder to obtain more stable and discriminative query inputs during the training phase, thereby improving the model's convergence speed and detection robustness.

[0231] Furthermore, this application designs a structure-aware weighted Hungarian matching strategy and a consistency loss function, taking into account the structural characteristics of road damage targets, such as their slender, irregular, and densely distributed distribution. During target matching, in addition to considering classification scores and positional bias, it further incorporates the spatial structural features, directional features, and prediction consistency information of the targets to achieve more reasonable matching of different prediction results. This approach effectively mitigates matching ambiguities between small targets, dense targets, and adjacent damage targets, reduces training noise caused by incorrect matching, and thus further improves the model's detection stability and generalization ability in complex UAV scenarios.

[0232] In summary, this application, through the synergistic effect of directional strip enhancement features, target content feature-driven noise query, and structure-aware matching mechanism, enables the detection network to more accurately perceive the spatial structure, directional features, and scale variation patterns of road damage targets. While ensuring inference efficiency, it significantly improves the detection accuracy and overall detection performance of small-scale, directional, and complex background damage targets from the perspective of UAVs, and has high engineering application value and promotion significance.

[0233] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0234] Corresponding to the road defect detection method from the perspective of a drone in the above embodiments, Figure 5 shows a structural block diagram of the road defect detection device 5 from the perspective of a drone provided in this application embodiment. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0235] Reference Figure 5 The road defect detection device 5 from the perspective of an unmanned aerial vehicle includes:

[0236] The detection module is used to input the acquired image to be detected into a pre-trained target detection model to obtain the road defect detection result corresponding to the image to be detected; the image to be detected is a road image from the perspective of an UAV.

[0237] The target detection model is trained from the initial detection model, which includes: a backbone network for extracting features from the input image to obtain image features; a Transformer encoding network for globally modeling the image features to obtain enhanced image features; and a Transformer decoding network for outputting the detection result corresponding to the input image based on the enhanced image features.

[0238] During the training of the initial detection model, the Transformer encoding network includes an ADQG module. The ADQG module is used to generate a noise query related to the content of the training samples based on the real target boxes of the training samples and the image features corresponding to the training samples. The Transformer decoding network is also used to perform a denoising task based on the noise query to obtain the real target boxes corresponding to the noise query, thereby assisting the initial detection model in convergence. The target detection model is obtained from the converged initial detection model with the ADQG module removed.

[0239] Optionally, the ADQG module is specifically used for:

[0240] For each training sample:

[0241] The encoded image features corresponding to the current training sample are split into multi-scale two-dimensional features;

[0242] Determine the local features of the target content corresponding to the true target box of the current training sample in the two-dimensional features at each scale;

[0243] The target content features are obtained by stitching together the local features of the target content at all scales;

[0244] Noise queries are generated based on the target content features and the true target bounding boxes to correspond to the current training samples.

[0245] Optionally, a noisy query corresponding to the current training sample is generated based on the target content features and the true target bounding box, including:

[0246] Generating perturbation information based on target content characteristics;

[0247] Determine the noisy reference box based on the perturbation information and the scale information corresponding to the real target box;

[0248] Noise queries are generated based on target content features and noise reference boxes to correspond to the current training samples.

[0249] Optionally, the loss function of the initial detection model includes the Hungarian matching cost function and the matching consistency loss function. The Hungarian matching cost function introduces scale constraints to characterize target scale differences, morphological constraints to characterize target shape differences, and directional constraints to characterize target orientation differences, based on the classification cost, location cost, and overlap cost. The matching consistency loss function is used to constrain the consistency between prediction results at different decoding stages.

[0250] The acquired image containing road cracks is converted into a block-level representation, and location information is introduced to obtain a feature sequence.

[0251] State modeling is performed on the feature sequence to obtain global semantic features;

[0252] Extract local detail features from the image to be segmented;

[0253] The global semantic features and the local detail features are fused to obtain the target fusion features;

[0254] Based on the target fusion features, the crack segmentation result corresponding to the image to be segmented is output.

[0255] Optionally, the Hungarian matching cost function is defined as:

[0256]

[0257] ,

[0258] ,

[0259] , ,

[0260] ,

[0261] The matching consistency loss function is:

[0262] .

[0263] Indicates the first The first real frame and the first The overall matching cost between prediction boxes; Indicates the actual target index; Indicates the target index for prediction; Indicates the first One real frame; Indicates the first One prediction box; , These represent the x and y coordinates of the center point of the actual bounding box, respectively. , These represent the width and height of the actual bounding box, respectively. , These represent the x and y coordinates of the center point of the prediction box, respectively. , These represent the width and height of the prediction box, respectively; Represents the weighting coefficient of the classification cost item; Indicates the first The first real goal and the first The classification cost among the prediction targets; Indicates the scale weighting factor; Indicates the first The area of ​​a real frame; This represents a very small constant, used to prevent the denominator from being zero; This represents the weight coefficient of the L1 position regression term; The L1 distance represents the positional parameter between the ground truth bounding box and the predicted bounding box; Represents the weighting coefficient of the IoU cost term; This represents the intersection-over-union ratio between the ground truth bounding boxes and the predicted bounding boxes; The weighting coefficient of the morphological consistency cost term; This represents the cost of morphological consistency matching between the ground truth bounding box and the predicted bounding box. Indicates the first Anisotropy coefficients of a real bounding box; Indicates the first Anisotropy coefficients of each prediction box; Indicates the first The second-order morphological matrix corresponding to each real frame; Indicates the first The second-order morphological matrix corresponding to each prediction box; Represents determinant operations; Represents trace operation; The weighting coefficients of the directional consistency cost term; This represents the cost of directional consistency matching between the ground truth bounding box and the predicted bounding box. Indicates the first The main orientation angle of a real bounding box; Indicates the first The main orientation angle of each prediction box; This represents the consistency loss function. Indicates the total number of decoding layers; Indicates the decoding layer index; Indicates the first The prediction box is in the... The output of the layer decoder; Indicates the first The prediction box is in the... The output of the layer decoder; This represents the L2 norm distance between the decoding outputs of two adjacent layers.

[0264] Optionally, the backbone network is equipped with a DSFEM module, which is used to extract orientation-aware features along the two directions of the feature map to model the spatial continuity features of strip-shaped disease targets.

[0265] Optionally, the DSFEM module is used to extract direction-aware features from the two directions of the input features to model the spatial continuity features of strip-shaped disease targets, including:

[0266] The input features are transformed to obtain the basic features;

[0267] Strip spatial attention operations are performed on the basic features in two directions to obtain the corresponding directional strip enhanced features;

[0268] Statistical processing is performed on the enhancement features of each direction strip to obtain the corresponding direction weights;

[0269] The corresponding directional strip enhancement features are weighted based on the directional weight, and the weighted directional strip enhancement features are fused to obtain the directional fusion features;

[0270] Based on preset learnable parameters, the directional fusion features and basic features are fused to obtain target enhancement features;

[0271] The target enhancement features are fused with the input features to obtain the output features corresponding to the input features.

[0272] Optionally, performing strip spatial attention operations on the basic features along two directions yields corresponding directional strip enhanced features, including:

[0273] In each direction:

[0274] Perform global average pooling on the basic features to obtain the pooling result;

[0275] The pooling results are sequentially subjected to convolution, attention, and reshaping operations to obtain the strip weights.

[0276] The basic features are expanded along the current direction using a sliding window to obtain multiple local stripes in the current direction;

[0277] Based on the strip weights, the local strips in the current direction after reshaping are weighted separately to obtain the local strip enhancement features in the current direction;

[0278] Summing all local strip enhancement features and reshaping the summation result yields the strip enhancement features in the current direction.

[0279] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0280] Figure 6 This is a schematic diagram of the physical layer structure of an electronic device provided in an embodiment of this application. Figure 6As shown, the electronic device 6 of this embodiment includes: at least one processor 60 ( Figure 6 The diagram shows only one processor, memory 61, and a computer program 62 stored in memory 61 that can run on at least one processor 60. When processor 60 executes computer program 62, it implements the steps in any of the above embodiments of the road defect detection method from the perspective of an unmanned aerial vehicle (UAV), for example... Figure 1 The steps shown are 110-.

[0281] The processor 60 may be a central processing unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0282] In some embodiments, memory 61 may be an internal storage unit of electronic device 6, such as a hard disk or memory of electronic device 6. In other embodiments, memory 61 may also be an external storage device of electronic device 6, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 6.

[0283] Furthermore, memory 61 may include both internal storage units and external storage devices of electronic device 6. Memory 61 is used to store operating devices, application programs, bootloaders, data, and other programs, such as program code for computer programs. Memory 61 can also be used to temporarily store data that has been output or will be output.

[0284] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0285] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0286] This application provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0287] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographic device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, such as a USB flash drive, a portable hard drive, a magnetic disk, or an optical disk.

[0288] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0289] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0290] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0291] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0292] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for detecting road defects from the perspective of an unmanned aerial vehicle (UAV), characterized in that, include: The acquired image to be detected is input into a pre-trained target detection model to obtain the road defect detection result corresponding to the image to be detected; The image to be detected is a road image from the perspective of an unmanned aerial vehicle (UAV). The target detection model is trained from an initial detection model, which includes: a backbone network for extracting features from the input image to obtain image features; a Transformer encoding network for globally modeling the image features to obtain enhanced image features; and a Transformer decoding network for outputting the detection result corresponding to the input image based on the enhanced image features. During the training of the initial detection model, the Transformer encoding network is equipped with an ADQG module. The ADQG module is used to generate a noise query related to the content of the training samples based on the ground truth bounding boxes of the training samples and the image features corresponding to the training samples. The Transformer decoding network is also used to perform a denoising task based on the noise query to obtain the ground truth bounding boxes corresponding to the noise query, thereby assisting the initial detection model in convergence. The target detection model is obtained by converging the initial detection model and removing the ADQG module. The ADQG module is specifically used for: For each training sample: The encoded image features corresponding to the current training sample are split into multi-scale two-dimensional features; Determine the local features of the target content corresponding to the true target box of the current training sample in the two-dimensional features at each scale; The target content features are obtained by stitching together the local features of the target content at all scales; Based on the target content features and the real target bounding box, a noise query corresponding to the current training sample is generated.

2. The road defect detection method as described in claim 1, characterized in that, The step of generating a noise query corresponding to the current training sample based on the target content features and the real target bounding box includes: Perturbation information is generated based on the target content features; A noise reference box is determined based on the perturbation information and the scale information corresponding to the real target box; Based on the target content features and the noise reference box, a noise query corresponding to the current training sample is generated.

3. The road defect detection method as described in claim 1, characterized in that, The loss function of the initial detection model includes a Hungarian matching cost function and a matching consistency loss function. The Hungarian matching cost function, based on the classification cost, location cost, and overlap cost, introduces scale constraints to characterize target scale differences, morphological constraints to characterize target shape differences, and directional constraints to characterize target orientation differences. The matching consistency loss function is used to constrain the consistency between prediction results at different decoding stages.

4. The road defect detection method according to any one of claims 1 to 3, characterized in that, The backbone network is equipped with a DSFEM module, which is used to extract directional sensing features along the two directions of the input features to model the spatial continuity features of the strip-shaped disease target. The input features are obtained based on the input image and are used to generate the image features.

5. The road defect detection method as described in claim 4, characterized in that, The DSFEM module is used to extract direction-aware features along both directions of the input features to model the spatial continuity features of strip-shaped disease targets, including: The input features are transformed to obtain the basic features; Perform strip spatial attention operations on the basic features along two directions to obtain corresponding directional strip enhanced features; Statistical processing is performed on the enhancement features of each direction strip to obtain the corresponding direction weights; The corresponding directional strip enhancement features are weighted based on the directional weight, and the weighted directional strip enhancement features are fused to obtain the directional fusion features; The directional fusion features and the basic features are fused based on preset learnable parameters to obtain target enhancement features; The target enhancement feature is fused with the input feature to obtain the output feature corresponding to the input feature.

6. The road defect detection method as described in claim 5, characterized in that, The step of performing strip spatial attention operations on the basic features along two directions to obtain corresponding directional strip enhanced features includes: In each direction: Global average pooling is performed on the basic features to obtain the pooling result; The pooling results are sequentially subjected to convolution, attention, and reshaping operations to obtain strip weights. The basic feature is expanded along the current direction using a sliding window to obtain multiple local stripes in the current direction; Based on the strip weights, the local strips in the current direction after reshaping are weighted respectively to obtain the enhancement features of each local strip in the current direction; Summing all the local strip enhancement features and reshaping the summation result yields the strip enhancement features in the current direction.

7. A road defect detection device from the perspective of an unmanned aerial vehicle (UAV), characterized in that, include: The detection module is used to input the acquired image to be detected into a pre-trained target detection model to obtain the road defect detection result corresponding to the image to be detected; the image to be detected is a road image from the perspective of an unmanned aerial vehicle (UAV). The target detection model is trained from an initial detection model, which includes: a backbone network for extracting features from the input image to obtain image features; a Transformer encoding network for globally modeling the image features to obtain enhanced image features; and a Transformer decoding network for outputting the detection result corresponding to the input image based on the enhanced image features. During the training of the initial detection model, the Transformer encoding network is equipped with an ADQG module. The ADQG module is used to generate a noise query related to the content of the training samples based on the ground truth bounding boxes of the training samples and the image features corresponding to the training samples. The Transformer decoding network is also used to perform a denoising task based on the noise query to obtain the ground truth bounding boxes corresponding to the noise query, thereby assisting the initial detection model in convergence. The target detection model is obtained by converging the initial detection model and removing the ADQG module. The ADQG module is specifically used for: For each training sample: The encoded image features corresponding to the current training sample are split into multi-scale two-dimensional features; Determine the local features of the target content corresponding to the true target box of the current training sample in the two-dimensional features at each scale; The target content features are obtained by stitching together the local features of the target content at all scales; Based on the target content features and the real target bounding box, a noise query corresponding to the current training sample is generated.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the road defect detection method from the perspective of an unmanned aerial vehicle as described in any one of claims 1 to 6.

9. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the road defect detection method from the perspective of an unmanned aerial vehicle as described in any one of claims 1 to 6.

Images