Unmanned vehicle inspection small target detection method based on efficient attention mechanism

Abstract

The application discloses an unmanned vehicle inspection small target detection method based on an efficient attention mechanism, which comprises four stages of image feature extraction, vocabulary embedding extraction, efficient attention encoding and cross attention decoding. Image feature extraction: a backbone network is used for feature extraction of an input image to generate a multi-scale feature map. Vocabulary embedding extraction: a defined category vocabulary table generates vocabulary embedding through a CLIP text encoder. Efficient attention encoding: the multi-scale feature map of the picture is processed through deep feature interaction, spatial attention guidance and multi-scale feature aggregation to obtain an image feature map that fuses visual context and multi-scale information. Cross attention decoding: the aggregated feature map and the vocabulary embedding are processed through cross attention fusion, IoU perception query and region text comparison to output a final detection result. Compared with the prior art, the application has the advantages of good prediction effect and good practicability.

Description

Technical Field

[0001] This invention relates to a method for detecting small targets during unmanned vehicle inspections, specifically a method for detecting small targets during unmanned vehicle inspections based on an efficient intention mechanism, used for target detection during routine inspections of dams. Background Technology

[0002] With the widespread application of unmanned vehicles in infrastructure monitoring fields such as dam inspection, higher comprehensive performance requirements have been placed on the small target detection systems they carry, including real-time performance, detection accuracy, and open recognition capabilities. In actual dam inspection tasks, the targets to be detected not only include conventional structural elements but may also include sudden or unknown abnormal targets as well as small targets. However, existing target detection methods are mostly based on the closed-set assumption and can only identify a limited number of categories predefined during the training phase. This makes it difficult to meet the needs of diverse and dynamically evolving target categories in the complex environment of dams, thus significantly limiting their intelligence and adaptability.

[0003] Most existing target detection techniques rely on a fixed set of categories and supervised learning using manually labeled data. This presents significant limitations when addressing the challenges of continuously expanding categories and scarce samples in dam inspections. On one hand, labeling target samples in dam scenarios is costly and highly specialized, severely restricting the scalability of traditional methods. On the other hand, closed-loop models struggle to identify new categories beyond the training data when faced with sudden structural anomalies or novel target detection needs, resulting in insufficient generalization ability, risk perception, and early warning capabilities.

[0004] In recent years, many advanced deep learning techniques have demonstrated powerful expressive capabilities in object detection, especially in handling high-resolution images and multi-class recognition tasks, making Transformer architectures and multimodal models a research hotspot. Commonly used structures in object detection models include convolutional Feature Pyramid (FPN) and Transformer models such as Deformable-DETR. In the feature extraction stage, the FPN structure extracts feature maps of different scales through multi-level convolutions, enhancing the model's ability to perceive small targets; while the Transformer structure utilizes a self-attention mechanism to model global dependencies, improving the richness of feature representation. However, while the multi-scale fusion introduced by FPN enhances multi-level semantics, it also leads to a sharp increase in computational complexity, which is detrimental to real-time deployment on edge devices. Simultaneously, the Transformer structure suffers from sequence expansion problems under high-resolution input, resulting in significant memory overhead and making it difficult to adapt to unmanned vehicle platforms with limited computing resources. On the other hand, visual-language pre-trained models (such as CLIP) have been proven to effectively endow detectors with open-vocabulary capabilities and improve the recognition level of unknown categories. However, in existing architectures, the alignment of image and text information is usually performed through simple splicing or knowledge distillation, ignoring the interactive details between visual semantics. This easily introduces background noise interference, weakens the semantic saliency of foreground targets, and thus affects detection accuracy. In addition, current methods lack adaptation mechanisms for scenes with small targets and complex backgrounds when processing images captured by unmanned vehicles, making it difficult to fully explore the complementary information between deep and shallow features, resulting in limited model performance in small target recognition and semantic alignment. The former makes it difficult for detectors to balance computational efficiency and small target detection capabilities, limiting real-time performance; the latter leads to semantic ambiguity in complex backgrounds, reduced image-text fusion effects, and affects the stability of the model in recognizing unknown targets. Summary of the Invention

[0005] Purpose of the Invention: Addressing the problems and shortcomings of existing technologies, this invention provides a method for small target detection in unmanned vehicle inspections based on an efficient attention mechanism. This method aims to improve small target detection capabilities and adapt to complex scenarios involving unmanned vehicles. It achieves deep alignment of image and text features to realize small target detection in unmanned vehicles.

[0006] Technical Solution: A method for small target detection in unmanned vehicle inspection based on an efficient attention mechanism, comprising an image feature extraction stage, a vocabulary embedding extraction stage, an efficient attention encoding stage, and a cross-attention decoding stage; the specific implementation steps are as follows: Step 1) In the image feature extraction stage, the original image is input into the backbone of the deep neural network for image feature extraction. The output feature maps {S3, S4, S5} of the last three layers of the backbone network are retained as multi-scale image feature representations, which correspond to spatial features of different resolutions and semantic depths respectively. In deep neural networks, the deep feature interaction module, the spatial attention guidance module, and the multi-scale feature aggregation module constitute the efficient attention fusion encoder in the efficient attention encoding stage.

[0007] In deep neural networks, the cross-attention fusion module, the IoU-aware query module, and the region text comparison module constitute the IoU-aware decoder in the cross-attention decoding stage.

[0008] Step 2) In the vocabulary embedding extraction stage, an open category vocabulary is constructed, and the category vocabulary is input into the pre-trained CLIP text encoder to convert the text into vocabulary embeddings with uniform dimensions and consistent distribution. Step 3) In the efficient attention encoding stage, the deep feature map S5 obtained in step 1) and the location are embedded into the input deep feature interaction module. Global context modeling is achieved through the self-attention mechanism, and the optimized feature map F5 is output. Step 4) In the efficient attention encoding stage, the spatial attention guidance module uses the category-independent mask generated by the shallow feature map S3 in step 1) and reconstructs it into a spatial attention weight matrix after processing by the average pooling layer and the 1×1 convolutional layer; S4 is concatenated with the feature map F5 upsampled by bilinear interpolation and then multiplied element-wise with the weight matrix to achieve feature enhancement, and finally outputs the intermediate feature map. Step 5) In the efficient attention encoding stage, the intermediate feature map obtained in step 4) is input into the multi-scale feature aggregation module. This module models spatial dimension correlation and channel dimension dependency through parallel spatial attention branch and channel attention branch respectively, and finally generates an output feature map with strong representation ability through weighted fusion strategy. Step 6) In the cross-attention decoding stage, the cross-attention fusion module fuses the output feature map of step 5) with the word embedding obtained in step 2) through a multi-head cross-attention mechanism to generate a flattened one-dimensional feature vector; Step 7) In the cross-attention decoding stage, the IoU-aware query module enables the deep neural network model to remove noise and adds IoU constraint terms through comparative denoising training, so that the denoised and optimized feature representation has both high classification confidence and accurate localization information. Step 8) In the cross-attention decoding stage, the region text comparison module calculates the similarity between the feature representation obtained in step 7) and the word embedding obtained in step 2), and integrates the bounding box regression to obtain the final target detection result.

[0009] Furthermore, in step 1), feature extraction is performed on the original image to generate a multi-scale spatial feature map. The specific steps are as follows: The original image is input into the backbone of a deep neural network, which adopts the CLIP image encoder structure and is structurally adapted. After feature extraction, the output feature maps of the last three layers of the backbone network are retained as multi-scale image feature representations, corresponding to spatial features of different resolutions and semantic depths. The resulting feature map set {S3, S4, S5} is input into an efficient attention fusion encoder to perform intra-scale interaction, spatial guided fusion, and multi-scale aggregation operations.

[0010] Furthermore, in step 2), the open category vocabulary is encoded into a lexical embedding, and the specific steps are as follows: Based on the target categories in the VisDroneZSD dataset and common target categories in dam inspection, an open category vocabulary is constructed, including training categories and potential unseen categories. The vocabulary can be in the form of words or phrases and has semantic discriminative power. This category vocabulary is input into a pre-trained CLIP text encoder to convert the text into word vector representations with uniform dimensions and consistent distribution. The word vector representations of all categories are combined to form a word embedding set, which serves as the language input for the overall model of the small target detection method and participates in subsequent image-text fusion and region matching processes.

[0011] Furthermore, in step 3), the deep feature interaction module: the input of this module is a multi-scale deep image feature map extracted by the backbone network; this module first flattens the feature map and performs position embedding by combining two-dimensional sine and cosine position encoding; then it models the contextual dependencies of the deep feature map through a single-layer Transformer encoder composed of a multi-head self-attention mechanism, a multi-layer perceptron and layer normalization; finally, it outputs an enhanced deep semantic feature map.

[0012] Furthermore, in step 4), the spatial attention guidance module: the input of this module is a shallow feature map and an enhanced deep semantic feature map; firstly, a single-channel mask is generated on the shallow feature map using a class-independent mask generator, and the mask is processed by average pooling and 1×1 convolution to obtain the spatial attention weight matrix; this module concatenates the upsampled deep feature map with the shallow feature map, and multiplies it element-wise with the spatial weights to achieve feature alignment and enhancement; the output is an intermediate fused feature map.

[0013] Furthermore, in step 5), the multi-scale feature aggregation module: the input of this module is the intermediate fused feature map; this module includes a spatial attention branch and a channel attention branch. The spatial branch uses multi-scale convolutional kernels (3×3, 5×5, 7×7) to process the feature map and fuses the resulting multiple convolutional feature maps to generate a spatial attention map. The channel branch uses global average pooling and non-linear activation to generate a channel attention map. The two attention maps enhance the target feature representation capability through a weighted fusion strategy; the output is a feature map enhanced with spatial and channel information.

[0014] Furthermore, the efficient attention encoding stage is specifically implemented by an efficient attention fusion encoder, which consists of a deep feature interaction module, a spatial attention guidance module, and a multi-scale feature aggregation module. The input of the efficient attention fusion encoder is the multi-scale image features extracted by the backbone network in step 1). First, context modeling is performed on the deep feature map, then spatial localization information from the shallow feature map is fused, and finally, the feature representation is enhanced through the multi-scale feature aggregation module; the output is an image feature map that fuses visual context and multi-scale information.

[0015] Furthermore, in step 6), the cross-attention fusion module: the input of this module is the image feature map output by the efficient attention fusion encoder and the word embedding output by the CLIP text encoder; the image features are mapped to query vectors, and the word embeddings are mapped to key and value vectors respectively; the semantic matching relationship between the image feature map and the word embedding is calculated through a multi-head cross-attention mechanism, and a foreground suppression mechanism is introduced to generate foreground probabilities based on similarity for noise suppression; the output is the image features fused with semantic features.

[0016] Furthermore, in step 7), the IoU-aware query module: the input of this module is the image features of the fused semantic features output by the cross-attention fusion module in step 6); a contrastive denoising training strategy is adopted to apply positive noise and negative noise respectively to generate positive and negative query samples; the module jointly evaluates the IoU score and classification confidence between the predicted bounding box and the ground truth box, and uses an IoU-aware query selection strategy to select the optimal Top-K queries; the output is a set of target queries that simultaneously have high IoU and high confidence.

[0017] Furthermore, in step 8), the region text comparison module takes the target query output by the IoU-aware query module in step 7) and the word embedding obtained in step 2) as input. The module performs linear projection of the image query vector through MLP and maps it to a unified semantic space with the word embedding. It constructs a region-text comparison loss and optimizes it by combining it with Varifocal Loss. It outputs the similarity score between each image region and various types of words, which is used for the determination and localization of image semantic labels.

[0018] Furthermore, the cross-attention decoding stage is specifically implemented by an IoU-aware decoder, which consists of a cross-attention fusion module, an IoU-aware query module, and a region-text comparison module. The input to the IoU-aware decoder is the image feature map output by the efficient attention fusion encoder and the CLIP word embedding; first, image-text fusion is performed to generate semantically enhanced features, then high-quality target features are jointly selected based on IoU and classification confidence, and finally, category recognition and bounding box regression are completed through region-text comparison loss; the output is the final detection result containing the detected category and bounding box position.

[0019] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described method for detecting small targets in unmanned vehicles based on an efficient attention mechanism.

[0020] A computer-readable storage medium storing a computer program that performs the above-described method for detecting small targets in unmanned vehicle inspections based on an efficient attention mechanism.

[0021] Beneficial Effects: Compared with existing technologies, the small target detection method for unmanned vehicle inspection based on an efficient attention mechanism proposed in this invention has the following advantages: By introducing intra-scale attention interaction and multi-scale attention aggregation modules, the feature representation of small targets in complex backgrounds is effectively enhanced. A lightweight feature fusion strategy is adopted to replace the traditional FPN structure, significantly reducing computational complexity and improving the real-time performance of the model on unmanned vehicles. The introduction of a CLIP text encoder and a word-guided cross-attention mechanism achieves deep fusion of image features and semantic word embeddings. Combined with an IoU-aware query strategy and regional text contrast loss, the model's recognition accuracy and generalization ability for unknown target categories are significantly improved. Attached Figure Description

[0022] Figure 1 This is a flowchart of a method according to an embodiment of the present invention; Figure 2 This is a method architecture diagram of an embodiment of the present invention. Detailed Implementation

[0023] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0024] Deep learning has attracted researchers' attention due to its outstanding performance in processing data. This invention combines deep learning's ability to model complex data distributions to establish a small target detection model for unmanned vehicles. A specific implementation method is provided below: A method for small target detection in unmanned vehicle inspection based on an efficient attention mechanism, the process of which is as follows: Figure 1 As shown, it includes: Step 1) In the image feature extraction stage, the original images (including but not limited to: images of the dam itself, images of the inspection area, images of small targets, etc.) acquired by unmanned vehicles during the dam inspection process are input into the backbone of the deep neural network. The backbone network adopts the CLIP image encoder structure and is structurally adapted. After feature extraction, the output feature maps of the last three layers of the backbone network are retained as multi-scale image feature representations, corresponding to spatial features of different resolutions and semantic depths. The resulting feature map set {S3, S4, S5} will be input into the efficient attention fusion encoder to perform intra-scale interaction, spatial guided fusion, and multi-scale aggregation operations.

[0025] Step 2) In the vocabulary embedding extraction stage, an open category vocabulary is constructed based on the target categories of the VisDroneZSD dataset and common target categories during dam inspection (e.g., equipment, cracks, seepage, etc.). This includes training categories, potential unseen categories, etc. The vocabulary can be in the form of words or phrases and has semantic discriminative power. The category vocabulary is input into the pre-trained CLIP text encoder to convert the text into word vector representations with uniform dimensions and consistent distribution. The word vector representations of all categories are used to form a word embedding set as the language input of the model, which participates in the subsequent image-text fusion and region matching processes.

[0026] Step 3) In the efficient attention encoding stage, the deep feature S5 obtained in Step 1) and its location are embedded into the input deep feature interaction module. Global context modeling is achieved through a self-attention mechanism, outputting the optimized feature map F5. Specific implementation details are as follows: The deep feature interaction module is implemented by a single-layer Transformer encoder.

[0027] The input is a feature map of an image, with a tensor dimension of . Here, B represents the batch size, C represents the number of channels, and H and W are the height and width of the feature map, respectively. To adapt to the Transformer encoder structure, the input features are first flattened into a sequence format X. flat :

[0028] After flattening, position encoding is performed using two-dimensional sine and cosine functions. Setting the feature channel dimension to d, the encoding for each position pos is as follows:

[0029] Among them PE （pos,2i） PE （pos,2i+1） The positional encoding is used to encode different locations, where i is the channel index. This positional encoding matrix is ​​added to the original input to form a position-sensitive feature representation X. embed =X flat +PE.

[0030] Subsequently, the position-embedded sequence X embed The input is a single-layer Transformer encoder. This encoder consists of a multi-head self-attention mechanism, a multilayer perceptron (MLP), and layer normalization.

[0031] The multi-head self-attention mechanism maps the embedded features into three matrices: query Q, key K, and value V.

[0032] Among them, W q W k W v For learnable weight parameters, d k For the attention dimension, a self-attention operation (Self-Att(Q,K,V)) is then performed.

[0033] K T Representation matrix K transpose, d k express K This operation can capture the correlation between any two spatial locations in the feature sequence and ensure that the attention distribution is in probabilistic form through the softmax function.

[0034] Residual connections and normalization operations, attention output Self-Att (Q,K,V) and original input X embed Add them together and process them using LayerNorm. Record the result of this operation as Z. The calculation formula is: Z = LayerNorm(X) embed +Self-Att(Q,K,V)) This step stabilizes the training process and preserves the original feature information.

[0035] Finally, the encoder further extracts nonlinear features MLP(Z) through a multilayer perceptron (MLP) module. This module contains two linear layers and a ReLU activation function, with the following structure: MLP(Z) = ReLU (ZW1 + b1) W2 + b2 Where W1 and b1 are the weight matrices and bias terms of the first linear layer, and W2 and b2 are the weight matrices and bias terms of the second linear layer, the extracted nonlinear features are concatenated with the input residuals and then normalized to obtain the final output Y: Y=LayerNorm(Z+MLP(Z)) This output feature is then restored to the original spatial structure for subsequent feature fusion operations.

[0036] Step 4) In the efficient attention encoding stage, the spatial attention guidance module uses the class-independent mask generated by the shallow feature S3 in Step 1), which is then processed by an average pooling layer and a 1×1 convolutional layer to reconstruct a spatial attention weight matrix; S4 is concatenated with the feature map F5 upsampled by bilinear interpolation and then multiplied element-wise with the weight matrix to achieve feature enhancement, and finally outputs an intermediate feature map. The specific implementation details are as follows: The spatial attention guidance module consists of depthwise separable convolutions, average pooling layers, and 1×1 convolutional layers.

[0037] The input is a shallow feature map. and deep semantic feature maps The output is the fused intermediate feature map. .

[0038] Depthwise separable convolutions are used to extract structural information from shallow features. To match the 8×8 pixel receptive field of the input image for each location in the shallow layer, the kernel size is set to 7×7 to avoid introducing redundant regions with excessively large kernels.

[0039] The average pooling layer downsamples the convolutional output by 2×2, aligning the mask features spatially with the upsampled deep feature map.

[0040] A 1×1 convolutional layer is used for channel compression and remapping, adjusting the single-channel mask to an attention weight map consistent with the fusion structure.

[0041] Output Spatial Attention Matrix This is used to perform element-wise weighted operations on the fused features.

[0042] Upsampling is performed on the deep feature map F5:

[0043] in, This is the upsampled deep feature map; `Upsample()` is the upsampling operation. Then... By concatenating S4 with S4 along the channel dimension, the concatenated feature is obtained. .

[0044] Subsequently, the features are concatenated with the attention matrix A. s Perform element-wise multiplication:

[0045] Among them, F mid This represents the intermediate feature map output by the attention guidance module, and ⊙ represents element-wise multiplication.

[0046] Step 5) In the efficient attention encoding stage, the intermediate feature map obtained in Step 4) is input into the multi-scale feature aggregation module. This module models spatial dimension correlation and channel dimension dependency through parallel spatial attention branches and channel attention branches, respectively. Finally, a weighted fusion strategy is used to generate an output feature map with strong representational capabilities. The specific implementation details are as follows: The multi-scale feature aggregation module receives the intermediate feature maps output by the spatial attention guidance module. As input, the output is the fused and enhanced feature map. .

[0047] The multi-scale feature aggregation module has three parallel branches in its internal structure: The spatial attention branch focuses on uncovering local dependencies and region importance between different spatial locations in an image. First, the input feature map F... mid Perform channel dimensionality reduction using 1×1 convolution Conv 1×1 Mapped to low-dimensional features F low : F low =Conv 1×1 (F mid ) Then, 3×3 convolution kernels Conv were used respectively. 3×3 5×5 convolution kernel Conv 5×5 and 7×7 convolution kernel Conv 7×7 Perform multi-scale convolution operations to obtain scale feature maps F at the corresponding scales. (3) 、 F (5) 、 F (7) : F (3) =Conv 3×3 (F low ), F (5) =Conv 5×5 (F low ), F (7) =Conv 7×7 (F low ) The above multi-scale features are weighted and fused to form a spatially enhanced feature map F. spa : Fspa =F (3) + F (5) + F (7) Spatially salient region information is extracted using max pooling and then processed by 7×7 convolution. 7×7 Generate spatial attention weight map A s : A s =σ(Conv 7×7 (MaxPool (F spa ))) MaxPool is the maximum pooling operation, and σ(⋅) represents the Sigmoid activation function, which is used to normalize the output to the [0,1] interval.

[0048] Finally, the spatial features are weighted by location to obtain the results. :

[0049] Channel attention branch modeling models the weight relationships between different channels, thereby improving the response strength of key semantic channels.

[0050] First, spatial information is compressed using Global Average Pooling (GAP), retaining only channel information F. gap : F gap =GAP(F mid )∈R B×C×1×1 Next, a channel attention mapping function A is constructed using two 1×1 convolutions and the non-linear activation function ReLU. c : A c =σ(Conv 1×1 (ReLU (Conv 1×1 (F gap )))) Here, σ(⋅) represents the Sigmoid activation function, used to normalize the output to the [0,1] interval. Then, a channel-wise weighted sum is performed on the original feature map to obtain... :

[0051] Finally, the results from the two augmentation branches are weighted and fused, and residual connections can be optionally added to maintain stable training.

[0052] Output F out This is the final feature map after multi-scale spatial-channel enhancement, which is used as input to the subsequent cross-attention fusion module.

[0053] Step 6) In the cross-attention decoding stage, the cross-attention fusion module fuses the image features obtained in step 5) with the word embeddings obtained in step 2) through a multi-head cross-attention mechanism to generate a flattened one-dimensional feature vector. The specific implementation details are as follows: The Cross-Attention Fusion Module (VCAF) takes two parts as input: an image feature tensor output from the efficient attention encoding stage. Lexical embedding matrix generated by CLIP text encoder N represents the number of category words, and d is the word embedding dimension.

[0054] Image features are converted into a two-dimensional sequence form through the Flatten operation before input. .

[0055] To perform image-text cross-attention calculation, image features are projected as query vectors (Q), and word embeddings are projected as key (K) and value (V) vectors, respectively: Q=F' W Q K=TW K V=TW V Among them, W Q W K W V ∈ These are trainable linear mapping weights.

[0056] The cross-attention mechanism calculates the attention weight Attn(Q,K) by measuring the similarity between the query and the key:

[0057] K T d represents the transpose of matrix K. k Denotes the dimension of K. This operation can capture the correlation between any two spatial locations in the feature sequence, and the softmax function ensures that the attention distribution is in probabilistic form. Subsequently, this weight is applied to the value vector V to obtain the feature F after image-text fusion. cross : F cross =Attn(Q,K)V Injecting all lexical semantics into all locations may introduce noise. To address this, a foreground suppression mechanism is introduced. First, the maximum similarity s between each location i in the image and all class j embeddings is calculated. i :

[0058] Where Q is the query vector and K is the key vector. Let s iTransformed into foreground probability p using the Sigmoid function. i ∈[0,1].

[0059] Finally, the features obtained after image-text fusion are residually connected to the features of the original image and then weighted and modulated to obtain the final result. The calculation formula is as follows:

[0060] To enhance the model's ability to model different semantic layers, the VCAF module employs a multi-head cross-attention mechanism (default 8 heads), which performs parallel head-by-head calculations on queries, keys, and values ​​and then concatenates them at the end.

[0061] Step 7) In the cross-attention decoding stage, the IoU-aware query is trained through contrastive denoising to remove noise from the model and incorporates an IoU constraint term, thereby ensuring that the obtained features simultaneously possess high classification confidence and accurate localization information. Specific implementation details are as follows: The IoU-aware query module combines a contrastive denoising training mechanism with an IoU-guided Top-K query selection rule. The input is the fused query vector sequence from the vocabulary-guided cross-attention fusion module, and the output is a sequence of queries from N... q The Top-K high-quality query subset selected from the queries.

[0062] This module first employs a contrastive denoising strategy, constructing positive and negative query samples during the training phase to guide the model in learning high-quality feature representations.

[0063] For each positive sample true bounding box b gt Add small perturbations (such as coordinate offsets) Generate positive noise query box b + :

[0064] Simultaneously generate negative noise query box b - The disturbance is larger (such as coordinate offset). ):

[0065] This enables the model to learn to treat positive noise as the target and negative noise as the background.

[0066] During the inference phase, joint IoU and classification confidence are used as filtering criteria to select high-quality targets from the query set output by the encoder. For each query q... i The model also predicts its bounding box. Text category score that matches it ; Calculate the IoU value between the predicted bounding box and the ground truth bounding box. i Construct joint selection weights Lcls+IoU (As a training objective):

[0067] Where: p i q is the semantic similarity score between the region and the text. i represents the positive and negative sample labels. When it is a positive sample, it is equal to the IoU between the predicted box and the ground truth box. Background samples are set to 0. i is the position index, and γ is the modulation index, which controls the influence of hard samples on the weight.

[0068] Finally, the top-K queries with both high IoU and high classification confidence were selected as candidate targets. Enter the subsequent modules.

[0069] Step 8) In the cross-attention decoding stage, the region text comparison module calculates the similarity between image features and word embeddings, and integrates bounding box regression to obtain the final target detection result, including the target's category information (such as cracks, seepage, collapse, etc.) and the target's spatial location (indicating the target's position in the image). Specific implementation details are as follows: The region text comparison module accepts Top-K image region features output from the query selection module and inputs them together with the word embeddings generated by the CLIP encoder. It learns the matching relationship between images and text through semantic comparison training.

[0070] First, a nonlinear transformation is performed on the image region features, transforming the positive sample q... i Mapping to the text semantic space, an MLP (Multi-Layer Perceptron) structure is used to obtain image features z. i text:

[0071] Then the image features z i With lexical embeddings j The similarity is calculated by performing an inner product, and a learnable scaling parameter α and a bias term β are introduced to obtain the region-text similarity score:

[0072] Where <, > denote the vector dot product, s i,j It is a region-text similarity score, representing the similarity between the i-th region and the j-th category.

[0073] To complete the positive and negative sample assignment, this invention employs the Hungarian matching algorithm, using region-text similarity s. i,j The cost matrix is ​​used to match the predicted region with the true category.

[0074] The model is trained using a loss function and then fine-tuned to complete the training. The final output is the detection category and bounding box location for each target region.

[0075] Loss function L total Specifically:

[0076] Where λ is the regularization coefficient. L is the L2 norm of the model weight parameters. contrast For region-text contrast loss, L varifocal To classify and locate joint losses.

[0077] The region-text contrast loss L contrast This is used to measure the semantic similarity between each visual region in an image and a category word, and is defined as:

[0078] Where N represents the total number of region queries, C represents the number of category terms, and y i s represents the true matching category index of the i-th visual region. ij τ represents the similarity between region i and category j, and τ is the temperature coefficient.

[0079] The IoU perception classification localization joint loss L varifocal This is used to enhance the classification accuracy of high-quality predicted samples and suppress low-IoU negative samples, and is defined as follows:

[0080] Where p is the region text similarity score, q represents the IoU value between the predicted bounding box and the ground truth box (for the foreground) or is 0 (for the background), α is the scaling factor that adjusts the negative sample loss, and γ is the modulation index; Obviously, those skilled in the art should understand that the steps of the unmanned vehicle inspection small target detection method based on an efficient attention mechanism or the modules of the unmanned vehicle inspection small target detection system based on an efficient attention mechanism described in the above embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.

Claims

1. A method for small target detection in unmanned vehicle inspection based on an efficient attention mechanism, characterized in that, It includes the image feature extraction stage, the word embedding extraction stage, the efficient attention encoding stage, and the cross-attention decoding stage; the specific implementation steps are as follows: Step 1) In the image feature extraction stage, the original image is input into the backbone of the deep neural network for image feature extraction. The output feature maps {S3, S4, S5} of the last three layers of the backbone network are retained as multi-scale image feature representations, which correspond to spatial features of different resolutions and semantic depths respectively. Step 2) In the vocabulary embedding extraction stage, an open category vocabulary is constructed, and the category vocabulary is input into the pre-trained CLIP text encoder to convert the text into vocabulary embeddings with uniform dimensions and consistent distribution. Step 3) In the efficient attention encoding stage, the deep feature map S5 obtained in step 1) and the location are embedded into the input deep feature interaction module. Global context modeling is achieved through the self-attention mechanism, and the enhanced deep semantic feature map F5 is output. Step 4) In the efficient attention encoding stage, the spatial attention guidance module uses the category-independent mask generated by the shallow feature map S3 in step 1) and reconstructs it into a spatial attention weight matrix after processing by an average pooling layer and a 1×1 convolutional layer; S4 is concatenated with the enhanced deep semantic feature map F5 after bilinear interpolation upsampling and then multiplied element-wise with the weight matrix to achieve feature enhancement, and finally outputs the intermediate fused feature map; Step 5) In the efficient attention encoding stage, the intermediate fused feature map obtained in step 4) is input into the multi-scale feature aggregation module. This module models the spatial dimension correlation and channel dimension dependency through parallel spatial attention branch and channel attention branch respectively, and finally generates an output feature map with strong representation ability through a weighted fusion strategy. Step 6) In the cross-attention decoding stage, the cross-attention fusion module fuses the output feature map of step 5) with the word embedding obtained in step 2) through a multi-head cross-attention mechanism to generate a flattened one-dimensional feature vector, which is the image feature with fused semantic features; Step 7) In the cross-attention decoding stage, the IoU-aware query module enables the deep neural network model to remove noise and adds IoU constraint terms through comparative denoising training. This allows the denoised and optimized feature representation to have both high classification confidence and accurate localization information, thus obtaining the target query set. Step 8) In the cross-attention decoding stage, the region text comparison module calculates the similarity between the feature representation obtained in step 7) and the word embedding obtained in step 2), and integrates the bounding box regression to obtain the final target detection result.

2. The method for detecting small targets in unmanned vehicle inspection based on an efficient attention mechanism according to claim 1, characterized in that, In step 1), feature extraction is performed on the original image to generate a multi-scale spatial feature map. The specific steps are as follows: The original image is input into the backbone of a deep neural network, which adopts the CLIP image encoder structure and is structurally adapted. After feature extraction, the output feature maps of the last three layers of the backbone network are retained as multi-scale image feature representations, corresponding to spatial features of different resolutions and semantic depths. The resulting feature map set {S3, S4, S5} will be input into an efficient attention fusion encoder to perform intra-scale interaction, spatial guided fusion, and multi-scale aggregation operations.

3. The method for detecting small targets in unmanned vehicle inspection based on an efficient attention mechanism according to claim 1, characterized in that, In step 2), the open category vocabulary is encoded into a vocabulary embedding. The specific steps are as follows: Based on the target categories in the VisDroneZSD dataset and the target categories in the dam inspection process, an open category vocabulary is constructed, including training categories and potential unseen categories. The vocabulary is in the form of words or phrases and has semantic discriminative power. This category vocabulary is input into the pre-trained CLIP text encoder to convert the text into word vector representations with uniform dimensions and consistent distribution. The word vector representations of all categories are combined to form a word embedding set, which serves as the language input for the overall model of the small target detection method and participates in the subsequent image-text fusion and region matching processes.

4. The method for detecting small targets in unmanned vehicle inspection based on an efficient attention mechanism according to claim 1, characterized in that, In step 3), the deep feature interaction module takes a multi-scale deep image feature map extracted by the backbone network as input. The module first flattens the feature map and embeds it into a position using two-dimensional sine and cosine position encoding. Then, it models the contextual dependencies of the deep feature map using a single-layer Transformer encoder consisting of a multi-head self-attention mechanism, a multilayer perceptron, and layer normalization. Finally, it outputs an enhanced deep semantic feature map.

5. The method for detecting small targets in unmanned vehicle inspection based on an efficient attention mechanism according to claim 1, characterized in that, In step 4), the spatial attention guidance module takes a shallow feature map and an enhanced deep semantic feature map as input. First, a single-channel mask is generated on the shallow feature map using a class-independent mask generator. The mask is then processed by average pooling and 1×1 convolution to obtain the spatial attention weight matrix. This module concatenates the upsampled deep semantic feature map with the shallow feature map and multiplies it element-wise with the spatial weights to achieve feature alignment and enhancement. The output is an intermediate fused feature map.

6. The method for detecting small targets in unmanned vehicle inspection based on an efficient attention mechanism according to claim 1, characterized in that, In step 5), the multi-scale feature aggregation module takes an intermediate fused feature map as its input. This module includes a spatial attention branch and a channel attention branch. The spatial attention branch processes the intermediate fused feature map using multi-scale convolutional kernels and fuses the resulting multiple convolutional feature maps to generate a spatial attention map. The channel attention branch uses global average pooling and non-linear activation to generate a channel attention map. The two attention maps enhance the target feature representation capability through a weighted fusion strategy. The output is a feature map enhanced with spatial and channel information.

7. The method for detecting small targets in unmanned vehicle inspection based on an efficient attention mechanism according to claim 1, characterized in that, The efficient attention encoding stage is specifically implemented by the efficient attention fusion encoder, which consists of a deep feature interaction module, a spatial attention guidance module, and a multi-scale feature aggregation module. The input of the efficient attention fusion encoder is the multi-scale image features extracted by the backbone network in step 1). First, the deep feature map is modeled in context, then the spatial positioning information in the shallow feature map is fused, and finally the feature expression is enhanced by the multi-scale feature aggregation module. The output is an image feature map that integrates visual context and multi-scale information; The cross-attention decoding stage is specifically implemented by the IoU-aware decoder, which consists of a cross-attention fusion module, an IoU-aware query module, and a region text comparison module. The input of the IoU-aware decoder is the image feature map output by the efficient attention fusion encoder and the CLIP word embedding. First, image-text fusion is performed to generate semantically enhanced features. Then, high-quality target features are jointly selected based on IoU and classification confidence. Finally, category recognition and bounding box regression are completed through region text comparison loss. The output is the final detection result containing the detection category and bounding box position.

8. The method for detecting small targets in unmanned vehicle inspection based on an efficient attention mechanism according to claim 2, characterized in that, In step 6), the cross-attention fusion module is input to the image feature map output by the efficient attention fusion encoder and the word embedding output by the CLIP text encoder. Image features are mapped to query vectors, and word embeddings are mapped to key and value vectors, respectively. The semantic matching relationship between image feature maps and word embeddings is calculated through a multi-head cross-attention mechanism, and foreground probabilities are generated based on similarity for noise suppression. The output is an image feature fused with semantic features.

9. The method for detecting small targets in unmanned vehicle inspection based on an efficient attention mechanism according to claim 1, characterized in that, In step 7), the IoU-aware query module takes as its input the image features of the fused semantic features output by the cross-attention fusion module in step 6. A contrastive denoising training strategy is adopted, applying positive and negative noise respectively to generate positive and negative query samples; the module evaluates the IoU score between the predicted bounding box and the ground truth box and the classification confidence jointly, and uses an IoU-aware query selection strategy to select the optimal Top-K queries; the output is a set of target queries that have both high IoU and high confidence.

10. The method for detecting small targets in unmanned vehicle inspection based on an efficient attention mechanism according to claim 1, characterized in that, In step 8), the region text comparison module takes the target query output by the IoU-aware query module in step 7) and the word embedding obtained in step 2) as input. The module performs linear projection on the target query using MLP and maps it to a unified semantic space with the word embedding. It constructs a region-text comparison loss and optimizes it by combining it with Varifocal Loss. It outputs the similarity score between each image region and various words, which is used for the determination and localization of image semantic labels.

Images