Unmanned vehicle inspection small target detection method based on efficient attention mechanism

Abstract

The invention discloses an unmanned vehicle inspection small target detection method based on an efficient attention mechanism. The method comprises four stages of image feature extraction, vocabulary embedding extraction, efficient attention coding and cross attention decoding. And image feature extraction: performing feature extraction on the input image by using the backbone network to generate a multi-scale feature map. And vocabulary embedding extraction: generating vocabulary embedding in the defined category vocabulary through a CLIP text encoder. And efficient attention coding: performing deep feature interaction, space attention guidance and multi-scale feature aggregation processing on the multi-scale feature map of the picture to obtain an image feature map fusing the visual context and the multi-scale information. And cross attention decoding: embedding the aggregated feature map and vocabulary, and outputting a final detection result through cross attention fusion, IoU perception query and regional text comparison processing. Compared with the prior art, the method has the advantages of good prediction effect, good practicability and the like.

Description

TECHNICAL FIELD

[0001] The application relates to an unmanned vehicle inspection small target detection method, in particular to an unmanned vehicle inspection small target detection method based on an efficient intention mechanism, which is used for target detection during dam daily inspection. BACKGROUND

[0002] With the wide application of unmanned vehicles in dam inspection and other infrastructure monitoring fields, higher real-time performance, detection accuracy and open recognition ability and other comprehensive performance requirements are put forward for the small target detection system carried by the unmanned vehicles. In actual dam inspection tasks, the detection targets not only cover conventional structural elements, but also may include sudden or unknown abnormal targets and small targets. However, the existing target detection methods are mostly based on closed set assumption and can only identify a limited number of categories predefined in the training stage, which is difficult to meet the needs of target category diversification and dynamic evolution in the complex dam environment, thereby being significantly restricted in intelligence and adaptability.

[0003] The target detection technology in the existing research is mostly established on the premise of fixed category set and relies on artificial labeled data for supervised learning, which has great limitations in dealing with the problems of continuous expansion of categories and scarcity of samples in dam inspection. On the one hand, the labeling cost of target samples in the dam scene is high and professional, which seriously restricts the scalability of traditional methods; on the other hand, in the face of sudden structural abnormalities or new target detection needs, the closed model is difficult to identify new category information outside the training data, resulting in insufficient performance of the system in generalization ability, risk perception and early warning ability.

[0004] In recent years, many advanced deep learning techniques have shown strong expressive power in the field of object detection, especially in handling high-resolution images and multi-class recognition tasks. The Transformer structure and multi-modal model have become a research hotspot. In the object detection model, the commonly used structures include the convolution-based feature pyramid network (FPN) and the Deformable-DETR Transformer model. In the feature extraction stage, the FPN structure extracts features of different scales through multi-level convolution, enhancing the model's perception of small targets. The Transformer structure uses self-attention mechanisms to model global dependencies, improving the richness of feature expression. However, the multi-scale fusion introduced by FPN, although enhancing multi-level semantics, also brings a sharp rise in computational complexity, which is not conducive to real-time deployment on edge devices. At the same time, the Transformer structure has a sequence expansion problem under high-resolution input, resulting in significant memory overhead, making it difficult to adapt to the limited computing resources of unmanned vehicle platforms. On the other hand, visual-linguistic pre-training models such as CLIP have been proven to effectively endow detectors with open vocabulary capabilities, improving the recognition level of unknown classes. However, in existing architectures, the alignment of image-text information is usually done through simple concatenation or knowledge distillation, ignoring the interaction details between visual semantics, which can easily introduce background noise interference, weaken the semantic saliency of foreground objects, and thus affect detection accuracy. In addition, current methods lack mechanisms to adapt to small targets and complex backgrounds when processing images taken by unmanned vehicles, making it difficult to fully exploit the complementary information between deep and shallow features, resulting in limited performance in small target recognition and semantic alignment. The former: makes it difficult to balance between computational efficiency and small target detection capability for the detector, limiting real-time performance; the latter: in complex backgrounds, semantic ambiguity reduces the effectiveness of image-text fusion, affecting the stability of the model in recognizing unknown targets. SUMMARY

[0005] The present application provides a small target detection method for unmanned vehicle inspection based on an efficient attention mechanism, aiming to improve small target detection capability and adapt to complex scenarios of unmanned vehicles. The method achieves deep alignment of image-text features and realizes small target detection for unmanned vehicles.

[0006] Technical solution: A small target detection method for unmanned vehicle inspection based on an efficient attention mechanism, including an image feature extraction stage, a vocabulary embedding extraction stage, an efficient attention encoding stage, and a cross-attention decoding stage. The specific implementation steps are as follows: Step 1) In the image feature extraction stage, the original image is input into a deep neural network backbone for image feature extraction, and the output feature maps {S3, S4, S5} of the last three layers of the backbone network are retained as multi-scale image feature representations, corresponding to spatial features of different resolutions and semantic depths, respectively; In the deep neural network, the deep feature interaction module, the spatial attention guide module, and the multi-scale feature aggregation module constitute an efficient attention fusion encoder in the efficient attention encoding stage.

[0007] In the deep neural network, the cross-attention fusion module, the IoU-aware query module, and the region-text contrast module constitute an IoU-aware decoder in the cross-attention decoding stage.

[0008] Step 2) In the vocabulary embedding extraction stage, an open-class vocabulary table is constructed, and the class vocabulary table is input into a pre-trained CLIP text encoder to convert the text into a unified dimension and consistent distribution vocabulary embedding. Step 3) In the efficient attention encoding stage, the deep feature map S5 obtained in step 1) and the position embedding are input into the deep feature interaction module, and global context modeling is realized through the self-attention mechanism, and an optimized feature map F5 is output. Step 4) In the efficient attention encoding stage, the spatial attention guide module uses the class-independent mask generated by the shallow feature map S3 in step 1), which is processed by an average pooling layer and a 1x1 convolution layer to reconstruct a spatial attention weight matrix; S4 is concatenated with the feature map F5 after bilinear interpolation upsampling, and the weight matrix is multiplied element by element to realize feature enhancement, and finally an intermediate feature map is output. Step 5) In the efficient attention encoding stage, the intermediate feature map obtained in step 4) is input into the multi-scale feature aggregation module, which models the spatial dimension correlation and channel dimension dependency through parallel spatial attention branches and channel attention branches, respectively, and finally generates an output feature map with strong representation ability through a weighted fusion strategy. Step 6) In the cross-attention decoding stage, the cross-attention fusion module fuses the output feature map in step 5) and the vocabulary embedding obtained in step 2) through a multi-head cross-attention mechanism to generate a flattened one-dimensional feature vector. Step 7) In the cross-attention decoding stage, the IoU-aware query module removes noise through contrastive denoising training, and adds an IoU constraint term, so that the feature representation after denoising and optimization has high classification confidence and accurate positioning information. Step 8) In the cross-attention decoding stage, the region-text contrast module calculates the similarity between the feature representation obtained in step 7) and the vocabulary embedding obtained in step 2), and integrates the bounding box regression to obtain the final target detection result.

[0009] Further, in step 1), the original image is feature extracted to generate a multi-scale spatial feature map, and the specific steps are as follows: The original image is input into a deep neural network backbone, which adopts a CLIP image encoder structure and is structurally adapted; after feature extraction, the last three layers of the backbone network are reserved as multi-scale image feature representations, corresponding to spatial features of different resolutions and semantic depths; the obtained feature set {S3, S4, S5} is input into an efficient attention fusion encoder to perform intra-scale interaction, spatial guidance fusion and multi-scale aggregation operations.

[0010] Further, the step 2) encodes the open category vocabulary into a vocabulary embedding, and the specific steps are as follows: Based on the target categories of the VisDroneZSD dataset and the target categories commonly seen in dam inspection, an open category vocabulary is constructed, including training categories, potential unseen categories, etc. The vocabulary form can be a word or a phrase, and has semantic distinguishability. The category vocabulary is input into a pre-trained CLIP text encoder to convert the text into a unified dimension and consistent word vector representation. The word vector representation of all converted category vocabularies is combined into a vocabulary embedding set as the language input of the overall model implementing the small target detection method, participating in the subsequent image-text fusion and region matching process.

[0011] Further, the deep feature interaction module in step 3) has the following steps: the input of the module is the multi-scale deep image feature map extracted by the backbone network; the module first flattens the feature map and performs position embedding combined with two-dimensional sine cosine position encoding; then it models the context dependency of the deep feature map through a single-layer Transformer encoder composed of multi-head self-attention mechanism, multi-layer perceptron and layer normalization; finally, it outputs an enhanced deep semantic feature map.

[0012] Further, the spatial attention guidance module in step 4) has the following steps: the input of the module is the shallow feature map and the enhanced deep semantic feature map; first, a class-independent mask generator is used to generate a single-channel mask for the shallow feature map, and the mask is subjected to average pooling and 1×1 convolution to obtain a spatial attention weight matrix; the module concatenates the upsampled deep feature map with the shallow feature map, and element-wise multiplies it with the spatial weight, realizing feature alignment and enhancement; the output is an intermediate fusion feature map.

[0013] Further, the multi-scale feature aggregation module in step 5) has the following steps: the input of the module is the intermediate fusion feature map; the module includes a spatial attention branch and a channel attention branch, the spatial branch uses multi-scale convolution kernels (3×3, 5×5, 7×7) to process the feature map and generates a spatial attention map after fusing the obtained multiple convolution feature maps, and the channel branch uses global average pooling and non-linear activation to generate a channel attention map; the two attention maps enhance the target feature expression ability through a weighted fusion strategy; the output is a feature map with enhanced spatial and channel information.

[0014] Further, the efficient attention encoding stage is specifically implemented by an efficient attention fusion encoder, which is composed of a deep feature interaction module, a spatial attention guidance module, and a multi-scale feature aggregation module. The input of the efficient attention fusion encoder is the multi-scale image features extracted by the backbone network in step 1), which first models the context of the deep feature map, then fuses the spatial positioning information in the shallow feature map, and finally enhances the feature expression through the multi-scale feature aggregation module. The output is an image feature map that fuses visual context and multi-scale information.

[0015] Further, the cross-attention fusion module in step 6) has the following characteristics: the input of the module is the image feature map output by the efficient attention fusion encoder and the word embedding output by the CLIP text encoder; the image feature is mapped into a query vector, and the word embedding is mapped into a key and value vector respectively; the semantic matching relationship between the image feature map and the word embedding is calculated through the multi-head cross-attention mechanism, and the foreground suppression mechanism is introduced to generate a foreground probability based on similarity for noise suppression; the output is an image feature that fuses semantic features.

[0016] Further, the IoU-aware query module in step 7) has the following characteristics: the input of the module is the image feature that fuses semantic features output by the cross-attention fusion module in step 6); a contrastive denoising training strategy is used to generate positive and negative query samples by applying positive and negative noise respectively; the module jointly evaluates the IoU score and the classification confidence between the predicted bounding box and the true box, and uses an IoU-aware query selection strategy to select the optimal Top-K query; the output is a target query set that simultaneously has high IoU and high confidence.

[0017] Further, the region-text contrast module in step 8) has the following characteristics: the input of the module is the target query output by the IoU-aware query module in step 7) and the word embedding obtained in step 2); the module projects the image query vector through an MLP, and maps it to the same semantic space as the word embedding; a region-text contrast loss is constructed and optimized in combination with Varifocal Loss; the output is a similarity score between each image region and each class of word, which is used to determine and locate the image semantic label.

[0018] Further, the cross-attention decoding stage is specifically implemented by an IoU-aware decoder, which is composed of a cross-attention fusion module, an IoU-aware query module and a region-text contrast module. The input of the IoU-aware decoder is the image feature map and the CLIP word embedding output by the efficient attention fusion encoder; semantic enhanced features are generated by image-text fusion, high-quality target features are screened based on the combination of IoU and classification confidence, and finally the category recognition and bounding box regression are completed through the region-text contrast loss; and the output is the final detection result containing the detection category and the bounding box position.

[0019] A computer device comprises a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor implements the above-mentioned unmanned vehicle inspection small target detection method based on an efficient attention mechanism when executing the computer program.

[0020] A computer readable storage medium stores a computer program for executing the above-mentioned unmanned vehicle inspection small target detection method based on an efficient attention mechanism.

[0021] Advantages: Compared with the prior art, the unmanned vehicle inspection small target detection method based on the efficient attention mechanism has the following advantages: by introducing the scale-in attention interaction and the multi-scale attention aggregation module, the feature expression of small targets in a complex background is effectively enhanced, the lightweight feature fusion strategy is used to replace the traditional FPN structure, the calculation complexity is greatly reduced, and the real-time performance of the model on the unmanned vehicle is improved. The CLIP text encoder and the word-guided cross-attention mechanism are introduced to realize the deep fusion of image features and semantic word embedding, and the IoU-aware query strategy and the region-text contrast loss are combined to significantly improve the recognition accuracy and generalization ability of the model to unknown category targets. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 The method flowchart of the embodiments of the present application is shown in the figure; Figure 2 The method architecture diagram of the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0023] The present application will be further illustrated below in conjunction with specific embodiments, which should be understood as merely illustrating the present application and not limiting the scope of the present application. After reading the present application, those skilled in the art can make various equivalent modifications to the present application, which all fall within the scope defined by the claims of the present application.

[0024] Deep learning has attracted the attention of researchers due to its excellent performance in processing data information. The present application combines the modeling capability of deep learning for complex data distribution to establish an unmanned vehicle small target detection model. A specific embodiment is provided as follows: The unmanned vehicle inspection small target detection method based on an efficient attention mechanism has a process as shown in Figure 1 , which includes the following steps: Step 1) In the image feature extraction stage, the original image (including but not limited to: image of the dam itself, image of the inspection area, small target image, etc.) obtained by the unmanned vehicle during the dam inspection process is input into the deep neural network backbone. The backbone network adopts the CLIP image encoder structure and is structurally adapted. After feature extraction, the output feature maps of the last three layers of the backbone network are retained as multi-scale image feature representations, which correspond to spatial features of different resolutions and semantic depths, respectively. The obtained feature map set {S3, S4, S5} is input into the efficient attention fusion encoder for scale interaction, spatial guidance fusion and multi-scale aggregation operations.

[0025] Step 2) In the vocabulary embedding extraction stage, an open class vocabulary table is constructed based on the target categories of the VisDroneZSD dataset and the target categories commonly seen during the dam inspection process (for example: equipment, cracks, water seepage, etc.), including training categories, potential unseen categories, etc. The vocabulary form can be a word or a phrase, and has semantic distinguishability. The category vocabulary table is input into the pre-trained CLIP text encoder to convert the text into a unified dimension and consistent distribution word vector representation. The word vector representation of all category vocabularies obtained constitutes a word embedding set as the language input of the model, participating in the subsequent image-text fusion and region matching process.

[0026] Step 3) In the efficient attention encoding stage, the deep feature S5 obtained in step 1) and the position embedding are input into the deep feature interaction module. Global context modeling is achieved through the self-attention mechanism, and the optimized feature map F5 is output. The specific implementation details are as follows: The deep feature interaction module is realized by a single-layer Transformer encoder.

[0027] The input is the feature map of the picture, and the tensor dimension is , where B represents the batch size, C represents the number of channels, H and W represent the height and width of the feature map, respectively. To adapt to the Transformer encoder structure, the input feature is first flattened into a sequence format X flat :

[0028] After flattening, position encoding is performed using the two-dimensional sine and cosine functions. Assuming that the feature channel dimension is d, the encoding of each position pos is as follows:

[0029] Among them PE （pos,2i） PE （pos,2i+1） The positional encoding is used to encode different locations, where i is the channel index. This positional encoding matrix is ​​added to the original input to form a position-sensitive feature representation X. embed =X flat +PE.

[0030] Subsequently, the position-embedded sequence X embed The input is a single-layer Transformer encoder. This encoder consists of a multi-head self-attention mechanism, a multilayer perceptron (MLP), and layer normalization.

[0031] The multi-head self-attention mechanism maps the embedded features into three matrices: query Q, key K, and value V.

[0032] Among them, W q W k W v For learnable weight parameters, d k For the attention dimension, a self-attention operation (Self-Att(Q,K,V)) is then performed.

[0033] K T Representation matrix K transpose, d k express K This operation can capture the correlation between any two spatial locations in the feature sequence and ensure that the attention distribution is in probabilistic form through the softmax function.

[0034] Residual connections and normalization operations, attention output Self-Att (Q,K,V) and original input X embed Add them together and process them using LayerNorm. Record the result of this operation as Z. The calculation formula is: Z = LayerNorm(X) embed +Self-Att(Q,K,V)) This step stabilizes the training process and preserves the original feature information.

[0035] Finally, the encoder further extracts nonlinear features MLP(Z) through a multilayer perceptron (MLP) module. This module contains two linear layers and a ReLU activation function, with the following structure: MLP(Z) = ReLU (ZW1 + b1) W2 + b2 where W1, b1 are the weight matrix and bias term of the first linear layer, W2, b2 are the weight matrix and bias term of the second linear layer, the extracted nonlinear features are connected with the input residual and normalized to obtain the final output Y: Y = LayerNorm(Z + MLP(Z)) The output feature is restored to the original spatial structure and used for subsequent feature fusion operations.

[0036] Step 4) In the efficient attention encoding stage, the spatial attention guidance module uses the class-independent mask generated by step 1) shallow feature S3, which is processed by the average pooling layer and the 1x1 convolution layer to reconstruct the spatial attention weight matrix; S4 is spliced with the feature map F5 after bilinear interpolation upsampling and multiplied by the weight matrix element by element to realize feature enhancement, and finally output the intermediate feature map, the specific implementation details are as follows: The spatial attention guidance module is composed of depth separable convolution, average pooling layer and 1x1 convolution layer.

[0037] The input is the shallow feature map and the deep semantic feature map , and the output is the fused intermediate feature map .

[0038] The depth separable convolution is used to extract structural information from the shallow feature. In order to match the 8x8 pixel receptive field of each position of the shallow layer corresponding to the input image, the convolution kernel size is set to 7x7 to avoid introducing redundant regions by too large convolution kernel.

[0039] The average pooling layer reduces the sampling of the convolution output by 2x2, so that the mask feature is aligned in space with the upsampled deep feature map.

[0040] The 1x1 convolution layer is used for channel compression and remapping, which adjusts the single-channel mask to the attention weight map consistent with the fusion structure.

[0041] The output spatial attention matrix is used for element-wise weighting operation on the fused feature.

[0042] The deep feature map F5 is upsampled:

[0043] wherein is the upsampled deep feature map, Upsample() is the upsample operation. Then is spliced with S4 in the channel dimension to obtain the spliced feature .

[0044] Subsequently, the splicing feature and the attention matrix A s Element-wise multiplication is performed:

[0045] where F mid represents the intermediate feature map output by the attention guidance module, and represents element-wise multiplication.

[0046] Step 5) In the efficient attention encoding stage, the intermediate feature map obtained in step 4) is input into a multi-scale feature aggregation module. This module models the spatial dimension correlation and channel dimension dependency through parallel spatial attention branches and channel attention branches, respectively, and finally generates an output feature map with strong representation ability through a weighted fusion strategy. The specific implementation details are as follows: The multi-scale feature aggregation module receives the intermediate feature map output by the spatial attention guidance module As input, the output is a feature map enhanced by fusion .

[0047] The internal structure of the multi-scale feature aggregation module includes three parallel branches: The spatial attention branch focuses on mining the local dependency and regional importance between different spatial positions in the image. First, the input feature map F mid is subjected to a channel dimension reduction operation using a 1x1 convolution Conv 1×1 to map it to a low-dimensional feature F low : F low = Conv 1×1 (F mid ) Then, 3x3 convolution kernel Conv 3×3 , 5x5 convolution kernel Conv 5×5 and 7x7 convolution kernel Conv 7×7 are used for multi-scale convolution operation to obtain scale feature maps F (3) 、 F (5) 、 F (7) : F (3) =Conv 3×3 (F low ), F (5) =Conv 5×5 (F low ), F (7) =Conv 7×7 (F low ) The above multi-scale features are weighted and fused to form a spatial enhanced feature map F spa : Fspa =F (3) + F (5) + F (7) The spatial salient region information is extracted using the max-pooling operation, and the 7x7 convolution Conv 7×7 The spatial attention weight map A s : A s =σ(Conv 7×7 (MaxPool (F spa ))) where MaxPool is the max-pooling operation, and σ(·) represents the Sigmoid activation function, which is used to normalize the output to the interval [0, 1].

[0048] Finally, the spatial features are weighted by position to obtain the result :

[0049] The channel attention branch models the weight relationship between different channels to enhance the response strength of key semantic channels.

[0050] First, the global average pooling (GAP) is used to compress the spatial information, and only the channel information F gap : F gap =GAP(F mid )∈R B×C×1×1 Then, two 1x1 convolutions and a nonlinear activation function RELU are used to construct the channel attention mapping function A c : A c =σ(Conv 1×1 (ReLU (Conv 1×1 (F gap )))) where σ(·) represents the Sigmoid activation function, which is used to normalize the output to the interval [0, 1]. Then, the original feature map is weighted channel by channel to obtain :

[0051] Finally, the results of the two enhanced branches are weighted and fused, and a residual connection can be optionally added to maintain stable training:

[0052] The output F out is the final feature map after multi-scale spatial-channel enhancement, which is used as input to the subsequent cross-attention fusion module.

[0053] Step 6) In the cross-attention decoding stage, the cross-attention fusion module fuses the image features obtained in step 5) and the word embeddings obtained in step 2) through a multi-head cross-attention mechanism to generate a flattened one-dimensional feature vector. The specific implementation details are as follows: The cross-attention fusion module (VCAF) inputs include two parts: the image feature tensor output by the efficient attention encoding stage The word embedding matrix generated by the CLIP text encoder N represents the number of category words, and d is the word embedding dimension.

[0054] The image features are converted into a two-dimensional sequence form through the Flatten operation before input .

[0055] To implement the image-text cross-attention calculation, the image features are projected into a query vector (Q), and the word embeddings are projected into key (K) and value (V) vectors respectively: Q=F' W Q , K=TW K , V=TW V Where W Q , W K , W V ∈ are trainable linear mapping weights.

[0056] The cross-attention mechanism calculates the similarity between the query and the key to obtain the attention weight Attn(Q,K):

[0057] K T represents the transpose of matrix K, and d k represents the dimension of K. This operation can capture the correlation between any two spatial positions in the feature sequence, and the softmax function ensures that the attention distribution is in the form of probability Subsequently, the weight is applied to the value vector V to obtain the feature F cross after image-text fusion: F cross =Attn(Q,K)V Injecting all word semantics into all positions may introduce noise. Therefore, a foreground suppression mechanism is introduced. First, calculate the maximum similarity s i between each position i in the image and all category embeddings:

[0058] Where Q is the query vector and K is the key vector. s iTransformed to foreground probability p by Sigmoid function i ∈ [0, 1].

[0059] Finally, the features after text-image fusion are connected with the original image features by residual connection and weighted modulation to obtain the results , the calculation formula is as follows:

[0060] To improve the modeling ability of the model for different semantic layers, the VCAF module adopts a multi-head cross-attention mechanism (default 8 heads) to perform parallel head calculation on the query, key, and value and splicing at the end.

[0061] Step 7) In the cross-attention decoding stage, the IoU-aware query enables the model to remove noise through contrastive denoising training and adds an IoU constraint term, so that the obtained features have high classification confidence and accurate positioning information. The specific implementation details are as follows: The IoU-aware query module combines the contrastive denoising training mechanism and the IoU-guided Top-K query screening rule. The input is the fusion query vector sequence of the vocabulary-guided cross-attention fusion module, and the output is the Top-K high-quality query subset selected from N q queries.

[0062] This module first uses the contrastive denoising strategy to construct positive and negative query samples in the training stage to guide the model to learn high-quality feature representations.

[0063] For each positive sample real box b gt , a small perturbation (such as coordinate offset ) is added to generate a positive noise query box b + :

[0064] At the same time, a negative noise query box b - is generated, with a larger perturbation (such as coordinate offset ):

[0065] so that the model learns to regard the positive noise as the target and identify the negative noise as the background.

[0066] In the inference stage, to select high-quality targets from the query set output by the encoder, joint IoU and classification confidence are used as the screening basis. For each query q i , the model simultaneously predicts its bounding box and the text category score that matches it; calculates the IoU value IoU i between its predicted box and the real box; and constructs the joint selection weight Lcls+IoU (As a training target):

[0067] where: p i is the semantic similarity score between the region and the text, q i is the positive and negative sample label, which is equal to the IoU of the predicted box and the true box when it is a positive sample, 0 for background samples, i is the position index, and γ is the modulation index, which controls the influence weight of difficult samples.

[0068] Finally, the Top-K queries with high IoU and high classification confidence are selected as candidate targets , which are input into the subsequent modules.

[0069] Step 8) In the cross-attention decoding stage, the region-text comparison module calculates the similarity between the image features and the word embeddings, and integrates the bounding box regression to obtain the final target detection results, including the class information of the target (such as cracks, water seepage, landslides, etc.) and the spatial position of the target (indicating the position of the target in the image), the specific implementation details are as follows: The region-text comparison module accepts the Top-K image region features output from the query selection module, and inputs them together with the word embeddings generated by the CLIP encoder to learn the matching relationship between the image and the text through semantic comparison training.

[0070] First, the image region features are nonlinearly transformed, and the positive samples q i are mapped into the text semantic space, which is realized by a double-layer perception machine (MLP) structure, to obtain the image features z i Text:

[0071] Then, the image features z i are calculated with the word embeddings t j to calculate the similarity, and the learnable scaling parameter α and the bias term β are introduced to obtain the region-text similarity score:

[0072] where <, > represents the inner product of vectors, s i,j is the region-text similarity score, and represents the similarity between the i-th region and the j-th class.

[0073] To complete the positive and negative sample assignment, the present application uses the Hungarian matching algorithm, taking the region-text similarity s i,j as the cost matrix to match the predicted region and the true class.

[0074] The model is trained using a loss function and fine-tuned, and the training of the model is completed, and finally the output is the detection category and boundary box position of each target region.

[0075] Loss function L total Specifically,

[0076] wherein, λ is a regularization coefficient, is the L2 norm of the model weight parameter, L contrast is the region-text contrast loss, L varifocal is the classification and positioning joint loss.

[0077] The region-text contrast loss L contrast is used to measure the semantic similarity between each visual region in the image and the category vocabulary, and is defined as:

[0078] wherein, N represents the total number of region queries, C represents the number of category vocabularies, y i represents the true matching category index of the i-th visual region, s ij represents the similarity between region i and category j, and τ is a temperature coefficient.

[0079] The IoU-aware classification and positioning joint loss L varifocal is used to strengthen the classification accuracy of high-quality prediction samples and suppress low-IoU negative samples, and is defined as follows:

[0080] wherein, p is the region text similarity score, q represents the IoU value between the predicted boundary box and the true box (for foreground) or 0 (background), α is a scaling coefficient for adjusting the loss of negative samples, and γ is a modulation index; Obviously, those skilled in the art should understand that each step of the unmanned vehicle inspection small target detection method based on the efficient attention mechanism of the above-mentioned embodiments of the present application or each module of the unmanned vehicle inspection small target detection system based on the efficient attention mechanism can be realized by a general computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. Alternatively, they can be realized by program codes executable by a computing device, so that they can be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described can be executed in an order different from that shown here, or they can be manufactured into individual integrated circuit modules, or multiple modules or steps among them can be manufactured into a single integrated circuit module to realize. Thus, the embodiments of the present application are not limited to any specific combination of hardware and software.

Claims

1. A method for detecting small targets in unmanned vehicle inspection based on an efficient attention mechanism, characterized in that, The method comprises an image feature extraction stage, a vocabulary embedding extraction stage, an efficient attention encoding stage and a cross-attention decoding stage. Step 1) In the image feature extraction stage, the original image is input into a deep neural network backbone for image feature extraction, and the last three layers of the backbone network are retained as multi-scale image feature representations, corresponding to spatial features of different resolutions and semantic depths. Step 2) In the vocabulary embedding extraction stage, an open-class vocabulary table is constructed, and the class vocabulary table is input into a pre-trained CLIP text encoder to convert the text into vocabulary embeddings with unified dimensions and consistent distribution. Step 3) In the efficient attention encoding stage, the deep feature map S5 obtained in step 1) and the position embedding are input into a deep feature interaction module, and global context modeling is realized through a self-attention mechanism to output an optimized feature map F5. Step 4) In the efficient attention encoding stage, the spatial attention guide module generates a class-independent mask using the shallow feature map S3 obtained in step 1), which is processed by an average pooling layer and a 1x1 convolution layer to reconstruct a spatial attention weight matrix. The S4 and the feature map F5 upsampled by bilinear interpolation are concatenated and multiplied element by element with the weight matrix to realize feature enhancement, and finally output an intermediate feature map. Step 5) In the efficient attention encoding stage, the intermediate feature map obtained in step 4) is input into a multi-scale feature aggregation module, which models spatial dimension correlation and channel dimension dependency through parallel spatial attention branches and channel attention branches, and finally generates an output feature map with strong representation ability through a weighted fusion strategy. Step 6) In the cross-attention decoding stage, the cross-attention fusion module fuses the output feature map of step 5) and the vocabulary embedding obtained in step 2) through a multi-head cross-attention mechanism to generate a flattened one-dimensional feature vector. Step 7) In the cross-attention decoding stage, the IoU-aware query module removes noise through contrastive denoising training, and adds an IoU constraint term, so that the feature representation after denoising and optimization has high classification confidence and accurate positioning information. Step 8) In the cross-attention decoding stage, the region text comparison module calculates the similarity between the feature representation obtained in step 7) and the vocabulary embedding obtained in step 2), and integrates the bounding box regression to obtain the final target detection result.

2. The unmanned vehicle inspection small target detection method based on the efficient attention mechanism of claim 1, wherein, In step 1), the original image is feature-extracted to generate multi-scale spatial feature maps, and the specific steps are as follows: The original image is input into a deep neural network backbone, which adopts a CLIP image encoder structure and is structurally adapted. After feature extraction, the last three layers of the backbone network are retained as multi-scale image feature representations, corresponding to spatial features of different resolutions and semantic depths. The obtained feature map set {S3, S4, S5} is input into an efficient attention fusion encoder for scale-intra interaction, spatial-guided fusion and multi-scale aggregation operations.

3. The unmanned vehicle inspection small target detection method based on the efficient attention mechanism of claim 1, characterized in that, In step 2), the open-class vocabulary table is encoded into vocabulary embeddings, and the specific steps are as follows: Based on the target categories of the VisDroneZSD dataset and the target categories in the dam inspection process, an open category vocabulary is constructed, including training categories and potential unseen categories, in the form of words or phrases, with semantic distinction; the category vocabulary is input into the pre-trained CLIP text encoder to convert the text into a unified dimension and consistent distribution of word vector representation; the word vector representation of all converted category words is combined into a word embedding set as the language input of the overall model implementing the small target detection method, participating in the subsequent image-text fusion and region matching process.

4. The unmanned vehicle inspection small target detection method based on the efficient attention mechanism of claim 1, characterized in that, The deep feature interaction module in the step 3): the input of the module is the multi-scale deep image feature map extracted by the backbone network; the module first flattens the feature map and combines two-dimensional sine cosine position coding for position embedding; then the context dependency of the deep feature map is modeled through a single-layer Transformer encoder composed of a multi-head self-attention mechanism, a multi-layer perceptron and a layer normalization; finally, the enhanced deep semantic feature map is output.

5. The unmanned vehicle inspection small target detection method based on the efficient attention mechanism of claim 1, characterized in that, The spatial attention guiding module in the step 4): the input of the module is the shallow feature map and the enhanced deep semantic feature map; first, a single-channel mask is generated for the shallow feature map using a class-independent mask generator, and the mask is subjected to average pooling and 1x1 convolution to obtain a spatial attention weight matrix; the module splices the up-sampled deep feature map and the shallow feature map, and element-wise multiplies them with the spatial weight, realizing feature alignment and enhancement; The output is an intermediate fusion feature map.

6. The unmanned vehicle inspection small target detection method based on the efficient attention mechanism of claim 1, characterized in that, The multi-scale feature aggregation module in the step 5): the input of the module is the intermediate fusion feature map; the module includes a spatial attention branch and a channel attention branch, the spatial branch uses multi-scale convolution kernels to process the feature map and fuses the obtained multiple convolution feature maps to generate a spatial attention map, and the channel branch uses global average pooling and non-linear activation to generate a channel attention map; the two attention maps enhance the target feature expression ability through a weighted fusion strategy; The output is a feature map with enhanced spatial and channel information.

7. The unmanned vehicle inspection small target detection method based on the efficient attention mechanism of claim 1, characterized in that, The efficient attention encoding stage is specifically implemented by an efficient attention fusion encoder, which is composed of a deep feature interaction module, a spatial attention guiding module and a multi-scale feature aggregation module; the input of the efficient attention fusion encoder is the multi-scale image feature extracted by the backbone network in step 1), which first models the context of the deep feature map, then fuses the spatial positioning information in the shallow feature map, and finally enhances the feature expression through the multi-scale feature aggregation module; The output is an image feature map that fuses visual context and multi-scale information; The cross-attention decoding stage is specifically implemented by an IoU-aware decoder, which is composed of a cross-attention fusion module, an IoU-aware query module and a region-text contrast module. The input of the IoU-aware decoder is the image feature map output by the efficient attention fusion encoder and the CLIP vocabulary embedding; semantic enhanced features are generated by image-text fusion, high-quality target features are selected based on the joint filtering of IoU and classification confidence, and finally the category recognition and bounding box regression are completed through the region-text contrast loss; and the output is the final detection result containing the detection category and the bounding box position. 8.The unmanned vehicle inspection small target detection method based on the efficient attention mechanism of claim 1, characterized in that, The cross-attention fusion module in step 6): the input of the module is the image feature map output by the efficient attention fusion encoder and the vocabulary embedding output by the CLIP text encoder; The image features are mapped into query vectors, and the vocabulary embedding is mapped into key and value vectors, respectively; the semantic matching relationship between the image feature map and the vocabulary embedding is calculated through the multi-head cross-attention mechanism, and the foreground suppression mechanism is introduced to generate foreground probability based on similarity for noise suppression; and the output is the image feature with fused semantic features. 9.The unmanned vehicle inspection small target detection method based on the efficient attention mechanism of claim 1, characterized in that, The IoU-aware query module in step 7): the input of the module is the image feature with fused semantic features output by the cross-attention fusion module in step 6); A contrastive denoising training strategy is adopted to generate positive and negative query samples by applying positive and negative noise respectively; the module jointly evaluates the IoU score and the classification confidence between the predicted bounding box and the real box, and selects the optimal Top-K query by using the IoU-aware query selection strategy; and the output is a target query set with high IoU and high confidence.

10. The unmanned vehicle inspection small target detection method based on the efficient attention mechanism of claim 1, wherein, The region-text contrast module in step 8): the input of the module is the target query output by the IoU-aware query module in step 7) and the vocabulary embedding obtained in step 2); the module linearly projects the image query vector through the MLP, and maps the vocabulary embedding to the unified semantic space; the region-text contrast loss is constructed and optimized in combination with the Varifocal Loss; and the output is the similarity score between each image region and each class of vocabulary, which is used for determining and positioning the image semantic label.

Images