Robot inspection defect detection method based on cross attention and multi-dimension measurement

Abstract

The application discloses a robot inspection defect detection method based on cross attention and multi-dimension measurement, designs a small sample target detection model, collects dam images through robot autonomous inspection, extracts defect areas in the images by using a cross convolution attention method, classifies the defect areas through multi-dimension measurement learning, and realizes high-precision and high-efficiency defect detection work.The application aims at the problems of catastrophic forgetting and excessive model parameters in the existing small sample target detection method, designs a cross convolution module and a multi-dimension measurement learning module, effectively alleviates the catastrophic forgetting of the model, and greatly reduces the model parameters.Combined with the robot autonomous inspection technology, the method significantly improves the recognition efficiency and accuracy of dam concrete surface defects, and realizes the intelligentization and automation of dam safety monitoring.

Description

TECHNICAL FIELD

[0001] The application relates to a robot inspection defect detection method based on cross-attention and multi-dimension measurement, and belongs to the technical field of engineering defect image classification. BACKGROUND

[0002] Reservoir dam projects in China play an irreplaceable important role in flood control, hydropower generation, irrigation, etc. These water conservancy infrastructures are not only the key nodes for regulating the spatial and temporal distribution of water resources, but also the important hubs for optimizing water resource allocation, preventing water disasters, and protecting ecological environment. They are indispensable components for maintaining the safety of river flood control system and ensuring the safety of people's lives and property. In the safety guarantee system of reservoir dams, regular inspection and safety monitoring have always been one of the core links.

[0003] However, although dam safety monitoring is crucial for maintaining the safety of national water conservancy infrastructure, existing dam defect monitoring methods cannot meet the actual needs. Traditional monitoring methods require manual construction of observation platforms and rely on professional operators for monitoring, which is not only inefficient but also easily affected by subjective factors. In order to overcome these difficulties, new methods use target detection deep learning models in the field of computer vision to identify concrete defects in dam monitoring video images.

[0004] However, existing target detection methods face many difficulties in obtaining concrete surface defect samples. On the one hand, the dam structure is complex, with various types of defects such as cracks and water seepage, and it requires a lot of manpower and resources to collect samples covering all types of defects. On the other hand, due to the influence of climate, environment, engineering construction, etc., it is difficult to obtain clear and accurately labeled defect image samples.

[0005] In order to solve the above problems, small sample learning methods are introduced into the target detection model of dam safety monitoring. However, the current small sample target detection model faces the problem of overfitting to new class samples, which leads to the catastrophic forgetting problem of the model's detection accuracy for base classes. At the same time, the parameter size of the current small sample target detection model is very large, making it difficult to deploy the model to the edge devices of the monitoring system. SUMMARY

[0006] Invention purpose: In view of the problems of catastrophic forgetting of base class memory in the training process of small sample target detection model and large model parameter quantity in the existing concrete dam surface defect identification work, the application provides a robot inspection defect detection method based on cross attention and multi-dimension measurement, a small sample target detection model based on cross attention and multi-dimension measurement is designed in the defect detection part, which is called target detection model, so that the model can identify defects in the query image through a small amount of support samples, at the same time, the catastrophic forgetting problem is relieved, and the defect detection accuracy of the model is improved.

[0007] At the same time, the parameter size of the target detection model is effectively reduced by designing the cross convolution module and the multi-dimension measurement module, the inference speed of the model is improved, and the model can better adapt to the edge equipment such as the inspection robot in the dam monitoring system.

[0008] Technical scheme: A robot inspection defect detection method based on cross attention and multi-dimension measurement comprises the following steps:

[0009] Step 1) Obtain the defect images of concrete surface cracks, water seepage and other categories in the dam inspection process through the inspection robot, pre-process the defect images and label the defects, arrange the defects of the same category into a support image set, and construct a concrete dam defect detection data set;

[0010] Step 2) Real-time acquisition of dam inspection images as query images, design a target detection model based on cross attention and multi-dimension measurement for defect detection of the query images, the target detection model uses a pre-trained convolutional neural network as a backbone network, extracts features of the support image set and the query image in step 1), obtains feature maps of support features and query features, and integrates the features through a lightweight feature encoder;

[0011] Step 3) Calculate the weight of the integrated support features obtained in step 2), and perform weighted summation on the defect features of each category to obtain a support prototype, which contains the visual features of the support image;

[0012] Step 4) Fill the query features obtained in step 2), use the support prototype obtained in step 3) as a convolution kernel to perform cross convolution on the filled query features, obtain an attention map of the query features, use the attention map to enhance the query features, and then input the enhanced query features into a region proposal network to generate candidate regions;

[0013] Step 5) Input the support prototype of each category obtained in step 2) and the candidate regions of the query image into three different dimension measurement modules of high-level semantic features, global information and local information to obtain target classification results of three dimensions;

[0014] Step 6) Weighted sum of the classification results of each dimension obtained in step 5) to obtain the final prediction result, including the location, type and confidence of the target defect.

[0015] The concrete dam defect detection data set construction of step 1) includes the following processes:

[0016] 1-1) Collect different types of concrete surface defect images from image data in the inspection process of the concrete dam, and label each type of defect in the defect image, including the location and defect type;

[0017] 1-2) Classify and filter the labeled defect images, exclude images with unclear defect features or insufficient clarity, and ensure that the number of defect images in each category is as balanced as possible and not less than 30, to obtain a defect detection data set;

[0018] 1-3) Divide the defect image data set into support set, query set and test set by random allocation.

[0019] The feature extraction and feature integration process of step 2) includes the following steps:

[0020] 2-1) Use Residual Network ResNet-50 as the backbone network for feature extraction, input the support image and query image, and pass through the residual convolution of ResNet-50 to obtain multi-scale support features and query features;

[0021] 2-2) Align the region features of k support features S i ,i∈{1,2,…,k} of the same category, so that their shape becomes s×s, and perform the same operation on the query feature to obtain Q;

[0022] 2-3) Use a lightweight feature encoder to integrate the support features S i and the query feature Q, and integrate the features with shape s×s to obtain S i ′ and Q ′ . Next, S i ′ and Q ′ are deformed and converted into vectors with shape , where r=s×s is the resolution and d is the number of channels. The purpose of feature integration is to cope with the pose changes of the detected objects in the query image. By integrating the features of the surrounding pixels, the influence of object pose changes on feature extraction is reduced. The feature encoder includes two 1×1 convolution layers, one 1×5 and one 5×1 convolution layer. The first two layers are used for adaptive feature space conversion of the target detection model, and the last two layers are used to integrate the information of the surrounding pixels while reducing the influence of noise.

[0023] wherein the support aggregation process of step 3) to obtain the support prototype comprises the following steps:

[0024] 3-1) The integrated support features obtained in step 2) are sequentially subjected to weight calculation through two fully connected layers, the input dimension of the first fully connected layer is rd, and the output dimension is d; the input dimension of the second fully connected layer is d, and the output dimension is 1. Then the obtained calculation result is applied to the Softmax function for scaling, limiting the sum to 1, to obtain the weight vector of each support feature

[0025] 3-2) The obtained weight vector is multiplied by the corresponding support feature S i ′ to obtain the weighted support feature;

[0026] 3-3) The weighted support features are summed to obtain the support prototype of the support class, and the specific formula is as follows:

[0027]

[0028] The size of the support prototype P is w p × h p × d, wherein w p represents the width of the support prototype, h p represents the height of the support prototype, and d represents the number of channels of the support prototype. The support prototype includes the semantic features and morphological features of each support image of the class.

[0029] wherein the cross-convolution process of step 4) comprises the following steps:

[0030] 4-1) In order to ensure that the size of the query feature Q ′ is unchanged before and after convolution, the support prototype P obtained in step 3) is subjected to padding operation on the query feature Q ′ obtained in step 2), and the padding operation fills 0 above, below, left and right of the query feature by default. The specific padding method is as follows:

[0031]

[0032]

[0033]

[0034]

[0035] wherein pad top , pad bottom , pad left , and pad rightrespectively represent the padding scale of the query feature in the up, down, left and right four directions, w represents the width of the support prototype, and h represents the height of the support prototype;

[0036] 4-2) Cross convolution operation is performed on the padded query feature using the support prototype P as the convolution kernel to obtain an attention map, and the cross convolution operation does not require additional training and memory parameters, and the cross convolution formula is as follows:

[0037] Y j =∑W j ×X j +b j ,j∈{1,2,…,n}

[0038] Where Y j represents the output feature of the jth class, W j represents the weight of the jth convolution kernel, X j represents the jth input feature map, i.e., the corresponding part of the query feature Q ′ , b j represents the bias term of the jth convolution operation, and n represents the number of support sample classes;

[0039] 4-3) Pixel-level multiplication is performed between the attention map and the original query feature to obtain an enhanced query feature;

[0040] Wherein, the multi-dimensional metric learning process of step 5) finally generates the following steps:

[0041] 5-1) The support prototype and the candidate region of the query image are input into a lightweight feature decoder, the decoder includes two 1x1 convolution layers and a Sigmoid layer, and the features of the decoded support prototype and the candidate region are obtained;

[0042] 5-2) The support prototype and the candidate region Q d are input into three heterogeneous lightweight metric modules, and the three metric modules of the defect detection method are respectively based on high-level semantic features, global information and local information, and three-dimensional classification probabilities are obtained. The multi-dimensional metric module mainly converts the feature vector in the feature space to the metric space. Because the feature space and the metric space have a large span, a large amount of computing resources are required for encoding in the conversion process. Unlike existing metric learning heads, the multi-dimensional metric module has a shallow feature encoder, which consumes very low performance and is targeted at different dimensional features, extracting high-level semantic information, global information and local information from the candidate region;

[0043] 5-3) In each metric module, the support prototype and the candidate region Q dThe input is first input into the encoder of the three dimensions of high-level semantics, global information and local information to obtain the specified encoding information and Q e , then and Q e are measured by the Euclidean distance to obtain the inter-class measurement result D c , c e {1, 2, …, n}, and Q d is input into the pseudo-range encoder to obtain the background measurement D 0 ;

[0044] 5-4) The inter-class measurement result D c is combined with the background measurement result D 0 , and the classification probability L i of the different dimension measurement modules is obtained by the distance decoder, i e {0, 1, …, n};

[0045] 5-5) The three measurement modules extract information from different dimensions of features, and the measurement module SMMod based on high-level semantic features, the measurement module GMMod based on global information and the measurement module LMMod based on local information pay attention to high-level semantic information, global semantic information and local information respectively, so as to perform more comprehensive classification. Due to the different attention information, the feature encoder structures of the three heads are different.

[0046] SMMod evaluates from the dimension of high-level semantics, and the feature encoder Semantic Encoder thereof includes two 3x3 convolution layers, a 1x1 convolution layer, a BatchNorm layer and a LeakyReLU layer. The two 3x3 convolutions are responsible for arranging the features around the pixels to generate high-level semantic features. The 1x1 convolution is responsible for integrating the features, the BatchNorm layer scales the features, and the LeakyReLU layer activates and filters irrelevant features.

[0047] GMMod evaluates from the dimension of global information, and the feature encoder thereof includes two fully connected layers and a LeakyReLU layer. The input of the first fully connected layer is sxsxd, and the output is 1024. The input of the second fully connected layer is 1024, and the output is 64. Finally, the LeakyReLU layer is activated.

[0048] LMMod evaluates from the dimension of local information, and the feature encoder thereof includes three 1x1 convolution layers, a BatchNorm layer and a LeakyReLU layer. Similar to SMMod, the first two 1x1 convolutions are responsible for arranging the features of each pixel, and the last 1x1 convolution is responsible for integrating the features. The BatchNorm layer scales the features, and the LeakyReLU layer activates the features.

[0049] 5-6) Set three learnable parameters, denoted as σ i , i ∈ {1, 2, 3}, which are the weights of the three metric modules. The three learnable parameters are converted into weight coefficients with a sum of 1 using the Softmax function. The classification results of the three metric modules are weighted and summed using the weight coefficients to obtain the final classification probability.

[0050] 5-7) Combine the candidate region position information generated by the region proposal network to obtain the position, type and confidence of the target defect in the query image.

[0051] The method further comprises a loss function for training and optimizing the target detection model, and the loss function comprises the following parts:

[0052] The loss function of the model is divided into four parts, which are the auxiliary loss in feature extraction, the regression loss in the cross-convolution module and the classification loss in the multi-metric module and the regression loss The regression loss of the two stages and is calculated using the smooth L1 loss, and the specific formula is as follows:

[0053]

[0054] Where x represents the difference between the predicted value and the true value.

[0055] The auxiliary loss L aux is mainly used to reduce the relevance of the support prototype and increase the distance between classes, and the specific formula is as follows:

[0056]

[0057] Where represents the support feature vector of class i, C represents the class set, represents the distance between the support feature vectors and .

[0058] The classification loss of the multi-metric module uses an improved N-Pair Loss, and the specific formula is as follows:

[0059]

[0060] Where Q represents the query feature, represents the mean of the positive sample query feature, represents the mean of the i-th class sample query feature.

[0061] In the classification loss In calculation, N-Pair Loss needs to be calculated for each metric module respectively, so the loss of three metric modules needs to be divided, and the specific formula is as follows:

[0062]

[0063] The total loss formula of the model is as follows:

[0064]

[0065] Beneficial effects: compared with the prior art, the target detection model based on cross convolution and multi-dimensional measurement is designed for the problem of catastrophic forgetting of base class memory and the problem of too large model parameter quantity in the training process of a small sample target detection model, the model first acquires a dam surface image by using a robot, then integrates the features of defects in support samples into a support prototype through a support aggregation module, takes the support prototype as a convolution kernel to perform cross convolution on a query image, generates an attention map and obtains a defect area, and through a multi-dimensional measurement learning module, the defect area is compared with the defect categories in the support samples to determine the defect type. The invention utilizes the class-independent feature extraction capability of cross convolution and the multi-dimensional measurement learning to greatly alleviate the catastrophic forgetting problem, improve the recognition accuracy of the model for concrete defects, and at the same time, through the lightweight model structure design, the parameter size of the model is effectively reduced, and the inference speed of the model is improved, so that it is more suitable for edge devices in the dam monitoring system. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 The method flowchart of the embodiment of the present application is shown in the figure;

[0067] Figure 2 The overall framework diagram of the small sample target detection model based on cross attention and multi-dimensional measurement of the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0068] The present application will be further illustrated below in conjunction with specific embodiments, and it should be understood that these embodiments are only used to illustrate the present application and not used to limit the scope of the present application, and after reading the present application, various equivalent modifications of the present application by those skilled in the art all fall within the scope defined by the claims attached hereto.

[0069] As shown in the figure, a robot inspection defect detection method based on cross attention and multi-dimensional measurement, the specific implementation process includes: Figure 1

[0070] ​Step 1: Collect defect images of the concrete surface during daily inspection of the dam by a patrol robot. The types of defects include cracks, water seepage, landslides, concrete damage, subsidence, and deformation. Label the location and type information of each defect in the collected defect images. The labeling work can be done manually or assisted by automatic labeling tools. Organize defect images of the same category into a support image set. Ensure that the number of support images for each defect category is not less than 30 and try to balance the number of images for each category. Preprocess the collected defect images, including image scale normalization, color space conversion, data augmentation, and other operations to improve image quality and enhance the generalization performance of the model.

[0071] Divide the organized defect detection dataset into support set, query set, and test set using random allocation. The support set is used for the feature aggregation module and cross-convolution module in training, the query set is used for the metric learning module in training, and the test set is used for final model performance evaluation. The specific data set division ratio can be adjusted according to the actual situation, usually support set: query set: test set = 6:2:2 or 7:2:1.

[0072] Step 2: Design a target detection model based on cross-attention and multi-dimensional metrics. The model uses a pre-trained ResNet-50 as the backbone network for feature extraction. Input the support image and query image into ResNet-50, and pass through a series of residual convolution layers to obtain multi-scale support features S i and query features Q.

[0073] To deal with the pose changes of the target in the query image, the target detection model designs a feature aggregation module. Feature aggregation aligns the k support features S i ,i∈{1,2,…,k} of the same category using regional feature alignment to unify them into a feature map with a shape of s×s. The query feature Q also undergoes the same operation. Input the support features and query features after regional feature alignment into the feature integration module. The feature integration module includes two 1×1 convolutions, one 1×5, and one 5×1 convolution. The first two layers are used for adaptive feature space conversion, and the last two layers are used to integrate the surrounding pixel information to enhance the robustness of the model to pose changes. Although convolution operations have translational invariance, they lack rotational invariance, making it difficult to accurately judge pose changes of the target. The model integrates the features of the surrounding pixels through the feature integration module to reduce the impact of pose changes on feature extraction. After the feature integration module, the integrated support features S i ′ and query features Q ′ are obtained, which are prepared for subsequent support aggregation and cross-convolution.

[0074] Input the integrated support features Si ′ is transformed into a vector, where s x s is the feature resolution, and d is the number of feature channels. This transformation is to facilitate the subsequent weight calculation and feature aggregation.

[0075] Step 3: From the k support features of the same category, extract the distinguishing features to form the support prototype of this category. The specific implementation process is as follows:

[0076] The integrated support features S i ′ are input into two fully connected layers in turn for weight calculation. The input dimension of the first fully connected layer is rd, and the output dimension is d; the input dimension of the second fully connected layer is d, and the output dimension is 1. The output result of the second fully connected layer is normalized by the Softmax function to obtain the weight vector of each support feature The sum of the weight vectors is 1.

[0077] The calculated weight vector is multiplied by the corresponding support feature S i ′ to obtain the weighted support feature. The weighted support features are summed to obtain the support prototype P of the support category. Since the support prototype contains the pose, shape, and other information of the support category, the model uses the support prototype as the convolution kernel, which can replace the learning process of high-level semantic features. The backbone network focuses more on the extraction of general features, thereby reducing the class-related parameters of the backbone network. The size of the support prototype P is w p x h p x d, where w p and h p represent the width and height of the support prototype, respectively, and d represents the number of channels. The support prototype P contains the semantic features and morphological features of each support image of the category and can be used as the visual feature prototype of the category.

[0078] This step includes an auxiliary loss function L aux for the feature extraction process. The auxiliary loss function is mainly used to reduce the relevance of the support prototype and increase the inter-class distance. The specific formula is as follows:

[0079]

[0080] where represents the support feature vector of category i, C represents the category set, represents the support feature vector and ​distance between the query feature and the support prototype. Step 4: using the support prototype generated in step 3, the query feature is enhanced by cross-convolution to improve the performance of target detection.

[0081] Specifically, step 4 includes:

[0082] In order to ensure that the size of the query feature Q ′ does not change before and after cross-convolution, it is necessary to pad Q ′ according to the support prototype P obtained in step 3. The padding formula is where pad top , pad bottom , pad left , and pad right represent the padding scale of the query feature in the up, down, left, and right directions, respectively.

[0083] After padding, the support prototype P obtained in step 3 is used as the convolution kernel to perform cross-convolution on the padded query feature Q ′ to obtain the attention map. Cross-convolution generates attention by performing convolution operations between feature maps. This process does not require additional training and does not involve parameters that need to be memorized. Compared with complex self-attention mechanisms such as Transformer, cross-convolution significantly reduces class-related parameters in the target detection model. Cross-convolution not only effectively alleviates the catastrophic forgetting problem caused by parameter conflicts in small sample target detection, but also enhances the adaptability to target pose and shape changes. The cross-convolution formula is: Y j =∑W j ×X j +b j ,j∈{1,2,…,n}. Where Y j represents the output feature of the jth class, W j represents the weight of the jth convolution kernel, X j represents the jth input feature map, i.e., the corresponding part of the query feature Q ′ , and b j represents the bias term of the jth convolution operation.

[0084] The obtained attention map is multiplied with the original query feature Q ′ pixel by pixel to obtain the enhanced query feature.

[0085] This step includes a regression loss function for the cross-convolution process The regression loss function is mainly used to improve the ability of the cross-convolution part to enhance the query feature. The regression loss function uses smooth L1 loss calculation, and the specific formula is as follows:

[0086]

[0087] where x represents the difference between the predicted value and the true value. Step 5: Utilize the support prototype and the candidate regions of the query image to perform classification through multiple heterogeneous lightweight metric modules, further improving the target detection model's recognition ability of the target. Finally, fuse the classification results of the three metric modules to obtain the final prediction results, including the position, type, and confidence of the target defect.

[0088] Specifically, step 5 includes:

[0089] First, input the support prototype generated in step 3 and the candidate regions Q of the query image generated in step 4 d into a lightweight feature decoder. The feature decoder contains two 1x1 convolution layers and a Sigmoid layer, which aims to reduce the dimension of the features and perform nonlinear transformation to obtain the decoded support prototype and the candidate region Q e .

[0090] Input the decoded support prototype and the candidate region Q e into three heterogeneous lightweight metric modules: the advanced semantic feature-based metric module SMMod, the global information-based metric module GMMod, and the local information-based metric module LMMod. These three metric modules respectively evaluate and classify the target from different dimensions. The feature encoder of SMMod contains two 3x3 convolutions, one 1x1 convolution, BatchNorm, and LeakyReLU, which are used to extract high-level semantic features. The feature encoder of GMMod contains two fully connected layers and LeakyReLU, which are used to extract global information features. The feature encoder of LMMod contains three 1x1 convolutions, BatchNorm, and LeakyReLU, which are used to extract local information features. For each metric module, the Euclidean distance measurement result D between the support prototype e and the candidate region Q c is calculated, c∈{1,2,…,n}, and the distance measurement result D 0 between the candidate region and the background is also calculated.

[0091] Convert the inter-class distance D c and the background distance D 0 into classification probabilities L i, i e {0, 1, …, n}. The distance decoder contains a learnable temperature coefficient γ to convert the distance to probability. Three metric modules SMMod, GMMod and LMMod extract feature information from different dimensions respectively, and then get the classification probability through the distance decoder. This multi-dimensional metric method can more accurately capture the characteristics of the target, thus improving the performance of detection and classification.

[0092] The multi-dimensional metric module sets three learnable parameters σ i , i e {1, 2, 3}, which correspond to the weights of the three metric modules SMMod, GMMod and LMMod respectively. The Softmax function is used to convert the three parameters into weight coefficients σ i with a sum of 1. The classification probabilities L SMMod , L GMMod and L LMMod of the three metric modules in step 5 are multiplied by the corresponding weight coefficients σ i , and then summed to get the final classification probability L.

[0093] The candidate region position information generated in step 4 is combined with the final classification probability L obtained in step 5 to obtain the position, type and confidence of the target defect in the query image.

[0094] This step includes the regression loss function and the classification loss function used in the multi-dimensional metric process. The regression loss function and the classification loss function are mainly used to improve the ability of the multi-dimensional metric process to classify the region of interest. The regression loss function uses the Smooth L1 loss function, and the specific formula is as follows:

[0095]

[0096] Where x represents the difference between the predicted value and the true value.

[0097] The classification loss function uses an improved N-Pair Loss, and the specific formula is as follows:

[0098]

[0099] Where Q represents the query feature, represents the mean of the positive sample query feature, and represents the mean of the i-th class sample query feature.

[0100] In the calculation of the classification loss , N-Pair Loss needs to be calculated for each metric module, so the loss needs to be divided into three metric modules, and the specific formula is as follows:

[0101]

[0102] The total loss formula of the target detection model is as follows:

[0103]

[0104] In the training process, the model parameters are optimized by minimizing the total loss L through the back propagation algorithm and the gradient descent algorithm, so as to improve the accuracy and robustness of defect detection.

[0105] Obviously, those skilled in the art should understand that the robot inspection defect detection method based on cross attention and multi-dimension measurement of the embodiments of the present application described above can be realized by general computing devices, which can be concentrated on a single computing device or distributed on a network composed of multiple computing devices, and optionally, they can be realized by program codes executable by computing devices, so that they can be stored in storage devices and executed by computing devices, and in some cases, the steps shown or described herein can be executed in different order, or they can be made into individual integrated circuit modules, or the multi-dimension modules or steps among them can be made into a single integrated circuit module. Thus, the embodiments of the present application are not limited to be combined with any specific hardware and software.

Claims

1. A method for robot inspection defect detection based on cross-attention and multi-dimension metrics, characterized in that, Comprise the following steps: Step 1) obtain the defect image of the concrete surface in the dam inspection process by the inspection robot, preprocess the defect image and label the defect, arrange the defects of the same category into a support image set, and construct a concrete dam defect detection data set; Step 2) real-time acquisition of dam inspection image as query image, a small sample target detection model based on cross attention and multi-dimension measurement is designed for defect detection of query image, referred to as target detection model, the target detection model uses a pre-trained convolutional neural network as the backbone network, and the support image set and the query image are feature extracted to obtain the feature maps of the support features and the query features, and the feature maps are integrated by the feature encoder to obtain the integrated support features; Step 3) the integrated support features are calculated respectively, and the weighted sum of the defect features of each category is obtained to obtain the support prototype, which contains the visual features of the support image; Step 4) the query feature is filled, the support prototype is used as the convolution kernel to cross-convolve the filled query feature to obtain the attention map of the query feature, the attention map is used to enhance the query feature, and then the enhanced query feature is input into the region proposal network to generate the candidate region; Step 5) the support prototype of each category and the candidate region of the query image are input into the measurement module of three different dimensions of high-level semantic features, global information and local information to obtain the target classification results of three dimensions; Step 6) the classification results of each dimension are weighted and summed to obtain the final prediction result, including the position, type and confidence of the target defect; The multi-dimension measurement process of step 5) final prediction result generation process comprises the following steps: 5-1) input the support prototype and the candidate region of the query image into the feature decoder, the decoder comprises two 1x1 convolution layers and a sigmoid layer to obtain the decoded support prototype and the feature of the candidate region; 5-2) input the support prototype and the candidate region into three heterogeneous lightweight measurement modules, which are a measurement module based on high-level semantic features, a measurement module based on global information and a measurement module based on local information, to obtain classification probabilities of three dimensions; 5-3) in each measurement module, the support prototype and the candidate region are first input into the encoder of three dimensions of high-level semantic, global information and local information to obtain high-dimensional features, and then the Euclidean distance measurement is performed on the high-dimensional features to obtain the inter-class measurement result, while the pseudo-distance encoder is used to measure the background of the query feature; 5-4) combine the inter-class measurement result and the background measurement result to obtain the classification probability of the different dimension measurement module through the distance decoder; 5-5) set three learnable parameters corresponding to the weights of the three measurement modules, convert the three learnable parameters into weight coefficients with a sum of 1, and use the weight coefficients to weight and sum the classification results of the three measurement modules to obtain the final classification probability; 5-6) combine the candidate region position information generated by the region proposal network to obtain the position, type and confidence information of the target defect in the query image.

2. The method of claim 1, wherein, The concrete dam defect detection data set construction of step 1) comprises the following process: 1-1) Collect different types of concrete surface defect images from image data in the inspection process of concrete dams, and label each type of defect in the defect image, including the location and defect type; 1-2) Classify and filter the labeled defect images to exclude images with unclear defect features or insufficient clarity, balance the number of defect images in each category, and obtain a defect detection dataset; 1-3) The defect image dataset is divided into support set, query set and test set by random allocation.

3. The method of claim 1, wherein, The feature extraction and feature integration process of step 2) includes the following steps: 2-1) Use a residual network as the backbone network for feature extraction, input the support image and the query image, and obtain multi-scale support features and query features through residual convolution; 2-2) Perform region feature alignment on k support features and query features of the same category to change their shape to s x s; 2-3) Use a feature encoder to integrate the support features and query features.

4. The method of claim 1, wherein, The support aggregation process of step 3) to obtain the support prototype includes the following steps: 3-1) Calculate the weight of the integrated support features, then scale the calculation results to limit their sum to 1, and obtain the weight vector of each support feature; 3-2) Multiply the obtained weight vector with the corresponding support feature to obtain the weighted support feature; 3-3) Sum the weighted support features to obtain the support prototype of the support category; the support prototype includes the semantic features and morphological features of each support image of the category.

5. The method of claim 3, wherein, The cross-convolution process of step 4) includes the following steps: 4-1) Fill the query feature according to the support prototype, and the fill operation defaults to filling 0 above, below, left and right of the query feature; 4-2) Use the support prototype as the convolution kernel to perform cross-convolution operation on the filled query feature to obtain the attention map, and the cross-convolution operation does not require additional training and memory parameters; 4-3) Perform pixel-level multiplication between the attention map and the original query feature to obtain the enhanced query feature.

6. The method of claim 1, wherein, The method further includes a loss function for training and optimizing the target detection model, and the loss function includes the following parts: Auxiliary loss of feature extraction The auxiliary loss calculates the sum of distances between all classes; the auxiliary loss is used to reduce the relevance of the support prototype and increase the distance between classes; The loss function of the cross-convolution-based region proposal network includes a regression loss ; The loss function of the classification head based on multi-dimension metrics includes a classification loss and a regression loss ; The regression loss of the region proposal network based on cross-convolution and the classification head based on multi-dimensional metrics uses a smooth F1 loss calculation; The classification loss of the classification head based on multi-dimensional metrics calculates N-Pair Loss for each metric module and adds the three loss results.

7. The method of claim 5, wherein, The metric module SMMod based on high-level semantic features, the metric module GMMod based on global information and the metric module LMMod based on local information respectively focus on high-level semantic information, global semantic information and local information; The metric module SMMod based on high-level semantic features evaluates from the dimension of high-level semantic features, The feature encoder of the semantic encoder includes two convolutional layers, one convolutional layer, one BatchNorm layer, and one LeakyReLU layer; the two convolutional layers are responsible for arranging the features around the pixels to generate high-level semantic features; the convolutional layer is responsible for integrating the features, the BatchNorm layer is responsible for scaling the features, and the LeakyReLU layer is responsible for activating the features and filtering irrelevant features; The global information-based metric module GMMod evaluates from a global dimension, The feature encoder includes two fully connected layers and one LeakyReLU layer; the first fully connected layer has an input dimension of and an output dimension of 1024, and the second fully connected layer has an input dimension of 1024 and an output dimension of 64; finally, the LeakyReLU layer is activated. The local information-based metric module LMMod evaluates from a local dimension, The feature encoder includes three convolutional layers, one BatchNorm layer, and one LeakyReLU layer; similar to the SMMod, the first two convolutional layers are responsible for arranging the features of each pixel, and the last convolutional layer is responsible for integrating the features, the BatchNorm layer is responsible for scaling the features, and the LeakyReLU layer is responsible for activating the features.

8. A computer device, comprising: The computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the above computer program to realize the steps of the robot inspection defect detection method based on cross attention and multi-dimension metric according to any one of claims 1-7.

9. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that: The computer program / instructions are executed by the processor to realize the steps of the robot inspection defect detection method based on cross attention and multi-dimension metric according to any one of claims 1-7.

Images