Bolt and pin defect detection method based on image features

Abstract

The invention discloses a bolt and pin defect detection method based on image features, and belongs to the technical field of image recognition, and the method comprises the steps: carrying out the preprocessing of a bolt and pin image photographed during the historical inspection of an unmanned plane through super-resolution reconstruction, and obtaining a training image set and a test image set; based on an SPD convolution module, improving YOLOV8 and combining with the training image set to construct and train a bolt pin detection model, and then inputting a test image set into the trained bolt pin detection model for testing to obtain a qualified bolt pin detection model; and finally, inputting a bolt and pin image shot during real-time inspection of the unmanned aerial vehicle into the qualified bolt and pin detection model for detection to obtain a detection result. The problem that in the prior art, the detection accuracy is poor in the bolt and pin image detection process is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a method for detecting bolt pin defects based on image features. Background Art

[0002] At present, the main detection methods for bolt pin defects include a pin defect detection method based on drone inspection and a pin defect damage detection method based on target detection; among them, the pin defect detection method based on drone inspection is to inspect the transmission line through drones, find out the abnormal state of bolt pins during the drone inspection, and report and eliminate the defects; the pin defect damage detection method based on target detection is to use big data analysis, machine learning or deep learning methods, and continuously learn from a large number of drone aerial photography pin defect images to establish a detection model; however, the above two methods are not very effective in detecting bolt pin defects. The following problems exist: 1) Bolt pins account for a very small proportion of drone aerial images and are typical small target detections. This means that during the detection process, the feature information of the bolt pins themselves and their surrounding areas is difficult to obtain, and slight positioning deviations can easily affect the detection accuracy, ultimately leading to poor detection accuracy of bolt pin defects; 2) The current mainstream target detection models, such as YOLOV5 and YOLOV8, are trained on public data sets of large-scale multi-category targets. The models and feature information obtained by training general detection models on this basis are not suitable for bolt pin defects, so defective objects are difficult to correctly locate and identify.

[0004] Chinese patent, publication number: CN 114445372 A, Publication Date: May 6, 2022, discloses a method for detecting pin fall-off of transmission lines based on drone image features, comprising the following steps: 1) constructing a connection fitting detector and training it; 2) constructing a pin fall-off detector and training it; 3) inputting the drone image into the trained connection fitting detector for first-stage target detection, detecting the connection fitting and outputting the marked external rectangular frame as the area of ​​interest; 4) cropping the drone image according to the external rectangular frame to obtain a connection fitting sub-image; 5) inputting the connection fitting sub-image into the trained pin fall-off detector for second-stage target detection, detecting bolts with pin fall-off and marking the external rectangular frame; 6) mapping the position of each external rectangular frame on the connection fitting sub-image back to the drone image; However, when detecting bolt pin defects, the invention only marks the connection fitting area where bolt pin defects may occur in advance, and does not consider that the connection fittings and bolt pins are both small targets in the drone aerial image, and thus cannot effectively obtain the feature information of the connection fittings and bolt pins themselves and their adjacent areas, resulting in poor detection accuracy. Summary of the invention

[0005] The purpose of the present invention is to address the problem of poor detection accuracy in the prior art when detecting bolt pin defects because the bolt pin, as a small target, occupies a very small proportion in the drone aerial image, and the feature information of the bolt pin itself and its neighboring area is difficult to obtain; a bolt pin defect detection method based on image features is proposed, and the bolt pin images taken by the drone are preprocessed by super-resolution reconstruction to obtain a training image set and a test image set, thereby improving the quality of the bolt pin images, exposing the image features clearly, and reducing the difficulty of obtaining image feature information, and using the SPD convolution module to improve YOLOV8, enhance the feature extraction capability of YOLOV8 for small targets, so that the improved YOLOV8 can accurately identify and extract the feature information of the bolt pin itself and its neighboring area in the drone aerial image; a bolt pin detection model is constructed by the improved YOLOV8 to detect the bolt pin images taken by the drone aerial, and accurate bolt pin defect detection results are obtained.

[0006] In a first aspect, a technical solution provided in an embodiment of the present invention is a bolt pin defect detection method based on image features, comprising the following steps: S1, collect bolt pin images taken by drone during historical inspections to build an original image set; S2. Preprocessing the original image set based on super-resolution reconstruction to obtain a training image set and a test image set; S3. Improve YOLOV8 based on the SPD convolution module and combine it with the training image set to build and train a bolt pin detection model; S4, inputting the test image set into the trained bolt pin detection model for testing to obtain a test result, and detecting the correctness of the test result to obtain a test result; S5, judging whether the bolt pin detection model is qualified based on the detection result, if not, jumping to S1, if yes, outputting a qualified bolt pin detection model; S6. Input the bolt pin image taken by the drone during real-time inspection into the qualified bolt pin detection model for detection, and output the detection result.

[0007] In this scheme, bolt pin images taken during historical drone inspections are collected and preprocessed with super-resolution reconstruction to obtain training image sets and test image sets, thereby standardizing a large number of bolt pin images and eliminating the influence of abnormal images and incomplete images on the detection results, effectively improving the efficiency and accuracy of bolt pin image detection; secondly, YOLOV8 is improved based on the SPD convolution module and combined with the training image set to construct and train a bolt pin detection model, which promotes the automation and intelligence of bolt pin image detection, and the use of a large number of images for training effectively improves the accuracy of the bolt pin detection model; when the test image set is used to detect the bolt pins, the detection efficiency and accuracy of the bolt pins are improved. After the pin detection model is verified and a qualified bolt pin detection model is output, it is ensured that the bolt pin detection model meets the needs of actual use, effectively preventing it from overfitting and underfitting, and the qualified bolt pin detection model obtained at this time has extremely high efficiency and accuracy when performing bolt pin image detection; finally, the bolt pin image taken by the drone during real-time inspection is input into the qualified bolt pin detection model for detection. The bolt pin detection model can accurately and quickly lock images with bolt pin defects, significantly improving the detection accuracy and efficiency of bolt pin images, thereby reducing the hidden dangers caused by defects in bolt pins.

[0008] As a preferred embodiment, in S2, the specific process of preprocessing the original image set based on super-resolution reconstruction to obtain the training image set and the test image set is: S21, using Real-ESRGAN technology to perform super-resolution reconstruction on the images in the original image set; S22, dividing the images that have completed super-resolution reconstruction into a training image set and a test image set.

[0009] In this solution, the Real-ESRGAN technology is used to perform super-resolution reconstruction on the images in the original image set, which can eliminate artifacts and noise in the images, while effectively improving the resolution and clarity of the images, significantly improving the image quality while effectively improving the accuracy of image detection.

[0010] Preferably, in S21, the specific process of using the Real-ESRGAN technology to perform super-resolution reconstruction on the images in the original image set is: S211, blurring the images in the original image set based on a Gaussian blur combined with a filter to obtain a blurred image set; S212, downsampling the images in the blurred image set based on an interpolation method to obtain a sampled image set; S213, adding noise to the images in the sampled image set based on the types of the images in the sampled image set to obtain a noise image set; S214, compressing the images in the noise image set based on image compression technology combined with filters to complete super-resolution reconstruction of the images in the original image set.

[0011] In this scheme, the images in the original image set are blurred based on Gaussian blur combined with filters to obtain a blurred image set, which can smooth and blur specific frequency details in the image, reduce noise and weaken unnecessary high-frequency details in the image; secondly, the images in the noisy image set are compressed based on image compression technology combined with filters, so that the bolt pin detection model can adapt to different image degradation conditions, greatly improving the robustness of image restoration.

[0012] Preferably, in S3, the specific process of constructing and training the bolt pin detection model based on the SPD convolution module to improve YOLOV8 and the training image set is as follows: S31. Improve YOLOV8 based on the SPD convolution module, and the YOLOV8 includes a Backbone module, a Neck module and a Head module; Specifically, the Backbone module is improved by using the SPD convolution module, and the Neck module is improved by using the SPD convolution module to obtain an improved YOLOV8; S32. Build a bolt pin detection model based on the improved YOLOV8; S33: training the constructed bolt pin detection model based on the training image set.

[0013] In this scheme, the SPD convolution module is used to improve YOLOV8 to obtain the bolt pin detection model, which enhances the detection ability of the bolt pin detection model for small targets and effectively reduces the loss of image features, thereby effectively improving the accuracy and efficiency of image detection of the bolt pin detection model.

[0014] Preferably, in S33, the specific process of training the constructed bolt pin detection model based on the training image set is as follows: S331, the bolt pin detection model performs dimension conversion on the images in the training image set and extracts features to obtain a feature map; S332, fusing the feature graphs, and identifying and classifying according to the fusion results of the feature graphs to obtain identification and classification results; S333. Determine whether the bolt pin detection model has converged based on the recognition and classification results. If so, end the training. If not, repeat S331-S333.

[0015] In this solution, the images in the training image set are dimensionally transformed, which greatly retains the image information and significantly enhances the image features; secondly, the feature maps are fused to more effectively utilize the feature information and reduce unnecessary computational redundancy; finally, it is determined whether the bolt pin detection model has converged, and the training of the model can be stopped at an appropriate time to avoid unnecessary waste of computing resources, while ensuring the stable performance of the bolt pin detection model.

[0016] Preferably, in S331, the bolt pin detection model performs dimension conversion on the images in the training image set and extracts features to obtain the feature map in the following specific process: A1. Input the training image set into the bolt pin detection model for downsampling to obtain a sub-image; A2, connecting the subgraphs into a whole; A3. Based on the connected subgraph combined with the non-strided convolutional layer, feature extraction is performed to obtain a feature map.

[0017] In this solution, the training image set is input into the bolt pin detection model for downsampling to obtain sub-images, so that the bolt pin detection model can adapt to input images of different sizes and dimensions, while improving the generalization ability and adaptability of the bolt pin detection model.

[0018] Preferably, in S332, the recognition and classification result specifically includes: Marking the images that are considered to be in normal state in the recognition and classification results as positive samples to obtain a positive sample atlas; At the same time, the images considered to be in abnormal states in the recognition and classification results are marked as negative samples to obtain a negative sample atlas.

[0019] In this scheme, the classification results of the image are specifically divided into positive sample atlases and negative sample atlases, which provides strong support for the subsequent judgment of whether the bolt pin detection model has converged, reduces unnecessary data processing in the calculation process, and effectively improves the efficiency of the bolt pin detection model.

[0020] Preferably, in S333, the specific process of judging whether the bolt pin detection model is converged based on the recognition and classification result is: The average precision mean of the bolt pin detection model is calculated based on the recognition and classification results, and a curve graph of the average precision mean is drawn for observation. If the curve graph shows that the average precision mean tends to be stable, it is judged that the bolt pin detection model has converged; otherwise, it is judged that the bolt pin detection model has not converged.

[0021] Preferably, in S4, the specific process of inputting the test image set into the trained bolt pin detection model for testing to obtain the test result, and detecting the correctness of the test result to obtain the test result is: S41, testing the test image set based on the trained bolt pin detection model to obtain test results, and dividing the test results into two sets: a positive sample set and a negative sample set; S42, performing visual comparison based on the positive sample atlas, finding images with correct test results and images with incorrect test results, and counting the number of the images with correct test results and the number of the images with incorrect test results respectively; At the same time, a visual comparison is performed based on the negative sample atlas to find the images with correct test results and the images with incorrect test results, and the number of the images with correct test results and the number of the images with incorrect test results are counted respectively.

[0022] In this scheme, the test results are divided into two sets: positive sample atlas and negative sample atlas, and the number of correctly tested images and the number of incorrectly tested images in each atlas are counted, which provides strong support for the subsequent judgment of whether the bolt pin detection model is qualified, reduces unnecessary data processing, and effectively improves the efficiency of the bolt pin detection model.

[0023] Preferably, in S5, the specific process of judging whether the bolt pin detection model is qualified based on the detection result is: S51, calculating the performance evaluation index of the bolt pin detection model based on the detection result; S52. Determine the performance of the bolt pin detection model based on the performance evaluation index. If the performance evaluation index meets the preset conditions, determine that the bolt pin detection model is qualified; otherwise, determine that the bolt pin detection model is unqualified.

[0024] In this solution, judging whether the bolt pin detection model is qualified ensures the stability of the bolt pin detection model finally output, thereby effectively improving the accuracy of the bolt pin detection model in detecting the bolt pin image.

[0025] The present invention has at least the following substantial effects: Firstly, the bolt pin images taken by UAV during historical inspections are collected and preprocessed with super-resolution reconstruction to obtain training image sets and test image sets, so that a large number of bolt pin images are standardized, and the influence of abnormal images and incomplete images on the detection results is eliminated, which effectively improves the efficiency and accuracy of bolt pin image detection; secondly, YOLOV8 is improved based on the SPD convolution module and combined with the training image set to construct and train the bolt pin detection model, which promotes the automation and intelligence of bolt pin image detection, and the use of a large number of images for training effectively improves the accuracy of the bolt pin detection model; when the test image set is used to detect the bolt pins, the detection efficiency and accuracy of the bolt pin detection model are improved, and the detection accuracy of the bolt pin detection model is improved; secondly, the bolt pin detection model is trained based on the training image set, which promotes the automation and intelligence of bolt pin image detection, and the training of a large number of images effectively improves the accuracy of the bolt pin detection model; when the test image set is used to detect the bolt pins, the detection efficiency and accuracy of the bolt pin detection model are improved, and the detection accuracy of the bolt pin detection model is improved; secondly, the bolt pin detection model is trained based on the SPD convolution module and the training image set is improved, which effectively improves the accuracy of the bolt pin detection model; secondly, the bolt pin detection model is trained based on the SPD convolution module and the SPD convolution module After the detection model is verified and a qualified bolt pin detection model is output, it is ensured that the bolt pin detection model meets the needs of actual use, effectively preventing it from overfitting and underfitting, and the qualified bolt pin detection model obtained at this time has extremely high efficiency and accuracy when performing bolt pin image detection; finally, the bolt pin image taken by the drone during real-time inspection is input into the qualified bolt pin detection model for detection. The bolt pin detection model can accurately and quickly lock images with bolt pin defects, significantly improving the detection accuracy and efficiency of bolt pin images, thereby reducing the hidden dangers caused by defects in bolt pins.

[0026] The above invention content is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Other features, objects and advantages of the present invention will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings. The drawings are only for the purpose of illustrating preferred embodiments and are not to be considered as limiting the present invention. Also, the same reference symbols are used throughout the drawings to represent the same parts.

[0028] Figure 1 It is a schematic diagram of the process of the bolt pin defect detection method based on image features of the present invention; Figure 2 It is a structural diagram of the bolt pin detection model of the present invention. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation method described herein is only an optimal embodiment of the present invention, which is only used to explain the present invention and does not limit the scope of protection of the present invention. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0030] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the operations (or steps) as sequential processes, many of the operations (or steps) therein can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the drawings; the process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0031] Embodiment 1: like Figure 1 As shown, this embodiment provides a bolt pin defect detection method based on image features, comprising the following steps: S1. Collect the bolt pin images taken by the drone during historical inspections to construct the original image set.

[0032] In this embodiment, the collected bolt and pin images come from inspection data collected during the drone aerial inspection process, and the inspection data is sorted to obtain an original image set, which includes two types of images: normal bolt images and pin missing images.

[0033] S2. Preprocessing the original image set based on super-resolution reconstruction to obtain a training image set and a test image set.

[0034] In one embodiment, in order to pre-process the original image set, this embodiment adopts a super-resolution reconstruction solution, which specifically includes the following steps: S21. Use Real-ESRGAN technology to perform super-resolution reconstruction on the images in the original image set.

[0035] In one embodiment, in order to solve the image reconstruction problem, this embodiment adopts a super-resolution reconstruction solution, which specifically includes the following steps: S211, blurring the images in the original image set based on a Gaussian blur combined with a filter to obtain a blurred image set; S212, downsampling the images in the blurred image set based on an interpolation method to obtain a sampled image set; S213, adding noise to the images in the sampled image set based on the types of the images in the sampled image set to obtain a noise image set; S214, compressing the images in the noise image set based on image compression technology combined with filters to complete super-resolution reconstruction of the images in the original image set.

[0036] In this embodiment, the Real-ESRGAN technology based on the improvement of the traditional first-order degradation model is used to perform super-resolution reconstruction, denoising and deblurring on the original image; the traditional first-order degradation model usually only considers one degradation operation and ignores multiple degradation operations existing at the same time. The traditional first-order degradation model is shown in the following formula: Where D(y) represents the degradation process, y represents the input image, k represents the blur function, ↓ r represents the downsampling factor, n represents the noise, [] JPEG Indicates compressing the result into JPEG format; The Real-ESRGAN technology is a high-order degradation model upgraded from the traditional first-order degradation model. This high-order degradation model uses multiple repeated degradation processes, each of which represents a classic degradation model, as shown in the following formula: The above formula is actually a repeated operation of the first-order degradation model, where each D represents the execution of a first-order degradation. The main idea is to simulate the degradation process of real images through blurring, downsampling, noise addition and JPEG compression. First, in the blurring stage, isotropic and anisotropic Gaussian blur kernels are applied to the original image, causing loss of details and clarity to simulate the blurring effect of the real world. Secondly, the blurred image is downsampled using randomly selected bilateral interpolation, bilinear interpolation, regional interpolation and other methods to further reduce the resolution of the image, reduce details and clarity, and simulate the actual resolution reduction effect. Finally, noise is added according to the image type. For color images, Gaussian noise and noise following Poisson distribution are added to simulate the environment. In addition, sinc filters are used in both the initial blurring step and the final synthesis step. This filter is highly selective in the frequency domain and responds differently to signals of different frequencies, as shown in the following equation: Where (i, j) is the nuclear coordinate, ω c is the cut-off frequency, J1 is the first-order Bessel function of the first kind; Therefore, in the blurring stage of the image, the sinc filter can be used to reduce noise or weaken unnecessary high-frequency details in the image to improve the effect of subsequent processing tasks; and in the synthesis step of the image, the sinc filter can be used to enhance the details of the image. In order to cover a wider range of degradation scenarios, the order of applying the sinc filter and JPEG compression is randomly exchanged. This is because different degradation methods may have different effects on the image. Sometimes it may be more effective to apply the sinc filter first and then JPEG compression, while in other cases it may be the opposite. Therefore, randomly exchanging the order of these two steps helps the model better adapt to different degradation situations and improve the robustness and effect of image restoration. In summary, the sinc filter can smooth and blur specific frequency details in an image, thereby reducing the details and clarity of the image.

[0037] S22, dividing the images that have completed super-resolution reconstruction into a training image set and a test image set.

[0038] In this embodiment, the super-resolution reconstructed images are divided into 10 parts on average, of which the training image set accounts for 8 parts and the test image set accounts for 2 parts.

[0039] S3. Improve YOLOV8 based on the SPD convolution module and combine it with the training image set to build and train a bolt pin detection model.

[0040] In one embodiment, in order to construct a bolt pin detection model, this embodiment adopts the SPD convolution module to improve the YOLOV8 solution, which specifically includes: S31. Improve YOLOV8 based on the SPD convolution module, and the YOLOV8 includes a Backbone module, a Neck module and a Head module; Specifically, the Backbone module is improved by using the SPD convolution module, and the Neck module is improved by using the SPD convolution module to obtain an improved YOLOV8; S32. Build a bolt pin detection model based on the improved YOLOV8; S33: training the constructed bolt pin detection model based on the training image set.

[0041] In one embodiment, in order to train the bolt pin detection model, this embodiment adopts a training image set solution, which specifically includes the following steps: S331. The bolt pin detection model performs dimension conversion on the images in the training image set and extracts features to obtain a feature map.

[0042] In a real-time example, in order to extract image features, this embodiment adopts a dimension conversion solution, which specifically includes the following steps: A1. Input the training image set into the bolt pin detection model for downsampling to obtain a sub-image; A2, connecting the subgraphs into a whole; A3. Based on the connected subgraph combined with the non-strided convolutional layer, feature extraction is performed to obtain a feature map.

[0043] In this embodiment, the bolt pin detection model generalizes the original image conversion technology to down-sample the feature map inside and throughout the convolution to more effectively process and capture feature information. The SPD convolution module combined with this down-sampling method can enable the model to better adapt to input images of different scales and sizes, while improving the generalization ability and adaptability of the model, thereby improving the performance and effect of the model. The specific operation process is as follows: Consider an intermediate feature map X of arbitrary size S×S×C1, and cut out a series of sub-feature maps as follows: f 0,0 =X[0:S:scale,0:S:scale],f 1,0 =X[1:S:scale,0:S:scale],… f scale-1,0 =X[scale-1:S:scale,0:S:scale]; f 0,1 =X[0:S:scale,1:S:scale],f 1,1 ,...,f scale-1,1 =X[scale-1:S:scale,1:S:scale];; f 0,scale-1 =X[0:S:scale,scale-1:S:scale],f 1,scale-1 ,… f scale-1,scale-1 =X[scale-1:S:scale,scale-1:S:scale]. In general, given any original feature map X, its sub-feature map f x,yIt can be formed by dividing all entries X(i,j) by the scaling factors i+x and j+y, so each sub-feature map downsamples X by the scaling factor to obtain the corresponding sub-feature map, that is, the sub-map; next, the sub-feature maps are connected along the channel dimension to obtain a feature map X'. Compared with the original feature map X, the spatial dimension of the feature map X' is reduced by a scaling factor, while the channel dimension is increased by a scaling factor. The SPD convolution module converts the feature map X(S,S,C1) into an intermediate feature map: After the SPD convolution module feature transformation layer, a non-strided convolution layer with a C2 filter is added for further transformation, where C2 < scale 2 C1, the specific changes are as follows:

[0044] S332: Fusing the feature graphs, and identifying and classifying according to the fusion results of the feature graphs to obtain identification and classification results.

[0045] In one embodiment, in order to facilitate finding the recognition results, this embodiment adopts a classification recognition result solution, which specifically includes: Marking the images that are considered to be in normal state in the recognition and classification results as positive samples to obtain a positive sample atlas; At the same time, the images considered to be in abnormal states in the recognition and classification results are marked as negative samples to obtain a negative sample atlas.

[0046] S333. Determine whether the bolt pin detection model has converged based on the recognition and classification results. If so, end the training. If not, repeat S331-S333.

[0047] In one embodiment, in order to determine whether the bolt pin detection model is converged, this embodiment adopts an average accuracy value solution, which specifically includes: The average precision mean of the bolt pin detection model is calculated based on the recognition and classification results, and a curve graph of the average precision mean is drawn for observation. If the curve graph shows that the average precision mean tends to be stable, it is judged that the bolt pin detection model has converged; otherwise, it is judged that the bolt pin detection model has not converged.

[0048] In this embodiment, the calculation formula of the average precision mean is: Wherein, mAP is the mean average precision, N is the total number of categories of detected targets, and AP is the average precision; and the value of the mean average precision is calculated according to the above formula to draw a curve graph.

[0049] In this embodiment, the SPD convolution module is introduced in YOLOv8 to replace the standard convolution layer and pooling layer, and the SPD convolution module is used to enhance the feature network's ability to extract features for small targets, and the following is obtained: Figure 2 The bolt pin detection model shown in the figure; the SPD convolution module is a convolutional neural network module in deep learning, which can reduce the computational complexity of the network while processing spatial information. The core idea is to convert the spatial dimension of the input data into a depth dimension so as to better capture and understand the local structure of the input; compared with the standard convolution layer and pooling layer in YOLOv8, the SPD convolution module is different in that it emphasizes the processing of spatial information. The standard convolution layer and pooling layer extract features in space through sliding windows, while the SPD module converts the spatial dimension before performing the convolution operation. This conversion increases the model's sensitivity to the input spatial structure, making it more likely that the model will extract more complex feature patterns. At the same time, the SPD convolution module can better extract and express local features in the image by converting the spatial dimension into the depth dimension, thereby improving the model's ability to understand the image content; Specifically, the control Figure 2 It can be seen that the YOLOv8 is divided into Backbone, Neck and Head detection heads; wherein the Backbone includes a Conv convolution module and a C2F feature extraction module, the Neck includes an SPPF pooling module, an Upsample upsampling module, a Conv convolution module, a Concat merging module and a C2F feature extraction module, and the Head detection head includes a Detect detection module; and the SPD convolution module is achieved by improving the Conv convolution module to improve YOLOv8, thereby obtaining the bolt pin detection model; When the image data is input into the bolt pin detection model, it first enters the Backbone part for convolution operation. At this time, the Conv convolution block improved by the SPD convolution module will first convert the dimension of the image data, and then perform the convolution operation, which facilitates the C2F feature extraction module in the next step to more completely extract the image feature map. After the above convolution operation is repeated several times, the feature maps of different levels obtained in different cycles are input into the Neck bottleneck part; when the feature maps of different levels enter the Neck bottleneck part, the Upsample upsampling module will start to sample up layer by layer from the bottom feature map, during which the Conv convolution module will adjust The number of channels of the upsampled feature map is kept consistent with that of the low-level feature map. The Concat merging module fuses the upsampled feature map of each layer with the corresponding low-level feature map to supplement the spatial information and enhance the semantics. Subsequently, the Conv convolution block improved by the SPD convolution module transfers features upward layer by layer starting from the bottom-level feature map, and fuses the feature map of each layer with the high-level feature map through the Concat merging module and the C2F feature extraction module. At the same time, the fused feature map is input to the Detect detection module of the Head detection head for target detection and classification, so as to obtain the final image detection and classification result.

[0050] S4. Input the test image set into the trained bolt pin detection model for testing to obtain a test result, and detect the correctness of the test result to obtain a test result.

[0051] In one embodiment, in order to detect the correctness of the test result, this embodiment adopts a visualization solution, which specifically includes the following steps: S41, testing the test image set based on the trained bolt pin detection model to obtain test results, and dividing the test results into two sets: a positive sample set and a negative sample set; S42, performing visual comparison based on the positive sample atlas, finding images with correct test results and images with incorrect test results, and counting the number of the images with correct test results and the number of the images with incorrect test results respectively; At the same time, a visual comparison is performed based on the negative sample atlas to find the images with correct test results and the images with incorrect test results, and the number of the images with correct test results and the number of the images with incorrect test results are counted respectively.

[0052] S5. Determine whether the bolt pin detection model is qualified based on the detection result. If not, jump to S1. If yes, output a qualified bolt pin detection model.

[0053] In one embodiment, in order to determine whether the bolt pin detection model is qualified, this embodiment adopts a performance evaluation index scheme, which specifically includes the following steps: S51, calculating the performance evaluation index of the bolt pin detection model based on the detection result; S52. Determine the performance of the bolt pin detection model based on the performance evaluation index. If the performance evaluation index meets the preset conditions, determine that the bolt pin detection model is qualified; otherwise, determine that the bolt pin detection model is unqualified.

[0054] In this embodiment, the performance evaluation indicators include Precision, Recall, AP (average precision) and mAP (mean average precision). The calculation formula of the performance indicators is: Among them, P is Precision, R is Recall, AP is Average Precision, mAP is Mean Average Precision, TP is the number of positive samples correctly identified as positive samples, FP is the number of negative samples incorrectly identified as positive samples, FN is the number of positive samples incorrectly identified as negative samples, and N is the total number of categories of detected targets; The preset conditions include precision greater than 0.9, recall greater than 0.9, average precision greater than 0.9 and average precision mean greater than 0.9. When the performance evaluation index meets the above preset conditions, the bolt pin detection model is judged to be qualified, otherwise it is unqualified.

[0055] S6. Input the bolt pin image taken by the drone during real-time inspection into the qualified bolt pin detection model for detection, and output the detection result.

[0056] The embodiments of the present invention have at least the following technical effects: Firstly, the embodiments of the present invention collect bolt pin images taken by drones during historical inspections and pre-process them in combination with super-resolution reconstruction to obtain training image sets and test image sets, so that a large number of bolt pin images are standardized, and the influence of abnormal images and incomplete images on the detection results is eliminated, thereby effectively improving the efficiency and accuracy of bolt pin image detection; secondly, YOLOV8 is improved based on the SPD convolution module and combined with the training image set to construct and train a bolt pin detection model, which promotes the automation and intelligence of bolt pin image detection, and the use of a large number of images for training effectively improves the accuracy of the bolt pin detection model; when the test image set is used to detect the bolt pins, the detection efficiency and accuracy of the bolt pins are effectively improved. After the pin detection model is verified and a qualified bolt pin detection model is output, it is ensured that the bolt pin detection model meets the needs of actual use, effectively preventing it from overfitting and underfitting, and the qualified bolt pin detection model obtained at this time has extremely high efficiency and accuracy when performing bolt pin image detection; finally, the bolt pin image taken by the drone during real-time inspection is input into the qualified bolt pin detection model for detection. The bolt pin detection model can accurately and quickly lock images with bolt pin defects, significantly improving the detection accuracy and efficiency of bolt pin images, thereby reducing the hidden dangers caused by defects in bolt pins.

[0057] The above specific embodiments are preferred embodiments of the present invention, and are not intended to limit the specific implementation scope of the present invention. The scope of the present invention includes but is not limited to the present specific embodiments. All equivalent changes made in accordance with the shape, structure, and method of the present invention are within the protection scope of the present invention.

Claims

1. A bolt pin defect detection method based on image features, characterized in that: The method comprises the following steps: S1, collect bolt pin images taken by drone during historical inspections to build an original image set; S2. Preprocessing the original image set based on super-resolution reconstruction to obtain a training image set and a test image set; S3. Improve YOLOV8 based on the SPD convolution module and combine it with the training image set to build and train a bolt pin detection model; S4, inputting the test image set into the trained bolt pin detection model for testing to obtain a test result, and detecting the correctness of the test result to obtain a test result; S5, judging whether the bolt pin detection model is qualified based on the detection result, if not, jumping to S1, if yes, outputting a qualified bolt pin detection model; S6. Input the bolt pin image taken by the drone during real-time inspection into the qualified bolt pin detection model for detection, and output the detection result.

2. The method for detecting bolt pin defects based on image features according to claim 1 is characterized in that: In S2, the specific process of preprocessing the original image set based on super-resolution reconstruction to obtain the training image set and the test image set is: S21, using Real-ESRGAN technology to perform super-resolution reconstruction on the images in the original image set; S22, dividing the images that have completed super-resolution reconstruction into a training image set and a test image set.

3. The method for detecting bolt pin defects based on image features according to claim 2 is characterized in that: In S21, the specific process of using the Real-ESRGAN technology to perform super-resolution reconstruction on the images in the original image set is: S211, blurring the images in the original image set based on a Gaussian blur combined with a filter to obtain a blurred image set; S212, downsampling the images in the fuzzy image set based on an interpolation method to obtain a sampled image set; S213, adding noise to the images in the sampled image set based on the types of the images in the sampled image set to obtain a noise image set; S214, compressing the images in the noise image set based on image compression technology combined with filters to complete super-resolution reconstruction of the images in the original image set.

4. The method for detecting bolt pin defects based on image features according to claim 1, characterized in that: In S3, the specific process of constructing and training the bolt pin detection model based on the SPD convolution module to improve YOLOV8 and the training image set is as follows: S31. Improve YOLOV8 based on the SPD convolution module, and the YOLOV8 includes a Backbone module, a Neck module and a Head module; Specifically, the Backbone module is improved by using the SPD convolution module, and the Neck module is improved by using the SPD convolution module to obtain an improved YOLOV8; S32. Build a bolt pin detection model based on the improved YOLOV8; S33: training the constructed bolt pin detection model based on the training image set.

5. The method for detecting bolt pin defects based on image features according to claim 4 is characterized in that: In S33, the specific process of training the constructed bolt pin detection model based on the training image set is as follows: S331, the bolt pin detection model performs dimension conversion on the images in the training image set and extracts features to obtain a feature map; S332, fusing the feature graphs, and identifying and classifying according to the fusion results of the feature graphs to obtain identification and classification results; S333. Determine whether the bolt pin detection model has converged based on the recognition and classification results. If so, end the training. If not, repeat S331-S333.

6. The method for detecting bolt pin defects based on image features according to claim 5 is characterized in that: In S331, the bolt pin detection model performs dimension conversion on the images in the training image set and extracts features to obtain the feature map in the following specific process: A1. Input the training image set into the bolt pin detection model for downsampling to obtain a sub-image; A2, connecting the subgraphs into a whole; A3. Based on the connected subgraph combined with the non-strided convolutional layer, feature extraction is performed to obtain a feature map.

7. The method for detecting bolt pin defects based on image features according to claim 5 is characterized in that: In S332, the identification and classification results specifically include: Marking the images that are considered to be in normal state in the recognition and classification results as positive samples to obtain a positive sample atlas; At the same time, the images considered to be in abnormal states in the recognition and classification results are marked as negative samples to obtain a negative sample atlas.

8. The method for detecting bolt pin defects based on image features according to claim 5 is characterized in that: In S333, the specific process of judging whether the bolt pin detection model is converged based on the recognition and classification result is as follows: The average precision mean of the bolt pin detection model is calculated based on the recognition and classification results, and a curve graph of the average precision mean is drawn for observation. If the curve graph shows that the average precision mean tends to be stable, it is judged that the bolt pin detection model has converged; otherwise, it is judged that the bolt pin detection model has not converged.

9. The method for detecting bolt pin defects based on image features according to claim 1, characterized in that: In S4, the specific process of inputting the test image set into the trained bolt pin detection model for testing to obtain the test result, and detecting the correctness of the test result to obtain the test result is: S41, testing the test image set based on the trained bolt pin detection model to obtain test results, and dividing the test results into two sets: a positive sample set and a negative sample set; S42, performing visual comparison based on the positive sample atlas, finding images with correct test results and images with incorrect test results, and counting the number of the images with correct test results and the number of the images with incorrect test results respectively; At the same time, a visual comparison is performed based on the negative sample atlas to find the images with correct test results and the images with incorrect test results, and the number of the images with correct test results and the number of the images with incorrect test results are counted respectively.

10. The method for detecting bolt pin defects based on image features according to claim 1, characterized in that: In S5, the specific process of judging whether the bolt pin detection model is qualified based on the detection result is as follows: S51, calculating the performance evaluation index of the bolt pin detection model based on the detection result; S52. Determine the performance of the bolt pin detection model based on the performance evaluation index. If the performance evaluation index meets the preset conditions, determine that the bolt pin detection model is qualified; otherwise, determine that the bolt pin detection model is unqualified.

Images