Visual recognition method for missing bolts in pipe gallery under the condition of robot mobile shooting

Abstract

The present invention discloses a visual recognition method for bolt missing in a pipe gallery under the condition of robot mobile shooting. The method corrects the red channel of the bolt missing visual recognition data set, uses a median filter and a DnCNN combined model to remove impulse noise and Gaussian noise, and on this basis, performs grayscale transformation calculation, cepstrum compression operation and centering operation on the blurred image, performs edge detection operation and Radon transformation on the obtained image, and correctly estimates the blur angle; calculates the peak value and blur angle by utilizing the peak symmetry of the first-order differential autocorrelation and the cepstrum three-dimensional graph; constructs a point spread function for the blur parameter, and uses the Richason-Lucy algorithm to iterate the blurred image for multiple times, thereby restoring the image. The training set and the validation set are input into the improved Yolov8 model to prove the effectiveness of the constructed model. The present invention can achieve the purpose of improving calculation efficiency and prediction accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of underground comprehensive pipeline corridor structure monitoring in actual operation, and relates to a method for identifying missing bolts in an underground comprehensive pipeline corridor, and specifically to a method for visually identifying missing bolts in a pipeline corridor under the condition of robot mobile shooting. Background Art

[0002] The existing means of identifying missing bolts in underground comprehensive pipeline corridors mostly adopts manual inspection methods, which have problems such as long inspection time, high detection cost, poor visual accuracy and easy omission of defects. The existing visual recognition method is limited by low detection accuracy and many misjudgments of targets, and cannot achieve real-time detection of missing bolts in the entire underground comprehensive pipeline corridor. Summary of the invention

[0003] In order to solve the problems of uneven illumination, more target interferences, low precision, slow processing speed, etc. in the existing visual recognition method for missing bolts in the pipe gallery, which leads to the imperfection of the visual recognition method for missing bolts in the pipe gallery, the present invention provides a visual recognition method for missing bolts in the pipe gallery under the condition of robot mobile shooting. The method uses channel compensation, CLAHE processing, median filter combined with DnCNN combined model to remove noise, restore blurred images and improve Yolov8, which can achieve the purpose of improving calculation efficiency and prediction accuracy, thereby providing effective guarantee for the safe operation and maintenance of underground comprehensive pipe gallery structure during the operation cycle.

[0004] The objective of the present invention is achieved through the following technical solutions:

[0005] A visual recognition method for missing bolts in a pipe gallery under robot mobile shooting conditions comprises the following steps:

[0006] Step 1: Use the inspection robot to capture and collect image data of the bolts and bolt holes of the underground integrated pipe gallery under mobile shooting conditions; manually annotate the collected image data using labelimg annotation software, and convert the labels into Yolo format to construct a bolt missing visual recognition dataset;

[0007] Step 2: The red channel of the annotated data set is corrected based on the image enhancement method of channel compensation, and the image is rotated, mirrored, adjusted for brightness, contrast, and CLAHE processing to improve the data type under mobile shooting conditions with excessive or insufficient illumination and contrast. The expanded data set is divided into a training set and a validation set. A median filter and a residual learning denoising convolutional neural network DnCNN combined model is used to remove the impulse noise and Gaussian noise generated by the robot under mobile shooting conditions due to motion bumps, poor collection illumination, high temperature, or high transmittance.

[0008] Step 3: Perform grayscale transformation calculation, cepstrum compression calculation and centering operation on the blurred images under the mobile shooting condition in the training set and the validation set in step 2; perform edge detection calculation on the obtained images, and then perform Radon transform to eliminate the interference information caused by the cross bright line, so as to correctly estimate the blur angle of the blurred images under the mobile shooting condition; calculate the peak value and obtain the blur scale by using the first-order difference autocorrelation and the peak symmetry of the cepstrum three-dimensional graph; construct a point spread function for the blur parameter, use the Richardson-Lucy algorithm to iterate it multiple times to restore the blurred images under the mobile shooting condition, and construct new training set and validation set;

[0009] Step 4: Replace the Conv module with the GSConv module in the Yolov8 network backbone to ensure that the accuracy remains unchanged and the operation speed of the visual recognition model under mobile shooting conditions is improved; replace the C2f module with the VoVGSCSP module in the head end, and add the CAFM attention module and the EMA attention module to improve the model's attention to small targets such as bolt holes, and build an improved Yolov8 model;

[0010] Step 5: According to steps 3 and 4, the training set is input into the improved Yolov8 model for training to improve the evaluation index of bolt hole visual recognition under mobile shooting conditions; according to the training results, the validation set is verified using the best weight best.pt to prove the effectiveness of the constructed model and output the bolt hole detection results.

[0011] Compared with the prior art, the present invention has the following advantages:

[0012] The present invention can effectively improve the efficiency and accuracy of visual recognition of missing bolts in pipe gallery under the condition of robot mobile shooting, and is suitable for target detection and monitoring of missing bolts in underground comprehensive pipe gallery structures during the operation cycle. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a flow chart of the visual recognition method for missing bolts in the pipe gallery under the condition of robot mobile shooting.

[0014] Figure 2 The image of the dataset.

[0015] Figure 3 This is the picture after data enhancement.

[0016] Figure 4 Manually labeled using labelimg labeling software.

[0017] Figure 5 Schematic diagram of the CAFM attention module.

[0018] Figure 6 Schematic diagram of the EMA attention module.

[0019] Figure 7 Schematic diagram of the GSConv convolution module.

[0020] Figure 8 Schematic diagram of the VoVGSCSP module.

[0021] Fig. 9 Schematic diagram of the Yolov8+GSConv+SlimNeck+CAFM+EMA model.

[0022] Fig.10 is the precision-recall curve for the training set.

[0023] Fig.11 is the precision-recall curve for the validation set. DETAILED DESCRIPTION

[0024] The technical solution of the present invention is further described below in conjunction with the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention should be included in the protection scope of the present invention.

[0025] The present invention provides a visual recognition method for bolt missing in a pipe gallery under the condition of robot mobile shooting. The method uses an inspection robot to shoot and collect image data under the condition of mobile shooting for an underground comprehensive pipe gallery structure with bolt missing in an operating environment, and manually annotates the image data through annotation software to construct a visual recognition data set for bolt missing. Secondly, the red channel of the data set is corrected based on the image enhancement algorithm and CLAHE processing of channel compensation, and the data type under different illumination is improved. The median filter and the residual learning denoising convolutional neural network DnCNN combined model are used to remove the impulse noise and Gaussian noise generated by the robot under the condition of mobile shooting. On this basis, the blurred image is subjected to grayscale transformation calculation, cepstrum compression operation and centering pair operation, the obtained image is subjected to edge detection operation, and then Radon transformation is performed to eliminate the interference information caused by the cross bright line, so as to correctly estimate the blur angle. The peak value and blur angle are calculated by utilizing the peak symmetry of the first-order differential autocorrelation and the cepstrum three-dimensional graph. A point spread function is constructed for the blur parameter, and the blurred image is iterated multiple times using the Richason-Lucy algorithm to restore the image. In the basic framework of Yolov8, GSConv+SlimNeck lightweight convolution modules, CAFM and EMA attention mechanism modules are added to improve the operation speed and the attention to small targets such as bolt holes, and a visual recognition algorithm for missing bolts in the pipeline corridor under the condition of robot mobile shooting based on improved Yolov8 is constructed. The training set and validation set are input into the improved Yolov8 model to prove the effectiveness of the constructed model. The improved Yolov8 model is experimentally compared with other Yolov8 models to evaluate the constructed visual recognition method for missing bolts in the pipeline corridor under mobile shooting conditions.

[0026] like Figure 1 As shown, the specific steps include:

[0027] Step 1: Use the inspection robot to capture and collect image data of the bolts and bolt holes of the underground integrated pipe gallery under mobile shooting conditions ( Figure 2 ); manually label the collected image data using labelimg labeling software ( Figure 4 ), and convert the labels into Yolo format to build a bolt missing visual recognition dataset.

[0028] The specific steps are as follows:

[0029] Step 1: Use the inspection robot to move every 2 meters in the underground utility corridor to capture image data of bolts and bolt holes. The image data includes images generated by factors such as different lighting, different noises, and mobile shooting bumps.

[0030] Step 1 and 2: Use labelimg annotation software to manually annotate the collected image data of bolts and bolt holes, divide them into two categories: bolts and bolt holes missing, and output them as a data set in Pascal VOC format. Use the VOC to Yolo format program to convert the VOC data set into a Yolo data set to construct a bolt missing visual recognition data set.

[0031] Step 2: The red channel of the labeled data set is corrected based on the image enhancement method of channel compensation, and the red channel is rotated, mirrored, adjusted for brightness, contrast, and CLAHE processing to improve the data type under the conditions of high or low light and contrast under mobile shooting conditions ( Figure 3 ), the expanded data set is divided into a training set and a validation set; a median filter and a residual learning denoising convolutional neural network DnCNN combined model is used to remove the impulse noise and Gaussian noise generated by the robot under mobile shooting conditions due to motion bumps, poor collection lighting, high temperature or high transmittance. The specific steps are as follows:

[0032] Step 21: Rotate, mirror, increase brightness, and enhance contrast of the missing bolt visual recognition dataset constructed in step 1, and add corresponding label files at the same time to increase the data type under mobile shooting conditions with excessive or insufficient lighting and contrast, thereby expanding the dataset.

[0033] Step 22: Correct the red channel based on the green channel. The formula is as follows:

[0034]

[0035] Where: Y b1 (x) and Y b2 (x) is the red channel value before and after compensation; Y g (x) is the green channel value of the image; k is the gain coefficient for adjusting the red channel compensation effect, and k is 0.3; It is the ratio of the red and green channels. The red color is corrected with the deviation of the two channels as a reference, so that the red loss of each pixel can be dynamically compensated.

[0036] Step 2 and 3: Convert the image to YUV color space and perform global histogram equalization on the brightness component Y. The YUV color space stores brightness information and chromaticity information separately, and then specifies the color of the pixel, where Y is the brightness, that is, the grayscale value, and U and V are the chromaticity, which are used to describe the color and saturation of the image. The conversion relationship between it and the RGB model is:

[0037]

[0038] Where: Y, U, V and R, G, B are the components of YUV and RGB color space respectively. Perform global histogram equalization on the brightness component Y, the formula is as follows:

[0039]

[0040] Where: S k is the pixel value after equalization, r k is the image value in the original image, T(r k ) is r k The new pixel value after histogram equalization, L is the total number of gray levels, p r (r j ) is the pixel value r in the original image j The proportion, n j is the number of pixels with gray level j in the original image, and N is the total number of pixels in the image. The mapping relationship is obtained by calculating the sum of the frequencies of the first k pixel values, and the pixel value r in the original image is k Mapped to the equalized pixel value S k . is the cumulative distribution function, indicating that the gray level is less than or equal to r k The probability of the pixel value appearing in the image. Multiply the cumulative distribution function by (L-1) so that the new pixel value S k Mapped to the grayscale range from 0 to (L-1). After global histogram equalization, lower frequency pixel values ​​are mapped to lower new pixel values, while higher frequency pixel values ​​are mapped to higher new pixel values, so that the brightness value distribution is more balanced and the brightness distribution is adjusted.

[0041] Step 24: In order to better preserve the local details and features of the image and improve the contrast of the local area, the CLAHE algorithm is performed. The specific steps are as follows:

[0042] (1) Divide the image into m×n sub-blocks;

[0043] (2) Evenly distribute the number of pixels in each sub-block to each gray level, and obtain the average number of pixels N distributed to each gray level. a , the calculation formula is as follows:

[0044]

[0045] Where: N x and N y N is the number of pixels of each sub-block in the x and y directions respectively; g is the gray level of the sub-block. In fact, the clipping threshold T is:

[0046] T=N c ·Na (5)

[0047] Where: N c is the cropping limit coefficient, which limits the number of pixels N evenly distributed at each gray level a N c times does not exceed the threshold T.

[0048] (3) Based on the threshold T, the grayscale histogram of each sub-block is cropped and the redundant pixels are divided into each grayscale level of the grayscale histogram:

[0049]

[0050] Where: A is the number of pixels allocated to each gray level; S is the total number of pixels cropped. The following process is used for pixel redistribution:

[0051]

[0052] Where: P(i) is the number of pixels at the i-th gray level before redistribution.

[0053] Due to the uneven distribution of pixel values ​​in the original image and the characteristics of the redistribution algorithm, it is difficult to ensure that all pixels can be accurately mapped to the new grayscale. Therefore, it is necessary to cyclically allocate pixels until the remaining pixels are 0 to obtain a new histogram.

[0054] (4) Perform equalization processing on the histogram of each sub-block.

[0055] (5) Taking the grayscale value of the center point of each sub-block as the reference value, the bilinear interpolation method is used to process the pixels of different sub-blocks in the image to eliminate the block effect of the image.

[0056] Step 25: Select the RGB image and extract the red, green and blue channels separately. Find the noise in each channel and remove the noise in the channel by using median filtering.

[0057] Using filters with 3×3 and 5×5 window sizes, the maximum and minimum values ​​of other pixels are calculated by excluding the central pixels of the 3×3 and 5×5 windows. A median filter is applied to detect noisy and non-noisy pixels.

[0058] (1) When the value of the central pixel is equal to 0 or 255, it is considered to be a noise pixel, and a denoising adaptive median filter is used to suppress high-density noise to replace the noise-free pixels, and edge preservation is used for the edges and details of the image, while non-local mean filtering is used to improve noise suppression and edge detail preservation.

[0059] (2) If the pixel value of the center pixel is neither 0 nor 255, it is considered as a noise-free pixel and is selected by switching the filter. This is because salt and pepper noise changes the pixel value to 0 or 255 in each channel of RGB. Among them, P center pixels and the window size generate P points for averaging the noise pixels. The weighted sum of the P center pixels is:

[0060]

[0061] Where: P ( ' i,j) is a noise-free pixel; W (i,j) YesP i and P j The weighted sum of and the weight measured by distance; P (i,j) is the pixel value at (i, j).

[0062] Step 26: In the median filtering process of step 25, an image with Gaussian noise is obtained, and the denoising convolutional neural network DnCNN is used to further denoise the image. DnCNN is used to denoise the image and the following conditions are met:

[0063] (1) Obtain a latent clean image through end-to-end deep CNN residual learning to remove Gaussian noise. The image with Gaussian noise is y = x + v, which is the noisy observation of DnCNN.

[0064] (2) Speed ​​and performance for noise-free images are achieved through batch normalization and residual learning during convolutional neural network learning.

[0065] (3) The network depth D is (2D+1)x(2D+1), the size of the convolution filter is 3×3, and there is no pooling layer.

[0066]

[0067] Where: f (i,j) is the convolution window coefficient at position (i, j); d (i,j) is the pixel value at position (i, j); q is the window size; B is the output of the convolution.

[0068]

[0069] Where: w and h are the width and height; p and s are the padding and stride of the filter; f (i-1,j-1) is the size of the previous window.

[0070] (4) Using L2 regularization, the square of the coefficient magnitude is added as a penalty term to the loss function Loss. L2 regularization avoids overfitting during training and improves the accuracy of the model.

[0071]

[0072] In the formula: A is the true value; B is the predicted value; is the L2 regularization coefficient, which is found by iterative updates using the optimizer algorithm. λ should be greater than 0 to avoid facing least squares regression.

[0073] The input of DnCNN is a noisy image with Gaussian and impulse noise, followed by window selection. The channel selection part is used to detect the red, blue and green channels in parallel, and the channel condition part is used to detect noisy pixels and noise-free pixels in the channel. By replacing the improved median filter, salt and pepper noise is removed, and the DnCNN convolutional neural network is used to remove Gaussian noise from the input and give an output noise-free image.

[0074] Step 3: Perform grayscale transformation, cepstrum compression and centering operations on the blurred images under mobile shooting conditions in the training set and validation set in step 2; perform edge detection on the obtained images, and then perform Radon transform to eliminate the interference information caused by the cross bright lines to correctly estimate the blur angle of the blurred images under mobile shooting conditions; calculate the peak value and obtain the blur scale by using the first-order differential autocorrelation and the peak symmetry of the cepstrum three-dimensional graph; construct a point spread function for the blur parameters, and use the Richardson-Lucy algorithm to iterate it multiple times to restore the blurred images under mobile shooting conditions, and construct new training sets and validation sets. The specific steps for restoring blurred images under mobile shooting conditions are as follows:

[0075] Step 31: After the linear convolution operation is performed on the original clear image f(x, y) and the degradation function h(x, y), the output obtained under the interference of noise n(x, y) is the motion blurred image g(x, y). Therefore, the restoration of the original clear image is the inverse process of formula (13). The key steps are to accurately estimate the blur coefficient of the degradation function and reduce the interference of random noise on image restoration.

[0076] g(x,y)=h(x,y)×f(x,y)+n(x,y) (12)

[0077] Where: f(x,y) is the original clear image; h(x,y) is the degradation function; n(x,y) is the noise; g(x,y) is the motion blurred image.

[0078] The image convolution operation can be transformed into the frequency domain through Fourier transform, and the convolution in the spatial domain can be simplified to the product in the frequency domain, reducing the complexity of the operation. Then, the inverse Fourier transform is calculated to obtain the inverse spectrum for subsequent solution.

[0079] Step 32: From the cepstrum characteristics, we can know that when estimating the blur angle in the motion blurred image, as long as we first determine the direction of the stripes in the cepstrum, we can basically determine the direction of the motion blur. After the inverse Fourier transform, the cepstrum operation is performed, and then compressed and centered. It can be found that each stripe is based on the central stripe as the geometric center, showing a gradually weakening and symmetrical distribution. However, there are other interference features in the image. The Canny operator is used to binarize and Radon transform the centered compressed cepstrum image, and convert it into bright stripes with multiple rules. The motion blur angle can be obtained by taking the maximum value of the angle of the binarized spectrum image.

[0080] Step 3: For motion blurred images, the three-dimensional inverse spectrum can reflect two inherent characteristics of the image: original characteristics and blurred characteristics. Among them, the low-amplitude components can well reflect the characteristics of the blur system. At the same time, the value of the blur scale can be calculated by the characteristic that the value of the blur scale is half of the distance between different negative peak points. Calibrate the center point of the motion blurred image inverse spectrum to zero, and the negative peak point on the left is T gt (k1, m1), the negative peak point on the right is T gt (k2, m2), then the calculation formula of the fuzzy scale d is:

[0081]

[0082] Step 3 and 4: The fuzzy features present a Poisson distribution. By iterating the image using the maximum likelihood estimation algorithm, the image will converge and eventually a clearer image will be obtained. The iterative restoration process is as follows:

[0083]

[0084] Step 4: Replace the Conv module with the GSConv module in the Yolov8 network backbone ( Figure 7 ) to ensure that the accuracy remains unchanged while improving the speed of visual recognition model operation under mobile shooting conditions; replace the C2f module with the VoVGSCSP module in the head end ( Figure 8 ), and add the CAFM attention module ( Figure 5 ) and the EMA attention module ( Figure 6 ), improve the model's attention to small targets such as bolt holes, and build an improved Yolov8 model ( Fig. 9 ). The specific steps are as follows:

[0085] Step 41: Shuffle the channel dense convolution (SC) and channel sparse convolution (DSC), and use the shuffle operation to infiltrate the information generated by the SC operation into every part of the information generated by the DSC. The uniform mixing strategy allows the information from the SC to be completely mixed into the output of the DSC by uniformly exchanging local feature information on different channels. Use linear operations to perform the Shuffle task to establish the GSConv lightweight convolution module. Replace the Conv convolution layer in the Yolov8 backbone network with the established GSConv lightweight convolution module to increase the operation speed while ensuring the accuracy.

[0086] Step 42: Add the VoVGSCSP module, CAFM attention module and EMA attention module to the neck of the Yolov8 basic model framework. The specific steps are as follows:

[0087] 1) Use the one-shot aggregation method to build a VoVGSCSP module as a neck to reduce the computational complexity and inference time of the detector, but maintain accuracy. GSConv completes the task of improving the operation speed, while the VoVGSCSP module reduces the inference time and maintains the accuracy.

[0088] 2) The CAFM attention mechanism consists of a local branch and a global branch. The self-attention mechanism is adopted in the global branch to capture broader feature information, while the local branch focuses on extracting local information for comprehensive extraction.

[0089] In the local branch, in order to enhance cross-channel interaction and promote information integration, 1×1 convolution is first used to adjust the channel dimension. After that, a channel transformation operation is performed to further mix the channel information. The channels are redistributed. Subsequently, the output tensors of each group are concatenated along the channel dimension to generate a new output tensor. The features are then extracted using 3×3×3 convolutions. The local branch can be expressed as:

[0090] F Conv =W 3×3×3 (CS(W 1×1 (Y))) (15)

[0091] Where: F Conv is the output of the local branch; W 1×1 is a 1×1 convolutional layer; W 3×3×3 is a 3×3×3 convolutional layer; CS is a channel shuffle operation; Y is the input feature.

[0092] In the global branch, first 1×1 convolution and 3×3 depthwise convolution generate query (Q), key (K) and value (V), resulting in three shapes of Next, Q is reshaped into And K is shaped as Then, by and The interaction of Reduces the computational burden, instead of calculating the size of Output F in the global branch att Defined as:

[0093]

[0094] Where: α is a learnable scaling parameter used to control the and The magnitude of the matrix multiplication.

[0095] Finally, the CAFM module calculates the output calculation:

[0096] F out =F att +F Conv (17)

[0097] 3) Based on the grouping structure, the sequential processing method of CA is modified, and only two convolution kernels will be placed in parallel sub-networks respectively, one of which is a 1×1 convolution kernel processed in the same way as shown in CA, and the other is a 3×3 convolution kernel, which reshapes part of the channel dimension into the batch dimension to avoid some form of dimensionality reduction through convolution, thereby establishing the EMA attention mechanism module. The specific steps are as follows:

[0098] (1) For any given input feature Figure X ∈R C×H×W EMA divides x into G sub-features along the channel dimension for learning different semantics, where the group type can be represented by X∈[X0,X1,···,X G-1 ] indicates that X i ∈R C / / G×H×W , without losing generality, we make G much smaller than C and assume that the learned attention weight descriptor will be used to strengthen the feature representation of the region of interest in each sub-feature.

[0099] (2) The G group is reshaped and permuted into the batch dimension, and the input tensor of shape C / / G×H×W is redefined. On the one hand, a similar processing method to CA is adopted to connect the two encoded features in the image height direction so that they share the same 1×1 convolution output factorized into two vectors on the 1×1 branch, and then two nonlinear Sigmoid functions are used to fit a two-dimensional binomial distribution on the linear convolution. In order to achieve different cross-channel interaction features between the two parallel routes of the 1×1 branch, the two channel attention maps within each group are combined by simple multiplication. On the other hand, the 3×3 branch convolution captures local cross-channel interactions to expand the feature space. In this way, EMA not only encodes inter-channel information to adjust the importance of different channels, but also retains the precise spatial structure information into the channel.

[0100] (3) Two tensors are introduced, one of which is the output of the 1×1 branch and the other is the output of the 3×3 branch. Then, 2D global average pooling is used to encode the global spatial information in the output of the 1×1 branch. The output of the smallest branch will be directly converted to the corresponding dimensional shape before the channel feature joint activation mechanism, i.e., R1 1×C / / G ×R3 C / / G×H×W , the 2D global pooling operation is expressed as:

[0101]

[0102] It is designed for encoding global information and modeling long-range dependencies. To improve computational efficiency, the natural nonlinear function softmax of the 2D Gaussian map is used for the output of the 2D global average pooling to fit the linear transformation. The first spatial attention map is obtained by multiplying the output of the above parallel processing with the matrix dot product operation. To observe this, it collects spatial information of different scales in the same processing stage, and we also use 2D global average pooling to encode global spatial information in the 3×3 branch. In channel features such as R1 1×C / / G ×R3 C / / G×H×W Before the joint activation mechanism, the 1×1 branch is directly converted to the corresponding dimension shape. After that, the second spatial attention map that retains the entire precise spatial position information is derived. Finally, the output feature map within each group is calculated as a set of two generated spatial attention weight values, followed by a Sigmoid function. It captures pixel-level pairwise relationships and highlights the global context of all pixels. The final output of EMA is the same size as x, which is effective for stacking into the current architecture.

[0103] Step 5: According to step 3 and step 4, the training set is input into the improved Yolov8 model for training to improve the evaluation index of bolt hole visual recognition under mobile shooting conditions; according to the training results, the validation set is verified using the best weight best.pt to prove the effectiveness of the constructed model and output the bolt hole detection results; the improved Yolov8 model is experimentally compared with other Yolov8 models to evaluate the performance of the constructed visual recognition model for missing bolts in the pipeline corridor under mobile shooting conditions. The specific steps are as follows:

[0104] Step 51: Input the training set into the improved Yolov8 model for training to improve the evaluation indicators of bolt hole visual recognition under mobile shooting conditions. The evaluation indicators include:

[0105] Precision:

[0106]

[0107] Recall:

[0108]

[0109] F1-Score:

[0110]

[0111] Step 52: Based on the training results, the validation set is verified using the best weight best.pt to prove the effectiveness of the constructed model and output the bolt hole detection results.

[0112] Step 53: Use the improved Yolov8 model to conduct experimental comparisons with other Yolov8 models to evaluate the performance of the constructed visual recognition model for missing bolts in the pipeline corridor under mobile shooting conditions.

[0113] The following test is used to verify the effect of the present invention:

[0114] This experiment uses the data collected from the underground utility corridor as an example to verify the effectiveness of the method. The data is collected by real-time mobile shooting using a patrol robot as the data source for the experimental verification.

[0115] The details of this test are as follows:

[0116] The inspection robot is used to capture the image data of the bolts and bolt holes of the underground integrated pipe gallery under mobile shooting conditions ( Figure 2 ). Manually label the collected image data using labelimg labeling software ( Figure 4 ), and convert the labels into Yolo format to build a bolt missing visual recognition dataset.

[0117] The image enhancement method based on channel compensation corrects the red channel of the annotated data set, and rotates, mirrors, adjusts brightness, contrast and CLAHE processing to improve the data type under mobile shooting conditions with high or insufficient lighting and contrast ( Figure 3 ), and the expanded dataset is divided into a training set and a validation set. A median filter and a residual learning denoising convolutional neural network (DnCNN) combined model is used to remove the impulse noise and Gaussian noise generated by the robot under mobile shooting conditions due to motion bumps, poor collection lighting, high temperature or high transmittance.

[0118] The blurred images under the condition of mobile shooting in the training set and the validation set were subjected to grayscale transformation calculation, cepstrum compression operation and centering operation; the obtained images were subjected to edge detection operation and Radon transformation to eliminate the interference information caused by the bright cross lines, so as to correctly estimate the blur angle of the blurred images under the condition of mobile shooting. The peak value was calculated and the blur scale was obtained by using the peak symmetry of the first-order difference autocorrelation and the cepstrum three-dimensional graph. The point spread function was constructed for the blur parameters, and the Richardson-Lucy algorithm was used to iterate it multiple times to restore the blurred images under the condition of mobile shooting, and a new training set and validation set were constructed.

[0119] In the Yolov8 network backbone, the Conv module is replaced with the GSConv module to ensure that the speed of the visual recognition model under mobile shooting conditions is improved while maintaining the same accuracy. In the head end, the C2f module is replaced with the VoVGSCSP module, and the CAFM attention module and the EMA attention module are added to improve the model's attention to small targets such as bolt holes, and a visual recognition method for missing bolts in the pipeline corridor under robot mobile shooting conditions based on the improved Yolov8 is constructed.

[0120] The training set is input into the improved Yolov8 model for training to improve the evaluation index of bolt hole visual recognition under mobile shooting conditions ( Fig.10 ). According to the training results, the validation set is verified using the best weight best.pt to prove the effectiveness of the constructed model and output the bolt hole detection results ( Fig.11 ). The improved Yolov8 model is experimentally compared with other Yolov8 models to evaluate the performance of the constructed visual recognition model for missing bolts in the pipeline gallery under mobile shooting conditions.

[0121] The experimental results of the training set show that compared with the Yolov8+GSConv+SlimNeck+CAFM+EMA model, the other models are not very ideal in detecting bolts and bolt holes. Compared with the Yolov8 model, the accuracy of bolts (Bolts) of the Yolov8+GSConv+SlimNeck+CAFM+EMA model remains basically unchanged, the accuracy of bolt holes (Missing) is improved by 7.5%, and the average precision mAP@0.5 is improved by 3.8%. In summary, the Yolov8+GSConv+SlimNeck+CAFM+EMA model performs best in terms of training set accuracy and average accuracy.

[0122] The experimental results of the validation set show that compared with the Yolov8 model, the Yolov8+GSConv+SlimNeck+CAFM+EMA model has an improvement of 1.5% in bolt accuracy, 1.5% in bolt hole accuracy, and 1.5% in average precision mAP@0.5. In summary, the Yolov8+GSConv+SlimNeck+CAFM+EMA model has the best performance in bolt accuracy, bolt hole accuracy, and average accuracy of the validation set.

[0123] The present invention uses channel compensation, CLAHE processing, median filter combined with DnCNN combined model to remove noise, restore blurred images and improve Yolov8, which can achieve the purpose of improving calculation efficiency and prediction accuracy, thereby providing effective guarantee for the safe operation and maintenance of underground comprehensive pipe gallery structure during the operation period.

Claims

1. A visual recognition method for missing bolts in a pipe gallery under robot mobile shooting conditions, characterized in that The method comprises the following steps: Step 1: Use the inspection robot to capture and collect image data of the bolts and bolt holes of the underground integrated pipe gallery under mobile shooting conditions; manually annotate the collected image data using labelimg annotation software, and convert the labels into Yolo format to construct a bolt missing visual recognition dataset; Step 2: The red channel of the annotated data set is corrected based on the image enhancement method of channel compensation, and the image is rotated, mirrored, adjusted for brightness, contrast, and CLAHE processing to improve the data type under mobile shooting conditions with excessive or insufficient illumination and contrast. The expanded data set is divided into a training set and a validation set. A median filter and a residual learning denoising convolutional neural network DnCNN combined model is used to remove the impulse noise and Gaussian noise generated by the robot under mobile shooting conditions due to motion bumps, poor collection illumination, high temperature, or high transmittance. Step 3: Perform grayscale transformation calculation, cepstrum compression calculation and centering operation on the blurred images under the moving shooting condition in the training set and the validation set in step 2; The obtained image is subjected to edge detection operation and then Radon transform to eliminate the interference information caused by the bright cross lines, so as to correctly estimate the blur angle of the blurred image under the condition of mobile shooting; By using the peak symmetry of the first-order difference autocorrelation and the cepstrum three-dimensional graph, the peak value is calculated and the blur scale is obtained; the point spread function is constructed for the blur parameter, and the Richardson-Lucy algorithm is used to iterate it multiple times to restore the blurred image under the moving shooting condition, and a new training set and validation set are constructed; The specific steps for restoring blurred images under mobile shooting conditions are as follows: Step 31: After the linear convolution operation is performed on the original clear image f(x, y) and the degradation function h(x, y), the output obtained under the interference of noise n(x, y) is the motion blurred image g(x, y): g(x,y)=h(x,y)×f(x,y)+n(x,y) (1) Where: f(x,y) is the original clear image; h(x,y) is the degradation function; n(x,y) is the noise; g(x,y) is the motion blurred image; Step 32: Use the Canny operator to binarize and Radon transform the center-compressed cepstrum image, convert it into bright stripes with multiple regularities, and take the maximum value of the angle of the binarized spectrum image to obtain the motion blur angle; Step 3: Calibrate the center point of the motion blurred image cepstrum to zero, and the negative peak point on the left is T gt (k1, m1), the negative peak point on the right is T gt (k2, m2), then the calculation formula of the fuzzy scale d is: Step 3 and 4: The blurred features present a Poisson distribution. By iterating the image using the maximum likelihood estimation algorithm, a clearer image is finally obtained. The iterative restoration process is as follows: Step 4: Replace the Conv module with the GSConv module in the Yolov8 network backbone to ensure that the accuracy remains unchanged and the operation speed of the visual recognition model under mobile shooting conditions is improved; replace the C2f module with the VoVGSCSP module in the head end, and add the CAFM attention module and the EMA attention module to improve the model's attention to small targets such as bolt holes, and build an improved Yolov8 model; Step 5: According to steps 3 and 4, the training set is input into the improved Yolov8 model for training to improve the evaluation index of bolt hole visual recognition under mobile shooting conditions; according to the training results, the validation set is verified using the best weight best.pt to prove the effectiveness of the constructed model and output the bolt hole detection results.

2. The visual identification method for missing bolts in a pipe gallery under the condition of robot mobile shooting according to claim 1 is characterized in that The specific steps of step one are as follows: Step 1: Use the inspection robot to move every 2 meters in the underground utility corridor to take pictures of bolts and bolt holes. The picture data includes images generated by different lighting, different noises, and moving shooting bumps; Step 1 and 2: Use labelimg annotation software to manually annotate the collected image data of bolts and bolt holes, divide them into two categories: bolts and bolt holes missing, and output them as a data set in PascalVOC format. Use the VOC to Yolo format program to convert the VOC data set into a Yolo data set to construct a bolt missing visual recognition data set.

3. The visual identification method for missing bolts in a pipe gallery under the condition of robot mobile shooting according to claim 1 is characterized in that The specific steps of step 2 are as follows: Step 21: Rotate, mirror, increase brightness, enhance contrast of the missing bolt visual recognition dataset constructed in step 1, and add the corresponding label file at the same time; Step 22: Correct the red channel based on the green channel. The formula is as follows: Where: Y b1 (x) and Y b2 (x) is the red channel value before and after compensation; Y g (x) is the green channel value of the image; k is the gain coefficient for adjusting the red channel compensation effect; It is the ratio of red and green channels; Step 2 and 3: Convert the image to YUV color space and perform global histogram equalization on the brightness component Y. The YUV color space stores brightness information and chromaticity information separately, and then specifies the color of the pixel, where Y is brightness, that is, grayscale value, and U and V are chromaticity, which are used to describe the color and saturation of the image. The conversion relationship between it and the RGB model is: Where: Y, U, V and R, G, B are the components of YUV and RGB color space respectively; the brightness component Y is subjected to global histogram equalization, and the formula is as follows: Where: S k is the pixel value after equalization, r k is the image value in the original image, T(r k ) is r k The new pixel value after histogram equalization, L is the total number of gray levels, p r (r j ) is the pixel value r in the original image j The proportion, n j is the number of pixels with gray level j in the original image, N is the total number of pixels in the image, is the cumulative distribution function, indicating that the gray level is less than or equal to r k The probability of the pixel value appearing in the image; Step 24: In order to better preserve the local details and features of the image and improve the contrast of the local area, the CLAHE algorithm is performed; Step 25: Select the RGB image and extract the red, green and blue channels respectively, find the noise in each channel, and eliminate the noise in the channel by using median filtering; Step 26: In the median filtering process of step 25, an image with Gaussian noise is obtained, and the image is further denoised using the denoising convolutional neural network DnCNN.

4. The method for visually identifying missing bolts in a pipe gallery under robot mobile shooting conditions according to claim 3 is characterized in that The specific steps of the CLAHE algorithm are as follows: (1) Divide the image into m×n sub-blocks; (2) Evenly distribute the number of pixels in each sub-block to each gray level, and obtain the average number of pixels N distributed to each gray level. a , the calculation formula is as follows: Where: N x and N y N is the number of pixels of each sub-block in the x and y directions respectively; g is the gray level of the sub-block; The clipping threshold T is: T=N c ·N a (8) Where: N c is the cropping limit coefficient, which limits the number of pixels N evenly distributed at each gray level a N c times does not exceed the threshold T; (3) Based on the threshold T, the grayscale histogram of each sub-block is cropped and the redundant pixels are divided into each grayscale level of the grayscale histogram: Where: A is the number of pixels divided equally into each gray level; S is the total number of pixels cropped; The following process is used for pixel redistribution: Where: P(i) is the number of pixels at the i-th gray level before redistribution; (4) performing equalization processing on the histogram of each sub-block; (5) Taking the grayscale value of the center point of each sub-block as the reference value, the bilinear interpolation method is used to process the pixels of different sub-blocks in the image to eliminate the block effect of the image.

5. The visual identification method for missing bolts in a pipe gallery under the condition of robot mobile shooting according to claim 3 is characterized in that In step 25, filters with 3×3 and 5×5 window sizes are used to exclude the central pixels of the 3×3 and 5×5 windows, calculate the maximum and minimum values ​​of other pixels, and apply a median filter to detect noisy and noise-free pixels, where: (1) When the value of the central pixel is equal to 0 or 255, it is considered to be a noise pixel, and a denoising adaptive median filter is obtained to suppress high-density noise to replace the noise-free pixels, edge preservation is used for the edges and details of the image, and non-local mean filtering is used to improve noise suppression and edge detail preservation; (2) If the pixel value of the center pixel is neither 0 nor 255, it is regarded as a noise-free pixel and the noise-free pixel is selected by switching the filter, where P center pixels and the window size generate P points for averaging the noise pixels. The weighted sum of the P center pixels is: Where: P′ (i,j) is a noise-free pixel; W (i,j) YesP i and P j The weighted sum of and the weight measured by distance; P (i,j) is the pixel value at (i, j).

6. The method for visually identifying missing bolts in a pipe gallery under robot mobile shooting conditions according to claim 3 is characterized in that In step 26, applying DnCNN to denoise the image satisfies the following conditions: (1) Obtain a potential clean image through end-to-end deep CNN residual learning to remove Gaussian noise. The image with Gaussian noise is y = x + v, which is the noisy observation of DnCNN; (2) Speed ​​and performance for noise-free images achieved through batch normalization and residual learning during convolutional neural network learning; (3) The network depth D is (2D+1)x(2D+1), the size of the convolution filter is 3×3, and there is no pooling layer: Where: f (i,j) is the convolution window coefficient at position (i, j); d (i,j) is the pixel value at position (i, j); q is the window size; B is the output of the convolution; Where: w and h are the width and height; p and s are the padding and stride of the filter; f (i-1,j-1) is the size of the previous window; (4) Using L2 regularization, the square of the coefficient magnitude is added as a penalty term to the loss function Loss: In the formula: A is the true value; B is the predicted value; is the L2 regularization coefficient, λ is greater than 0.

7. The method for visually identifying missing bolts in a pipe gallery under robot mobile shooting conditions according to claim 1 is characterized in that The specific steps of step 4 are as follows: Step 41: Shuffle the channel-dense convolution SC and the channel-sparse convolution DSC, and use the shuffle operation to infiltrate the information generated by the SC operation into every part of the information generated by the DSC. The uniform mixing strategy allows the information from the SC to be completely mixed into the output of the DSC by uniformly exchanging local feature information on different channels. Use linear operations to perform the Shuffle task, thereby establishing a GSConv lightweight convolution module, and replace the Conv convolution layer in the Yolov8 backbone network with the established GSConv lightweight convolution module; Step 42: Add the VoVGSCSP module, CAFM attention module and EMA attention module to the neck of the Yolov8 basic model framework.

8. The visual identification method for missing bolts in a pipe gallery under the condition of robot mobile shooting according to claim 7 is characterized in that The specific steps of step 42 are as follows: 1) Use a one-time aggregation method to build a cross-stage partial network VoVGSCSP module as a thin neck; 2) The CAFM attention mechanism consists of a local branch and a global branch, where: In the local branch, 1×1 convolution is first used to adjust the channel dimension. After that, a channel transformation operation is performed to further mix the channel information, the channels are redistributed, and the output tensors of each group are connected along the channel dimension to generate a new output tensor. Subsequently, 3×3×3 convolution is used to extract the features. The local branch is expressed as: F Conv =W 3×3×3 (CS(W 1×1 (Y))) (15) Where: F Conv is the output of the local branch; W 1×1 is a 1×1 convolutional layer; W 3×3×3 is a 3×3×3 convolutional layer; CS is a channel shuffle operation; Y is the input feature; In the global branch, first a 1×1 convolution and a 3×3 depthwise convolution generate the query Q, key K, and value V, resulting in three shapes Next, Q is reshaped into And K is shaped as Then, by and The interaction of Output F in the global branch att Defined as: Where: α is a learnable scaling parameter used to control the and The magnitude of the matrix multiplication of ; Finally, the CAFM module calculates the output calculation: F out =F att +F Conv (17) 3) Based on the grouping structure, the sequential processing method of CA is modified, and only two convolution kernels will be placed in parallel sub-networks respectively, one of which is a 1×1 convolution kernel processed in the same way as shown in CA, and the other is a 3×3 convolution kernel, which reshapes part of the channel dimension into the batch dimension, thereby establishing the EMA attention mechanism module.

9. The visual identification method for missing bolts in a pipe gallery under the condition of robot mobile shooting according to claim 8 is characterized in that The specific steps of the EMA attention mechanism module are as follows: (1) For any given input feature map X∈R C×H×W EMA divides x into G sub-features along the channel dimension for learning different semantics, where the group type is X∈[X0,X1,···,X G-1 ] indicates that X i ∈R C / / G×H×W , without losing generality, let G be much smaller than C, and assume that the learned attention weight descriptor will be used to strengthen the feature representation of the region of interest in each sub-feature; (2) The G group is reshaped and permuted into the batch dimension, and the input tensor with the shape of C / / G×H×W is redefined. On the one hand, the two encoded features are connected in the image height direction so that they share the same 1×1 convolution output factorization into two vectors on the 1×1 branch, and two nonlinear Sigmoid functions are used to fit the two-dimensional binomial distribution on the linear convolution. In order to achieve different cross-channel interaction features between the two parallel routes of the 1×1 branch, the two channel attention maps within each group are combined by simple multiplication; on the other hand, the 3×3 branch convolution captures local cross-channel interactions to expand the feature space; (3) Two tensors are introduced, one of which is the output of the 1×1 branch and the other is the output of the 3×3 branch. 2D global average pooling is used to encode the global spatial information in the output of the 1×1 branch. The output of the smallest branch will be directly converted to the corresponding dimensional shape before the channel feature joint activation mechanism, i.e., R1 1×C / / G ×R3 C / / G×H×W , the 2D global pooling operation is expressed as: The first spatial attention map is obtained by multiplying the output of the above parallel processing with the matrix dot product operation; 2D global average pooling is used to encode global spatial information in the 3×3 branch, and the channel feature R1 1×C / / G ×R3 C / / G×H×W Before the joint activation mechanism, the 1×1 branch is directly converted to the corresponding dimensional shape. After that, the second spatial attention map that retains the entire precise spatial position information is derived. Finally, the output feature map within each group is calculated as a set of two generated spatial attention weight values, and then the Sigmoid function is used to capture the pixel-level pairwise relationship and highlight the global context of all pixels.

Images